I’m setting up automated offsite backups from my NSLU2 to Amazon S3. With suprisingly little effort, I’ve managed to get a tool called s3sync running on the “slug” (as it’s known). s3sync is a Ruby script, so in order to run it, I had to install Ruby, which in turn meant that I had to replace the slug’s firmware with a different version of Linux, called Unslung. Once all of this was done, I just had to set up the appropriate directory structures and certificates so that the sync tool could use SSL, and write a simple upload/download script. All of this worked pretty much as advertised in the tools’ respective documentation – for the details, see the previous posts in this series.

My final step had been to set up a cron job to run the upload script, but it had failed, not logging anything. In order to debug, I ran the upload script directly from the command line, and left it to run overnight, copying a large set of directories to S3.

13 hours later, it had completed. From the jet3St Cockpit, I checked how much data was present in the bucket; it told me I had 1.61Gb, split over 2774 items. This seemed a little on the low side, but I had to get back to my workstation to be sure. And there, the same program told me that I had 1.71Gb, split over 2770 items. Checking the console showed that the command had, it thought, succeeded with no errors – but and the directory that was meant to be synced claimed to be about 4Gb in size!

A quick investigation showed that there were certainly files missing from S3. I decided to see what would happen if I ran it again – would it start uploading where it left off?

I suspect sorting this problem out may take a certain amount of poking around over a number of days, so I won’t post again in this series until I’ve found the solution.

[Update] Still hard at work on this; it looks like there’s a problem with s3sync making it cut out after some amount of transfer, so I’m trying to diagnose the problem – which is tricky when each exeperiment takes 24 hours :-/ Final results will be posted here when I have them.

I’m setting up automated offsite backups from my NSLU2 to Amazon S3. With suprisingly little effort, I’ve managed to get a tool called s3sync running on the “slug” (as it’s known). s3sync is a Ruby script, so in order to run it, I had to install Ruby, which in turn meant that I had to replace the slug’s firmware with a different version of Linux, called Unslung. Once all of this was done, I just had to set up the appropriate directory structures and certificates so that the sync tool could use SSL, and write a simple upload/download script. All of this worked pretty much as advertised in the tools’ respective documentation – for the details, see the previous posts in this series.

My final step in my last post was to set up a cron job to synchronise quite a lot of data up to S3 overnight; here is the line from the crontab file:

42 22 * * * root /home/s3sync/upload.sh &> /tmp/s3sync.log

Checking this morning brought some bad news. Nothing had been written to the log file, and the bucket I’d set up to receive the backup on S3 had only 6Mb of data – as compared to a total of 4Gb+ that was there to be backed up.

Clearly something had gone wrong.

I figured it was best to try again, this time trying to eliminate whatever problem had occurred with the cron job by simply running the backup script from a command prompt. After all, I had run the script from a command line previously, and had seen some useful logging information.

This time it at least seemed to be logging something:

-bash-3.1# /home/s3sync/upload.sh
Create node Giles

…

I left it for an hour or so, after which it had uploaded 141.25Mb, logging all the while. Clearly there was (a) something wrong with the way I had set up logging from the crontab, and (b) something had interrupted it when it had run the previous night. After a little thought, I came to the conclusion that it might not ba a great idea to have something in the crontab that could take multiple hours to run; there could well be a limit, at least in the version of the cron daemon that lives on the NSLU2, and the sync process might have been killed before it was able to sync its output to the log file. That said, I could find no mention of such a thing on the obvious page on the NSLU2-Linux site. I decided to ask the site’s mailing list, to see if anyone knew for sure if this was the answer; in the meantime, I watched the sync from the command line as it reached 683 items and 270Mb.

I’m setting up automated offsite backups from my NSLU2 to Amazon S3. With suprisingly little effort, I’ve managed to get a tool called s3sync running on the “slug” (as it’s known). s3sync is a Ruby script, so in order to run it, I had to install Ruby, which in turn meant that I had to replace the slug’s firmware with a different version of Linux, called Unslung. All of this worked pretty much as advertised in the tools’ respective documentation – for the details, see the previous posts in this series.

Having confirmed that s3sync worked as I’d expect it to, I needed to install it in a sensible place – I’d previously just put it in /tmp – set it up so that I could use SSL to encrypt the data while it was on its way to Amazon, and then write a script to synchronise at least one of the directories I want backed up. I’d then be able to test the script, schedule it, test the scheduling, and then I’d be done!

First things first – I was getting annoyed with not having some of my favourite packages installed on the slug, so:

# ipkg install less
Installing less (394-2) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/less_394-2_armeb.ipk
Installing ncursesw (5.5-1) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/ncursesw_5.5-1_armeb.ipk
Installing ncurses (5.5-1) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/ncurses_5.5-1_armeb.ipk
Configuring less
Configuring ncurses
Configuring ncursesw
# ipkg install bash
Installing bash (3.1-1) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/bash_3.1-1_armeb.ipk
Installing readline (5.1-1) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/readline_5.1-1_armeb.ipk
Configuring bash
Configuring readline
# ls /opt/bin/bash
# /opt/bin/bash
bash-3.1#

So, I edited /etc/passwd to make /opt/bin/bash the shell for root, logged out, then logged back in again.

OK, the next task was to installing s3sync somewhere sensible: I felt that /home/s3sync was a good enough place for the s3sync script itself and my own shell scripts, so I put everything there:

-bash-3.1# cd /home
-bash-3.1# mkdir s3sync
-bash-3.1# cd s3sync
-bash-3.1# mv /tmp/s3sync/* .
-bash-3.1# ls
HTTPStreaming.rb README.txt      README_s3cmd.txt S3.rb           S3_s3sync_mod.rb
S3encoder.rb     s3cmd.rb        s3sync.rb        s3try.rb        thread_generator.rb
-bash-3.1#

Next, it was necessary to install some root certificates so that it could use SSL to transfer data. Working from John Eberly’s post on how he set up s3sync, I did the following:

-bash-3.1# mkdir certs
-bash-3.1# cd certs
-bash-3.1# wget http://mirbsd.mirsolutions.de/cvs.cgi/~checkout~/src/etc/ssl.certs.shar
Connecting to mirbsd.mirsolutions.de[85.214.23.162]:80
-bash-3.1# sh ssl.certs.shar
x - 00869212.0
x - 052e396b.0
x - 0bb21872.0

…

x - f4996e82.0
x - f73e89fd.0
x - ff783690.0
-bash-3.1#

And now I could put in scripts to upload to S3, based on John Eberly’s:

-bash-3.1# cat > upload.sh
#!/opt/bin/bash
# script to sync local directory up to s3
cd /home/s3sync
export AWS_ACCESS_KEY_ID=<my key ID>
export AWS_SECRET_ACCESS_KEY=<my secret key>
export SSL_CERT_DIR=/home/s3sync/certs
./s3sync.rb -r --ssl --delete "/user data/Giles/Catalogue" <my key ID>.Backups:/remotefolder
-bash-3.1# chmod 700 upload.sh

The chmod was required to stop non-root users (of whom I naturally have hordes on the slug :-) from being able to read the private key. Better to be safe than sorry. The directory I was syncing is a very small subdirectory of the area I want to back up to S3.

Next, a download script:

-bash-3.1# cat > download.sh
#!/opt/bin/bash
# script to sync "directory" down from s3
cd /home/s3sync
export AWS_ACCESS_KEY_ID=<my key ID>
export AWS_SECRET_ACCESS_KEY=<my secret key>
export SSL_CERT_DIR=/home/s3sync/certs
./s3sync.rb -r --ssl --delete <my key ID>:/remotefolder/Catalogue/ /downloads/
-bash-3.1# chmod 700 download.sh
-bash-3.1#

Next, I created the <my key ID>.Backups bucket using jets3t Cockpit, and then ran the upload script:

-bash-3.1# ./upload.sh
-bash-3.1#

A quick check confirmed that the data had been uploaded. However, I found myself thinking – I’d like the tool to log a bit more than that. s3sync’s usage said that there was a “-v” option to run it in verbose mode, so I set that in the upload script and reran it. There was still no output, but I suspected that that was simply because there were no changes to upload… so I deleted the data from S3 using jets3t Cockpit, and reran. This time I got output:

-bash-3.1# ./upload.sh
Create node 19_Ejiri.jpg
Create node 22_Okabe.jpg
Create node 29_The_Original_Hachiman_Shrine_at_Suna_Village.jpg
Create node 47_Kameyama.jpg
-bash-3.1#

Time to test the download script (adding the -v to it first):

-bash-3.1# mkdir /downloads/
-bash-3.1# ./download.sh
Create node 19_Ejiri.jpg
Create node 22_Okabe.jpg
Create node 29_The_Original_Hachiman_Shrine_at_Suna_Village.jpg
Create node 47_Kameyama.jpg
-bash-3.1# ls -lrt /downloads/
-rwxrw----    1 guest    everyone   578008 Nov 14 22:30 19_Ejiri.jpg
-rwxrw----    1 guest    everyone   607822 Nov 14 22:30 22_Okabe.jpg
-rwxrw----    1 guest    everyone   563472 Nov 14 22:30 29_The_Original_Hachiman_Shrine_at_Suna_Village.jpg
-rwxrw----    1 guest    everyone   681194 Nov 14 22:31 47_Kameyama.jpg
-bash-3.1# ls -lrt /user\ data/Giles/Catalogue/
-rwxrw----    1 guest    everyone   607822 Mar 17  2005 22_Okabe.jpg
-rwxrw----    1 guest    everyone   578008 Mar 17  2005 19_Ejiri.jpg
-rwxrw----    1 guest    everyone   681194 Mar 17  2005 47_Kameyama.jpg
-rwxrw----    1 guest    everyone   563472 Mar 17  2005 29_The_Original_Hachiman_Shrine_at_Suna_Village.jpg

Hooray! So, finally, I decided to try syncing up my entire “user data” share on an cron job, set to execute very soon. I modified the upload.sh script to point to the correct directory, and then edited /etc/crontab, adding a line saying:

42 22 * * * root /home/s3sync/upload.sh &> /tmp/s3sync.log

And then I waited until 10:42pm by the slug’s time (which, incidentally, seemed to have drifted a minute or so since the previous evening). At 10:42pm, I checked what processes were running:

-bash-3.1# ps auxww
  PID TTY     Uid        Size State Command
    1         root       1212   S   /bin/init
    2         root          0   S   [keventd]

…

 1628 ttyp1   root       2100   S   -bash
 1715         root       2036   S   /opt/bin/bash /home/s3sync/upload.sh
 1716         root      12856   S   /opt/bin/ruby ./s3sync.rb -v -r --ssl --del
 1718 ttyp1   root       1984   R   ps auxww
-bash-3.1#

Excellent. The logfile was there; nothing had been written yet, but checking the bucket showed that data was already being copied up. My best guess was that the logfile would be flushed at a later point.

At this point, all I could really do was wait – so it was time to leave the slug for the day, ready to check the next. If everything had synchronised up correctly – and a download to another machine worked – then I would be able to say that I’d completed the project :-)

Next: Scheduling part 2

I’m setting up automated offsite backups from my NSLU2 to Amazon S3. With suprisingly little effort, I’ve managed to get a tool called s3sync running on the “slug” (as it’s known). s3sync is a Ruby script, so in order to run it, I had to install Ruby, which in turn meant that I had to replace the slug’s firmware with a different version of Linux, called Unslung. All of this worked pretty much as advertised in the tools’ respective documentation – for the details, see the previous posts in this series.

As all of the pieces were in place, I next needed to do some simple tests to make sure it could handle the kind of files I wanted it to back up. In particular, I wanted it to be able to handle deep directory hierarchies, and to remember user and group ownership and file permissions.

The first step was to create some test files.

# cd /tmp
# mkdir testdata
# cd testdata
# mkdir directorynumber1
# cd directorynumber1
# mkdir directorynumber2
# cd directorynumber2

…

# cd directorynumber21
# pwd
/tmp/testdata/directorynumber1/directorynumber2/directorynumber3/directorynumber4/directorynumber5/directorynumber6/directorynumber7/directorynumber8/directorynumber9/directorynumber10/directorynumber11/directorynumber12/directorynumber13/directorynumber14/directorynumber15/directorynumber16/directorynumber17/directorynumber18/directorynumber19/directorynumber20/directorynumber21
# cat > file000
000
# chmod 000 file000
# cat > file644
644
# chmod 644 file644
# cat > file777
777
# chmod 777 file777
# chown guest:nobody file777
# chown bin:administrators file000
# ls -lrt
----------    1 bin      administ        4 Nov 14  2006 file000
-rw-r--r--    1 root     root            4 Nov 14  2006 file644
-rwxrwxrwx    1 guest    nobody          4 Nov 14  2006 file777
#

So, I had some files with differing permissions and ownership, at the bottom of a directory hierarchy with over 350 characters in it – I had a vague impression that there might be a 200-character key limit on S3, and I’m always worried about 255-character limits, so 350 seemed like a sensible test length; if a system can manage 350, it can probably manage much larger figures, up to 32,767 or so… Anyway, the next step was to sync the whole thing up to S3:

# cd /tmp/s3sync/
# ./s3sync.rb -r /tmp/testdata <my key ID>.Test:yetanotherprefix
#

A quick check with jets3t Cockpit confirmed that everything was uploaded with appropriate-looking keys, and also with properties specifying decent-looking integer owner, group and permission values. This looked good – no key-length limit issues. However, there was only one way to be absolutely sure that it was working:

# ./s3sync.rb -r &ltmy key ID>.Test:yetanotherprefix/testdata/ /tmp/copytestdata
#

(Note the positions of the slashes, etc. – the full syntax for s3sync can take a while to work out, but the README documents it well if you take the time to read it…)

And then, to confirm that it’s OK:

# cd /tmp/copytestdata/directorynumber1/directorynumber2/directorynumber3/directorynumber4/directorynumber5/directorynumber6/directorynumber7/directorynumber8/directorynumber9/directorynumber10/directorynumber11/directorynumber12/directorynumber13/directorynumber14/directorynumber15/directorynumber16/directorynumber17/directorynumber18/directorynumber19/directorynumber20/directorynumber21/
# ls -lrt
-rw-r--r--    1 root     root            4 Nov 14 01:03 file644
----------    1 bin      administ        4 Nov 14 01:03 file000
-rwxrwxrwx    1 guest    nobody          4 Nov 14 01:03 file777
#

…which all looked correct!

So now I knew that s3sync would work from the NSLU2 to Amazon S3, that the file attributes I cared about were being persisted, and that deep directory hierarchies were not a problem. The next step would have to be to get it working with full SSL, as I don’t really want my private data flying over the public Internet unencrypted, and then to put the whole thing into a shell script and schedule a cron job to sync daily.

Next: SSL, and scheduling part 1

I’m setting up automated offsite backups from my NSLU2 to Amazon S3. The tool I need to use to make this happen is called s3sync; it’s a Ruby script, so in order to run it, I had to work out some way of installing Ruby. In order to do that, I had to replace the slug’s firmware with a different version of Linux, called Unslung; once that was done, getting Ruby up and running wasn’t too tricky. The next step was to get s2sync itself to work.

I started by getting the s3sync script itself copied over to the slug. It’s a gzipped tar archive, and is available from a web server – so the obvious tools to use were gunzip, tar, and wget. Conveniently, these were already installed as part of Unslung (or perhaps the original firmware):

# gunzip
gunzip: compressed data not read from terminal.  Use -f to force it.
# wget
BusyBox v1.00 (2006.04.11-01:22+0000) multi-call binary

Usage: wget [-c|--continue] [-q|--quiet] [-O|--output-document file]
                [--header 'header: value'] [-Y|--proxy on/off] [-P DIR] url

wget retrieves files via HTTP or FTP

Options:

…

# tar
BusyBox v1.00 (2006.04.11-01:22+0000) multi-call binary

Usage: tar -[czjZxtvO] [-f TARFILE] [-C DIR] [FILE(s)] ...

Create, extract, or list files from a tar file.

Options:

…

#

So, I created a temporary directory, downloaded, and unpacked the script:

# cd /tmp
# mkdir s3sync
# cd s3sync
# wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz
Connecting to s3.amazonaws.com[72.21.206.42]:80
# gunzip s3sync.tar.gz
# tar xf s3sync.tar
# ls
HTTPStreaming.rb README.txt      README_s3cmd.txt S3.rb
S3_s3sync_mod.rb S3encoder.rb    s3cmd.rb         s3sync.rb
s3sync.tar       s3try.rb         thread_generator.rb
#

The Ruby files, I noted, were not marked as executable, so I fixed that. I also remembered from when I was installing Ruby on the slug that the location of the interpreter installed by the standard Unslung package was not on the path – it was /opt/bin/ruby, and so the “#!” lines at the start of the scripts would probably need to be changed to reflect that. I checked the start of the s3sync scripts, and noticed that all of the top-level ones used the following first line:

#!/usr/bin/env ruby

This looked a bit odd to me – I’ve used env to list the environment, but never as a launcher for an interpreter. However, a quick poke around made me comfortable that it was just a way of avoiding putting an explicit path to the interpreter into the script file. As /usr/bin/env did not exist on the slug yet – though perhaps I could have installed it – I decided to modify the scripts to refer to the location of the ruby command on the machine.

The next steps were to set up the access key ID and the secret key, just as before:

# export AWS_ACCESS_KEY_ID=<my key ID>
# export AWS_SECRET_ACCESS_KEY=<my key>

…and to try running the script as a test, once more to synchronising the script’s own directory into the bucket I’d previously set up on S3 (with a different prefix for the keys to the one I used in my original test).

# ./s3sync.rb -r . <my key ID>.Test:adifferentprefix
./S3encoder.rb:42:in `iconv': invalid encoding ("UTF-8", "ISO-8859-1") (Iconv::InvalidEncoding)
        from ./S3encoder.rb:42:in `escape'
        from ./S3.rb:138:in `list_bucket'
        from ./s3sync.rb:21:in `map'
        from ./S3.rb:138:in `each'
        from ./S3.rb:138:in `map'
        from ./S3.rb:138:in `list_bucket'
        from ./s3try.rb:51:in `send'
        from ./s3try.rb:51:in `S3try'
        from ./s3sync.rb:244:in `s3TreeRecurse'
        from ./s3sync.rb:293:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:226:in `new'
        from ./s3sync.rb:226:in `main'
        from ./s3sync.rb:631
#

Oh dear. Well, I’d wanted to learn Ruby for some time, so here was a great incentive. The line causing the error, line 42 in S3encoder.rb, read:

result = Iconv.iconv("UTF-8", @nativeCharacterEncoding, string).join if @useUTF8InEscape

A bit of Googling around led to a (the?) Ruby documentation site, where from the page describing the Iconv class, it was clear that this call was a way of asking the runtime environment to convert the string in the variable string from the UTF-8 character set to whatever charset was specified in the variable (or perhaps field?) nativeCharacterEncoding. A few lines higher up, nativeCharacterEncoding appeared to be being set to “ISO-8859-1″, which made sense, especially given the error message.

However, this seemed strange – after all, UTF-8 is pretty much the standard character set for most new applications and systems, and ISO-8859-1, aka Latin-1, is the charset that predated it (and is used for most HTML). Still, the slug is a small embedded system – so perhaps, I thought, it might lack certain charsets? Might it be something dreadful like ASCII-only?

I decided to read through the list of available packages, to see if there was something obvious that needed to be installed – an “essential-charsets” package or something like that:

# ipkg list
abook - 0.5.5-1 - Abook is a text-based addressbook program designed to use with mutt mail client.
adduser - 1.1.3-6 - a multi-call binary for login and user account administration
adns - 1.3-2 - Asynchronous resolver library and DNS resolver utilities.

…

gambit-c - 4.0b20-1 - A portable implementation of Scheme.
gawk - 3.1.5-1 - Gnu AWK interpreter
gconv-modules - 2.2.5-5 - Provides gconv modules missing from the firmware.  These are used by glibc's iconv() implementation.

Now that last one looked promising – after all, as the Ruby documentation said:

Iconv is a wrapper class for the UNIX 95 iconv() function family, which translates string between various encoding systems.

I gave it a go:

# ipkg install gconv-modules
Installing gconv-modules (2.2.5-5) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/gconv-modules_2.2.5-5_armeb.ipk
Configuring gconv-modules
#

…and tried running the command again:

# ./s3sync.rb -r . <my key ID>.Test:adifferentprefix
S3 command failed:
list_bucket <my key ID>.TEST max-keys 200 prefix adifferentprefix/. delimiter /
With result 403 Forbidden
S3 ERROR: #
./s3sync.rb:249:in `+': can't convert nil into Array (TypeError)
        from ./s3sync.rb:249:in `s3TreeRecurse'
        from ./s3sync.rb:293:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:226:in `new'
        from ./s3sync.rb:226:in `main'
        from ./s3sync.rb:631
#

This was fantastic news! Although it had not synced, it had clearly contacted S3, and had been refused access – so the charset problem was, it appeared, solved.

Now, back when I tried to get s3sync to work on my Ubuntu box, I’d discovered that it would refuse to sync when the machine’s local time was skewed from the S3 server’s time. I’d foolishly forgotten to check the slug’s time before trying this sync, so before trying anything else I decided to check that it was OK:

# date
Tue Nov 14 02:49:31 GMT 2006

D’oh. It was 11:43pm on Monday 13 November when I typed that. So, I fixed it and tried again:

# date 11132344
Mon Nov 13 23:44:00 GMT 2006
# ./s3sync.rb -r . <my key ID>.Test:adifferentprefix
#

Which looked pretty good. I checked the S3 server, using the jets3t Cockpit tool that I’d used before, and lo and behold – the files were there!

So, I now had successfully used s3sync to synchronise a directory from my NSLU2 up to Amazon S3 – which was the main point of this project. While there was still a little bit of work to do – for example, making sure it worked with reasonably deep directory hierarchies, checking user/group ownership and permissions were persisted, setting up encryption, and setting up a cron job to automate the backup – the most tricky and experimental part of the work was done.

Next: File attributes, and deep directory hierarchies.

I’ve discovered that in order to get automated offsite backups from my NSLU2 to Amazon S3, I have to get it to run Ruby so that it can run s3sync. Installing Ruby required the slug’s firmware to be upgraded to a new version of Linux, called Unslung, so I did that – and I also installed s3sync on a Ubuntu machine as a dry run. Both of these worked out OK, so the next step was to get the Ruby language itself running under the Unslung firmware – and importantly, to make sure that Ruby had the OpenSSL package installed; the latter had proven non-obvious under regular Linux, so I was expecting problems on Unslung, which is, after all, a cut-down version of the operating system.

The first thing was to work out how to install new packages on an Unslung system in the first place. The last section of the installation README gives a link to this page, which is a good starting point. Going from the instructions there, I made sure that the gateway and DNS server were correctly set up on the slug’s web admin pages, then ran ipkg update from a telnet window logged into the unslung slug. This seemed to work just fine – appropriate messages about downloading, inflating and updating things went by. The next recommended steps were to run ipkg install unslung-feeds, and then ipkg update again. I’m sure there is a very good reason for this, but I’ve no idea what it is – for now, I just followed the instructions, bleating occasionally. Nothing obviously bad happened, and it looked like it managed to install some further lists of software.

The next step was to run ipkg list to see what I could now install. And wow, there were a lot of packages. Most importantly for this project, there was the ruby one, so…

# ipkg install ruby
Installing ruby (1.8.5-1) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/ruby_1.8.5-1_armeb.ipk
Configuring ruby
#

That looked promising – but…

# ruby --version
ruby: No such file or directory

Hmmm. So, how to find out where it’s been installed? Well, ipkg --help is apparently an unrecognised option, but when I tried it, it forced a printout of the options for ipkg anyway. This made it clear that I should run ipkg files ruby to find out where everything was, and from there I found out that it had installed the binary in /opt/bin/ – which is not unreasonable.

# /opt/bin/ruby --version
ruby 1.8.5 (2006-08-25) [armv5b-linux]

Looking good! Come to thing of it, that’s a more recent version of Ruby than the default for Ubuntu Linux :-)

Now, as I said earlier in this post, one worry I had was the difficulty of installing the Ruby OpenSSL libraries, which are a requirement for s3sync. I had noticed something relevant-looking as the list of files for the package went past earlier, so decided to double-check:

# ls /opt/lib/ruby/1.8/openssl/
bn.rb           buffering.rb    cipher.rb       digest.rb       ssl.rb
x509.rb

This was pretty promising – not a sure thing, but it looked good. So how to be sure – or, at least reasonably sure – that nothing was missing? Well, going back to the error message I got when s3sync failed on my Ubuntu machine (which, at the time, lacked an OpenSSL library for Ruby) I noted that I’d had the error

./S3.rb:25:in `require': no such file to load -- openssl (LoadError)

So, it sounded to me like the “require” command in Ruby loads libraries – perhaps somewhat like import in Python or Java, or using in C#. A quick grep through the s3sync source seemed to confirm this – the first few non-comment lines of S3.rb ran:

require 'base64'
require 'cgi'

…and then, shortly after, there was:

require 'openssl'

So, I could reasonably comfortably posit that if a Ruby script containing just that last line would run happily on the slug, s3sync should be able to run. On that basis, I created such a file and ran it:

# cd /tmp
# ls
# cat > test.rb
require 'openssl'
# /opt/bin/ruby test.rb
/opt/lib/ruby/1.8/armv5b-linux/openssl.so: libssl.so.0.9.7: cannot open shared object file: No such file or directory - /opt/lib/ruby/1.8/armv5b-linux/openssl.so (LoadError)
        from /opt/lib/ruby/1.8/openssl.rb:17
        from test.rb:1:in `require'
        from test.rb:1
#

Now, from this I suspected that the Ruby system had its own SSL stuff installed, and it was just the operating system’s shared library that was missing. In retrospect, this was not obvious – after all, the file that was missing was in /opt/lib/ruby. But, without noticing that, I decided to try installing any non-Ruby OpenSSL package that was out there:

# ipkg list | grep ssl
alac-decoder - 0.1.0-2 - A decoder for the apple lossless file format
flac - 1.1.2-4 - FLAC is a free lossless audio codec.  This package contains the codec libraries and the command-line tools flac and metaflac.
openssl - 0.9.7d-5 - Openssl provides the ssl implementation in libraries libcrypto and libssl, and is needed by many other applications and librari
perl-io-socket-ssl - 0.999-1 - IO-Socket-SSL - Nearly transparent SSL encapsulation for IO::Socket::INET
perl-net-ssleay - 1.30-1 - Net_SSLeay - Perl extension for using OpenSSL
wget-ssl - 1.10.2-3 - A network utility to retrieve files from the Web
# ipkg install openssl
Installing openssl (0.9.7d-5) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/openssl_0.9.7d-5_armeb.ipk
Configuring openssl
#

…and then tried running the script again:

# /opt/bin/ruby test.rb
#

Which looked perfect. Perhaps someone can tell me how that all worked… or perhaps I’ll work it out myself later. But for now, onward! The next step was clearly to copy over s3sync, and to try it out on a simple directory. And that’s one for tomorrow.

Next: Running s3sync on an Unslung NSLU2.

I’ve discovered that in order to get automated offsite backups from my NSLU2 to Amazon S3, I have to get it to run Ruby so that it can run s3sync. I’ve installed the latter on a Ubuntu machine as a dry run, and it all looks good – so now I need to get Ruby onto the slug, and the first step in that direction is to install Unslung.

I won’t detail what you have to do to install Unslung, as that’s likely to change, and the last thing I want is for someone to land here in a few months time, follow my out-of-date instructions, and destroy their hardware. So, suffice it to say – I followed the instructions at the Unslung New User’s Guide and in the README file that came with the download – and I followed them exactly. I’ve supported enough software to know that most problems encountered by experienced users come from overconfidence and not following instructions… so I don’t want to be one of the people I’ve cursed in the past.

All that said, I did encounter one problem that required a quick detour; once I had done the initial checks, and tried to upload the Unslung firmware from the NSLU2’s web admin pages, I got an error dialog saying “upgrade: no enough free space” (sic). A quick poke around with Google made it clear that some people had fixed similar-sounding problems using a tool called “EraseAll” to clear out the slug’s flash RAM prior to doing an upgrade – but a quick check at the NSLU2-Linux site made it clear that they thought that using this tool was a terrible idea! Further investigation showed that there was a FAQ on the NSLU2-Linux site regarding similar (but different) messages, saying that out of memory errors when flashing

may be due to a SLUG using an older Linksys FW. My 2nd SLUG had the .24 version and faced this issue even after a factory reset and going straight to the download page. Linksys has a help page that will walk you through upgrading to the .63 FW. Once I did this, upgrading to the Unslung FW was per the install guide.

My slug is very old – its firmware was also the antediluvian V2.3R24 – so I kind of suspected that was the problem… I decided to upgrade to the latest Linksys firmware, and went to the page linked from the FAQ. Worryingly, that page said that you should use EraseAll, which I was feeling quite nervous of – so before trying to follow its instructions, I decided to see what would happen if I used the regular web-based interface on the NSLU2 to upgrade to the new V2.3R63 firmware (which is linked at the bottom of the page). To my delight, that seemed to work – at least the admin pages said that it was now version 2.3R63 – so I powered the unit down, plugged my drive back in, powered up again, and checked that the disk was still accessible. It was, so I started the Unslung install procedure from the README once more, from the top. This time, it seemed to work – the process completed without errors, the unit rebooted, and when (after a nail-biting few minutes) it came up, I checked the web interface, and it reported that its firmware version was V2.3R63-uNSLUng-6.8-beta.

Trying to avoid becoming cocky, I completed the instructions in the README, and updated the NSLU2-Linux Wiki to reflect my experiences). Once all of this was done, I had “unslung” my NSLU2 to the drive USB drive attached to it – or, in less jargonny terms, I had changed the unit’s settings so that no longer did it boot entirely from the build-in flash memory, afterwards mounting the USB drive separately for sharing, but instead it used the built-in memory as some kind of low-level bootstrap system to mount the USB disk as the root filesystem, and then booted from that. That probably sounds more complicated than it is – all I really needed to do was robotically follow the instructions in the README.

So, now that’s all installed, the next step is to install Ruby.

Next: Installing Ruby on an Unslung NSLU2.

I now know that in order to get automated offsite backups from my NSLU2 to Amazon S3, I have to get it to run Ruby so that it can run s3sync. I want to back everything up first, which is going to take some time – so while that’s happening, I’ll get both Ruby and s3sync installed on an Ubuntu Linux machine as a dry run.

Firstly, I need to download s3sync from its homepage, and unzip/untar it into a convenient directory.

The next step is to install Ruby. The machine I am using to test this is running Ubuntu 6.10 (”Edgy Eft”), and to install standard packages on it you use a tool called Synaptic Package Manager. This has somewhat conflicting information regarding Ruby. The basic package – called, unsurprisingly, ruby – claims to be version 1.8.2, and s3sync requires 1.8.4 or higher. There is also a ruby1.8 package, which appears to be version 1.8.4-5ubuntu1.1, so that sounds like it might be a good one to use. However, it states in its notes that “on Debian, Ruby 1.8 is provided as separate packages. You can get full Ruby 1.8 distribution by installing following packages. ruby1.8 ruby1.8-dev” etc. Most of the listed packages do not show up in the package manager. To make things even more confusing, the ruby package appears to depend on the ruby1.8 one, which implies that it actually is version 1.8.4… To keep things simple, I installed the ruby package, and then typed ruby ----version at the command line; it told me that it was version 1.8.4. OK.

The next step was to see if let’s see if I could sync something with my S3 account. John Eberly has kindly blogged the details of how he set up s3sync on his system, and it’s clear that setting up a bucket on S3 in advance using a separate application is a good idea; so, I downloaded jets3t Cockpit as he suggests and tried it out. (Note – you need Java 1.4 or higher installed to run it – as an old Java hand using a newish machine, I was surprised to notice that I’d not yet installed it on my workstation. That’s what switching to IronPython does to you.)

The jets3t Cockpit is a pretty simple app – you can set up multiple logins in a “saved login” screen, or you can log in directly using the Access Key ID/Secret Access Key that Amazon provide. Once you’re logged in, it is pretty trivial to create a new bucket – and, because all S3 buckets exist in the same namespace (even across users!), it sensibly suggests bucket names that start with your user ID. So, I created a bucket called <my-access-key-id>.Test

Now, let’s see if we can backup the s3sync install directory itself over to S3. Firstly, s3sync needs to know the access key ID and secret key:

# export AWS_ACCESS_KEY_ID=<my key ID>
# export AWS_SECRET_ACCESS_KEY=<my key>

And now, we can run the sync:

# ./s3sync.rb -r . <my key ID>.Test:aprefix

The first time I tried this, it failed – the Ruby files are not +x… Grumble. OK, trying again:

# chmod +x *.rb
# ./s3sync.rb -r . <my key ID>.Test:aprefix

Hmmm. I got an error message:

./S3.rb:25:in `require': no such file to load -- openssl (LoadError)

Interesting. Wild guesswork followed: I know that the default package list of Ubuntu does not contain the full list of available packages – just those that are strictly Open Source (for some value of “Open Source”). I had a vague memory from somewhere, some time back, that OpenSSL has some odd licensing restrictions. Perhaps the Ruby SSL package is not part of the standard Ruby package because of this? That might also be the case with the other parts of the Ruby distribution mentioned in the note to the ruby package I mentioned above – which would explain why they were missing from the package manager.

Working on this basis, I checked my Settings/Repositories dialog in the package manager, and noted that it was only looking at the “main” source of software – “Canonical supported Open Source software”. I set the checkboxes for the “universe” and “multiverse” sources as well – for community-maintained packages and those with copyright restrictions – and searched for OpenSSL packages. Bang – I got the libopenssl-ruby1.8 package, which had not previously been visible. I installed it and tried again…

…and the result was better, but still not great – this time I got a 403 “Access forbidden” error. My first guess was that I must have mistyped either the access key ID or the key itself…. but further inspection (and a copy/paste to make absolutely sure) showed that that was not the case. Another thought occured to me – it might be a problem with simultaneous access from jets3t Cockpit and from s3sync, so I tried shutting the former down – but to no avail.

OK, time to investigate. A quick check on the thread where the script was announced showed that one other person was getting the same response (you can see his comment if you search for “403″), but it was because he’d not replaced the string “bucket” in the sample command line with his bucket name – whoops. However, his second post, a little further down, gives an example of a call to a lower-level script that simply sends an S3 command and shows the result. I decided to try something like that:

# ./s3cmd.rb -dv list <my key ID>.Test 200

…and I got a message saying that there is too much of a clock skew between the “request time” and the current time. A quick check, and – aha! – my Ubuntu box’s clock is out by half an hour O_o. Fixing that, and then re-running the list command made it work!

# ./s3cmd.rb -dv list <my key ID>.Test 200
list <my key ID>.Test: 200  {}
--------------------
200
Trying command list_bucket <my key ID>.Test max-keys 200
prefix   with 100 retries left
Response code: 200
#

So, I retried the s3sync, and it returned with no output – a fine Unixy way of saying that it thinks it succeeded. I checked what I had stored in the bucket on on S3 using jets3t Cockpit…. and w00t! It was all there, with sensible keys.

Next, I waited for a few minutes, then ran the command again to see if it really was synchronising rather than simply copying. Of course, it was hard to be sure – but the command returned much more quickly than last time, and the file timestamps on S3 didn’t change their modified time – which would seem to imply that it didn’t try to re-write them.

As a final check, I touched one of the files, and synced again… but it was just as quick and there was no update to the modified time. Thinking that perhaps it was using hashes rather than the update time ^[1], I tried changing the file using vi and syncing again… and this time the command took slightly longer to return, and the modified time updated.

At that point I decided I was confident that s3sync was working as it should – and conveniently, the backup had completed. So it was time for the next step – installing the Unslung firmware on the slug.

Next: Installing Unslung.

^[1] After perusing the thread on the Amazon site more closely, I discovered that yes, it does use hashes – it compares the MD5 of the file with a hash stored on S3 that happens to be the MD5.

In order to get automated offsite backups from my NSLU2 to Amazon S3, I’ve determined I need to use s3sync, a Ruby script. Obviously, this means that I need to get Ruby running on the “slug”.

As I noted earlier, the standard firmware will not support Ruby, so the first step is going to have to be to install new firmware. The matrix of possibilities on the NSLU2-Linux site lists a bunch. My gut instinct is to stay as close to the original firmware – to the left of the matrix – as possible. I’ve been using Linux for a long time now – on and off since 1992 – but I’ve never really got to a serious kernel-recompiling porting-it-to-a-ZX81 level with it. So let’s keep things as simple as possible.

Conveniently, it looks like Unslung, the most simple replacement firmware, and one that is compatible with the original firmware – hopefully meaning that I won’t need to do anything like reformat the disk I currently have attached – has some level of support for Ruby. At least, there is a Ruby package in the list of “cross packages”. Brief investigation implies that I may have to compile the package myself – hence the “cross” – but hopefully that won’t be too tricky.

Before doing anything else, I need to safeguard my data. Although it’s all backed up on the IBackup system I mentioned earlier, it’s better to be safe than sorry – so, the first step is to copy it all over to my main workstation’s hard drive.

Once this is done, it’s time to install. There is a new user’s guide on the NSLU2-Linux site with strict instructions to follow the README, which lives next to the binaries on Slug-Firmware.net – they have a complicated click-through license that I shan’t deep-link past. The README is very detailed, and is full of dire warnings about the consequences of not following it.

As it will take a while to back up the drive, let’s try out s3sync on a regular Linux box while we wait.

Next: s3sync on Ubuntu.

I am trying to get my NSLU2 to back itself up automatically to Amazon S3. I currently know that in order to get this to work, the “slug” (as it’s affectionally known) will need to be upgraded with new firmware – basically, a new version of Linux. Just which of the many competing firmwares is appropriate will depend on the software I use to do the sync, and so it’s time to work through the various options presented in Jeremy Zawodny’s post and the comments people have left there.

• Firstly, we have s3sync. This appears to be hosted at the S3 site, which is promising. However, it’s written in Ruby. I have nothing against the language – indeed, it’s next on the list of languages I want to learn – but it’s not precisely lightweight, and so would almost certainly mean that I cannot fit the complete boot image for the OS into its firmware; some stuff will need to go on the disk, and that feels somehow… inelegant. If the other options have the same constraint, then maybe this will be the one to go for. But for now I’ll see if there’s anything more lightweight.
• Backup Manager looks like a nice tool for building full backups, which can then be pushed offsite. It’s written in Bash and Perl – the latter meaning that it would probably require as much stuff to be installed on the machine as Ruby would, so a bit of a problem there. When you combine this with the fact that it’s not really designed for simple synchronisation with offsite systems, it doesn’t sound quite like the kind of thing I’m looking for.
• S3DAV does just what you’d expect it to do – it creates a WebDAV interface to your S3 account. This looks interesting. It’s written in Java, so would require quite a lot of stuff to be installed on the NSLU2 (I’m beginning to see a pattern here) and would require some kind of rsync-to-WebDAV software on top, however – so potentially more work than, say s3sync. Worth keeping an eye on, though, especially because…
• …of Duplicity, which uses the rsync algorithm to produce “bandwidth-efficient” archives. Unfortunately, it’s not quite ready for prime-time: it “should continue to work fine until you depend on it for your business or to protect important personal data”. Again, one to check at a later date, perhaps.
• Similarly, Fuse looks like it might be usable in some time – but “[n]o warranties and if you screw up then don’t blame me, this is pre-alpha code meaning it might not work or worse screw up your system.” So probably not safe for non-PC hardware.
• Sync2S3 looks like a nice tool, but is closed-source Windows-only.
• …as is S3 Backup…
• …and BigSafeBox…
• …and Altexa…
• …JungleDisk does support Linux, but it’s closed-source with a binary distro – so I doubt it could be persuaded to work on the ARM processor on the NSLU2…
• …and finally, SuperSync seems to be designed for music files only.

So, for my fairly low-level simple needs, it looks like there’s one clear winner – s3sync. This is going to need stuff to be installed on the NSLU2, but that’s true of any other option I saw.

The next steps? To play with s3sync on a regular Linux box so that I can work out how it is meant to operate normally, and to get a Ruby platform up and running on the NSLU2. The latter is likely to be the more difficult, so I’ll start looking at that first.

Next: Upgrading the firmware – a first look.