Monthly Archives: November 2006

Drop that zero…

30 November 2006

Sometimes the .NET Clipboard.GetData() method returns a MemoryStream full of zeros, even when you set it to something else just microseconds before, and even though nothing else is running that you’d expect to be putting stuff on the clipboard. In some of the unit tests that we have at Resolver, this happens as often as once in fifty calls – which can cause problems when you’re running the tests on an integration machine all day, every day.

We were looking around on the ‘net, trying to fix this, when we discovered that Google is a Vanilla Ice fan :-)

Back again

29 November 2006

Things have been busy for the last week or so, especially at Resolver, where we are moving inexorably toward a public launch. Normal blogging service will be resumed shortly.

Project: Automated offsite backups for an NSLU2 – part 13

17 November 2006

Previously in this series: Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8, Part 9, Part 10, Part 11, Part 13.

I’m setting up automated offsite backups from my NSLU2 to Amazon S3. With suprisingly little effort, I’ve managed to get a tool called s3sync running on the “slug” (as it’s known). s3sync is a Ruby script, so in order to run it, I had to install Ruby, which in turn meant that I had to replace the slug’s firmware with a different version of Linux, called Unslung. Once all of this was done, I just had to set up the appropriate directory structures and certificates so that the sync tool could use SSL, and write a simple upload/download script. All of this worked pretty much as advertised in the tools’ respective documentation – for the details, see the previous posts in this series.

My final step had been to set up a cron job to run the upload script, but it had failed, not logging anything. In order to debug, I ran the upload script directly from the command line, and left it to run overnight, copying a large set of directories to S3.

13 hours later, it had completed. From the jet3St Cockpit, I checked how much data was present in the bucket; it told me I had 1.61Gb, split over 2774 items. This seemed a little on the low side, but I had to get back to my workstation to be sure. And there, the same program told me that I had 1.71Gb, split over 2770 items. Checking the console showed that the command had, it thought, succeeded with no errors – but and the directory that was meant to be synced claimed to be about 4Gb in size!

A quick investigation showed that there were certainly files missing from S3. I decided to see what would happen if I ran it again – would it start uploading where it left off?

I suspect sorting this problem out may take a certain amount of poking around over a number of days, so I won’t post again in this series until I’ve found the solution.

[Update] Still hard at work on this; it looks like there’s a problem with s3sync making it cut out after some amount of transfer, so I’m trying to diagnose the problem – which is tricky when each exeperiment takes 24 hours :-/ Final results will be posted here when I have them.

Project: Automated offsite backups for an NSLU2 – part 12

16 November 2006

Previously in this series: Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8, Part 9, Part 10, Part 11.

I’m setting up automated offsite backups from my NSLU2 to Amazon S3. With suprisingly little effort, I’ve managed to get a tool called s3sync running on the “slug” (as it’s known). s3sync is a Ruby script, so in order to run it, I had to install Ruby, which in turn meant that I had to replace the slug’s firmware with a different version of Linux, called Unslung. Once all of this was done, I just had to set up the appropriate directory structures and certificates so that the sync tool could use SSL, and write a simple upload/download script. All of this worked pretty much as advertised in the tools’ respective documentation – for the details, see the previous posts in this series.

My final step in my last post was to set up a cron job to synchronise quite a lot of data up to S3 overnight; here is the line from the crontab file:

42 22 * * * root /home/s3sync/upload.sh &> /tmp/s3sync.log

Checking this morning brought some bad news. Nothing had been written to the log file, and the bucket I’d set up to receive the backup on S3 had only 6Mb of data – as compared to a total of 4Gb+ that was there to be backed up.

Clearly something had gone wrong.

I figured it was best to try again, this time trying to eliminate whatever problem had occurred with the cron job by simply running the backup script from a command prompt. After all, I had run the script from a command line previously, and had seen some useful logging information.

This time it at least seemed to be logging something:

-bash-3.1# /home/s3sync/upload.sh
Create node Giles

I left it for an hour or so, after which it had uploaded 141.25Mb, logging all the while. Clearly there was (a) something wrong with the way I had set up logging from the crontab, and (b) something had interrupted it when it had run the previous night. After a little thought, I came to the conclusion that it might not ba a great idea to have something in the crontab that could take multiple hours to run; there could well be a limit, at least in the version of the cron daemon that lives on the NSLU2, and the sync process might have been killed before it was able to sync its output to the log file. That said, I could find no mention of such a thing on the obvious page on the NSLU2-Linux site. I decided to ask the site’s mailing list, to see if anyone knew for sure if this was the answer; in the meantime, I watched the sync from the command line as it reached 683 items and 270Mb.

MSBuild WTF: “The error was:”

15 November 2006

Here’s a fun one for anyone who uses msbuild (at least, v2.0.50727). Create a project file like this:

<Project DefaultTargets="Foo" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="Foo">
        <Exec Command = "echo Hello!" />
  </Target>
</Project>

From a command prompt, run the project; you will get a the effect you would expect.

Now replace the word “Hello” with “The error was: something random”. Run it again.

C:\\Dev\\Resolver>msbuild foo.proj
Microsoft (R) Build Engine Version 2.0.50727.42
[Microsoft .NET Framework, Version 2.0.50727.42]
Copyright (C) Microsoft Corporation 2005. All rights reserved.

Build started 15/11/2006 17:50:22.
__________________________________________________
Project "C:\\Dev\\Resolver\\foo.proj" (default targets):

Target Foo:
    echo The error was: something random
    EXEC : The error was: something random
    C:\\Dev\\Resolver\\foo.proj(3,9): error MSB3073: The command "echo The error was: something random" exited with code -1.
Done building target "Foo" in project "foo.proj" -- FAILED.

Done building project "foo.proj" -- FAILED.

Build FAILED.
EXEC : The error was:
C:\\Dev\\Resolver\\foo.proj(3,9): error MSB3073: The command "echo The error was: something random" exited with code -1.
    0 Warning(s)
    2 Error(s)

Time Elapsed 00:00:00.09

C:\\Dev\\Resolver>

Fuzzyman and I bumped into this one at work today; our continuous integration server, which watches our Subversion repository and checks out, builds, and tests any code changes it sees, had reported a failure despite the fact that none of the tests had failed. It turned out that one test was quite innocently printing out the text “The error was: ” followed by some logging information; it wasn’t an error at all. As far as I can tell, the statement that the echo command exited with code -1 is absolute nonsense.

This behaviour is not documented anywhere that we were able to find; I can only assume it was added for some specific purpose in the context of Visual Studio…

Project: Automated offsite backups for an NSLU2 – part 11

14 November 2006

Previously in this series: Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8, Part 9, Part 10.

I’m setting up automated offsite backups from my NSLU2 to Amazon S3. With suprisingly little effort, I’ve managed to get a tool called s3sync running on the “slug” (as it’s known). s3sync is a Ruby script, so in order to run it, I had to install Ruby, which in turn meant that I had to replace the slug’s firmware with a different version of Linux, called Unslung. All of this worked pretty much as advertised in the tools’ respective documentation – for the details, see the previous posts in this series.

Having confirmed that s3sync worked as I’d expect it to, I needed to install it in a sensible place – I’d previously just put it in /tmp – set it up so that I could use SSL to encrypt the data while it was on its way to Amazon, and then write a script to synchronise at least one of the directories I want backed up. I’d then be able to test the script, schedule it, test the scheduling, and then I’d be done!

First things first – I was getting annoyed with not having some of my favourite packages installed on the slug, so:

# ipkg install less
Installing less (394-2) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/less_394-2_armeb.ipk
Installing ncursesw (5.5-1) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/ncursesw_5.5-1_armeb.ipk
Installing ncurses (5.5-1) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/ncurses_5.5-1_armeb.ipk
Configuring less
Configuring ncurses
Configuring ncursesw
# ipkg install bash
Installing bash (3.1-1) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/bash_3.1-1_armeb.ipk
Installing readline (5.1-1) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/readline_5.1-1_armeb.ipk
Configuring bash
Configuring readline
# ls /opt/bin/bash
# /opt/bin/bash
bash-3.1#

So, I edited /etc/passwd to make /opt/bin/bash the shell for root, logged out, then logged back in again.

OK, the next task was to installing s3sync somewhere sensible: I felt that /home/s3sync was a good enough place for the s3sync script itself and my own shell scripts, so I put everything there:

-bash-3.1# cd /home
-bash-3.1# mkdir s3sync
-bash-3.1# cd s3sync
-bash-3.1# mv /tmp/s3sync/* .
-bash-3.1# ls
HTTPStreaming.rb README.txt      README_s3cmd.txt S3.rb           S3_s3sync_mod.rb 
S3encoder.rb     s3cmd.rb        s3sync.rb        s3try.rb        thread_generator.rb
-bash-3.1#

Next, it was necessary to install some root certificates so that it could use SSL to transfer data. Working from John Eberly’s post on how he set up s3sync, I did the following:

-bash-3.1# mkdir certs
-bash-3.1# cd certs
-bash-3.1# wget http://mirbsd.mirsolutions.de/cvs.cgi/~checkout~/src/etc/ssl.certs.shar
Connecting to mirbsd.mirsolutions.de[85.214.23.162]:80
-bash-3.1# sh ssl.certs.shar
x - 00869212.0
x - 052e396b.0
x - 0bb21872.0

x - f4996e82.0
x - f73e89fd.0
x - ff783690.0
-bash-3.1#

And now I could put in scripts to upload to S3, based on John Eberly’s:

-bash-3.1# cat > upload.sh
#!/opt/bin/bash
# script to sync local directory up to s3
cd /home/s3sync
export AWS_ACCESS_KEY_ID=<my key ID>
export AWS_SECRET_ACCESS_KEY=<my secret key>
export SSL_CERT_DIR=/home/s3sync/certs
./s3sync.rb -r --ssl --delete "/user data/Giles/Catalogue" <my key ID>.Backups:/remotefolder
-bash-3.1# chmod 700 upload.sh

The chmod was required to stop non-root users (of whom I naturally have hordes on the slug :-) from being able to read the private key. Better to be safe than sorry. The directory I was syncing is a very small subdirectory of the area I want to back up to S3.

Next, a download script:

-bash-3.1# cat > download.sh
#!/opt/bin/bash
# script to sync "directory" down from s3
cd /home/s3sync
export AWS_ACCESS_KEY_ID=<my key ID>
export AWS_SECRET_ACCESS_KEY=<my secret key>
export SSL_CERT_DIR=/home/s3sync/certs
./s3sync.rb -r --ssl --delete <my key ID>:/remotefolder/Catalogue/ /downloads/
-bash-3.1# chmod 700 download.sh
-bash-3.1#

Next, I created the <my key ID>.Backups bucket using jets3t Cockpit, and then ran the upload script:

-bash-3.1# ./upload.sh
-bash-3.1# 

A quick check confirmed that the data had been uploaded. However, I found myself thinking – I’d like the tool to log a bit more than that. s3sync’s usage said that there was a “-v” option to run it in verbose mode, so I set that in the upload script and reran it. There was still no output, but I suspected that that was simply because there were no changes to upload… so I deleted the data from S3 using jets3t Cockpit, and reran. This time I got output:

-bash-3.1# ./upload.sh
Create node 19_Ejiri.jpg
Create node 22_Okabe.jpg
Create node 29_The_Original_Hachiman_Shrine_at_Suna_Village.jpg
Create node 47_Kameyama.jpg
-bash-3.1#

Time to test the download script (adding the -v to it first):

-bash-3.1# mkdir /downloads/
-bash-3.1# ./download.sh
Create node 19_Ejiri.jpg
Create node 22_Okabe.jpg
Create node 29_The_Original_Hachiman_Shrine_at_Suna_Village.jpg
Create node 47_Kameyama.jpg
-bash-3.1# ls -lrt /downloads/
-rwxrw----    1 guest    everyone   578008 Nov 14 22:30 19_Ejiri.jpg
-rwxrw----    1 guest    everyone   607822 Nov 14 22:30 22_Okabe.jpg
-rwxrw----    1 guest    everyone   563472 Nov 14 22:30 29_The_Original_Hachiman_Shrine_at_Suna_Village.jpg
-rwxrw----    1 guest    everyone   681194 Nov 14 22:31 47_Kameyama.jpg
-bash-3.1# ls -lrt /user\ data/Giles/Catalogue/
-rwxrw----    1 guest    everyone   607822 Mar 17  2005 22_Okabe.jpg
-rwxrw----    1 guest    everyone   578008 Mar 17  2005 19_Ejiri.jpg
-rwxrw----    1 guest    everyone   681194 Mar 17  2005 47_Kameyama.jpg
-rwxrw----    1 guest    everyone   563472 Mar 17  2005 29_The_Original_Hachiman_Shrine_at_Suna_Village.jpg

Hooray! So, finally, I decided to try syncing up my entire “user data” share on an cron job, set to execute very soon. I modified the upload.sh script to point to the correct directory, and then edited /etc/crontab, adding a line saying:

42 22 * * * root /home/s3sync/upload.sh &> /tmp/s3sync.log

And then I waited until 10:42pm by the slug’s time (which, incidentally, seemed to have drifted a minute or so since the previous evening). At 10:42pm, I checked what processes were running:

-bash-3.1# ps auxww
  PID TTY     Uid        Size State Command
    1         root       1212   S   /bin/init
    2         root          0   S   [keventd]

 1628 ttyp1   root       2100   S   -bash
 1715         root       2036   S   /opt/bin/bash /home/s3sync/upload.sh
 1716         root      12856   S   /opt/bin/ruby ./s3sync.rb -v -r --ssl --del
 1718 ttyp1   root       1984   R   ps auxww
-bash-3.1#

Excellent. The logfile was there; nothing had been written yet, but checking the bucket showed that data was already being copied up. My best guess was that the logfile would be flushed at a later point.

At this point, all I could really do was wait – so it was time to leave the slug for the day, ready to check the next. If everything had synchronised up correctly – and a download to another machine worked – then I would be able to say that I’d completed the project :-)

Next: Scheduling part 2

Project: Automated offsite backups for an NSLU2 – part 10

14 November 2006

Previously in this series: Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8, Part 9.

I’m setting up automated offsite backups from my NSLU2 to Amazon S3. With suprisingly little effort, I’ve managed to get a tool called s3sync running on the “slug” (as it’s known). s3sync is a Ruby script, so in order to run it, I had to install Ruby, which in turn meant that I had to replace the slug’s firmware with a different version of Linux, called Unslung. All of this worked pretty much as advertised in the tools’ respective documentation – for the details, see the previous posts in this series.

As all of the pieces were in place, I next needed to do some simple tests to make sure it could handle the kind of files I wanted it to back up. In particular, I wanted it to be able to handle deep directory hierarchies, and to remember user and group ownership and file permissions.

The first step was to create some test files.

# cd /tmp
# mkdir testdata
# cd testdata
# mkdir directorynumber1
# cd directorynumber1
# mkdir directorynumber2
# cd directorynumber2

# cd directorynumber21
# pwd
/tmp/testdata/directorynumber1/directorynumber2/directorynumber3/directorynumber4/directorynumber5/directorynumber6/directorynumber7/directorynumber8/directorynumber9/directorynumber10/directorynumber11/directorynumber12/directorynumber13/directorynumber14/directorynumber15/directorynumber16/directorynumber17/directorynumber18/directorynumber19/directorynumber20/directorynumber21
# cat > file000
000
# chmod 000 file000
# cat > file644
644
# chmod 644 file644
# cat > file777
777
# chmod 777 file777
# chown guest:nobody file777
# chown bin:administrators file000
# ls -lrt
----------    1 bin      administ        4 Nov 14  2006 file000
-rw-r--r--    1 root     root            4 Nov 14  2006 file644
-rwxrwxrwx    1 guest    nobody          4 Nov 14  2006 file777
#

So, I had some files with differing permissions and ownership, at the bottom of a directory hierarchy with over 350 characters in it – I had a vague impression that there might be a 200-character key limit on S3, and I’m always worried about 255-character limits, so 350 seemed like a sensible test length; if a system can manage 350, it can probably manage much larger figures, up to 32,767 or so… Anyway, the next step was to sync the whole thing up to S3:

# cd /tmp/s3sync/
# ./s3sync.rb -r /tmp/testdata <my key ID>.Test:yetanotherprefix
#

A quick check with jets3t Cockpit confirmed that everything was uploaded with appropriate-looking keys, and also with properties specifying decent-looking integer owner, group and permission values. This looked good – no key-length limit issues. However, there was only one way to be absolutely sure that it was working:

# ./s3sync.rb -r <my key ID>.Test:yetanotherprefix/testdata/ /tmp/copytestdata
#

(Note the positions of the slashes, etc. – the full syntax for s3sync can take a while to work out, but the README documents it well if you take the time to read it…)

And then, to confirm that it’s OK:

# cd /tmp/copytestdata/directorynumber1/directorynumber2/directorynumber3/directorynumber4/directorynumber5/directorynumber6/directorynumber7/directorynumber8/directorynumber9/directorynumber10/directorynumber11/directorynumber12/directorynumber13/directorynumber14/directorynumber15/directorynumber16/directorynumber17/directorynumber18/directorynumber19/directorynumber20/directorynumber21/
# ls -lrt
-rw-r--r--    1 root     root            4 Nov 14 01:03 file644
----------    1 bin      administ        4 Nov 14 01:03 file000
-rwxrwxrwx    1 guest    nobody          4 Nov 14 01:03 file777
#

…which all looked correct!

So now I knew that s3sync would work from the NSLU2 to Amazon S3, that the file attributes I cared about were being persisted, and that deep directory hierarchies were not a problem. The next step would have to be to get it working with full SSL, as I don’t really want my private data flying over the public Internet unencrypted, and then to put the whole thing into a shell script and schedule a cron job to sync daily.

Next: SSL, and scheduling part 1

Project: Automated offsite backups for an NSLU2 – part 9

14 November 2006

Previously in this series: Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8.

I’m setting up automated offsite backups from my NSLU2 to Amazon S3. The tool I need to use to make this happen is called s3sync; it’s a Ruby script, so in order to run it, I had to work out some way of installing Ruby. In order to do that, I had to replace the slug’s firmware with a different version of Linux, called Unslung; once that was done, getting Ruby up and running wasn’t too tricky. The next step was to get s2sync itself to work.

I started by getting the s3sync script itself copied over to the slug. It’s a gzipped tar archive, and is available from a web server – so the obvious tools to use were gunzip, tar, and wget. Conveniently, these were already installed as part of Unslung (or perhaps the original firmware):

# gunzip
gunzip: compressed data not read from terminal.  Use -f to force it.
# wget
BusyBox v1.00 (2006.04.11-01:22+0000) multi-call binary

Usage: wget [-c|--continue] [-q|--quiet] [-O|--output-document file]
                [--header 'header: value'] [-Y|--proxy on/off] [-P DIR] url

wget retrieves files via HTTP or FTP

Options:

# tar
BusyBox v1.00 (2006.04.11-01:22+0000) multi-call binary

Usage: tar -[czjZxtvO] [-f TARFILE] [-C DIR] [FILE(s)] ...

Create, extract, or list files from a tar file.

Options:

#

So, I created a temporary directory, downloaded, and unpacked the script:

# cd /tmp
# mkdir s3sync
# cd s3sync
# wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz
Connecting to s3.amazonaws.com[72.21.206.42]:80
# gunzip s3sync.tar.gz
# tar xf s3sync.tar
# ls
HTTPStreaming.rb README.txt      README_s3cmd.txt S3.rb           
S3_s3sync_mod.rb S3encoder.rb    s3cmd.rb         s3sync.rb
s3sync.tar       s3try.rb         thread_generator.rb
#

The Ruby files, I noted, were not marked as executable, so I fixed that. I also remembered from when I was installing Ruby on the slug that the location of the interpreter installed by the standard Unslung package was not on the path – it was /opt/bin/ruby, and so the “#!” lines at the start of the scripts would probably need to be changed to reflect that. I checked the start of the s3sync scripts, and noticed that all of the top-level ones used the following first line:

#!/usr/bin/env ruby

This looked a bit odd to me – I’ve used env to list the environment, but never as a launcher for an interpreter. However, a quick poke around made me comfortable that it was just a way of avoiding putting an explicit path to the interpreter into the script file. As /usr/bin/env did not exist on the slug yet – though perhaps I could have installed it – I decided to modify the scripts to refer to the location of the ruby command on the machine.

The next steps were to set up the access key ID and the secret key, just as before:

# export AWS_ACCESS_KEY_ID=<my key ID>
# export AWS_SECRET_ACCESS_KEY=<my key>

…and to try running the script as a test, once more to synchronising the script’s own directory into the bucket I’d previously set up on S3 (with a different prefix for the keys to the one I used in my original test).

# ./s3sync.rb -r . <my key ID>.Test:adifferentprefix
./S3encoder.rb:42:in `iconv': invalid encoding ("UTF-8", "ISO-8859-1") (Iconv::InvalidEncoding)
        from ./S3encoder.rb:42:in `escape'
        from ./S3.rb:138:in `list_bucket'
        from ./s3sync.rb:21:in `map'
        from ./S3.rb:138:in `each'
        from ./S3.rb:138:in `map'
        from ./S3.rb:138:in `list_bucket'
        from ./s3try.rb:51:in `send'
        from ./s3try.rb:51:in `S3try'
        from ./s3sync.rb:244:in `s3TreeRecurse'
        from ./s3sync.rb:293:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:226:in `new'
        from ./s3sync.rb:226:in `main'
        from ./s3sync.rb:631
#

Oh dear. Well, I’d wanted to learn Ruby for some time, so here was a great incentive. The line causing the error, line 42 in S3encoder.rb, read:

result = Iconv.iconv("UTF-8", @nativeCharacterEncoding, string).join if @useUTF8InEscape

A bit of Googling around led to a (the?) Ruby documentation site, where from the page describing the Iconv class, it was clear that this call was a way of asking the runtime environment to convert the string in the variable string from the UTF-8 character set to whatever charset was specified in the variable (or perhaps field?) nativeCharacterEncoding. A few lines higher up, nativeCharacterEncoding appeared to be being set to “ISO-8859-1”, which made sense, especially given the error message.

However, this seemed strange – after all, UTF-8 is pretty much the standard character set for most new applications and systems, and ISO-8859-1, aka Latin-1, is the charset that predated it (and is used for most HTML). Still, the slug is a small embedded system – so perhaps, I thought, it might lack certain charsets? Might it be something dreadful like ASCII-only?

I decided to read through the list of available packages, to see if there was something obvious that needed to be installed – an “essential-charsets” package or something like that:

# ipkg list
abook - 0.5.5-1 - Abook is a text-based addressbook program designed to use with mutt mail client.
adduser - 1.1.3-6 - a multi-call binary for login and user account administration
adns - 1.3-2 - Asynchronous resolver library and DNS resolver utilities.

gambit-c - 4.0b20-1 - A portable implementation of Scheme.
gawk - 3.1.5-1 - Gnu AWK interpreter
gconv-modules - 2.2.5-5 - Provides gconv modules missing from the firmware.  These are used by glibc's iconv() implementation.

Now that last one looked promising – after all, as the Ruby documentation said:

Iconv is a wrapper class for the UNIX 95 iconv() function family, which translates string between various encoding systems.

I gave it a go:

# ipkg install gconv-modules
Installing gconv-modules (2.2.5-5) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/gconv-modules_2.2.5-5_armeb.ipk
Configuring gconv-modules
#

…and tried running the command again:

# ./s3sync.rb -r . <my key ID>.Test:adifferentprefix
S3 command failed:
list_bucket <my key ID>.TEST max-keys 200 prefix adifferentprefix/. delimiter /
With result 403 Forbidden
S3 ERROR: #
./s3sync.rb:249:in `+': can't convert nil into Array (TypeError)
        from ./s3sync.rb:249:in `s3TreeRecurse'
        from ./s3sync.rb:293:in `main'
        from ./thread_generator.rb:79:in `call'
        from ./thread_generator.rb:79:in `initialize'
        from ./thread_generator.rb:76:in `new'
        from ./thread_generator.rb:76:in `initialize'
        from ./s3sync.rb:226:in `new'
        from ./s3sync.rb:226:in `main'
        from ./s3sync.rb:631
#

This was fantastic news! Although it had not synced, it had clearly contacted S3, and had been refused access – so the charset problem was, it appeared, solved.

Now, back when I tried to get s3sync to work on my Ubuntu box, I’d discovered that it would refuse to sync when the machine’s local time was skewed from the S3 server’s time. I’d foolishly forgotten to check the slug’s time before trying this sync, so before trying anything else I decided to check that it was OK:

# date
Tue Nov 14 02:49:31 GMT 2006

D’oh. It was 11:43pm on Monday 13 November when I typed that. So, I fixed it and tried again:

# date 11132344
Mon Nov 13 23:44:00 GMT 2006
# ./s3sync.rb -r . <my key ID>.Test:adifferentprefix
#

Which looked pretty good. I checked the S3 server, using the jets3t Cockpit tool that I’d used before, and lo and behold – the files were there!

So, I now had successfully used s3sync to synchronise a directory from my NSLU2 up to Amazon S3 – which was the main point of this project. While there was still a little bit of work to do – for example, making sure it worked with reasonably deep directory hierarchies, checking user/group ownership and permissions were persisted, setting up encryption, and setting up a cron job to automate the backup – the most tricky and experimental part of the work was done.

Next: File attributes, and deep directory hierarchies.

Christmas has come early

13 November 2006

I read Make 6 last week – a bit late, I know – and fell immediately in love with the idea of BEAM robotics, simple robots that can be build at home with simple tools, but display complex behaviour. Last Wednesday I placed an order with Solarbotics for the components I needed to build the “Trimet” robot described in the magazine, and today it was delivered – not at all bad for something that was shipped from Canada!

Given that my Dremel tool was delivered today as well – oddly enough, Tesco.com (a supermarket) had the best UK price I could easily find – it’s all I can do to hold myself back from building a robot or two this evening… but I’ll be strong, and will finish the NSLU2 project first.

Project: Automated offsite backups for an NSLU2 – part 8

12 November 2006

Previously in this series: Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7.

I’ve discovered that in order to get automated offsite backups from my NSLU2 to Amazon S3, I have to get it to run Ruby so that it can run s3sync. Installing Ruby required the slug’s firmware to be upgraded to a new version of Linux, called Unslung, so I did that – and I also installed s3sync on a Ubuntu machine as a dry run. Both of these worked out OK, so the next step was to get the Ruby language itself running under the Unslung firmware – and importantly, to make sure that Ruby had the OpenSSL package installed; the latter had proven non-obvious under regular Linux, so I was expecting problems on Unslung, which is, after all, a cut-down version of the operating system.

The first thing was to work out how to install new packages on an Unslung system in the first place. The last section of the installation README gives a link to this page, which is a good starting point. Going from the instructions there, I made sure that the gateway and DNS server were correctly set up on the slug’s web admin pages, then ran ipkg update from a telnet window logged into the unslung slug. This seemed to work just fine – appropriate messages about downloading, inflating and updating things went by. The next recommended steps were to run ipkg install unslung-feeds, and then ipkg update again. I’m sure there is a very good reason for this, but I’ve no idea what it is – for now, I just followed the instructions, bleating occasionally. Nothing obviously bad happened, and it looked like it managed to install some further lists of software.

The next step was to run ipkg list to see what I could now install. And wow, there were a lot of packages. Most importantly for this project, there was the ruby one, so…

# ipkg install ruby
Installing ruby (1.8.5-1) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/ruby_1.8.5-1_armeb.ipk
Configuring ruby
# 

That looked promising – but…

# ruby --version
ruby: No such file or directory

Hmmm. So, how to find out where it’s been installed? Well, ipkg --help is apparently an unrecognised option, but when I tried it, it forced a printout of the options for ipkg anyway. This made it clear that I should run ipkg files ruby to find out where everything was, and from there I found out that it had installed the binary in /opt/bin/ – which is not unreasonable.

# /opt/bin/ruby --version
ruby 1.8.5 (2006-08-25) [armv5b-linux]

Looking good! Come to thing of it, that’s a more recent version of Ruby than the default for Ubuntu Linux :-)

Now, as I said earlier in this post, one worry I had was the difficulty of installing the Ruby OpenSSL libraries, which are a requirement for s3sync. I had noticed something relevant-looking as the list of files for the package went past earlier, so decided to double-check:

# ls /opt/lib/ruby/1.8/openssl/
bn.rb           buffering.rb    cipher.rb       digest.rb       ssl.rb
x509.rb

This was pretty promising – not a sure thing, but it looked good. So how to be sure – or, at least reasonably sure – that nothing was missing? Well, going back to the error message I got when s3sync failed on my Ubuntu machine (which, at the time, lacked an OpenSSL library for Ruby) I noted that I’d had the error

./S3.rb:25:in `require': no such file to load -- openssl (LoadError)

So, it sounded to me like the “require” command in Ruby loads libraries – perhaps somewhat like import in Python or Java, or using in C#. A quick grep through the s3sync source seemed to confirm this – the first few non-comment lines of S3.rb ran:

require 'base64'
require 'cgi'

…and then, shortly after, there was:

require 'openssl'

So, I could reasonably comfortably posit that if a Ruby script containing just that last line would run happily on the slug, s3sync should be able to run. On that basis, I created such a file and ran it:

# cd /tmp
# ls
# cat > test.rb
require 'openssl'
# /opt/bin/ruby test.rb
/opt/lib/ruby/1.8/armv5b-linux/openssl.so: libssl.so.0.9.7: cannot open shared object file: No such file or directory - /opt/lib/ruby/1.8/armv5b-linux/openssl.so (LoadError)
        from /opt/lib/ruby/1.8/openssl.rb:17
        from test.rb:1:in `require'
        from test.rb:1
#

Now, from this I suspected that the Ruby system had its own SSL stuff installed, and it was just the operating system’s shared library that was missing. In retrospect, this was not obvious – after all, the file that was missing was in /opt/lib/ruby. But, without noticing that, I decided to try installing any non-Ruby OpenSSL package that was out there:

# ipkg list | grep ssl
alac-decoder - 0.1.0-2 - A decoder for the apple lossless file format
flac - 1.1.2-4 - FLAC is a free lossless audio codec.  This package contains the codec libraries and the command-line tools flac and metaflac.
openssl - 0.9.7d-5 - Openssl provides the ssl implementation in libraries libcrypto and libssl, and is needed by many other applications and librari
perl-io-socket-ssl - 0.999-1 - IO-Socket-SSL - Nearly transparent SSL encapsulation for IO::Socket::INET
perl-net-ssleay - 1.30-1 - Net_SSLeay - Perl extension for using OpenSSL
wget-ssl - 1.10.2-3 - A network utility to retrieve files from the Web
# ipkg install openssl
Installing openssl (0.9.7d-5) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/unslung/cross/openssl_0.9.7d-5_armeb.ipk
Configuring openssl
#

…and then tried running the script again:

# /opt/bin/ruby test.rb
#

Which looked perfect. Perhaps someone can tell me how that all worked… or perhaps I’ll work it out myself later. But for now, onward! The next step was clearly to copy over s3sync, and to try it out on a simple directory. And that’s one for tomorrow.

Next: Running s3sync on an Unslung NSLU2.