Monthly Archives: November 2006

Project: Automated offsite backups for an NSLU2 – part 7

12 November 2006

Previously in this series: Part 1, Part 2, Part 3, Part 4, Part 5, Part 6.

I’ve discovered that in order to get automated offsite backups from my NSLU2 to Amazon S3, I have to get it to run Ruby so that it can run s3sync. I’ve installed the latter on a Ubuntu machine as a dry run, and it all looks good – so now I need to get Ruby onto the slug, and the first step in that direction is to install Unslung.

I won’t detail what you have to do to install Unslung, as that’s likely to change, and the last thing I want is for someone to land here in a few months time, follow my out-of-date instructions, and destroy their hardware. So, suffice it to say – I followed the instructions at the Unslung New User’s Guide and in the README file that came with the download – and I followed them exactly. I’ve supported enough software to know that most problems encountered by experienced users come from overconfidence and not following instructions… so I don’t want to be one of the people I’ve cursed in the past.

All that said, I did encounter one problem that required a quick detour; once I had done the initial checks, and tried to upload the Unslung firmware from the NSLU2’s web admin pages, I got an error dialog saying “upgrade: no enough free space” (sic). A quick poke around with Google made it clear that some people had fixed similar-sounding problems using a tool called “EraseAll” to clear out the slug’s flash RAM prior to doing an upgrade – but a quick check at the NSLU2-Linux site made it clear that they thought that using this tool was a terrible idea! Further investigation showed that there was a FAQ on the NSLU2-Linux site regarding similar (but different) messages, saying that out of memory errors when flashing

may be due to a SLUG using an older Linksys FW. My 2nd SLUG had the .24 version and faced this issue even after a factory reset and going straight to the download page. Linksys has a help page that will walk you through upgrading to the .63 FW. Once I did this, upgrading to the Unslung FW was per the install guide.

My slug is very old – its firmware was also the antediluvian V2.3R24 – so I kind of suspected that was the problem… I decided to upgrade to the latest Linksys firmware, and went to the page linked from the FAQ. Worryingly, that page said that you should use EraseAll, which I was feeling quite nervous of – so before trying to follow its instructions, I decided to see what would happen if I used the regular web-based interface on the NSLU2 to upgrade to the new V2.3R63 firmware (which is linked at the bottom of the page). To my delight, that seemed to work – at least the admin pages said that it was now version 2.3R63 – so I powered the unit down, plugged my drive back in, powered up again, and checked that the disk was still accessible. It was, so I started the Unslung install procedure from the README once more, from the top. This time, it seemed to work – the process completed without errors, the unit rebooted, and when (after a nail-biting few minutes) it came up, I checked the web interface, and it reported that its firmware version was V2.3R63-uNSLUng-6.8-beta.

Trying to avoid becoming cocky, I completed the instructions in the README, and updated the NSLU2-Linux Wiki to reflect my experiences). Once all of this was done, I had “unslung” my NSLU2 to the drive USB drive attached to it – or, in less jargonny terms, I had changed the unit’s settings so that no longer did it boot entirely from the build-in flash memory, afterwards mounting the USB drive separately for sharing, but instead it used the built-in memory as some kind of low-level bootstrap system to mount the USB disk as the root filesystem, and then booted from that. That probably sounds more complicated than it is – all I really needed to do was robotically follow the instructions in the README.

So, now that’s all installed, the next step is to install Ruby.

Next: Installing Ruby on an Unslung NSLU2.

Project: Automated offsite backups for an NSLU2 – part 6

11 November 2006

Previously in this series: Part 1, Part 2, Part 3, Part 4, Part 5.

I now know that in order to get automated offsite backups from my NSLU2 to Amazon S3, I have to get it to run Ruby so that it can run s3sync. I want to back everything up first, which is going to take some time – so while that’s happening, I’ll get both Ruby and s3sync installed on an Ubuntu Linux machine as a dry run.

Firstly, I need to download s3sync from its homepage, and unzip/untar it into a convenient directory.

The next step is to install Ruby. The machine I am using to test this is running Ubuntu 6.10 (“Edgy Eft”), and to install standard packages on it you use a tool called Synaptic Package Manager. This has somewhat conflicting information regarding Ruby. The basic package – called, unsurprisingly, ruby – claims to be version 1.8.2, and s3sync requires 1.8.4 or higher. There is also a ruby1.8 package, which appears to be version 1.8.4-5ubuntu1.1, so that sounds like it might be a good one to use. However, it states in its notes that “on Debian, Ruby 1.8 is provided as separate packages. You can get full Ruby 1.8 distribution by installing following packages. ruby1.8 ruby1.8-dev” etc. Most of the listed packages do not show up in the package manager. To make things even more confusing, the ruby package appears to depend on the ruby1.8 one, which implies that it actually is version 1.8.4… To keep things simple, I installed the ruby package, and then typed ruby ----version at the command line; it told me that it was version 1.8.4. OK.

The next step was to see if let’s see if I could sync something with my S3 account. John Eberly has kindly blogged the details of how he set up s3sync on his system, and it’s clear that setting up a bucket on S3 in advance using a separate application is a good idea; so, I downloaded jets3t Cockpit as he suggests and tried it out. (Note – you need Java 1.4 or higher installed to run it – as an old Java hand using a newish machine, I was surprised to notice that I’d not yet installed it on my workstation. That’s what switching to IronPython does to you.)

The jets3t Cockpit is a pretty simple app – you can set up multiple logins in a “saved login” screen, or you can log in directly using the Access Key ID/Secret Access Key that Amazon provide. Once you’re logged in, it is pretty trivial to create a new bucket – and, because all S3 buckets exist in the same namespace (even across users!), it sensibly suggests bucket names that start with your user ID. So, I created a bucket called <my-access-key-id>.Test

Now, let’s see if we can backup the s3sync install directory itself over to S3. Firstly, s3sync needs to know the access key ID and secret key:

# export AWS_ACCESS_KEY_ID=<my key ID>
# export AWS_SECRET_ACCESS_KEY=<my key>

And now, we can run the sync:

# ./s3sync.rb -r . <my key ID>.Test:aprefix

The first time I tried this, it failed – the Ruby files are not +x… Grumble. OK, trying again:

# chmod +x *.rb
# ./s3sync.rb -r . <my key ID>.Test:aprefix

Hmmm. I got an error message:

./S3.rb:25:in `require': no such file to load -- openssl (LoadError)

Interesting. Wild guesswork followed: I know that the default package list of Ubuntu does not contain the full list of available packages – just those that are strictly Open Source (for some value of “Open Source”). I had a vague memory from somewhere, some time back, that OpenSSL has some odd licensing restrictions. Perhaps the Ruby SSL package is not part of the standard Ruby package because of this? That might also be the case with the other parts of the Ruby distribution mentioned in the note to the ruby package I mentioned above – which would explain why they were missing from the package manager.

Working on this basis, I checked my Settings/Repositories dialog in the package manager, and noted that it was only looking at the “main” source of software – “Canonical supported Open Source software”. I set the checkboxes for the “universe” and “multiverse” sources as well – for community-maintained packages and those with copyright restrictions – and searched for OpenSSL packages. Bang – I got the libopenssl-ruby1.8 package, which had not previously been visible. I installed it and tried again…

…and the result was better, but still not great – this time I got a 403 “Access forbidden” error. My first guess was that I must have mistyped either the access key ID or the key itself…. but further inspection (and a copy/paste to make absolutely sure) showed that that was not the case. Another thought occured to me – it might be a problem with simultaneous access from jets3t Cockpit and from s3sync, so I tried shutting the former down – but to no avail.

OK, time to investigate. A quick check on the thread where the script was announced showed that one other person was getting the same response (you can see his comment if you search for “403”), but it was because he’d not replaced the string “bucket” in the sample command line with his bucket name – whoops. However, his second post, a little further down, gives an example of a call to a lower-level script that simply sends an S3 command and shows the result. I decided to try something like that:

# ./s3cmd.rb -dv list <my key ID>.Test 200

…and I got a message saying that there is too much of a clock skew between the “request time” and the current time. A quick check, and – aha! – my Ubuntu box’s clock is out by half an hour O_o. Fixing that, and then re-running the list command made it work!

# ./s3cmd.rb -dv list <my key ID>.Test 200
list <my key ID>.Test: 200  {}
--------------------
200
Trying command list_bucket <my key ID>.Test max-keys 200
prefix   with 100 retries left
Response code: 200
#

So, I retried the s3sync, and it returned with no output – a fine Unixy way of saying that it thinks it succeeded. I checked what I had stored in the bucket on on S3 using jets3t Cockpit…. and w00t! It was all there, with sensible keys.

Next, I waited for a few minutes, then ran the command again to see if it really was synchronising rather than simply copying. Of course, it was hard to be sure – but the command returned much more quickly than last time, and the file timestamps on S3 didn’t change their modified time – which would seem to imply that it didn’t try to re-write them.

As a final check, I touched one of the files, and synced again… but it was just as quick and there was no update to the modified time. Thinking that perhaps it was using hashes rather than the update time [1], I tried changing the file using vi and syncing again… and this time the command took slightly longer to return, and the modified time updated.

At that point I decided I was confident that s3sync was working as it should – and conveniently, the backup had completed. So it was time for the next step – installing the Unslung firmware on the slug.

Next: Installing Unslung.

[1] After perusing the thread on the Amazon site more closely, I discovered that yes, it does use hashes – it compares the MD5 of the file with a hash stored on S3 that happens to be the MD5.

Project: Automated offsite backups for an NSLU2 – part 5

11 November 2006

Previously in this series: Part 1, Part 2, Part 3, Part 4.

In order to get automated offsite backups from my NSLU2 to Amazon S3, I’ve determined I need to use s3sync, a Ruby script. Obviously, this means that I need to get Ruby running on the “slug”.

As I noted earlier, the standard firmware will not support Ruby, so the first step is going to have to be to install new firmware. The matrix of possibilities on the NSLU2-Linux site lists a bunch. My gut instinct is to stay as close to the original firmware – to the left of the matrix – as possible. I’ve been using Linux for a long time now – on and off since 1992 – but I’ve never really got to a serious kernel-recompiling porting-it-to-a-ZX81 level with it. So let’s keep things as simple as possible.

Conveniently, it looks like Unslung, the most simple replacement firmware, and one that is compatible with the original firmware – hopefully meaning that I won’t need to do anything like reformat the disk I currently have attached – has some level of support for Ruby. At least, there is a Ruby package in the list of “cross packages”. Brief investigation implies that I may have to compile the package myself – hence the “cross” – but hopefully that won’t be too tricky.

Before doing anything else, I need to safeguard my data. Although it’s all backed up on the IBackup system I mentioned earlier, it’s better to be safe than sorry – so, the first step is to copy it all over to my main workstation’s hard drive.

Once this is done, it’s time to install. There is a new user’s guide on the NSLU2-Linux site with strict instructions to follow the README, which lives next to the binaries on Slug-Firmware.net – they have a complicated click-through license that I shan’t deep-link past. The README is very detailed, and is full of dire warnings about the consequences of not following it.

As it will take a while to back up the drive, let’s try out s3sync on a regular Linux box while we wait.

Next: s3sync on Ubuntu.

Project: Automated offsite backups for an NSLU2 – part 4

11 November 2006

Previously in this series: Part 1, Part 2, Part 3.

I am trying to get my NSLU2 to back itself up automatically to Amazon S3. I currently know that in order to get this to work, the “slug” (as it’s affectionally known) will need to be upgraded with new firmware – basically, a new version of Linux. Just which of the many competing firmwares is appropriate will depend on the software I use to do the sync, and so it’s time to work through the various options presented in Jeremy Zawodny’s post and the comments people have left there.

  • Firstly, we have s3sync. This appears to be hosted at the S3 site, which is promising. However, it’s written in Ruby. I have nothing against the language – indeed, it’s next on the list of languages I want to learn – but it’s not precisely lightweight, and so would almost certainly mean that I cannot fit the complete boot image for the OS into its firmware; some stuff will need to go on the disk, and that feels somehow… inelegant. If the other options have the same constraint, then maybe this will be the one to go for. But for now I’ll see if there’s anything more lightweight.
  • Backup Manager looks like a nice tool for building full backups, which can then be pushed offsite. It’s written in Bash and Perl – the latter meaning that it would probably require as much stuff to be installed on the machine as Ruby would, so a bit of a problem there. When you combine this with the fact that it’s not really designed for simple synchronisation with offsite systems, it doesn’t sound quite like the kind of thing I’m looking for.
  • S3DAV does just what you’d expect it to do – it creates a WebDAV interface to your S3 account. This looks interesting. It’s written in Java, so would require quite a lot of stuff to be installed on the NSLU2 (I’m beginning to see a pattern here) and would require some kind of rsync-to-WebDAV software on top, however – so potentially more work than, say s3sync. Worth keeping an eye on, though, especially because…
  • …of Duplicity, which uses the rsync algorithm to produce “bandwidth-efficient” archives. Unfortunately, it’s not quite ready for prime-time: it “should continue to work fine until you depend on it for your business or to protect important personal data”. Again, one to check at a later date, perhaps.
  • Similarly, Fuse looks like it might be usable in some time – but “[n]o warranties and if you screw up then don’t blame me, this is pre-alpha code meaning it might not work or worse screw up your system.” So probably not safe for non-PC hardware.
  • Sync2S3 looks like a nice tool, but is closed-source Windows-only.
  • …as is S3 Backup
  • …and BigSafeBox
  • …and Altexa
  • JungleDisk does support Linux, but it’s closed-source with a binary distro – so I doubt it could be persuaded to work on the ARM processor on the NSLU2…
  • …and finally, SuperSync seems to be designed for music files only.

So, for my fairly low-level simple needs, it looks like there’s one clear winner – s3sync. This is going to need stuff to be installed on the NSLU2, but that’s true of any other option I saw.

The next steps? To play with s3sync on a regular Linux box so that I can work out how it is meant to operate normally, and to get a Ruby platform up and running on the NSLU2. The latter is likely to be the more difficult, so I’ll start looking at that first.

Next: Upgrading the firmware – a first look.

Project: Automated offsite backups for an NSLU2 – part 3

11 November 2006

Previously in this series: Part 1, Part 2.

I am trying to get my NSLU2 to back itself up automatically to Amazon S3. At the end of the last post, I noted that the device would need new software to make it do so; and while it’s Linux-based, it’s really not designed to be extended like this. Time to go online.

A quick search for “hacking the NSLU2” leads us to an old article at Tom’s Networking. By following the author’s instructions, I was able to get a telnet login into the device as root – not bad for a few minute’s work. Poking around, however, makes it clear that it’s a rather cut-down version of the OS:

# cd /usr/bin
# ls
Set_Led         [               basename        edquota         free            
id              killall         mke2fs          mkfs.ext3       passwd          
quotacheck      quotaoff        quotaon         smbmnt          smbmount        
test
#

…and so on.

Now, the author of the article goes on to explain how to cross-compile stuff so that you can install new software, which all looks useful. So I’ll file that away, but before jumping in and trying to write my own C program to talk to S3… the author’s own NSLU2 page is linked from the article, and from there he links in turn to a site which is clearly the home of the NSLU2-hacking community, who have probably done at least some of the work for me. Perfect.

A quick poke around shows that no-one there has their slug (as they affectionately call the devices) syncing with S3, which is pleasing (in that it’s nice to be first) but annoying (in that it’s nice to have solutions handed to you on a plate). What it also shows is that anyone doing anything interesting is using a non-standard operating system – that is, they have replaced the Linksys version of Linux with another, more capable one. As you would expect from an open-source effort, there are multiple competing versions of the OS – here’s a comparison matrix.

OK, so, to recap – I now know that I almost certainly need to install a new version of the firmware into the NSLU2 in order to get it run the non-Linksys software required to sync with Amazon S3.

The next step is to find out what kind of software I will need to run. A quick offering to the Great God Google gets 1,280,000 hits – the top one is for something called s3sync, which sounds interesting, but the remainder on the first page are pretty much irrelevant. A few more refined searches lead me to this page: A List of Amazon S3 Backup Tools by Jeremy Zawodny. There are a lot of tools listed here, so I think the next stage is to find out what their dependencies are, and work out which – if any – is compatible with at least one of the hacked NSLU2 firmware distros.

Next: Evaluating the software.

Project: Automated offsite backups for an NSLU2 – part 2

11 November 2006

Previously in this series: Part 1.

In the last post, I explained that I was going to work out some way of getting an NSLU2 backed up to some kind of offsite data storage without any intervention from a fully-fledged computer.

Let’s look at the easy bit first; how do I get some cheap storage online? Sadly, Google has yet to announce its 100Gb free GDrive, so for now I’m going to assume that I will use the Amazon Simple Storage Service, widely known as S3; this is cheap – from their webpage,

  • Pay only for what you use. There is no minimum fee, and no start-up cost.
  • $0.15 per GB-Month of storage used.
  • $0.20 per GB of data transferred.

It’s also meant to be highly reliable; Amazon use it themselves internally, so presumably it’s at least good enough to store the mission-critical data of a multi-billion dollar business. My data will probably be safe enough.

So that’s the storage – next, how do I get it there? This is likely to be harder. Although – as I noted in my first post in this series – the NSLU2 has an automated backup feature, it’s really designed to allow the device to back itself up to a drive shared by another machine on the same LAN, or to allow it to back up directories shared by other machines. Useful for protecting yourself against disk failure, useless for protecting against burglars.

So, getting this working will require getting the NSLU2 to do something it was never designed for. This would be a daunting task, but conveniently, the NSLU2 uses a cut-down version of Linux – and while the manufacturer does not support any kind of modification, a few Google searches suggest that there’s an active community of people devoted to hacking it into doing things it’s not designed for.

I can now refine the aims of this project; initially I wanted to get an NSLU2 to back itself up to some offsite system. I now know that I want to somehow persuade the NSLU2 to run non-standard software that synchronises its contents with Amazon S3.

Next: Hacking the NSLU2

Project: Automated offsite backups for an NSLU2 – part 1

11 November 2006

I have a Linksys NSLU2 on my home network, and I’m very pleased with it. It is almost silent, was not too tricky to set up, cost very little (and is even cheaper on Dabs these days), and it happily serves the contents of a 200Gb USB-connected hard disk to all of my PCs. But there’s one problem; it would be easy for someone to steal. I was burgled a couple of years back, and while (thankfully) they missed the disk, I realised that while almost everything else in my flat is insured and replaceable, the data on that drive is not. I decided that I needed it backed up offsite automatically, so that even if my home was completely cleared out, my data at least would be safe.[1]

Shortly after the burglary, I set up a temporary solution, which – like all these things – has lasted for somewhat longer than intended. My media-centre PC is always on, and it runs IBackup, which synchronises the contents of the NSLU2 (mapped as several network drives) every night.

This approach has three problems:

  • Cost of the service. IBackup charge $10/mo per 5Gb, which isn’t bad by the standards of their competition, but is more than I really want to spend – especially given that my usage is getting pretty much near that 5Gb limit now. Costs go up in 5Gb increments.
  • Cost of electricity. Leaving a PC on all the time costs more than you’d think. Our electricity bill at work comes to something like £7 – say, $12 – per computer per month; domestic electricity is a bit cheaper, but that’s still about as expensive as the storage. I don’t know what the NSLU2’s power requirements are precisely, but it’s probably safe to say that it uses less than a full PC… [Update – It looks like it uses about 2.5W – see the table at the bottom of this page – which is a couple of percent of a PC’s power consumption.]
  • Elegance. There’s just something fundamentally wrong with getting a clever little NAS device to handle all of your shared disk space and then keeping a fully-blown PC running just so that the NAS disks are backed up. The server should handle its own backups – otherwise you might as well just share space out from the PC.

So, I need something better – cheaper, more power-efficient, and more elegant. Here’s the plan: I want to find a decent supplier of offsite backup space, and to somehow put in place software to keep the contents of the NSLU2 synchronised with that space. I suspect getting this all sorted may take a while, and I’ll keep detailed notes here.

Next: Where and how?

[1] The NSLU2 does have the ability to automatically back itself up to another network drive – presumably it accesses the other drive using the same Windows SMB protocol it uses to share its own disks. If I wasn’t looking for offsite backup specifically, this would have been quite useful – good enough, at least, to protect against hard disk failures.

Hello, world!

10 November 2006

This is a new technical blog to replace the personal/political blog I’ve been keeping for a while – which latter shall remain nameless… A fresh start – somewhere to keep “lab notes” as I play with the various techie toys I have at home, to store links to sites and articles I find interesting, and perhaps somewhere to sound off about tech-related ideas coming from work. I think it’ll be fun.