poppy the training box, part 1: the beginnings

For a while I've been planning to put together a separate machine for local LLM training. Until now, I've been using my desktop PC, perry. I have an RTX 3090 installed, and can get useful training runs done (most recently, a 163M-parameter GPT-2 small style LLM in JAX), but there are a couple of problems.

• perry is my daily driver. If he's doing a training run, then everything is just a little bit sluggish as CPU and GPU alike are busy.
• Although I don't play games often, it's annoying to have the option ruled out for days at a time.
• While the GPU is busy with a training run, I can't do other experiments in parallel -- for example, to scope out what the next step might be.

And relatedly to all of those: the two-day limit to the training runs I've been doing is something I set because that's the maximum amount of time I'm willing to have perry tied up. It would be really interesting to try longer training runs!

I also have longer-term plans; a multi-GPU box would be interesting to put together -- not just to have more power locally, but so that I could test larger-scale cloud multi-GPU training runs before starting to pay for expensive machines. US$15.92 an hour to rent a machine isn't a lot of money, but it adds up, especially if you're spending it while debugging parallelism issues.

And finally, I've always been interested in putting together a custom water-cooling loop in a PC. I've been building my own machines since 1995 or so, but never got round to that side of things. It sounds fun!

But despite all of those future plans, this is a fairly normal machine-building post -- how I repurposed an old PC, plugged in a second-hand RTX 3090 from eBay, tested it all, accidentally trained an LLM for 11 days, and almost cooked a CPU.

Over time, I expect to be posting more -- and more interesting -- build details. Let's think of this as establishing the baseline.

[ Read more ]

10Gb/s Ethernet: switching to a Broadcom SFP+ module

Back in April, I upgraded my home LAN to 10Gb/s. The in-wall cabling is CAT-6 or similar, so I had to use 10GBASE-T. Now, the router I'm using, and the switch in my study, provide 10Gb/s through SFP+ cages; that meant that they needed 10GBASE-T SFP+ modules in order to connect.

That kind of module is known to run hot -- sometimes too hot to actually work. The modules in reggie, the router, appeared to be running OK (see the linked post above for charts), but the one in nigel, the study switch, was a worrying 93C. I tried sticking some mini-heatsinks on it, which seemed to help a bit. But the weather got warmer, and eventually the module overheated. I lost access to the Internet from the study, and checking the metrics showed me this:

You can see that it's "flapping": the temperature gets up to a level where the module shuts itself down for its own protection -- about 95C, I think -- and then when it has recovered, it switches on again, the temperature rises, and the process repeats.

I was able to work around the problem by switching on the air conditioning in the study. But normally I only have it on when I'm in there, and keeping aircon on 24/7 just to keep the network working felt like the wrong solution.

It was time to switch to a more power-efficient SFP+ module.

[ Read more ]

10Gb/s Ethernet: using mini-heatsinks with a 10GBASE-T SFP+ module

In my last post I showed the somewhat-scary temperatures I was getting on the MikroTik 10GBASE-T SFP+ module I have plugged into nigel, the 10Gb/s switch I have in my study. As I mentioned then, the plan was to try using some of the mini-heatsinks that people use on Raspberry Pis, to see if that would help.

Here's how it went.

[ Read more ]

10Gb/s Ethernet: what I actually did to get it working in my home

Having learned enough about 10Gb/s Ethernet to be comfortable about setting it up in my house, it was time to bite the bullet: order it from the ISP, buy some kit, and get started.

I already had 2.5Gb/s working. The apartment has structured cabling -- each room has one or more RJ45 sockets in the wall, and there's a patch panel downstairs by our front door that has a matching patch socket for each wall socket. So when we moved in, I simply set things up so that there was a 2.5Gb/s switch down by the patch panel, and wired everything together there. Most of our stuff works over WiFi, of course, but I needed a wired backbone to connect the excessive number of computers in my study both to each other, and to the outside world.

What did I need to do?

[ Read more ]

10Gb/s Ethernet: what I had to (re)learn

My ISP recently started offering a 10Gb option, and my "shiny new thing!" Pavlovian response immediately kicked in. So of course, I had to upgrade the wired networking in my home -- which meant I had to learn a few things to get it all working, and relearn a bunch of stuff I'd forgotten over the years.

Wired networking for home and small offices hasn't really moved forward that much in the last 20-odd years. Back in 2006, gigabit Ethernet was standard for businesses, and most home users moved to it not long after. Perhaps due to the rise of WiFi for most "last few metres" connections, it's pretty much stagnated there, perhaps with a bit of a push towards 2.5Gb/s more recently.

But with faster ISP connections arriving, I think things are starting to become a bit more interesting. Even the fastest WiFi 7 connections are only able to get up to around 6Gb/s to a single device -- and that's in an ideal "super-fast machine sitting right next to the AP in a shielded lab" setup.

Here's what I had to drag up from my memory, and the new stuff I had to learn, in order to get this all working. I'll write about the background in this post, and then tomorrow I'll post about what I actually put in place.

[ Read more ]

New laptop!

Back at the end of October, I asked whether when I bought a new laptop I should get one with a Core 2 Duo T9600 or a i7 720QM. They both cost the same, and there seemed to be pros and cons to each.

After weighing things up, and particularly with the advice of Greg Bray in the comments there, I'd pretty much decided on the i7, so on 2 November I was ready to buy. I went to the Dell shop... and discovered that they'd added GBP400 or so onto the price for both models in the spec that I wanted. Welcome to the pre-Christmas price rise.

So I waited until early January, and finally the price for the i7 returned to where it had been (though the T9700 was still expensive). And last Thursday, the new machine arrived. i7 720QM quad-core at 1.6GHz (up to 2.8GHz when only one core is active), 4Gb RAM, ATI Mobility Radeon HD 4670 with 1Gb RAM (which can do 432 GigaFLOPS[!]), Windows 7 Pro 64 bit, 256Gb SSD, and a 16" RGBLED screen (if I understand that correctly, the backlight is an array of LEDs in groups of 3, and it adjusts the backlight colour in a screen segment to match the colour in that part of the screen). All for the same price as the cheapest 15" MacBook Pro.

It's lovely. The build quality is excellent -- the only comparable machines I've tried have been the MacBook Pros and the Air, which are nice but slower and considerably more expensive. (To be fair to Apple, the MacBook Pros are better-made than this Dell. But it's a close thing, and I don't think it's worth the extra. YMMV.) The screen is the best I've seen since I saw high-end CRTs on SGI Workstations back in the 90s.

Size and weight-wise, it's comparable to a MacBook 17" -- smaller, but in the same category. So I wouldn't recommend you get one as a netbook -- but as a desktop replacement, it really looks like it's going to be perfect.

The only thing that sucks is the face recognition login widget, which has so far managed to recognise me once. Out of 30 tries. But hey, it can be uninstalled.

New gadget: Nokia N900

I got an early Christmas present today! Robert was able to find a Nokia N900 at a very reasonable price, and Lola decided to get it for me as combined Christmas and birthday presents...

The is a fascinating device. Nokia bill it as a "mobile computer"; it has about the CPU power of a 10-15-year-old PC (say, a Pentium II), but also has a modern graphics processor, and it runs Maemo, Nokia's rebranded version of Debian Linux. Earlier this month, Robert showed me his one running his own port of Frotz, and demonstrated that it had a full version of Linux; I saw the video of it running WebGL later on the same evening, and fell in love :-)

One thing I've noticed, though, is that although you can get a full Debian install running, it takes a little work and there's no good step-by-step documentation. I'll post about that next.

Dear lazyweb: To i7 or not to i7?

So, my home laptop is finally on its last legs. I'm sure a full reinstall would help, but to be honest now that I'm playing with WebGL and now OpenCL (a new standard for number-crunching using graphics cards, which sounds like it has interesting Resolver One possibilities) has appeared, a Centrino Pentium M at 1.6Ghz and a crappy Intel graphics card really won't cut the mustard. Add on to that the fact that I'm dropping my home desktop machine and becoming a laptop-only person, and it's really time to change.

The most tempting lappy out there for me is the Dell Studio XPS 16, which is extremely high-specced, has a very well-reviewed screen, and has what looks to me like an underpriced SSD upgrade (£240 upgrades you from a 500GB 7,200rpm drive to a 256GB SSD). The only downside I've noted is that apparently they run quite hot, but so long as I keep it on the arm of my armchair or on the desk, the risks should be minimal.

My big question is, though, which processor? I don't want a fully-loaded system (largely because I can't afford it) so the choice appears to be between a Core 2 Duo T9600 (2.80Ghz, 1066MHz FSB, 6MB cache) or a i7 720QM (1.60Ghz, 1333MHz FSB, 6MB cache). They're pretty much the same price. The i7, when running single-threaded code, overclocks itself using Intel's "Turbo Boost" and runs at 2.8Ghz anyway. And the benchmarks are better for the i7, so while it's hard to work out what's likely to be best for my ultimate usage patterns, the i7 looks most likely to win out speedwise. OTOH it consumes more power, which means that (a) battery life will be worse and (b) the heat problems that people have mentioned will be worse. And I'd rather not have a laptop that turns into a small puddle of silicon while I'm using it, bringing the Thomas line to a premature close -- or that, alternatively, makes a noise like a jet fighter taking off every time I run something computationally non-trivial.

Sadly, googling for comparisons between the CPUs (even after filtering out all the spammy "shopping comparison sites") didn't really lead to anything useful to help me make a decision. So, does anyone reading here have any ideas?

(UPDATE I got the i7)

SSDs

I'm very tempted to switch to SSDs for my home machines' boot drives. Videos like this aren't entirely representative of what's easily affordable, but it's pretty impressive...

Dear lazyweb: what is is about Linux and WPA?

I don't think I really have enough of a readership for this blog to get an answer to this, but... perhaps someone passing by will know. Why is WPA support invariably so bad in Linux-based OSes?

Three examples:

1. Despite having been a Linux user since 1992, I held off on switching my primary laptop, an aging Fujitsu-Siemens machine, from Windows for many years because no distribution I could find supported WPA out of the box with any kind of user interface I was willing to spend the time learning. This is not because I'm lazy -- I'm willing to put in weeks to learn web server configuration details, and years to learn programming languages. These things are inherently difficult and cannot be reasonably simplified. But connecting to an encrypted wireless network? Sure, the underlying tech is as complex as it gets, but if Microsoft can make it easy to use then why did it take so long for anything usable to get into any Linux distro? Sigh. Well, anyway, I eventually tried out Ubuntu 7.10, Gutsy Gibbon, and was delighted to discover that its support was OK. Not as good as Windows XP's, but good enough for me to switch the old lappy over, which I did sometime last autumn, and have never regretted. So, standard defined in 2004 or so, decent support in mid-2007. Hmm.

2. So along came my new Eee, just before Christmas, and for some reason I have difficulties connecting to my work or home networks. Why? Well, I banged my head against it for a while, and eventually discovered that there was a bug in the scripts used for WPA connections that meant it could not connect if you had a key with a space in it. A bit of digging around on the Eee Wiki helped me sort that one out, but seriously...

3. Anyhow, I posted earlier on today that I'd not been able to connect my OLPC XO to my work or home networks. No prizes for guessing what the problem was... from the Laptop.org wiki (found via):

   WPA is not working from the User Interface in Build 650! Some OLPCs (including in the G1G1 program) cannot connect to wireless routers protected with the WPA protocol. Although the networks are visible from the OLPC, the stall when user enters a password.

   The first solution they suggest is to switch off WPA. Right. Sorry, but I'm not Bruce Schneier, and there's no way I'm switching to WEP. It's almost enough to send me into some kind of Fake Steve Jobsesque rant.

   Luckily, some kind soul has worked out a hack to work around the problem, so I'll try that. [UPDATE, posted from XO: it works!]

But seriously -- what is the problem? Linux is a great OS, but WPA support seems to be seriously messed up. It can't just be a driver problem, because both the XO and the Eee have OSes built for them by the hardware manufacturers. Does anyone out there know?