poppy the training box, part 1: the beginnings

For a while I've been planning to put together a separate machine for local LLM training. Until now, I've been using my desktop PC, perry. I have an RTX 3090 installed, and can get useful training runs done (most recently, a 163M-parameter GPT-2 small style LLM in JAX), but there are a couple of problems.

• perry is my daily driver. If he's doing a training run, then everything is just a little bit sluggish as CPU and GPU alike are busy.
• Although I don't play games often, it's annoying to have the option ruled out for days at a time.
• While the GPU is busy with a training run, I can't do other experiments in parallel -- for example, to scope out what the next step might be.

And relatedly to all of those: the two-day limit to the training runs I've been doing is something I set because that's the maximum amount of time I'm willing to have perry tied up. It would be really interesting to try longer training runs!

I also have longer-term plans; a multi-GPU box would be interesting to put together -- not just to have more power locally, but so that I could test larger-scale cloud multi-GPU training runs before starting to pay for expensive machines. US$15.92 an hour to rent a machine isn't a lot of money, but it adds up, especially if you're spending it while debugging parallelism issues.

And finally, I've always been interested in putting together a custom water-cooling loop in a PC. I've been building my own machines since 1995 or so, but never got round to that side of things. It sounds fun!

But despite all of those future plans, this is a fairly normal machine-building post -- how I repurposed an old PC, plugged in a second-hand RTX 3090 from eBay, tested it all, accidentally trained an LLM for 11 days, and almost cooked a CPU.

Over time, I expect to be posting more -- and more interesting -- build details. Let's think of this as establishing the baseline.

[ Read more ]