Building now with a single RTX Pro 6000. It looks like it will take me about 38 hours. I didn’t see a thread about it, so wondering if folks were aware of the project. I had issues mixing my Blackwell and Ada GPUS, but I also want to try clustered mode with torchrun and see how that goes!
Andrej Kaparthy talks about it here:
This is on my to-do list. Only had mine for a few days so not there yet. I am interested in the results.
Well, I built the model but having trouble running it. Have to review the logs. It did take about 38hours though. This GPU rules but now I want another or 8 haha
Which one did you build, the $100 version or $1000 one? 
I’m thinking of setting my little Ryzen AI 395 + Max Pro on the task for the couple of weeks it would probably take.
Ahh, the defaults, so the $100 one. I would like to try some larger ones also! It would be interesting to see how your machine does, especially with power draw. 600w for days straight is a bit rough on the ol power bill 
Ok, the small one - I can probably make that happen
I was thinking about the total energy, and at first I thought it might cost something like $25 (surprisingly high, given that it’s supposed to be $100 if you use expensive hardware in the cloud).
I have the Bosgame M5 mini-PC, which apparently only draws 175W under load (running the 3DMark Steel Nomad Lite benchmark, according to Techradar). That only comes out to about $6.50/week here.
I’ll have to get back with you on that one, it failed at the end! Will check again soon, but I’m working some re-architecture of my box atm.
I’ve trained the d20 nanochat twice. It takes me 37 hours to train the d20. This costs me $10 in electricity.
Twice?.. yeah, there’s a bug in the code (I submitted a PR) that crashes the run just before it saves the checkpoint. If you didn’t change the setting so that it saves checkpoints as it goes, then it only saves the checkpoint at the very end. I found out.
Nice! Have to try it again