I’ve always been skeptical of the buzz surrounding the current “AI” trends. To me, it was another corporate parlor trick to perform for investors or shareholders in an effort to generate as much profit as possible.
Nearly a year ago, I heard of a person who would run some of their chats with potential suitors through chatGPT to help them make decisions about who to involve themselves with. At the time, I had been grappling with my own personal burdons that I won’t bore anyone here with, and reached for chatGPT in an effort to help me better understand myself. As I engaged with the platform, my focus would shift to other topics; mostly of a technical or academic nature. I heard about offerings such as ollama and how a person could host a smaller version of these models locally and decided to give it at shot. After installing ollama and firing it up, I asked some questions about the technology itself. In college I was originally working on getting my Computer Engineering degree, but thanks to some undiagnosed ADHD, teaching myself the content was a challenge. Eventually, I switched my degree and moved on with my life, but I always kept an interest in Linux and open source work. I asked Llama (2?) if I could give it some of the No Starch Press epubs or pdfs I had purchased from one of the humble bundles to help me build a curriculum to match my learning style. It told me yes, but they needed to be in a specific format for it to be able to ingest them. I asked if it could help me do that, and it spat out a python script to do just that. I was amazed.
What followed I can only descibe by paraphrasing Carl Sagan’s quote, “If you want to bake an apple pie, you must first invent the universe.”. However, for me it’s, “if you want to learn python, you must first build a data center in your basement with second hand enterprise gear.”
I went back to chatGPT to ask it bow I could properly host an LLM with my own hardware. It said my AMD card was nice, but ROCm wasn’t quite supported yet and I should get something like an RTX 3060. I found a decent deal on a 3080, so I opted for that instead. Then, I saw the massive VRAM available on the Tesla P40, add to cart. But oh wait, if I want two GPUs in my desktop, I need more power, and so on.
Eventually, I bought an old Dell PowerEdge R630 for a song and a dance (shout out to ptnerd on eBay!). Dual 12c 24t xenon CPUs, 128gb DDR4 ECC RAM, dual 750w PSUs, dual 10gb SPF+ ports, 4x 300gb SAS drives. I knew these things were loud power hungry monsters, but I wasn’t prepared for just now bad they were 
. By the time I had it set up and was comfortably configuring proxmox, my bad habit of idly scrolling eBay revealed another interesting find, 16GB Tesla V100 SXM2 GPUs for about $100 each 
. I found a seller from china who had listings for dual SXM2 mezzanines complete with NVLink and photos of the nvidia-smi output to prove it. Adds to cart… Again…. I cobbled together the correct connections to adapt the SFF-8654 connectors to pcie via some risers and to my relief, it listed both V100s in the output of lspci in proxmox.
A decent amount of time had elapsed between most of these purchases, being the parent to two young kids will do that. But then I found it… A listing on eBay for a Gigabyte WRX80 SU08 IPMI board for $230(!). I had been eying threadrippers with envy for a while. The R630 sat unused because of how much of a drain it was to run and I thought this was my shot. The seller listed it because he had bought it used himself, but sheared the head off one of the captive torx screws on the CPU bracket. I took a chance and sourced a new bracket to replace the old one, along with a modestly priced Threadripper 3945XW. I decided to cannibalize the R630 to support this monster and reclaimed the RAM and EPPs from it, and plan to draw up a custom PCB to take the fans too. I was going to need a case to fit this franken build into, but didn’t want to break the bank. I took a chance on an 8 GPU mining frame made of 2020 aluminum extrusions, and as it turns out, aside from one offset standoff, it fit an eATX board .
With the hardware mostly wrangled, I began setting up my development environments and piecing together my software stack. I’ve been experimenting with NixOS throughout this journey and have become a full on zealot. My plan is to eventually configure it to operate like proxmox to silo off the various bits of hardware and use flakes to define the environments for them. The p40 helps with chunking and embeddings for RAG (eventually I hope to have several of these based on content), some inference and stable diffusion. The v100s aren’t quite fully implemented in the new (to me) build yet, but they are going to be my go-to for inference and fine-tuning.
I’ve already learned a lot through building this weirdly beautiful set of components that weren’t quite meant perform the roles they currently are, but I have so much more ahead of me. Incorporating ROCm to my stack via the AI PRO 9700 would be a very welcome trajectory in this fascinating path my life has taken.
Ultimately, I hope to use this system to; explore scientific data about the cosmos, ingest and understand snapshots of the system that the LLM itself inhabits, develop a persistent memory of the things I show it and maybe learn a little more about myself in the process.
