I’m not even sure if the word ‘homelab’ applies here. I’m almost starting from completely zero knowledge in this space.
I’ve been thinking about building my own server for a while so i can have a network storage, media server, and now local LLM which preferrably containerized but can access network if needed. Having said all that, I have no idea where to start, what to read to get this up and running, and what hardware is needed/recommended. I’ve been watching several homelab tutorials including l1tech channel but i feel like a lot of these only vaguely touch what i exactly want to do or scale is out of proportion. It would be very helpful if I could be guided to where to start, what software/hardware is needed, and what i can start with and how to scale this operation.
Quick edit: I don’t know if this helps at all, but I have experience in building my own computers, and i’ve taken a really basic DevOps course, Data Science course and a bit of Python. I don’t have any degree in this space at all, just all self taught basic skills.
You can get away with an old desktop that’s laying around or even a raspberry pi for network share and media server. I’ve seen some people run LLM’s on pi’s but I wouldn’t reccomend it unless your looking to dabble for cheap or have a very specific low overhead use case.
A larger LLM will require a bit more beef and you’ll probably want a GPU(NVDIA). But any recent modern desktop will run them as long as they are spec’d appropriately.
I currently just have a Ryzen 5700x system and a couple of raspberry pi’s for my setup.
Note: I don’t currently run an LLM’s as I found it be too resource intensive and with little benefit for what I would need it to do.
A quick question to determine will also be how much storage would you like? How much do you NEED to begin with? And what kind of backup and redundancy would you like to have?
I’m mostly looking to move my hdds that’s in my workstation over to network NAS. Ideally, even as a network drive, i want to access it from my computer like it’s locally attached. I have 4 14TB locally and want to expand up to 8 drives total, possibly 6 HDDs and 2 SSDs for caching.
for the local hosted Ai, I was looking into this AI starter kit. My current plan is to transfer my RTX3080Ti once I get a better GPU for when I start data science/machine learning course. Media server is really just an afterthought, the point was to be able to have LLM access and also possibly stable diffusion for my kid/wife who wants to use them regularily.
I use it quite a lot on my work (i work from home) for correcting grammar or just polishing my email. I’m also learning devops and data sicence/ML and I don’t have background in programming at all, so I rely heavily on LLM to help myself on programming part of the work. My Sister in law is in school and uses it quite a bit to check grammar. My son also wants to use stable diffusion for his hobby/artwork.
Uses are pretty basic but they were asking about buying pro licenses for chatGPT. if all of us have them then it will be 1200USD/1700CAD per year just for 4 licenses. This is why I just said F it, i’ll just host it myself.
Depending on whether you have the space and want to experiment, I’d say pick up a rack somewhere, maybe half height.
From there you can either get a rack mount case, I’d go 4U and that will allow you to fit a lot of 3.5” drive bays. If you already have a MoBo CPU combo you can use that is.
If you don’t, maybe you can pick up 3U Supermicro chassis and MoBo with 12 bays. That would also work. If and when you need more storage, you can go the DAS route
Regardless, you just need to get going and keep your eyes open for great deals. I can almost guarantee you that what you start with today isn’t what you’ll have running in 12 months time. That’s why I like the idea of a rack, because it’s easy to add machines until you’re ready to replace without your home getting cluttered
Should have specified before, but best I can do is a square on my Ikea box shelves. My initial thought was something like a Jonsbo N4 or similar sized case from Aliexpress. I live in a 1 bdrm apartment so space will be the biggest issue.
I also plan to put some quite fans and lower TDP parts where applicable so I don’t have a heater going on. For example, using 4070ti instead 3080ti for lower wattage.
DAS route sounds like a good option to explore as I would be the only person that needs to access the storage network. I plan to put this machine about 6ft away from my workstation so I can easily connect it via Lan. However, that would mean I need another LAN port on my workstation since I only have 1. I was originally looking at Tailscale but there seems to be limitation to file size you can transfer at once.
In that case I’d consider going for a board with 10Gbe (if you need that) and enough SATA ports because you’ll beed your one PCIe slot for the GPU.
For noise you might wanna consider going full SSD
A DAS would usually be connected to by SAS cable. In this case I’d say start with what you can fit in the case and once you outgrow that, get a NAS. If you choose a board with multiple ethernet ports you can directly connect to it (kind of like a DAS), or you can go through a switch
Would I need 10Gbe? could 2.5 or 5Gbe be enough if i’m just using hdd with SSD tiered caching?
Here’s a rough draft of my setup:
CPU: enough to run NAS/Unraid seamlessly + run media servers if needed
Mobo: MATX with ECC support
Ram: 2x32gb ECC memory
Storage (2x14tb, 2x10tb right now, with expansions later), 1-2 2TB SSD for tiered caching
GPU: 4070Ti or similar
Case: Silverstone CS382
I want this to function as a network storage I can directly access to my workstation as if it’s a local drive, serving webUI LLMs via localhost network using docker container or kubernetes.
Would love to hear what CPU + mobo combo would be a good recommendation. Also, do I REALLY need ECC for my setup? I feel like going JBOD without any backup/redundancy is already heresy as is, what difference does it make without ECC?
1gig is 125MB/s, 2.5gig is 312.5MB/s, 5gig is 625MB/s and 10gig is 1.25GB/s… You decide what you need. You might say 2.5gig is fast enough for the homelab, I say 10gig is fast enough to not be annoying
As for ECC, it many cases it’s just very nice to have. And in the case of RDIMMs they’re also very affordable used. UDIMM ECC is another story tho
ok so from my experience, i would suggest to separate your NAS from the rest of server, since it might need proper bandwidth( unless you plan for nic teaming and multiple 1GB connection on the server) also using truenas, you need to passthrough your HDD to the VM, this might be come in a little buggy or troublesome later… because if you loose your VM host and if you have not set it to passthrough, you will loose your data, and if you did the passthrough, you might still loose some controls, so it might be better if you had separate trunas server as host.
for the LLM VM you need passthrough gpu, you might want to use platform like xeon to have proper SR-IOV.
so all and all, you can use trunas as hypervisor ( limited hypervisor capability but you can have one server for all your works), or promox, do not go for esxi, the support is very weak at this point and you will encounter driver issues. as for the hardware recommendation, i recomend go for a xeon workstation, it has ecc and its cheap and you got pretty good stability, do not go for pc unless you are sure about the stability, and i don’t have any good experience with the recent years pc hardware. drawback is the power consumption
There’s only 2 purpose i’m trying to serve with this project. LLM self hosting and NAS. I wasn’t planning to use Proxmox or VMs. This is the kit i’m starting with for hosting Ollama in a docker container, and NAS i haven’t decided what to go with but I just want JBOD that I can freely add/remove drives without having to completely destroy/rebuild the data already there.
In my head, this would basically was a linux box that’s always on with mapped network drive, but now i’m wondering if i were to have truenas or unraid and LLM in the same box, i would need to have 2 separate VMs that will host both. is this correct?
There aren’t a lot of wrong answers, just ones that are more or less matched to your needs, time, and budget. Given your stated inexperience with the ops side, I’d advocate for keeping things cheaper and simpler until you experience enough to have your next set of opinions.
My opinion is currently that you should separate storage and compute (I went the other way with my current build…). If you combine them you either make tradeoffs or spend more money (or both).
That said, if you want one box and only have the two requirements, I’d say go with Unraid. It has the jbod-like experience you want, limited tunables, can run docker containers, and should be able to install drivers for your GPU.
Any motherboard that has the slot you need for a gpu and enough SATA ports will then work.
If you end up needing a GPU and a HBA at the same time, you will start needing to make tradeoffs. It will work and probably be fine, but cheap and fast and good don’t coexist.
If you separate the roles, consider one of the cheap industrial NIC+many SATA motherboards for your storage and then any decent desktop for your GPU/llm host.
Remember, you can always buy more later. (Evil laughter)
Thanks for a good set of advices. I didn’t want to go buy synology or similar NAS boxes because i felt like it costed an arm and a leg when I could make much more powerful machine myself. Then my wife asking for LLM was what made me think of doing this project. But I see why there would be a tradeoff if i wanted to do both at the same time.
Unraid sounds like a good start. I could then basically make a NAS box and run a docker container in it to start out, then separate that into a different machine for hosting LLM exclusively if I want to make a more serious RAG LLM host later down the line.
Will update once I have more concrete list of hardware.
if you want NAS, you need to pick a NAS OS, otherwise you could have a network shared Drive on a windows in network or linux, there you wont have any proper raid or redundency. i recommend you to use NAS OS, truenas is fine, really good. you could use truenas as hypervisor for you LLM VM as well. but truenas is not the best hypervisor, it could serve your purpose but you need to study about it. so these are your two options:
1- two separate machine ( that you dont want)
2- one server with truenas as hypervisor for NAS and proper HDD dedication to NAS, with a LLM VM.
3- one server with hypervisor ( potentially promox) and two separate VM, hdd passthrough to truenas and one VM is llm.
for the anwser of your question, yes you need two VMs to host both. you cant have good NAS on a linux that can run LLM, at least there is no recommendable solution for this. you could try find it. but NAS is a nasty thing… i wont recommend messing with data on a janky software, i would go for NAS OSs.
I’ve been looking for some hardwares and i’ve been leaning heavily into option #3. that seems to be the most flexible path if i want to do anything on top of the current goal. Eventually, I want to add few things like DNS addblocker, manage my password, and few other little projects on top.
Another big challenge i’m trying to solve is the space. Ideally I want to have the NAS and LLM VM in a single MATX case or smaller. Trying to find a case that’s big enough to shove all these hardware is proving to be pretty challenging.
I need to find a network switch that can do POE and preferrably 10gbe and maximize the data transfer between proxmox and main workstation. does it make sense to get a 10gbe rj45 pcie network card for main workstation + proxmox host? if so, what network switch should i go for?
