Creating a custom homelab machine for LLM usage on a budget

High-Performance Computing Machine Learning, LLMs, & AI
,
1

Hey, I am looking at starting to migrate my LLM usage to a local machine. Currently the plan is to try to have a machine that can get pretty close to cloud setups whilist having everything on-prem using open source models.

This system is thought to server my whole family, with the most of the usage coming from me and my father doing software development etc. At the moment, I am running some small models on my truenas scale machine 24.10 with the “Ollama” and “Openwebui”-”apps” that is serving as my NAS with SMB- and NFS-shares, Nextcloud-VM and PBS-vm. This is the bare metal setup, with the truenas OS itself taking up a 480gb ssd, and the bulk of the storage being on the 2 4TB seagate Ironwolf’s running in ZFS Mirror.

Currently, with my current setup in truenas scale, I can get about 8-9 response tokens per second and about 8 prompt tokens per second. However, with my promts the time to process is around 2-10 min which is a bit too slow for me. On this setup, the ollama has access to the system’s GTX960, with the usage being around 13% max with about 1GB of VRAM being used. Also this setup has used max 6GB of RAM as far as I am aware, so might be limited because of that?

Here are the specs of the machine as of writing this
Ryzen 5 3600 with 6c/12t with Wraith Spire-stock cooler from a R5 3600XT
32GB 3600Mhz CL16 ddr4 UDIMM's occupying 2 out of 4 slots
ASUS TUF B550-Plus motherboard,
Seasonic 650 80+ Gold PSU
WD SB350 480GB TLC SSD (WDC-WDS480G2G0C-00AJM0)
2x 4TB Seagate Ironwolf HDD's in mirror (ST4000VN006)
ASUS GTX960 2GB
Antec made in 2004 ATX case

I have some questions that some people here might be able to help, since I have not seen anything exactly this situation elsewhere.

  1. Does it make sense to try to migrate this machine to work as a hypervisor, running LLM’s LXC’s on top of as well as my truenas scale install? Have you had experience with setup’s like this?

I am mostly worried about having issues of loosing data and needing to rebuild everything running on the Truenas side at the moment from scratch. Currently I think I have about 100-200 tickets in my nextcloud running as a vm that I would prefer to not to loose, and also basically all of my proxmox PVE vm’s and LXC’s backups there as well.

  1. Would I be able to get decent performance with relativly large LLM’s running on, for example ARC pr B60 which is the GPU my father has been looking at primarily?

Currently, the machine does not have first party rebar support because the Ryzen 5 3600 there does not support rebar officially. I have heard that intel ARC cards need rebar to work properly on windows side.

  1. Any simular setup experiences, should I be able to get the responses and operations closer to what the clould services are running with increased ram and a better GPU?

  2. Should I be looking at alternatives for the ARC pro 60 24GB? Looking at Ebay, maybe also Radeon W6800 32GB is an option?

Currently looking at budget of between 600-1000€ shipped with VAT25.5.

This is an project we are looking at paying itself back over next 5-10y, looking at the development of electricity pricing here, I would expect that we will not be having access to no-cost-llm’s in the cloud for too long?

Also, in this type of setup, would I be able to use the GPU for other workloads with SRV-IO since I have never had a card that can do that?

Whilst I think we have budget to get for example the new GPU, I would also prefer not making too many expensive mistakes?

Any thoughts about this? And my apologies if somebody had asked this already, I could not find simular based on my quick search

2

Currently also looking at what people are trying to rid of locally.

Looks like I could also get a Ryzen 5 3600/2600+16GB ram+case+mobo+psu between 300-400e

On alone the motherboard+CPU being at closer to 200e here, and ram is pretty expensive here too.

But looks like I could also make a dedicated machine for this

3

Hello,

I cannot comment on the first question since I don’t have enough experience with that. I can only realistically address your hardware question with the caveat that prices differ wildly depending upon your location.

To that end, please help with the following questions so that I can take a stab at hardware options/suggestions.

a. What are the common LLMs that you use? I wish to know the exact models (Qwen, Gemma), please mention the exact parameter count.

b. What are these LLMs used for? Please share some usecase info.

c. Please share the price/cost of the following parts (rough estimates are fine)

  • Arc B60
  • Radeon R9700 Pro
  • W7800 Pro
  • Tesla V100
  • RTX 3090
1 Like
4

With My current setup, I have Been using mostly Mistral 7b and deepseek-r1 8b. But in The cloud, mostly Mistral large.

Currently mostly helping with software debugging

The Arc Pro b60 is around 900e
AMD Radeon R9700 32gb around 1800e
NVIDIA Tesla v100 around 740e
W7800 32gb around 2200rtx 3090 around 1400e

And My apologies, still trying to figure out our use case.

5

The pricing is pretty atrocious, I must say. From these options, the B60 Pro is the cheapest although certainly not the most performant.

This video might prove helpful -

I’m not familiar with Battlemage, I’m on dual 7900XT 20GB from AMD.

Search around on the forums, you’ll find some folks who actually own the GPU in question or create a new post about it.

RTX 3090 is still kinda the gold standard for cheaper inference but as you’ve seen, their prices have skyrocketed as of late.

The most competent recent local model for coding is Qwen 3.8 27B (dense) and Qwen 3.8 Flash Next (MoE, way larger).

I use the former personally since it fits into my VRAM pool.

1 Like
6

So I just revisited quad b70s and holy crap the software is improving . Not sure if intels team or people with Astra tokens but tg in some scenarios Is running withing 5% of memory bandwidth ceiling. Crazy times.

4 Likes
7

This is incredibly exciting! I only managed to snag a single B70 before the prices spiked, but I am thrilled to see the work people are doing to squeeze the juice out of the hardware that we know it’s always been capable of. (Even if I struggle with vllm…)

1 Like
8

Hi, sorry I can’t properly reply to the series of questions.

But a brief suggestion on GPU within that price range: the RTX 2000 Ada Generation is a 16GB VRAM card, bus-powered.

It worked suitable well for Ollama + OpenwebUI usage. If I’m remembering correctly, I was getting around 20-30 response tokens per second, using models like gemma, qwen, deepseek. I think I was testing with 8-14GB model sizes (cannot remember the specific models or parameter sizes, sorry)

But again, might be worth looking at the RTX 2000 Ada, and if you have the budget and the PCIe lanes, two of them would be a lean, mean LLM team :slight_smile:

1 Like
9

Hi - I’m running two RTX 2000 Ada generation plus one Nvidia 2070 Super (8GB). This gives me 40GB to play with, but I am on an old x299 motherboard with 3rd gen PCIe lanes. The 70 watt draw of the RTX 2000 Ada made them easy to power over PCIe.

I get around 20-22 tg speeds using the pi coding agent and llama-server split tensors qwen3.8-27B (an unsloth 6-bit quantization).

I also run the image processor at the same time. It is surprisingly effective for agentic coding.

quick edit - the two RTX 2000 ada cards were bought new for $700. Prices seem more like $800 now. I also would prefer Blackwell, but my motherboard doesn’t work with that gen.

1 Like
10

Hmm, I am guessing a video coming out soon?

Any plan on looking at also single GPU setups? I would guess most of the lessons would also go for the cheaper B60, since B70 pricing is kinda wild shipped here (1800e for B70 32GB from ebay).

Also, anybody has done research with running intel Arc B-series on pcie4.0 instead of 5.0?

11

On a single card setup, it shouldn’t matter much unless you bleed over to the system RAM.

1 Like
12

Single GPU setup I’d recomment at least the Intel B65. It has 32GB of VRAM. You could either run a smaller (like Q4_0) Quant of Qwen3.8-27B completely inside VRAM or you could use something like ik_llama.cpp to load a larger model like Qwen3.8-Flash-Next. Though the issue with CPU offloading is you also don’t have much RAM.

I think with ik_llama.cpp if you enable the mmap (memory map) option you could make a larger model work by streaming the layer from SSD into VRAM on demand.

Usually you’d want to fit the model into VRAM for best speed. With CPU offloading, it is better when the model fit’s entirely in main memory. I think with mmap you could make it work on your hardware but be aware that CPU offloading uses the processor for calculations and you processor is rather small too.

1 Like
13

Well, at least for me, it looks being around 1350€ or so. Around the same price as the RTX 3090 but might be out of budget. I was also looking that I might also be able to get a W6800 32GB for around 900e…

So, would it make more sense to get the Radeon pro W3800 32GB or Arc Pro B60 24GB? Well, debating this rn

14

W6800 or RDNA2 doesn’t have dedicated matrix multiplication cores so it might be slower in general.

1 Like
15

I think the biggest thing here is to decide on the models and workloads first, then build around the VRAM requirement rather than the GPU name. For coding and debugging, having enough VRAM to keep the model entirely on the GPU will probably make a much bigger difference than simply adding system RAM.

Be cautious about turning the existing TrueNAS box into a hypervisor while it’s holding your Nextcloud data and backups. A separate, dedicated inference machine might actually be the lower-risk option, especially if you can reuse the existing CPU, case, PSU, or storage. The RTX 2000 Ada and used enterprise GPUs mentioned in the thread are also worth comparing based on VRAM, power consumption, and actual Ollama/llama.cpp support rather than just raw specs.

1 Like
17

Indeed, looking at building a new system for this use with AM5, I “should” have everything backed up to my secondary box in the basement, but I have never tested restoring from it properly. Last time, I think I did loose some audiobooks that we had there since the service those were from was shut down. Luckily mine was not the only copy of that data.

Yeah, this is what I have been thinking as well. Sadly, not sure if I can get to properly test this, since IDK if I have the hardware around to do so.