This will be a great starting point. A few limitations to be aware of off the bat.
Check your motherboard. It must have the x16 slot split into x8 + x8 or you’re going to have a really bad time with the 2nd 5060.
Forcing a Gen5 GPU to run at Gen3 is already 25% of expected performance. The 5060 Ti uses 8 lanes, so finding a Gen3 x16 split to dual x8 + x8 would be the best vast scenerio. This Gen3 platform severely limits the GPU <> GPU parallelism you’ll need to get close to the dual 5060 Ti numbers you are seeing online.
Fret not though… If only doing Inference (chat/agents), you shouldn’t see much of an impact on TPS as ping as you have true dual x8 lanes to the CPU (not to a chipset). However, PPS will take a hit on the slower bus - this means when you upload a large PDF or point agents to a large codebases, things are going to take 4x+ longer due to the Gen3 interface to load.
This is a great starting point though and you should not be deterred as you’ll hardly notice until you get up to 10+ agents. Just go in with proper expectations.
The only thing to fret about is if you want to do Training on your own models. Then, yeah, get a PCIe 4.0 or 5.0 systems because that will crawl on Gen3 x8.
Don’t do this. Keep your homelab server separate from your Ai server. There’s also not much to backup as 1. Your dozens of 10GB-30GB models, each, are most likely going to be stale - and you can always pull the latest from Huggingface and 2. It’s just some config files a d docker-compose yaml to backup. Memory or states, but I have those in notebooks than backup a server.
The reason is sometimes you need to reset a GPU or some memory leak forces a slowdown and you need to reboot. You’ll want to try different GPUs and slots, ever upgrading, and your family will be pissed at all the downtime.
In our household, and at work, it’s best to isolate Ai servers from other purposes.
The good news is you can cheap out with other components.
My only recommendation is to get a little more high speed storage. If you plan on presenting a web interface to you and your family, you will want to setup hot swapping of models as they serve different purposes - especially with image and video creations that have different flavors and styles. High-speed storage will vastly reduce the load times to swap models. Our Gen5 M.2 takes about 5 seconds for a 12GB model. Gen3 M.2 would take about 15 seconds, and an SSD… I shudder to think what 8x slower than Gen3 would yield…
Agents won’t be swapping models; however, note that if you want to offer llama-swap or llama-server’s built in router, via an Open WebUI, then you’ll be disabling your agents - as they won’t be able to run if you got swap to an image generation model.
That’s basically the delima we all experience.
I approach things from a security perspective. Never trust a model, nor a Web UI, nor a model router. I isolate everything within Podman containers. If you are not versed in docker containers, that may be future education to invest into.
Next, let’s lay out your goals first….
Are you wanting a pretty ChatGPT chatbot replacement? Then focus on general deep thinking and instruct models for smart reasoning that will fit within 32GB. Unsloth’s Qwen3.6 27B Q4 would fit nicely across both GPUs, have full 262k context window, and still be able to load a nice Image generation library with what’s left over. I would fit something larger like Q5 or Q6 and how a swap the largest stable diffusion model I can find. I would also setup Qwen to skip thinking on quick chats.
Are you wanting to do more coding, homelab automation? Then focus on more Coder centric models. That Qwen3.6 is also very good, but there are other UD variants I would target first though. You also want a fast inference model for intellisense/lookups. So reserve 4GB vram for that model and context.
As far as software stack… it depends. Is this going to be a server, or a local workstation? If server, then you gotta learn containers and follow best practice guides out there for Podman, Containers and monitors. A shortcut could be to use Ramalama (it auto detects your hardware and sets up the best pipeline for the hardware, while also launching everything in secure Podman containers in the background.
If a workstation… There are now nearly a dozen good open source local Ai apps and frontends. Evaluate them all IMO (in docker containers, which are easy to purge if you don’t like it). LocalAi, AnythingLLM, Ollama, Unsloth Studio, LemonadeLLM, etc etc. these can also expose an OpenAI API for other apps and tools to hook into, just remember it’s a desktop app that requires your interaction. (I’m more a backend server and config guy, then setup a web app for other users).
Feel free to ping me. Everything changes like every 9 months anyways. Lol