Recommended models/Softwares for local AI with Ryzen 7 2700/64GB RAM/2xRTX 5060 Ti 16GB

I currently have 2 machines. One is my everything server running Proxmox on Ryzen 9 5950X, 128GB RAM, ZFS Storage. I am running linux for everything except Active Directory on Windows Server. I have a docker VM/TrueNAS VM and other servers running. I have around 64GB RAM Available.

My second machine is a new build using my old server hardware. It has a Ryzen 7 2700, 64GB RAM, and 1TB NVMe SSD. It is running Linux Mint currently with RTX 5060 Ti 16GB. I have another RTX 5060 Ti 16GB on the way. I am building out my local AI setup and would like some guidance. I am currently working on a testing setup where most everything is stored on the Ryzen 7 2700 machine, all running in Docker.

I am trying to decide what softwares I should be looking at to get an AI setup fully configured in the best way possible. I will probably also migrate everything other than the ollama to the 5950x machine so it will be easier to backup using already existing PBS.

Here is my current setup, all in docker on the Mint machine:

Ollama and OpenWebUI: OpenWebUI is managing the ollama, I have never done anything with ollama in the command line on this setup (I have once on a test system on previous laptop so have some understanding).

Chromadb has been installed on this machine as well and the environment variables to integrate with OpenWebUI but not sure how much it is being utilized since I did no configurations in the web, just configuring the environment variables.

I plan to rebuild everything on the Mint machine and just trying to get everything working the way I want so when I rebuild I start out clean and don’t have all the iterations just sitting there taking up space or having extra programs I forgot to uninstall or clean up.

Any guidance would be greatly appreciated.

This will be a great starting point. A few limitations to be aware of off the bat.

Check your motherboard. It must have the x16 slot split into x8 + x8 or you’re going to have a really bad time with the 2nd 5060.

Forcing a Gen5 GPU to run at Gen3 is already 25% of expected performance. The 5060 Ti uses 8 lanes, so finding a Gen3 x16 split to dual x8 + x8 would be the best vast scenerio. This Gen3 platform severely limits the GPU <> GPU parallelism you’ll need to get close to the dual 5060 Ti numbers you are seeing online.

Fret not though… If only doing Inference (chat/agents), you shouldn’t see much of an impact on TPS as ping as you have true dual x8 lanes to the CPU (not to a chipset). However, PPS will take a hit on the slower bus - this means when you upload a large PDF or point agents to a large codebases, things are going to take 4x+ longer due to the Gen3 interface to load.

This is a great starting point though and you should not be deterred as you’ll hardly notice until you get up to 10+ agents. Just go in with proper expectations.

The only thing to fret about is if you want to do Training on your own models. Then, yeah, get a PCIe 4.0 or 5.0 systems because that will crawl on Gen3 x8.

Don’t do this. Keep your homelab server separate from your Ai server. There’s also not much to backup as 1. Your dozens of 10GB-30GB models, each, are most likely going to be stale - and you can always pull the latest from Huggingface and 2. It’s just some config files a d docker-compose yaml to backup. Memory or states, but I have those in notebooks than backup a server.

The reason is sometimes you need to reset a GPU or some memory leak forces a slowdown and you need to reboot. You’ll want to try different GPUs and slots, ever upgrading, and your family will be pissed at all the downtime.

In our household, and at work, it’s best to isolate Ai servers from other purposes.

The good news is you can cheap out with other components.

My only recommendation is to get a little more high speed storage. If you plan on presenting a web interface to you and your family, you will want to setup hot swapping of models as they serve different purposes - especially with image and video creations that have different flavors and styles. High-speed storage will vastly reduce the load times to swap models. Our Gen5 M.2 takes about 5 seconds for a 12GB model. Gen3 M.2 would take about 15 seconds, and an SSD… I shudder to think what 8x slower than Gen3 would yield…

Agents won’t be swapping models; however, note that if you want to offer llama-swap or llama-server’s built in router, via an Open WebUI, then you’ll be disabling your agents - as they won’t be able to run if you got swap to an image generation model.

That’s basically the delima we all experience.

I approach things from a security perspective. Never trust a model, nor a Web UI, nor a model router. I isolate everything within Podman containers. If you are not versed in docker containers, that may be future education to invest into.

Next, let’s lay out your goals first….

Are you wanting a pretty ChatGPT chatbot replacement? Then focus on general deep thinking and instruct models for smart reasoning that will fit within 32GB. Unsloth’s Qwen3.6 27B Q4 would fit nicely across both GPUs, have full 262k context window, and still be able to load a nice Image generation library with what’s left over. I would fit something larger like Q5 or Q6 and how a swap the largest stable diffusion model I can find. I would also setup Qwen to skip thinking on quick chats.

Are you wanting to do more coding, homelab automation? Then focus on more Coder centric models. That Qwen3.6 is also very good, but there are other UD variants I would target first though. You also want a fast inference model for intellisense/lookups. So reserve 4GB vram for that model and context.

As far as software stack… it depends. Is this going to be a server, or a local workstation? If server, then you gotta learn containers and follow best practice guides out there for Podman, Containers and monitors. A shortcut could be to use Ramalama (it auto detects your hardware and sets up the best pipeline for the hardware, while also launching everything in secure Podman containers in the background.

If a workstation… There are now nearly a dozen good open source local Ai apps and frontends. Evaluate them all IMO (in docker containers, which are easy to purge if you don’t like it). LocalAi, AnythingLLM, Ollama, Unsloth Studio, LemonadeLLM, etc etc. these can also expose an OpenAI API for other apps and tools to hook into, just remember it’s a desktop app that requires your interaction. (I’m more a backend server and config guy, then setup a web app for other users).

Feel free to ping me. Everything changes like every 9 months anyways. Lol

Thank you. I was planning on leaving the Ollama and GPU’s on the Mint computer and having everything else run in a docker VM on the 5950x system. This is what one of my AI chats really recommended. Basically let the powerhouse CPU server do the server stuff and leave the Mint computer to do the Ollama. I am currently running OpenWebUI, Ollama, Chromadb all in docker containers all on the Mint computer. I am digging through all the features in OpenWebUI and trying to figure it out. I am having trouble figuring out how to configure everything correctly. I have built a few tools to have it review my network and servers to give me the status. Have also uploaded the documentation for tools in OpenWebui and a few known working tools for it to use as a baseline. Haven’t messed with it since so not sure if it works yet.

I plan on having n8n, using it for HTML coding, home automation, general chatting, researching topics. My motherboard is a B450 Tomahawk and it has a PCIe 3 x16 and PCIe 2 x16. Was planning on using those but getting full PCIe x8 on each card would be nice. I do plan on upgrading to a 5900X and B550 in the future. This would allow me to keep my 64GB RAM and not pay the ridiculous DDR5 prices.

Also the computer although running mint will be basically a server. Just using mint because I may decide to game on it in the future. The SSD is a Gen 3 NVMe.

Actually just looked at the specs and may be worse than I thought. The PCIe x16 slot at the top only supports x8 with 2nd Gen Ryzen and PCIe x16 slot at bottom only supports x4 (PCIE 2.0) with just the GPU’s installed and with my 2.5Gb adapter installed drops to x2. May need that B550 and 5900x sooner than later.

1 Like