LLM under Proxmox with GPU passthrough?

High-Performance Computing Machine Learning, LLMs, & AI
1

Hello fellow nerds,

I have a Proxmox machine with a 5950X and 128GB of DDR4. I am interested in putting a 3090 in, passing it through to a VM, and running Strata with Qwen 3.8 Flash Next IQ3_S.

Are there any known blockers for this? Or potential Proxmox related performance bottlenecks?

Thanks!

2

Don’t believe so, should work just fine

My local agents are all setup in a VM on my proxmox server and it was a pretty seamless experience. I am getting a slightly higher average TPS than expected as well

1 Like
3

Yeah worked fine for me with Nvidia.

You could even do drivers on the host and pass through paths to machines that need the card. (I’ll call it method 1)

I did method 1 for Jellyfin transcodes in the past. You can also pass through the entire pcie device (method 2) if you’d like a vm to get it exclusively, I’ve done that too for a virtual gaming pc.

I haven’t tried cuda specifically but I think it’s at a higher level of the stack and should work.

So tl;dr: +1 to @ItTakes2ToMango

4

I use unraid, and have my 3090 in a vllm container. Then my agents hit that url and port to facilitate. Tailscale is based.