I’m setting up a self-hosted AI experimentation and learning environment on a Proxmox cluster node and would like advice on the best architecture for flexibility and performance.
The primary goal of this setup is to act as a learning platform and possibly scale to something more robust in the future. The limitations of the hardware are clear and it is not expected to be able to run all GPU assisted tools at the same time, but changing between tools should be as simple and seamless as possible.
Hardware
- CPU: AMD Ryzen 9 9950X (16 cores, 32 threads)
- RAM: 192 GB DDR5
- STORAGE: Local hi end consumer NVMe
- GPU: NVIDIA RTX 5090 32GB (planned to be shared across workloads)
Goals
- Host Ollama and OpenWebUI for LLM experimentation
- Run additional AI tools like olmocr and potentially Stable Diffusion or similar for image generation
- Provide a flexible learning platform for AI development and experimentation
- Share the GPU between multiple concurrent tasks without needing to reboot or reconfigure
- Provide Local AI-model API access to services such as home assistant and n8n
- Maintain high stability of the Proxmox host, which also runs other lightweight VMs
Deployment Models I’m Considering for the AI-side of the deployment
-
Docker directly on the Proxmox node
- Pros: Simple, fast setup, direct access to host GPU via NVIDIA Container Toolkit, low overhead
- Cons: Mixed workloads on host OS; security/config isolation concerns; unclear how well GPU usage can be isolated or scheduled across containers
-
LXC containers with GPU access
- Pros: Better workload isolation than Docker alone, lower overhead than VMs
- Cons: NVIDIA GPU support with LXC is non-trivial; unclear if this setup is stable for heavy AI workloads (LLMs, image generation)
-
Dedicated VM with full GPU passthrough
- Pros: Strong isolation, clean snapshots and rollback, ideal for fully sandboxed environments
- Cons: GPU passthrough locks the card to one VM; sharing between VMs is difficult unless vGPU/MIG is used (and may require NVIDIA licensing)
Questions
- What’s the most effective and future-proof architecture for running mixed AI workloads with GPU sharing in mind?
- Are there best practices for GPU sharing across multiple services (Docker/LXC/VM) using a single RTX 5090?
- Is it viable to run the AI stack via Docker containers on the host, and still keep things reasonably sandboxed and resilient?
- How would you organize LLM services vs. image generation tools to optimize GPU scheduling and availability?
Any input from people running similar self-hosted AI platforms would be greatly appreciated.
1 Like
I am trying to decide between running docker on the proxmox host directly or running LXC containers. Does anyone have ideas on wich one is the better approach?
To my knowledge this isn’t possible, if you are using gpu pass through I don’t think you can share your gpu to multiple docker but happy to be proven wrong.
Here is what I know from my very limited experiement with proxmox.
When you boot you need to pass your gpu through to a vm/docker… else it won’t be initiated and stay in some “boot mode” where the fans spins even if it isn’t hot. once you connect it to a vm it gets initated and leaves that “boot mode”.
Yes I guess, why do you want to keep it sandboxed? Do you have any security concern or are you looking for reliability in case one of your docker fails?
From my understanding, on baremetal, when you spin a docker and pass it a gpu it is the same as setting cuda_visible_device to you virtual env.
I quite proxmox and use virtual envs and docker on baremetal (I’m on arch if you want to know everything).
To be more explicit I dev in virtual env then build my dockers.
I made a custom orchestrator with various docker container (llm, image gen, wisper, embeddings…).
This orchestrator is my main api, I send it request, it monitors and logs my gpu usage, docker logs, jobs… In my request I give it my workflow_name and data. (each workflow is attached to a docker image with its env args)
By monitoring gpu usage against workflow I can track how much vram a workflow needs…
I’d love to open source it but this is really a dirty WIP. Hope it gives you inspiration.
I’m not sure if I’m recommending this, but it’s a thing I’ve been thinking about.
I really like proxmox and VM isolation, except for the mess of GPU pass through with consumer cards.
But I’ve also been playing with Talos for k8s. And Talos can run on bare metal with district-maintained drivers and container toolkit. It can even run in single node non-HA “cluster” mode. So that might be the very easiest way to manage an experiment server with shared resources and easy isolation.
Maybe it’s a dumb idea, but it might be fun.
I have a similar setup, and I’m using Archlinux as a host O/S with vfio passthrough to VMs. One advantage of using VMs as the host for my workloads is that I don’t even need to install drivers in the hypervisor. I can also easily switch VMs to experiment with software using the same card, just not at the same time.
I do have one VM on which I host ollama, openwebui, Tabby, openhands, etc. Since it’s just me, I don’t really have many issues with multiple users running workloads, but I do share a bit with family and friends.
I was previously just using the host machine with docker, but I found that it can become quite messy, and although it’s a bit more of a hassle and possibly decreases performance, the convience of isolated environments makes up for it.
I do use containers on some of my VMs, btw.
I have a similar HW spec to what you are looking at, except only 96GB ram and I have a 4060 Ti in addition to the 5090.
I have my own separate Proxmox server (w/o a fancy GPU). I run ollama off my machine with the GPUs and let various Proxmox VM/LXCs consume the Ollama API over HTTPs with locally signed certificates (served via nginx).
I read that it was tricky to do Proxmox on top of GPUs correctly, I opted for just letting the separate GPU server pass Ollama to the Proxmox VMs/containers.
I haven’t used olmocr but I have run ComfyUI directly on the GPU server. Again if you configure nginx with locally signed certificates, other machines can access all the AI tooling over HTTPs locally.