Got tired of seeing local LLM toolchains, agent dependencies, and scattered Python/Node environments pollute the host OS. Also wanted an architecture that doesn’t rely on cloud API black boxes or vendor-locked SaaS.
I put together a provisioning specification focused on a strict “Zero-Pollution” workflow for running local LLMs, Open WebUI, and dev agents inside isolated rootless containers while maintaining direct GPU acceleration.
Core Architecture & Hardening:
Host Integrity: Host OS remains strictly clean. All runtimes, toolchains, and agents are encapsulated within dedicated container volumes.
Hardware Passthrough: Direct mapping of /dev/kfd and /dev/dri for bare-metal AMD/ROCm acceleration without host-level driver breakage.
Rootless & Daemonized: Managed via rootless Podman compose / systemd user services (loginctl enable-linger) for persistent, automated multi-container orchestration.
Isolated Storage: Ephemeral runtimes mapped to dedicated named volumes with explicit SELinux (:Z) boundaries.
Initial prototype started around Distrobox, but has evolved toward fully locked-down native OCI/Podman specifications for air-gapped and headless appliance setups.
Published the initial specification and compose templates to GitHub:
Curious to hear thoughts from anyone running isolated bare-metal AI workflows or hardening rootless container pipelines on Fedora/RHEL.
AI’m building a harness that isolates every tool call, runs only under isolation, and if you have inference on device can be air-gapped. It’s built in Rust, uses a variety of virtualization and containerization techniques with fully user configurable actions/tools/skills/prompts etc…
This has the ability to use cloud providers, CLI tool providers, local inference first and foremost, 100% auditable(outside of locked away hashed traces, working on that) and is currently tuned on my machine for 64K context models ~8B each.
I already run claude in a VM as they tend to wander in auto-mode(reading just about the entire home folder in one pass) but the leftover processes is a big factor in why I designed this. Building a gvisor crate had hundreds of containers laying i background and halted the system. The work there led to a long path of building out a modular rust harness.
I’ve documented it well and it has full specification for each module in it, eventually I’ll publish it as an example of my OSC spec.
Not a cachyOS or fedora user(Artix and Lubunut for me) but we have plans to test run the harness through a variety of OS’s and I’m positive I could use test drivers once it’s published.
Due to the freedom models tend to expect, I prefer a wide open settings VM with agents, they can only hurt till the last snapshot.
Totally agree on the agent wandering issue. Giving an agentic LLM raw access to the host $HOME is an absolute nightmare, and zombie container accumulation from improper sandboxing is real.
My approach with Distrobox/Podman here was focused on solving bare-metal AMD ROCm passthrough (/dev/kfd, /dev/dri) while achieving zero-pollution on the host libraries, especially keeping it turnkey for immutable OS / kiosk appliances.
Your Rust-based harness sounds solid—handling tool call isolation at the crate level is a great angle. Looking forward to your OSC spec release.
Ai’m also in this mindset after getting whacked 2-3 months ago by a drive by python/node dependancy that jacked my harness.
my design is similar. the harness has no OS/cli access, it can only call MCP tools. runs in a container somewhere. dont care much about it. just a website behind nginx.
the MCPs are likewise in podman as unprivileged containers and only do what they specifically do.
the inference engine is bare metal running vllm with no access to anything aside from the local nfs over rdma mount for models/datasets and the container registry for images to run. also it has access to a blob storage endpoint for pulling in multimedia like video/image messgaes so they dont need to pass through the model router…
overall im quite happy with this arrangement. i download stuff to the nfs mount and container repo. i have my dev agents build containerz and publish them to the same repo. containers can mount NFS folders for very limited access to data, etc.
Getting hit by supply-chain dependency attacks is the ultimate wake-up call. Your segmented architecture makes absolute sense for a rack-scale setup—especially delegating tools strictly to unprivileged Podman MCP containers while keeping bare-metal vLLM fed over RDMA/NFS.
Over here in Korea, I’m tackling the exact same zero-host-pollution philosophy, but compressed into standalone, turnkey edge appliances. Currently working through a series of deployment projects to deliver pre-configured, air-gapped SLM nodes on top of immutable Fedora (bootc / Silverblue) for public sector and industrial closed networks.
Different physical scale, but the exact same conclusion: giving untrusted agent loops or messy package managers raw access to the host rootfs is asking for a disaster.