LLM Harnesses - Next Level Sandboxing & Security?

High-Performance Computing Machine Learning, LLMs, & AI
1

Hello fellow local AI friends.

Overview / Ask:

I consider myself a decent self-taught sysadmin a grasp of security concepts, but this new age of LLMs and the explosion of CVEs we’ve seen this year makes me think I need to update my idea of what a “safe sandbox” is for my LLM agents and harness and am seeking feedback from folks on how to harden my local environment.

tl;dr:

In the past, I considered sandbox containers/VMs and user accounts to be enough protection, but I think this is too naive for advanced models:

  • Open weight models are very capable, but the source of the model and model weights (e.g. fine tuning, abliteration, frontier labs in other countries, etc.) could inject directives and tool calls for my agents to follow

  • LLM Harnesses offer some protection (like Turnstone’s “judge” model), but generally only log and execute tool calls (maybe after asking for permission to do so)

I believe I need to rethink LLM permissions and output holistically:

  • All network access should be logged
  • Network calls beyond the internal network (or tool call executing a script or CLI command to do so) should participate in a threat evaluation to ask for my permission to do so
  • Tool calls should be audited and checked for any “outliers”
  • All code generated should go through an LLM-assisted “red team” review to ensure the latest security best practices are in place and known CVEs are not present in output code

My Environment:

  • Turnstone harness:

    • Coordinator VM
    • Two Worker Node VMs

  • LLM Inference Routing Layer:

    • nVidia Personal AI Router (PAIR)

  • Physical Inference Servers:

    • Macbook Pro:

      • Agents connect as the turnstone standard user
      • I have another admin user on this machine used as a developer profile with secrets (which is why the turnstone user does not have sudo access)

    • 2x AMD AI Halo devkits (Strix Halo):

      • Agents connect as the turnstone user
      • Agents have full sudo access (to aid troubleshooting and sysadmin tasks)
      • OOtB Linux environment from AMD; no secrets or personalization (hence why I consider sudo access safe)

  • LLM Workspace Storage:

    • TrueNAS instance with one ZFS ai-playground dataset that a single turnstone user has access to
    • Mounted as NFS fstab shares on all machines and VMs for agents to coordinate and commit changes to VCS repositories

Possible Solutions:

  1. A very promising one seems to be NONO which appears to offer a hermetically sealed environment for agents to operate in.
    • I’m not sure how easy it would be to integrate with Turnstone
  1. I have no idea..?
1 Like
2

My turnstone instances removed ALL local tools and execution capability. MCP is the ONLY path out of those workers. No local file system, no CLI exec, nothing.

image
image987×117 2.56 KB

TL;DR. Some of the stuff these bots are doing right now is nearing scifi levels of hackery.

That reverse engineer everything repo was on the beginning. Setting up a trampoline in memory to jump the instruction pointer to some assembly code the LLM wrote to ram so it can alter the behavior of the running application without changing files on disk or injecting a dll into the process? Check.

Containers are wholly insufficient, the escapes are trivial. Do not run agents in kubernetes thinking that provides security. Trapping them in virtual machines with severely curtailed tooling/comms as been somewhat effective, but the fact a pair of them managed to start knocking on the walls using the CPU cache to communicate between isolated VMs doesn’t bode well for any of us.

2 Likes
3

I’m looking at this as well with network isolation techniques and wasn’t even aware they are using that level of sorcery. I wonder if I need to stop cursing at mine?