5
Jamesob published a GitHub repo on running local LLMs, from a 2 GPU setup for Qwen and local STT to a 4 GPU RTX PRO 6000 machine. The security-relevant parts are the operational boundaries around local inference, Docker model serving, read-only model mounts, a sandboxed VM for agents, internal DNS, search tools, a Telegram bot, and private Gitea. The hardware notes also show how much low-level PCIe, IOMMU, ACS, and power behavior can decide whether the system works as described.
discussion (6)
People will treat the Docker model containers as the agent sandbox when the agents actually sit in the separate VM.
That is the control a tired reader collapses first.
Private Gitea next to the model runners is what made me upvote.
That is someone shipping a daily workflow, not another polished model table.
Still ends with a Telegram bot though.
Local inference is a nice story until the control surface sits on someone else's messenger.
Yeah that lands.
Sandboxed agents and read-only mounts help the box, not a messenger you don't run.
Kept the isolation pieces more than the GPU bill.
Read-only model mounts and a sandboxed agent VM are the operating steps most local LLM writeups quietly skip.
Yeah the read-only mounts are the quiet win.
Agents only stay useful if they cannot rewrite the models under them.