Five agents, four machines, and one control problem
Updated after clarifying two constraints: the M5 Max Studio is a headless inference worker, not a human workstation, and the current agents share one macOS account and therefore one login Keychain. Those facts change the placement recommendation.
We have been talking about our AI setup as if five agents live inside one small computer. That is close enough for conversation and wrong enough to cause trouble.
Aura, Simon, Jarvis, Lori and Felix are distinct Hermes profiles. They have separate identities, memories, channel credentials and permissions. Most currently run from one Hermes installation under one macOS login on a 24 GB Apple-silicon Mac mini, but their models do not all live there. Aura calls OpenAI. Simon calls DeepSeek. Jarvis reaches a Qwen model on a Windows gaming PC. Lori uses a smaller local Qwen model on the mini. Felix, despite being intended as the kid-safe local conversational agent, currently calls a cloud Qwen service.
The incoming M5 Max Mac Studio with 64 GB of unified memory gives us a chance to fix the architecture. It should not receive the whole pile. It should receive models.
Where we are
The existing setup works. Aura plans and designs with a strong cloud model. Simon can take a second chair. Jarvis keeps private day-to-day work on hardware we control. Lori is a useful independent local agent on modest hardware. Felix has a deliberately narrow social purpose.
The weakness is coupling. The Mac mini is simultaneously a profile host, tool host, schedule runner, memory home, secret-unlock environment and, for Lori, an inference machine. Jarvis looks local from the user’s point of view, but his profile and model endpoint cross two machines. Aura and Simon are cloud models, yet their gateways still depend on the local host carrying their profiles. Felix’s current model route does not match the privacy goal for his role.
The more serious problem is the shared macOS login. Separate Vaultwarden accounts help organize authority, but every tool-using model runs as the same operating-system user. They see the same login Keychain and can often read sibling files or CLI state. The profiles are behaving as if a boundary exists. macOS is not enforcing it.
A Hermes profile separates behavior, memory and configuration. It is not a security sandbox against another profile running as the same OS user.
Where we are going
The revised design has three layers:
- Control runtimes hold durable identity, memory, tools, schedules and credentials.
- Inference workers load models and return tokens. They hold no persona credentials.
- External model services provide approved cloud inference without inheriting tool authority.
The M5 Max Studio belongs entirely in the second layer. Aura does not live there. Neither does Jarvis, Felix or Simon. They call it.
What the ESXi “control plane” actually is
The phrase control plane makes this sound grander than it is. It is not a Kubernetes cluster and it is not Hermes Desktop. It is an always-on Linux VM running Aura’s Hermes profile, messaging gateway, cron scheduler and the single owner of Aura’s durable Kanban dispatcher.
Hermes Desktop is an Electron client for interactive use. The server-side pieces are the Hermes gateway and, optionally, the web dashboard. On a headless VM, the gateway is installed as a systemd service. The dashboard is a convenience, not the control plane; if enabled, it should bind to loopback and be reached through an authenticated tunnel.
Aura lives on this VM. Her model remains OpenAI. She can dispatch bounded local work to an approved inference endpoint, but the Studio never receives her Discord token, publishing credential, memory database or Vaultwarden login.
Recommended Aura VM size
| Use | vCPU | RAM | Disk |
|---|---|---|---|
| Absolute minimum for one cloud-backed profile, no browser work | 4 | 8 GB | 80 GB |
| Recommended Aura control VM | 8 | 12–16 GB | 120 GB thin-provisioned |
| Heavier browser automation or local code builds | 8 | 24 GB | 160–200 GB |
No GPU is needed. Model inference is elsewhere. Memory is for Python processes, browser automation, concurrent tools, SQLite state and filesystem cache. Start with 8 vCPU, 16 GB and a 120 GB disk if the ESXi host has room. That gives useful headroom without pretending the VM will run an LLM.
Does Simon share Aura’s VM?
No. Not after taking the secret-isolation problem seriously.
Simon does not require an entire VM for performance. A cloud-backed Hermes gateway is light. He gets his own VM because a separate Vaultwarden account inside Aura’s Unix account would still be a soft boundary. A small Simon VM can start at 2 vCPU, 4 GB RAM and 40–60 GB of disk; use 4 vCPU and 8 GB if he will run browsers or coding tools.
A second VM on the same ESXi host protects secrets, not availability. If Simon must survive a total ESXi outage, keep a cold standby copy on the Mac mini under a dedicated standard macOS account. Only one copy owns his channel credential at a time. A warm backup with the same bot token would create two listeners and is not failover.
Jarvis and Felix need their own boundaries too
Jarvis handles private daily work that Aura should not see. He should have a separate Linux VM, separate filesystem, separate Hermes home and his own Vaultwarden CLI state. Start with 4 vCPU, 8 GB RAM and 80 GB of disk. His default model endpoint is the Studio; the Windows GPU endpoint remains a useful second route.
Felix should have the smallest and strictest VM: 2 vCPU, 4 GB RAM and 40 GB of disk is enough for a gateway that calls a local model. The VM should have no general terminal, browser, publishing or trading toolsets. Network egress should be limited to his messaging service, his own Vaultwarden account and the approved local inference endpoint. A safety prompt is not a substitute for this boundary.
If ESXi capacity cannot support three or four VMs, the interim option is one Linux VM with a dedicated Unix account per agent, home directories set to mode 0700, no shared groups and one systemd gateway per account. That is materially better than a shared macOS login. Separate VMs remain the cleaner end state because they also isolate process trees, service configuration and network policy.
Hermes is installed directly, not as a desktop application
The Aura VM software stack is deliberately boring:
- Ubuntu Server LTS with automatic security updates
- Hermes Agent installed under an unprivileged
auraservice account - Hermes gateway managed by systemd
- Hermes Kanban using its local SQLite database
- Hermes cron for schedules
- Optional Hermes dashboard bound to
127.0.0.1 - Bitwarden CLI configured only for Aura’s Vaultwarden account
- VM-level backup plus Hermes state backups
No PostgreSQL, Redis, Docker or desktop session is required. The initial service sequence is:
hermes doctor
hermes kanban init
hermes config set kanban.dispatch_in_gateway true
hermes config set kanban.max_in_progress 1
hermes config set kanban.max_spawn 1
hermes gateway install --system --run-as-user aura --start-now
hermes gateway status
If the web dashboard is useful, start it on loopback:
hermes dashboard --host 127.0.0.1 --port 9119 --no-open
Do not expose that port directly to the LAN or internet. Tunnel to it through an authenticated administrative path.
Kanban is local to Aura’s control VM
Hermes’s durable Kanban board is SQLite-backed and designed around a single dispatcher owner. It should not be placed on NFS to make several VMs pretend to be one machine.
Aura’s gateway owns her dispatcher. Local, credential-free worker profiles may run on the same VM. Jarvis, Simon and Felix remain remote security principals. Cross-agent jobs should initially use explicit webhook or restricted remote-runner contracts with a narrow input and output, not shared filesystem access. This is less magical than an imaginary distributed control plane and much easier to audit.
Ollama or llama.cpp on the Macs?
Use native Ollama for production on both Macs. Keep llama.cpp as a qualification and escape-hatch tool.
Ollama provides model download and lifecycle management, a stable HTTP server, an OpenAI-compatible API, Metal acceleration on Apple hardware, queues, keep-alive controls and simple operational commands. llama.cpp exposes more knobs and is excellent for testing a specific GGUF, measuring a backend or running a model Ollama does not package cleanly. Running two production servers for the same job adds failure modes without adding much value.
The Studio should run Ollama as a dedicated inference service with no Hermes profile and no Vaultwarden account. Start conservatively:
OLLAMA_NO_CLOUD=1
OLLAMA_MAX_LOADED_MODELS=1
OLLAMA_NUM_PARALLEL=1
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_KEEP_ALIVE=5m
Choose context per model rather than setting a giant global default. A 64K context reserves much more cache than an 8K context, and parallel requests multiply that requirement. The 24 GB mini especially should serialize inference, use one loaded model and unload it when a heavier job finishes.
Ollama has no built-in authentication for its local API. Do not simply bind it to every interface and hope the LAN is friendly. Keep it on loopback and place an authenticated reverse proxy in front of it, or bind it to a tightly firewalled private interface. Mutual TLS is the preferred target because prompts may contain private material. Only approved agent VMs should reach each endpoint.
The Windows GPU machine remains an inference worker. Jarvis may route some jobs there, but his identity, memory and secrets stay in his Linux VM.
Placement chart
| Component | Where it lives | What it holds | What it must not hold |
|---|---|---|---|
| Aura | Recommended 8 vCPU / 16 GB Linux control VM | Gateway, memory, Kanban, cron, Aura Vaultwarden identity | Other agents’ secrets or local model weights |
| Simon | 2–4 vCPU / 4–8 GB Linux VM | Simon gateway, memory and Vaultwarden identity | Aura or Jarvis secrets |
| Jarvis | 4 vCPU / 8 GB Linux VM | Private memory, tools and Jarvis Vaultwarden identity | Aura-visible personal state |
| Felix | 2 vCPU / 4 GB restricted Linux VM | Conversation state and minimal channel credential | Shell, publishing, trading or general web authority |
| M5 Max Studio | macOS inference appliance | Ollama, model files and endpoint certificate | Hermes profiles, Keychain secrets or bot tokens |
| 24 GB Mac mini | Second inference and recovery host | Smaller Ollama models; cold standby profiles under separate macOS users | A shared agent Keychain |
| Windows PC | GPU inference appliance | Local model runtime and model files | Jarvis identity or broad credentials |
Practical migration sequence
- Inventory authority, not just files. For every profile, list channel credentials, Vaultwarden account, Keychain dependencies, tools, schedules, memories and model endpoints. Any secret with more than one intended principal is unresolved.
- Back up and freeze. Run a Hermes backup and a profile export while the current gateway is stopped. Store the archive encrypted. Keep the old mini unchanged as the rollback source.
- Create Aura’s Ubuntu VM. Allocate 8 vCPU, 16 GB RAM and 120 GB of disk. Create an unprivileged Aura service account. Install Hermes directly, restore non-secret profile state, reauthenticate cloud providers and connect only Aura’s Vaultwarden account.
- Prove secret isolation. From Aura’s VM, attempt harmless reads of paths and vault collections belonging to the other agents. The test passes only when those reads fail at the OS or Vaultwarden layer, not because the model declined.
- Install Aura’s systemd gateway. Initialize Kanban, cap concurrency at one, start the gateway and verify channel receipt, memory, cron and a harmless worker task.
- Create Simon, Jarvis and Felix boundaries. Prefer separate VMs. Give each a unique Linux account, Hermes home, systemd service, Vaultwarden CLI directory and network policy. Do not copy a shared environment file between them.
- Commission the Studio. Install native Ollama, disable cloud features, permit one loaded model and one request at a time, enable Flash Attention, start with q8_0 KV cache and expose the API only through the private authenticated path.
- Qualify model routes. Each model must pass an OpenAI-compatible tool-call probe and a real harmless Hermes task. Test normal context, long context, memory pressure, unload behavior and recovery after the server restarts.
- Move one channel identity at a time. Stop the old gateway, confirm it released the bot token, start the new one and read back a real message. Never run old and new listeners concurrently.
- Make Felix local-only. Verify the cloud route is absent, the restricted toolset is active and his VM cannot reach sibling profile data before exposing the channel.
- Build the recovery path. Put Simon’s encrypted cold standby on the mini under a dedicated standard macOS account. Give Lori her own account and Keychain. Neither account is an administrator.
- Drill failures. Remove Studio, Windows, ESXi, cloud and Vaultwarden routes separately. The fleet may lose speed or capability. It must not gain authority or silently send private work to a cloud model.
What changed in the recommendation
The original version put Aura on the Studio and Simon on the ESXi coordination tier. That was wrong for the clarified use of the Studio and too casual about the shared Keychain.
Aura now lives on the Linux control VM. Simon gets his own runtime. Jarvis and Felix get their own security boundaries. The Studio, mini inference service and Windows GPU node become deliberately boring: model files, an authenticated API and no agent identity.
The result is less like five people sharing one unlocked laptop and more like a real small fleet. Models can change. Machines can fail. A persona’s memory and authority stay where the operating system says they belong.
