Marc Manley

The Hybrid Local AI Rig: Running Hermes and Qwen on Apple Silicon

A self-hosted Hermes agent gateway (NousResearch’s hermes-agent, run via Docker Compose) sits at the center of my daily setup. Its default reasoning model is Qwen, running locally via LM Studio on an M2 Max MacBook Pro with 64 GB of RAM, reached over the LAN. Hermes calls it as an OpenAI-compatible endpoint rather than hitting a cloud API for everyday turns. Cloud keys (Anthropic, OpenAI) remain configured alongside it, which makes this a hybrid setup rather than a pure local rig — a local model as the daily driver, cloud available as a fallback or for specific subsystems.

The local model stack

Inside LM Studio on the MacBook, the runtime is configured from Apple Silicon out:

  • Model: qwen/qwen3.6-35b-a3b, in Apple MLX format with 6-bit quantization (arch qwen3_5_moe — a mixture-of-experts model, the “A3B” in the name), taking up 29.09 GB on disk.
  • Quantization trade-offs: 6-bit is a deliberate middle ground — smaller and faster than 8-bit, noticeably more accurate than 4-bit.
  • Capabilities: the model card flags Vision, Tools, and Reasoning, which is what lets Hermes drive it through an actual function-calling and tool-use loop instead of plain chat completion.
  • API compatibility: LM Studio exposes three surfaces side by side — an OpenAI-compatible endpoint on port 1234 (what Hermes actually speaks), an Anthropic-compatible one, and its own v1 REST API for programmatic model management (GET /api/v1/models, POST /api/v1/chat, POST/GET /api/v1/models/{load,download,download/status}).

Hermes connects to this local endpoint via its config.yaml:

model:
  provider: lmstudio
  base_url: http://192.168.4.64:1234/v1
  default: qwen/qwen3.6-35b-a3b

Notably, model.provider is one of the config values Hermes hot-reloads live — mtime-cached, re-read every turn — so switching between the local Qwen model and a cloud model doesn’t require restarting the gateway. That’s a small operational detail that turns out to matter a lot in daily use: “local by default, cloud on demand” only works well if flipping between them is actually cheap. The rest of the config leans the same direction — STT defaults to a local Whisper base model (with OpenAI’s whisper-1 as fallback), TTS via local Piper. Local-first, cloud-as-escape-hatch, end to end.

Telegram as the interface

Telegram is a bundled Hermes plugin that auto-enables the moment TELEGRAM_BOT_TOKEN is set — there’s no separate enabled: true flag. Access control is entirely TELEGRAM_ALLOWED_USERS (or the dev-only TELEGRAM_ALLOW_ALL_USERS); without it, Hermes silently denies every sender. Worth calling out explicitly: a bot token alone makes the bot reachable by anyone on Telegram who finds it. The allowlist is the only thing standing between that and an open bot.

One operational gotcha cost real debugging time: docker compose restart does not re-read .env — it reuses the environment snapshot baked in at container creation. Only docker compose up -d (which triggers a recreate when it detects env or config drift) actually picks up new tokens or allowlist entries. That’s standard Compose behavior, not specific to Hermes, but easy to miss because config.yaml changes do apply live — which trains you to assume .env behaves the same way. It doesn’t.

A second workload, same hardware: al-Khizanah

The same MacBook and the same LM Studio server also back a second, unrelated project: al-Khizanah (الخزانة, “the repository” — the classical term for a scholar’s private book collection), a fully offline RAG search engine over a personal library of Arabic and English Islamic texts. No cloud calls, by design.

  • Stack: BGE-M3 embeddings (1024-dim) plus bge-reranker-v2-m3 for reranking, LanceDB as the vector store, PyTorch on MPS for the embedding/reranking compute, and generation via the local Qwen3-30B-A3B model in LM Studio — a close sibling of the model serving Hermes, same MoE family.
  • Deliberately small: four scripts — inventory, extract text, build index, search — with a standing rule against growing it into a framework, a web UI, or a Docker setup. “This is five scripts and it should stay five scripts.”

Measured on the M2 Max: Surya OCR runs about 4.7s/page on ordinary prose and 7.8s/page on dense typography; indexing roughly 984 passages takes about 100 seconds. The current corpus is 5 works and 984 passages, scoring 5/5 on the project’s own retrieval eval set.

What actually worked

  • The local/cloud model swap is real and low-friction. Flipping model.provider in config.yaml takes effect on the next turn, no restart — the kind of thing that sounds like a nice-to-have on paper and turns out to matter a lot in daily use.
  • Bind-mounting an external project into the agent’s container, rather than duplicating code inside Hermes, keeps automation pipelines normal, independently-testable projects while still letting Hermes operate them.
  • Telegram earns its keep as an “ambient” interface once the underlying tool is solid — voice notes, “run the newsletter,” “send this as a one-off” — Hermes reliably executes already-built pipelines this way.
  • Two very different workloads — an agentic chat assistant and an offline RAG pipeline — both hold up on the same local model family, which is a reasonable proof point for treating a 35B MoE model on a MacBook as a genuine daily driver rather than a novelty.

There’s a second half to this story about what didn’t go smoothly — silent failures in the newsletter pipeline, and two very different philosophies for how to guardrail an agent against them — which is worth its own post: Debugging the Invisible.