Marc Manley

Hermes / Qwen / Middle Ground / al-Khizanah — Deployment Summary & Assessment

Draft research notes for a blog post — pulled from build/debug history across the Hermes, Middle Ground Newsletter, and al-Khizanah projects, plus a confirmed LM Studio screenshot and al-Khizanah’s own CLAUDE.md.

The stack, in one paragraph

A self-hosted Hermes agent gateway (NousResearch’s hermes-agent, run via Docker Compose) sits at the center. Its default reasoning model is Qwen running locally via LM Studio on an M2 Max MacBook Pro (64 GB RAM), reached over the LAN — Hermes calls it as an OpenAI-compatible endpoint rather than hitting a cloud API for everyday turns. Cloud keys (Anthropic, OpenAI) are still configured alongside it, so the setup is a hybrid, not a pure local rig — local model as the daily driver, cloud as fallback/for other subsystems. Telegram is the primary chat interface into this agent, allowlisted to a single user. On top of that runtime sits a purpose-built automation: a weekly (and on-demand) Mailchimp newsletter pipeline for Middle Ground (muslim.center), which Hermes operates via cron and via Telegram commands, but which was built separately in a real dev environment rather than by asking Hermes to build itself a new feature. The same Mac and the same local LM Studio server also back a second, unrelated project — al-Khizanah, an offline RAG search system over a personal library of Arabic and English Islamic texts — which turns out to be a useful point of comparison for how differently “AI guardrails” can be designed.

Component 1 — Hermes gateway + Qwen on the MacBook

  • config.yaml’s model block: provider: lmstudio, base_url: http://192.168.4.64:1234/v1, default: qwen/qwen3.6-35b-a3b. LM Studio’s local server exposes an OpenAI-compatible API on port 1234 on the MacBook; Hermes (in its own container) just points at that IP.
  • Notably, model.provider is one of the config values Hermes hot-reloads live (mtime-cached, re-read per turn) — so switching between the local Qwen model and a cloud model doesn’t require restarting the gateway. That’s a genuinely nice operational property for a “daily driver is local, but I want cloud on demand” setup.
  • The rest of the config leans the same direction: STT defaults to a local Whisper base model (OpenAI whisper-1 as fallback), TTS via local Piper. The design intent throughout is “local first, cloud as an escape hatch,” not just for the LLM.
  • Confirmed from a screenshot of LM Studio’s Developer / Local Server tab on the MacBook:
    • Model: qwen/qwen3.6-35b-a3b — MLX format, 6-bit quantization, arch qwen3_5_moe (i.e. a mixture-of-experts model, the “A3B” in the name), 29.09 GB on disk.
    • MLX is Apple’s own ML framework, built for Apple Silicon — so this is a from-Apple-Silicon-out setup, not a GGUF/llama.cpp model repurposed for the Mac. 6-bit quantization is a deliberate middle ground: smaller/faster than 8-bit, more accurate than 4-bit.
    • Capabilities flagged in the model card: Vision, Tools, Reasoning — so it’s not a text-only chat model; it supports tool-calling and image input, which matters given Hermes drives it through an agentic tool-use loop.
    • Server status: Running, reachable at http://192.168.4.64:1234 — matches the base_url in Hermes’s config.yaml exactly, confirming the two really are wired together as described above (not just theoretically compatible).
    • LM Studio’s local server exposes three compatibility surfaces side by side: its own LM Studio API, an OpenAI-compatible one (what Hermes’s provider: lmstudio actually speaks), and — notably — an Anthropic-compatible one too, plus a new REST API v1 (GET /api/v1/models, POST /api/v1/chat, POST/GET /api/v1/models/{load,download,download/status}) for managing models programmatically.
    • The Developer Logs pane shows live tool-schema traffic (a write_file function definition, JSON-schema parameters) — i.e. this instance is actively being driven through function/tool calling, not just plain chat completion. There’s also an mcp.json button in the toolbar, meaning LM Studio itself has MCP server config exposed at the app level, separate from whatever MCP tools Hermes brings to the conversation.
    • Worth noting for the blog: the model card literally reads “Qwen3.6 35B A3B” while the API identifier is qwen/qwen3.6-35b-a3b — good concrete detail if you want to show, not just tell, what “running a 35B MoE model locally on a laptop” looks like in practice.

Component 2 — Telegram as the interface

  • Telegram is a bundled Hermes plugin that auto-enables the moment TELEGRAM_BOT_TOKEN is set — no separate enabled: true flag.
  • Access control is entirely TELEGRAM_ALLOWED_USERS (or the dev-only TELEGRAM_ALLOW_ALL_USERS); without it, Hermes silently denies every sender and logs “No env user allowlists configured.” Worth calling out in a blog post: a bot token alone makes the bot reachable by anyone on Telegram who finds it — the allowlist is the only thing standing between that and an open bot.
  • Operational gotcha that cost real debugging time: docker compose restart does not re-read .env — it reuses the environment snapshot baked in at container creation. Only docker compose up -d (which triggers a recreate when it detects env/config drift) actually picks up new tokens or allowlist entries. This is standard Compose behavior, not Hermes-specific, but easy to miss because config.yaml changes do apply live, which makes you assume .env behaves the same way.

Component 3 — The Mailchimp newsletter pipeline for Middle Ground

  • generate_newsletter.py, a standalone project kept deliberately separate from the Hermes codebase, later bind-mounted read-write into the Hermes container so Hermes could operate it without a workaround.
  • Pipeline: scrapes muslim.center (Jumuʿah khatib/date, three “Learning” cards, latest podcast episode, latest blog post), pulls an optional freeform “extras” item (markdown + image) from a shared Google Drive folder via a service-account credential, renders everything into a Mailchimp-compatible HTML template (careful to preserve the VML/MSO Outlook button hack), and either dry-runs locally or pushes a real Mailchimp draft campaign.
  • Later extended with a --one-off mode for ad hoc sends (prayer reminders, event announcements) — same Drive-extras mechanism, a separate lighter-weight template, its own Mailchimp draft title.
  • Visual/typography rules were deliberately split into two documents: a full audit of the live site’s design system, and a condensed “email-safe” token doc with an explicit “these have no email equivalent, don’t use them” list (hover states, backdrop-blur, CSS transitions, ARIA landmarks). That split existed specifically because handing the full web-design doc to an LLM risked it reaching for browser-only techniques that silently don’t render in an inbox.
  • Hermes operates this pipeline through two Telegram-facing skills (weekly, one-off) that shell into a dedicated Python venv built inside the container’s persistent data volume, and a real cron job (Thursdays 10am Pacific) for the unattended weekly send.

Component 4 — al-Khizanah: the same local-AI stack, a very different discipline

A second, standalone project — al-Khizanah (الخزانة, “the repository,” the classical term for a scholar’s private book collection) — runs on the same MacBook, against the same LM Studio server, but for a completely different purpose: a fully offline RAG (retrieval-augmented generation) search engine over Marc’s personal library of Arabic and English Islamic texts. No cloud services, no API calls, by design.

  • Hardware, confirmed directly from its own docs: M2 Max, 64 GB RAM. (This is very likely the same physical machine serving Hermes over the LAN — al-Khizanah’s own CLAUDE.md points generation at http://localhost:1234/v1, i.e. LM Studio on that same machine, called from a session running locally on the Mac itself rather than over the network.)
  • Stack: BGE-M3 embeddings (1024-dim) + bge-reranker-v2-m3 for reranking, LanceDB as the vector store, PyTorch on MPS (Apple’s Metal backend) for the embedding/reranking compute, and generation via the local Qwen3-30B-A3B model in LM Studio — a close sibling of the model serving Hermes, same MoE family.
  • Pipeline is deliberately small: four scripts — inventory, extract text, build index, search — and the project’s own rules explicitly forbid growing it into a framework, a web UI, or a Docker setup. “This is five scripts and it should stay five scripts.”

Why this is worth including in the post — the guardrail contrast. Where Hermes’s write restrictions turned out to be an opaque, silent black box (see below — it looked like progress was happening in the Telegram transcript while nothing was persisted), al-Khizanah’s CLAUDE.md is a masterclass in the opposite approach: every guardrail is explicit, human-authored, and reasoned through in plain language, e.g.:

  • Never guess an author attribution. The project’s single hardest rule: classical-text author/title metadata is hand-labeled by Marc only. A guessed attribution is explicitly called out as worse than a blank one — “invisible downstream,” it “silently corrupts every author-filtered query.” The agent is allowed to flag inconsistencies but never fill in a value itself.
  • Stop, don’t self-heal, if you corrupt something. If the hand-labeled metadata file is ever found wiped or wrong — even by the agent’s own hand, mid-task — the instruction is to stop and tell Marc immediately, not attempt to reconstruct it from context (which “risks re-introducing exactly the silent corruption this rule exists to prevent”).
  • Test on a subset before any slow, expensive, or hard-to-undo operation, and actually read the sample output before scaling up — with a concrete example (Surya OCR measured at ~4.7s/page, used to extrapolate a 322-page book to ~25 minutes) rather than a vague “be careful.”
  • Long jobs run in Marc’s own terminal, not the assistant’s session — the assistant writes the exact nohup/caffeinate command for him to paste and watch with tail -f, rather than blocking a chat session on an hours-long index build.
  • A documented failure mode caught early: browser “print to PDF” of web articles often has no real text layer, so extraction silently yields only headers/URLs while the actual content is lost — passing an inventory check as “text” while containing nothing. Named in the doc with a real example (a 10-page article that extracted to 1,275 characters of Safari print furniture).
  • Explicit scope discipline: “if something feels like it needs a sixth [script], say so and explain why before writing it” — the opposite instinct from feature-creep.

The throughline for the post: two projects, same person, same local hardware, same class of model — one where the safety rails are the agent’s own opaque internals (and failed silently), one where the safety rails are hand-written by the non-developer owner because he understood exactly where an LLM would get a classical text wrong. The second approach produced fewer surprises.

Two more pieces from al-Khizanah worth folding in, in brief:

  • The project formalized its own governance, then the AI reviewed its own governance framework — and said so. Beyond the CLAUDE.md rules, al-Khizanah has a written “Proposal Review Framework” (impact/reversibility triggers a structured evaluation — problem framing, feasibility, operability, simplicity, evidence — before something becomes a standing decision, with the AI evaluating and Marc as sole decision authority). When the framework was reviewed against itself, the review opened by naming the conflict of interest outright (“I wrote this, and I am now also reviewing it — that is the exact blind spot this framework exists to catch”) and mostly landed on “Accept with Modification” rather than rubber-stamping its own work — including flagging that the new framework roughly tripled the conceptual overhead of the version it replaced, which is a direct violation of the project’s own stated preference for simplicity over bureaucracy, and that some of the new rules had quietly gotten too abstract for an explicitly non-developer audience. That’s a good, concrete example for the post of self-critique done well rather than performed — worth a paragraph, not a deep dive.
  • The exhaustive-search design is a clean, deliberate answer to the “silent failure” theme running through the rest of this doc. Ordinary semantic search answers “what’s relevant” with a ranked top-K list; al-Khizanah’s --exhaustive flag exists because a classical-text library often needs a completeness answer instead (“show me every place Ibn al-Qayyim discusses niyyah”), which a similarity ranking can’t honestly give — there’s no way to know if a relevant 11th passage got cut. So it runs a full lexical sweep (guaranteed-complete) and a semantic pass (suggestive, not complete) and keeps the two results visibly separate rather than merging them into one list. The README states the reasoning plainly: silently merging them “would make the tool feel more complete than it is — which for attributing a scholar’s words is worse than being obviously incomplete.” That’s the same instinct as the “never guess an author” rule, applied to search results instead of metadata — a nice one-sentence contrast against the Hermes/newsletter stories, where the failure mode was exactly a tool looking successful while quietly not being.

(Measured on the M2 Max, for a concrete performance data point if useful: Surya OCR at ~4.7s/page on ordinary prose, ~7.8s/page on dense typography; indexing ~984 passages in ~100 seconds. Current corpus: 5 works, 984 passages, 5/5 on the project’s own retrieval eval set.)

Assessment — what actually worked

  • The hybrid local/cloud model swap is real and low-friction. Flipping model.provider in config.yaml takes effect on the next turn, no restart. That’s the kind of thing that sounds like a nice-to-have on paper and turns out to matter a lot in daily use.
  • Bind-mounting an external project into the agent’s container, rather than duplicating code inside Hermes, kept the newsletter pipeline as a normal, independently-testable Python project while still letting Hermes run it.
  • Telegram as an “ambient” interface earns its keep once the underlying tool is solid — voice notes, “run the newsletter,” “send this as a one-off” — Hermes reliably executes already-built pipelines this way.
  • The end result is a real, low-maintenance win: a weekly newsletter for a mosque community that runs unattended on a cron job and can be operated ad hoc over Telegram, versus hand-building each send in Mailchimp.
  • al-Khizanah’s hand-written guardrails held up. Explicit, reasoned rules (“never guess an attribution,” “stop and tell me if metadata looks corrupted,” “test on a subset and read the output before scaling”) produced predictable behavior on a domain — classical Arabic/Islamic scholarship — where a silent wrong guess is genuinely costly, not just inconvenient.

Assessment — the honest friction (probably the more interesting half for a blog)

  • Hermes could not safely build a nontrivial new feature for itself via chat. Asked directly, over Telegram, to add the one-off-send capability, it tried to implement it as an internal Hermes “skill” and hit two guardrails: direct file writes are restricted to a safe root (/opt/data), so a patch against the actual project file outside that root was denied; and skill_manage refuses autonomous/background edits to user-owned skills that aren’t “curator-managed.” Both failures were silent — Hermes kept responding conversationally, asking clarifying questions, sounding like it was making progress, while nothing was actually persisted to disk. This only surfaced by checking agent.log directly and noticing the target file’s mtime hadn’t moved.
    • These guardrails are almost certainly intentional — you don’t want an LLM agent rewriting its own capability surface unsupervised — but the practical lesson is sharper: a chat agent’s transcript is not evidence that anything happened. The feature ended up being built the normal way, directly in the project, with the agent that had real file tools and no such restriction.
    • This suggests a fairly clean division of labor: Hermes/Telegram is for operating already-built tools day to day; nontrivial feature work happens in a proper dev session.
  • Silent correctness bugs in the newsletter renderer, twice in one day. The markdown-to-HTML conversion for the freeform “extras” content broke in two different ways that both passed a shallow “did the script exit cleanly” check: first, body markdown wasn’t being converted at all (raw ##/**/[text](url) went out in a live Mailchimp preview, caught only because the raw preview text got pasted back for inspection); later, **bold** and links worked but *italic* had no matching regex branch and passed through as literal asterisks. Neither failure produced an error — the script “succeeded” both times.
    • The fix wasn’t just patching the regex — it was adding a standing rule that any run has to grep the actual rendered HTML for leftover raw markdown syntax before being reported as clean, rather than trusting a zero exit code.
  • Infrastructure footguns that look like application bugs. A host-built Python virtualenv, mounted into the container, appeared to work — it ran without error — but was actually resolving to the container’s own system Python with none of the venv’s packages installed, because the venv’s python3 symlink pointed at a bare interpreter name rather than an absolute path. No exception was raised; it just silently used the wrong interpreter. The fix was building a separate venv inside the container’s own persistent volume, not reusing anything built on the host.
  • Config and secrets don’t share reload semantics, and that inconsistency is exactly the kind of thing that eats an evening: config.yaml is re-read live per turn, but .env is only loaded at container creation, and enabling a brand-new platform (vs. just changing an existing one’s credentials) additionally requires a real process restart because platform adapters are instantiated once at gateway startup, not reconciled continuously.

A few possible framings for the actual post

  1. “Running a hybrid local/cloud agent day to day” — what it’s actually like to have Qwen on an M2 Max MacBook Pro as your default model with cloud APIs as backup, and whether the local model holds up for real use across two very different workloads (an agentic chat assistant and a RAG pipeline).
  2. “Two guardrail philosophies, same hardware” — Hermes’s opaque, silent write-restrictions vs. al-Khizanah’s explicit, human-reasoned rules. This is probably the strongest single thread in the material: same owner, same local model family, wildly different outcomes when the agent hit a boundary.
  3. “Debugging automation you can’t fully see fail” — three separate silent-failure stories on the Hermes/newsletter side (skill writes, markdown rendering, the venv symlink) that all share the same shape: no error, wrong behavior, only caught by manually inspecting real output. Good contrast against al-Khizanah’s “stop and tell me” rule for the same class of problem.
  4. “What a non-developer’s AI guardrails look like when they’re right” — al-Khizanah’s CLAUDE.md as a case study in what good guardrails actually look like: not “restrict the agent because AI is risky” in the abstract, but rules that trace directly back to a real, named failure mode (guessed attributions, browser-printed PDFs with no text layer) the owner understood because he knows the domain.
  5. “Watching the AI grade its own homework” — al-Khizanah’s self-review of its own governance framework, conflict of interest named up front, mostly “Accept with Modification” rather than rubber-stamping, including a self-flagged violation of the project’s own simplicity principle. Short, but a genuinely rare thing to have a real transcript of.
  6. The concrete payoffs — a real automated newsletter serving an actual community (Middle Ground / muslim.center), operable over Telegram; and a private, offline search engine over a personal Islamic library that never leaves the Mac, with a search mode (--exhaustive) deliberately designed not to look more complete than it is. Good to anchor the more abstract lessons above to things that actually shipped and work.

Note: this doc reflects what’s recorded in build/debug history for the Hermes and newsletter projects, the LM Studio screenshot confirming the MacBook-side model setup, and al-Khizanah’s own CLAUDE.md. Still worth adding from your own experience: why Qwen specifically for each use case (vs. other local options), real-world tokens/sec or context-window behavior under load on the M2 Max, and how al-Khizanah’s search results have actually held up in practice.