Building automation with AI agents introduces a specific failure mode: workflows that pass a standard exit-code check, or a calm-sounding chat transcript, while quietly failing in production. Nothing throws. Nothing looks wrong. The only way to catch it is to go look at the actual output.
The pipeline: Middle Ground newsletter
To automate weekly and ad hoc email campaigns for Middle Ground (muslim.center), I built generate_newsletter.py — an independent Python tool, bind-mounted into a self-hosted Hermes agent container rather than folded into Hermes’s own codebase. It:
- Scrapes
muslim.centerfor Jumuʿah khatib/date, three “Learning” cards, the latest podcast episode, and the latest blog post. - Pulls an optional freeform “extras” item — markdown plus an image — from a shared Google Drive folder via a service-account credential.
- Renders everything into email-safe HTML, carefully preserving the VML/MSO Outlook button hack, and pushes a real Mailchimp draft campaign.
- Runs unattended via cron on Thursdays at 10am Pacific, and via an ad hoc
--one-offmode triggered over Telegram (prayer reminders, event announcements).
To keep rendering predictable across email clients, the visual rules were deliberately split into two documents: a full audit of the live site’s design system, and a condensed “email-safe” token doc with an explicit “these have no email equivalent, don’t use them” list — hover states, backdrop-blur, CSS transitions, ARIA landmarks. That split exists specifically because handing an LLM the full web-design doc risks it reaching for browser-only techniques that silently don’t render in an inbox.
Three silent failures
1. Hermes couldn’t build its own feature — and didn’t say so. Asked directly, over Telegram, to add the one-off-send capability, Hermes tried to implement it as an internal “skill” and hit two guardrails: direct file writes are restricted to a safe root (/opt/data), so a patch against the real project file outside that root was denied; and skill_manage refuses autonomous edits to user-owned skills that aren’t “curator-managed.” Both failures were silent — Hermes kept responding conversationally, asking clarifying questions, sounding like it was making progress, while nothing was actually persisted to disk. I only caught it by checking agent.log directly and noticing the target file’s mtime hadn’t moved.
These guardrails are almost certainly intentional — you don’t want an agent rewriting its own capability surface unsupervised. But the practical lesson is sharper than “guardrails exist”: a chat transcript is not evidence that anything happened. The feature ended up getting built the normal way, in a real dev session, by the agent that had unrestricted file tools.
2. The markdown renderer broke twice in one day, and both times “succeeded.” The markdown-to-HTML conversion for the freeform “extras” content failed in two different ways, and neither produced an error. First, body markdown wasn’t being converted at all — raw ##, **, and [text](url) went out in a live Mailchimp preview, caught only because I happened to paste the raw preview text back in for inspection. Later, after that fix, **bold** and links worked but *italic* had no matching regex branch, so it passed through as literal asterisks. The script exited cleanly both times.
The fix wasn’t just patching the regex — it was adding a standing rule that any run has to grep the actual rendered HTML for leftover raw markdown syntax before being reported as clean, rather than trusting a zero exit code.
3. A working virtualenv wasn’t. A Python virtualenv built on the host, then mounted into the container, ran without error — but was silently resolving to the container’s own system Python with none of the venv’s packages installed, because the venv’s python3 symlink pointed at a bare interpreter name instead of an absolute path. No exception, no warning — it just quietly used the wrong interpreter. The fix was building a separate venv inside the container’s own persistent volume, rather than reusing anything built on the host.
The shape of the problem
All three share the same shape: no error, wrong behavior, caught only by manually inspecting real output. That’s a bad match for how these systems get evaluated day to day — a clean exit code and a fluent chat transcript both look like success.
It’s a useful contrast against a second project running on the same hardware — al-Khizanah, an offline RAG search tool with an explicit “stop and tell me” rule: if hand-labeled metadata is ever found wiped or wrong, even by the agent’s own hand mid-task, the instruction is to stop immediately rather than try to reconstruct it from context. That rule exists for exactly this failure mode — an agent quietly doing the wrong thing and continuing as if nothing happened. Hermes’s guardrails were the agent’s own opaque internals; al-Khizanah’s were written by hand, by someone who knew exactly where things could go silently wrong. The second approach produced fewer surprises.