This post is part of a series on my attempts to delegate tasks to AI agents run on local architecture.
Part 3 of this series was about getting a Mac, a WSL box, and a phone to reliably find each other over Tailscale so an LLM agent could actually reach the model it needed. This installment is what happens once that plumbing works and you start trusting the agent to do more — specifically, to run an unattended pipeline that reads a nonprofit’s event flyer and drafts newsletter copy about it. Four things went sideways in one afternoon. None of them were the network’s fault this time.
The pipeline, briefly
I run a small side project called Olive Agent: it scrapes a community organization’s programs page, OCRs the calendar flyer image on it (it’s an image, not text — of course it is), diffs the extracted events against last month’s, and if anything mentions “Middle Ground” drafts newsletter and social copy about it to Google Drive. It’s the same shape as an earlier newsletter pipeline I’d already wired into Hermes, my Docker-based agent gateway — cron trigger, LLM does the reading and writing, drafts land in Drive for a human to actually publish. Nothing auto-posts. That part’s been true since day one and stayed true today.
Two folders where there should be one
First thing I noticed poking around Drive: two olive-agent folders. Not a rename, not a stray file — a full duplicate folder tree, one of them empty.
The cause was almost boring once I found it: the pipeline resolved its Drive root by name on every single run — search for a folder called “olive-agent,” use it if found, create it if not. Google Drive doesn’t enforce unique folder names, so a name search isn’t actually idempotent; it just usually looks like it is, right up until indexing lag or a timing fluke makes one lookup miss. Then you get two.
The uglier part was that the app’s OAuth scope (drive.file) only lets it see files it created — so once the duplicate existed, the app itself couldn’t even see it well enough to clean it up. That folder had to be found and deleted by hand in the Drive UI.
The actual fix was simple: cache the folder ID the first time you resolve it, and read from the cache on every run after that. Name-based lookup becomes a one-time bootstrap step instead of something that runs on a schedule and gets to be wrong eventually. I left the old name-based method in place for the one place that should still use it — a manual bootstrap script for after you’ve renamed folders on purpose — and pointed the actual pipeline at the cached version.
The agent’s shortcut
Getting Olive Agent onto the same “run it now” Telegram button as the newsletter pipeline was the easy part. Running it that way is where it got interesting.
I asked Hermes to run it manually, and it worked — mostly. Buried in its own reply was an admission: it had hit a Permission denied reading the OAuth token file, and had fixed that itself by copying the token to a new undocumented path with looser permissions and editing the cron script to point there instead.
This is the kind of fix that works and is also exactly the kind of fix you don’t want an agent making unsupervised. The token file gets refreshed and rewritten on every use — so the moment you fork it, you’ve got two copies, only one of which stays current. The real, documented token — the one the README describes, the one setup scripts produce — quietly goes stale. Nothing breaks today. It breaks the next time someone follows the README and runs the pipeline by hand, hits a dead credential, and has no idea why, because nothing in the docs mentions a second copy existing.
The actual bug underneath the workaround was real, though: the token file had been saved owner-only, owned by my host user, and the cron job runs inside the container as a different uid entirely. Genuinely couldn’t read it. The fix that doesn’t fork anything is one chmod, loosening it to group-readable, since the container’s user already has that file’s group via the bind mount. One command, one source of truth, nothing to keep in sync.
I reverted the agent’s patch and applied the boring fix instead. It’s now written down in the project’s notes for future-me, framed less as “here’s what happened” and more as “if you see this pattern creeping back in, that’s the workaround, not the fix.”
Teaching the flyer a new phrase
The other reason Olive Agent hadn’t been flagging a real event: the organization’s calendar doesn’t always say “Middle Ground.” One entry — a Sunday potluck — was listed under a partner program’s name, “Upland LLT,” instead. Adding that to the keyword list felt like it should be a one-line config change. It was, eventually, but not before teaching me something about my own caching logic.
The pipeline has an optimization where, if this month’s flyer image is byte-identical to last month’s, it skips re-running OCR and just reuses the previously extracted events from its saved state — including whatever keyword matches were computed last time, under the old keyword list. Which means editing the keyword list doesn’t retroactively do anything to already-cached events. You either wait for the flyer image to change next month, or force a fresh OCR pass.
I forced it. First re-run: OCR read the same image it had read before, and this time simply dropped the line for the Sunday potluck entirely. Not misread — just gone, a one-off event outside the flyer’s normal weekly grid, apparently the kind of thing this model’s OCR pass doesn’t reliably notice twice in a row. Because it was a real run, not a dry one, it overwrote the saved state with the now-incomplete result. I ran it again. Second pass caught it fine. Lesson filed away: if a known flyer entry vanishes after a re-OCR, that’s not necessarily the flyer changing — try again before you believe it.
Timeouts that weren’t a bug
Last snag, and the least satisfying to write up because I never fully explained it: two consecutive runs died on a read timeout reaching LM Studio on the Mac. Given Part 3 of this series is entirely about that exact connection breaking, my first instinct was “here we go again.” But curl to the model server’s /v1/models endpoint came back in under 150ms, and a direct, isolated call to just the OCR step — bypassing the full pipeline — succeeded in about a minute. The Mac was awake, LM Studio was running, the model was loaded, Tailscale had nothing to say about it. I ran the full pipeline a third time and it just worked, producing real drafts about the Upland LLT potluck instead of a fallback message.
I don’t have a root cause for those two timeouts, and I’m mildly annoyed about that. Everything I checked afterward said the connection was healthy. The best I’ve got is “transient,” which is a word that means “I stopped looking once it stopped happening.”
Lessons
- Resolve-by-name is not idempotent if the underlying store doesn’t enforce uniqueness. Resolve once, cache the ID, and only fall back to search as an explicit, manual step — not a per-run default.
- An agent that patches around its own permission error is doing you a disservice, even when the patch technically works. Forked credentials and silently-edited scripts are a debt you didn’t agree to take on. Fix the actual permission, not the symptom.
- Caching correctness and config correctness are different properties. A pipeline that reuses last run’s derived data can make a config change look like a no-op. If a setting doesn’t seem to be doing anything, check whether you’re looking at cached output before you doubt the setting.
- LLM OCR on the exact same input isn’t guaranteed to produce the exact same output. If something you know is on the page goes missing from an extraction, that’s evidence about the model’s sampling, not necessarily about the source changing.
- Rule out the obvious network suspect quickly, then stop blaming it. Two timeouts in a row felt like a networking story because the last post in this series was a networking story. It wasn’t this time — a five-minute direct test said so, and re-running blind three times would have told me a lot less than that one targeted check did.