Marc Manley

Domain-First Guardrails: Building al-Khizanah for Classical Scholarship

al-Khizānah (الخزانة, “the repository,” the classical Arabic term for a scholar’s private book collection) is a fully offline retrieval-augmented generation (RAG) search engine built over a personal library of Arabic and English Islamic texts.

The system runs on an M2 Max MacBook Pro with 64 GB RAM, querying a local LM Studio server (http://localhost:1234/v1) with zero cloud services or external API calls by design. While it shares underlying local infrastructure with daily-driver chat agents, al-Khizānah was built under an entirely different engineering discipline. Where general agent runtimes often rely on opaque, silent permission checks that look productive while failing to persist state, al-Khizānah’s rules are explicit, human-authored, and reasoned through in plain language.

The Offline Architecture and Performance

The pipeline is intentionally minimal, implemented across four core scripts—inventory, text extraction, index generation, and search. The project documentation explicitly forbids turning it into a sprawling framework, adding a web UI, or wrapping it in Docker: “This is five scripts and it should stay five scripts. If something feels like it needs a sixth, say so and explain why before writing it”.

  • Embeddings & Reranking: BGE-M3 (1024-dimension) paired with bge-reranker-v2-m3 running via PyTorch on Apple’s Metal backend (MPS).
  • Vector Store: LanceDB.
  • Generation: Local Qwen3-30B-A3B (a Mixture-of-Experts architecture) served directly through LM Studio.
  • Corpus & Throughput: On an initial corpus of 5 works across 984 passages, indexing takes ~100 seconds on the M2 Max, scoring 5/5 on the project’s retrieval evaluation suite.
  • OCR Benchmarks: Surya OCR processes ordinary prose at ~4.7 seconds per page and dense typography at ~7.8 seconds per page.

Hand-Authored Guardrails in CLAUDE.md

In classical Arabic and Islamic scholarship, a silent error is far more dangerous than an outright crash. The guardrails documented in al-Khizānah’s CLAUDE.md trace back to concrete failure modes understood by the domain owner:

  • Never Guess an Author Attribution: Classical text author and title metadata is hand-labeled only. A guessed attribution is invisible downstream and silently corrupts every author-filtered query. A blank field is obvious and fixable; an unverified guess is poison. The assistant may flag inconsistencies, but it is barred from populating missing metadata.
  • Stop, Don’t Self-Heal: If a hand-labeled metadata file is found wiped, truncated, or corrupted mid-task—even by the agent’s own operations—the instruction is to stop and notify the user immediately. Attempting to automatically reconstruct metadata from context re-introduces the exact silent corruption the rule exists to prevent.
  • Pre-Flight Subsets and Inspect Real Output: Before running slow, expensive, or hard-to-undo operations, the agent must test on a tiny sample and force inspection of the output. Rather than relying on a vague “be careful” directive, the system uses benchmarked measurements—such as extrapolating Surya OCR’s ~4.7s/page rate to estimate a 322-page book at ~25 minutes—before committing compute.
  • Run Heavy Compute in the User’s Terminal: Long-running index builds or OCR passes must never block interactive chat sessions. The assistant is required to output the exact nohup/caffeinate terminal command for the user to execute and monitor with tail -f.
  • Catching Phantom PDF Layers: Browser-based “Print to PDF” web captures often lack a real text layer, passing simple inventory checks while containing zero retrievable content. In one documented instance, a 10-page article extracted down to just 1,275 characters of Safari print furniture (headers and URLs). The pipeline now explicitly validates actual body text density.

Standard RAG implementations rely purely on semantic similarity, returning a top-$K$ ranked list answering “what is relevant?” In classical research, scholars often need completeness: “Show me every place Ibn al-Qayyim discusses niyyah (intention)”. A standard vector ranking cannot guarantee whether an eleventh, critically relevant passage was cut off.

al-Khizānah addresses this with a dedicated --exhaustive search mode: