Marc Manley

Building Musnad Middle Ground: Phases 1–4

Musnad Middle Ground is a new section of the Middle Ground website designed to become a structured repository of the aḥādīth cited, quoted, discussed, or paraphrased across Middle Ground khuṭbahs, classes, blog posts, Q&A posts, podcasts, and other teaching material.

The goal is not simply to create a list of aḥādīth. Each canonical ḥadīth will eventually include its Arabic matn, an English translation, narrator information, source and citation data, and a list of every Middle Ground resource in which it has been used. Over time, this will allow a single ḥadīth to function as a durable source asset connected to the wider Middle Ground knowledge base.

The project has now completed four infrastructure phases.

Phase 1 — Canonical Data Model and Validation

The first phase established the Musnad’s basic data architecture.

Because Middle Ground is a hand-authored static HTML site with no CMS, database, or existing structured data layer, the decision was made to store each canonical ḥadīth as its own JSON file.

Canonical records use permanent opaque IDs such as:

mm-h-0001
mm-h-0002

These IDs are intentionally independent of collection name, ḥadīth number, Arabic wording, or slug so that a record’s identity does not change if its citation or wording is corrected later.

The schema was designed to distinguish several layers that must never be conflated:

  • the canonical ḥadīth itself;
  • what Middle Ground originally published or cited;
  • what later independent verification establishes;
  • the places where Middle Ground has used the ḥadīth.

A validator was also built to enforce the schema, detect malformed records, duplicate IDs, duplicate slugs, invalid status values, broken relationships, and inconsistent verification states.

Main challenge

One early design mistake was treating a complete published citation as if it were already “verified.”

That would have collapsed the distinction between:

what the Middle Ground page says

and:

what Musnad Middle Ground has independently confirmed.

Fix

The verification standard was tightened.

A citation can be complete, specific, and even linked directly to a source while still remaining only:

citations.as_published

It does not become:

citations.verified

until a separate human verification process confirms it.

Phase 2 — Historical Candidate Extraction

The second phase addressed the historical corpus.

The goal was to scan existing Middle Ground content and identify likely aḥādīth without automatically turning them into canonical Musnad records.

This became more complicated than a simple HTML scrape.

Middle Ground’s existing markup uses the same general blockquote structure for Qurʾān, ḥadīth, scholarly quotations, and other cited material. Some aḥādīth are fully cited. Others have a narrator but no collection. Some are completely unattributed. Others are paraphrased in prose rather than quoted directly.

The extractor therefore uses two layers:

broad deterministic discovery
        ↓
conservative classification
        ↓
staging for review

Candidate passages are staged separately from canonical records and receive temporary IDs such as:

mm-c-0001

The first full scan of the current corpus examined 38 content files and produced 48 candidate passages:

20 prophetic_hadith
15 uncertain
12 quran
1 scholar_statement

No historical candidate was automatically promoted into the canonical Musnad.

Main challenges

The largest problem was false-positive detection.

For example, the word “Muslim” initially acted as a strong signal because it could refer to Ṣaḥīḥ Muslim. On a Muslim website, however, the ordinary word “Muslim” appears everywhere.

This flooded the extractor with irrelevant passages.

A second major challenge was distinguishing Qurʾān quotations from ḥadīth when the HTML structure was nearly identical.

Fixes

The signal rules were narrowed so that “Muslim” only acts as a collection signal in more specific forms such as:

Ṣaḥīḥ Muslim
Muslim no. 11

Qurʾān-specific signals were added to both discovery and classification.

The extractor was also given explicit regression tests for fully cited bilingual aḥādīth, uncited aḥādīth, narrator-only material, paraphrased narrations, Qurʾān immediately adjacent to ḥadīth, and Arabic/English blocks representing the same narration.

The result is deliberately conservative: ambiguity is staged for review rather than guessed.

Phase 3 — Human Review, Matching, and Promotion

The third phase built the human-review workflow.

Detection alone is not enough. Once the extractor identifies a probable ḥadīth, a human still needs to decide whether the candidate should be dismissed, deferred, linked to an existing canonical record, or promoted as a new canonical record.

Review decisions are stored separately from staging so that a future extraction run cannot erase human judgment.

The review system can now:

  • list and filter candidates;
  • inspect a candidate in detail;
  • compare it against existing canonical records;
  • dismiss false positives;
  • defer unresolved cases;
  • link a historical occurrence to an existing ḥadīth;
  • promote a reviewed candidate into a new canonical record;
  • perform dry runs before any canonical write.

Candidate-to-canonical matching uses several levels of evidence. Strong evidence includes exact collection + number and exact normalized Arabic wording. Fuzzy text similarity is only a suggestion.

The system never automatically decides that two narrations are the same canonical ḥadīth simply because their meanings are similar.

Main challenges

A serious identity problem surfaced during testing.

Candidate IDs were originally assigned relative to the current global maximum. That meant rescanning one file after unrelated corpus changes could cause an unchanged candidate to receive a different mm-c-* ID.

That is unacceptable once human review records exist.

A second issue was usage identity. Deduplicating only by source filename would prevent the same ḥadīth from being represented twice if it appeared in two different places within the same article.

Fixes

Candidate identity was redesigned around persistent content-addressed fingerprints.

A candidate’s identity now derives from stable occurrence content rather than scan order or line number.

The same fingerprint is also used to detect review drift and to support stable usage identity.

Usage records now have an occurrence-aware usage_id, allowing two genuinely distinct uses of the same ḥadīth in one source file to coexist.

Phase 4 — Scholarly Verification and Publication Readiness

The fourth phase created a separate scholarly verification workflow.

This phase preserves four distinct layers:

what Middle Ground published
        ↓
human-reviewed canonical identity
        ↓
independently verified source
        ↓
publication readiness

A new verification command can now record:

  • canonical Arabic text;
  • canonical English translation;
  • verified collection and citation;
  • narrator information;
  • source relationships;
  • citation conflicts;
  • verification notes;
  • review state.

Most importantly, scholarly verification and public presentation are not treated as the same thing.

A record can be internally valid while still not being ready for a public Musnad page.

New bilingual publication requirement

Every public Musnad entry will eventually be required to contain:

  1. the Arabic matn;
  2. the English translation directly beneath it.

Internal records may remain incomplete while they are still under review, but public Musnad pages will not be generated unless both languages are present.

The system also now records provenance for both Arabic and English so that the project can distinguish, for example, between wording quoted historically by Middle Ground, wording taken from an independently verified source, a Middle Ground translation, and a reviewer translation.

Main challenge

The key issue in Phase 4 was avoiding a second kind of conflation:

“verified” does not automatically mean “ready to publish.”

A source may be correctly verified while the English translation is still missing, or the canonical Arabic wording has not yet been selected.

Fix

Publication readiness is computed separately rather than stored as a second state that could become stale.

A readiness check now evaluates whether a record has the data required for eventual public rendering.

Where the Project Stands Now

The infrastructure for Musnad Middle Ground is now largely in place.

The system can:

  • scan the historical Middle Ground corpus;
  • identify likely ḥadīth occurrences;
  • stage them without altering canonical data;
  • preserve source provenance;
  • distinguish Qurʾān and other false positives;
  • review candidates;
  • link repeated usages;
  • promote new canonical records;
  • independently verify sources;
  • preserve historical citation discrepancies;
  • enforce bilingual publication requirements;
  • determine whether a canonical record is ready to publish.

What has not happened yet is equally important:

  • the historical corpus has not been mass-adjudicated;
  • the 20 probable ḥadīth candidates have not been automatically promoted;
  • the current real records have not been mass-verified;
  • no public Musnad pages have yet been generated.

That restraint is intentional.

Phase 5 — Public Musnad Pages

Phase 5 will move from infrastructure to visitor-facing design.

Before the full page generator is built, the plan is to complete a small pilot group of approximately three to five real aḥādīth through the complete verification process.

Those pilot records will help determine what a finished Musnad entry actually needs to look like.

The working page anatomy is expected to include:

  • Musnad ID;
  • an editorial descriptive title;
  • Arabic matn;
  • English translation immediately underneath;
  • narrator;
  • verified source;
  • citation or verification notes when needed;
  • topics;
  • a prominent Used at Middle Ground section linking back to every khuṭbah, class, post, podcast, or other resource in which the ḥadīth appears.

That final section is central to the project.

Without it, Musnad Middle Ground would risk becoming simply another ḥadīth database.

The real value is the relationship layer:

ḥadīth
↕
Middle Ground teaching
↕
topics
↕
classes
↕
khuṭbahs
↕
podcasts

Phase 5 will therefore focus on how to turn the verified data model into a clear, readable, searchable public experience while preserving the scholarly guardrails established in Phases 1–4.