Corpus round-trip and integrity
The markdown corpus and the graph are two representations of the same knowledge, and the code treats their agreement as a contract: one shared edge derivation, one shared section-header map, a reconcile pass that prunes drift, and 32 integrity checks that fail loud.
- How markdown becomes graph, and how the graph becomes markdown again. One measured caveat (2026-08-01): the markdown round-trip is faithful for rules and methodology, but NOT for the whole graph. Rebuilding from
bible/drops the 62 Abstraction nodes (they live inbible/abstractions.json, a separate materialization path) and re-derives a different RELATED_TO edge set. The faithful whole-graph restore is the Cypher dump:writ export-cypher/writ import-cypher. - Which edges are recomputed on every ingest, and why export deliberately leaves them out.
- What reconcile deletes, and what it is forbidden from touching.
- How the Abstraction summary layer is generated with zero LLM involvement.
- Diagram convention: the dark green rounded block is where the flow starts (START or INPUT); orange blocks are outcomes; numbered steps show reading order.
4.1The round-trip cycle
Ingest (ingest_path, writ/graph/methodology_ingest.py:168) parses two source formats: RULE-START blocks for coding rules and YAML front-matter for methodology nodes. Everything validates through the per-type Pydantic models before any write, node writes are batched with uniqueness constraints applied fail-loud, and edges are derived by one pure function. Export (writ/export.py) renders the graph back to the same two formats; the section-header map both directions use is a single shared constant (SECTION_HEADERS, schema.py:721), so a header rename cannot silently break the cycle.
flowchart LR
BIBLE(["START and END of the cycle:<br/>bible/ markdown corpus"])
subgraph ING["ingest_path"]
P["1. Parse<br/>RULE-START blocks + YAML front-matter"]
V["2. Validate<br/>per-type Pydantic models"]
W["3. Batch node writes<br/>+ uniqueness constraints, fail-loud"]
E["4. derive_edges<br/>one pure derivation, 4 sources"]
end
NEO[("Neo4j")]
RECON["reconcile<br/>deletes stale nodes and edges,<br/>clears stale managed properties.<br/>Records and graph-first proposals exempt"]
subgraph EXP["Export"]
R1["5a. rule_to_markdown<br/>RULE-START render,<br/>declared edges only"]
R2["5b. node_to_yaml_frontmatter<br/>methodology render"]
end
BIBLE -->|"ingest"| P --> V --> W --> NEO
W --> E --> NEO
BIBLE -->|"compare"| RECON
RECON -->|"prune"| NEO
NEO -->|"export"| R1 --> BIBLE
NEO -->|"export"| R2 --> BIBLE
classDef entry fill:#0B5E55,stroke:#083F39,color:#FFFFFF,font-weight:bold
classDef finish fill:#F6EEDF,stroke:#B4560F,color:#6B3D14,font-weight:bold
classDef hub fill:#DCE9E5,stroke:#0B5E55,stroke-width:2px
class BIBLE entry
class NEO hub
4.2Edge derivation: one function, four sources
derive_edges (methodology_ingest.py:651) is a pure function that produces the exact edge set a clean ingest would create. The reconcile oracle and the writer both call it, by design: two independent derivations drifting apart was the silent-failure class this closed. Dangling references, unknown ids or cross-project ids, are counted and reported, never silently resolved (methodology_ingest.py:720-767).
flowchart LR
SRCP(["START: parsed source<br/>nodes and edges"])
D1["1. Declared edges<br/>front-matter edges: lists<br/>+ RULE-START ### Edges sections"]
D2["2. RELATED_TO<br/>rule-id cross-references<br/>found in Rule prose"]
D3["3. BELONGS_TO<br/>from each node's category field"]
D4["4. BELONGS_TO<br/>child Category to parent Category"]
UNW["Single UNWIND transaction<br/>label-scoped index seeks<br/>dangling references counted"]
NEO[("Neo4j")]
SRCP --> D1 --> UNW
SRCP --> D2 --> UNW
SRCP --> D3 --> UNW
SRCP --> D4 --> UNW
UNW --> NEO
classDef entry fill:#0B5E55,stroke:#083F39,color:#FFFFFF,font-weight:bold
classDef finish fill:#F6EEDF,stroke:#B4560F,color:#6B3D14,font-weight:bold
classDef hub fill:#DCE9E5,stroke:#0B5E55,stroke-width:2px
class SRCP entry
class NEO hub
The export asymmetry is intentional
- Only declared edges are written back to markdown; the two export-derived types (RELATED_TO, BELONGS_TO;
writ/export.py:62,DERIVED_EDGE_TYPES) are excluded so a re-import re-derives them instead of freezing them into source. ABSTRACTS edges are outside this mechanism entirely: they belong to the compression pipeline and rebuild frombible/abstractions.json(section 4.3), not from the markdown export. - Plain ingest is upsert-only.
reconcile()(methodology_ingest.py:500) is the separate pruning pass: it compares source against graph and deletes what source no longer contains. - Reconcile has hard limits: nodes whose provenance is
recordor graph-first (proposed,graduation_pending) are exempt from deletion, and runtime properties like the frequency counters can never be cleared (RUNTIME_EXEMPT_PROPS,schema.py:742). - A timestamp file supports staleness detection: export records when it ran, and a check compares that against the last graph write (
export.py:454-486).
4.3Compression: a disposable summary layer
When the retrieval budget is tight, Abstraction nodes stand in for whole clusters of rules (chapter 2, summary mode). The layer is generated fully deterministically, with no LLM: run_compression (writ/compression/abstractions.py:31) embeds every non-mandatory rule, clusters twice, keeps the better result, and each cluster's summary is simply the statement of the rule nearest the cluster centroid.
flowchart LR
START(["START: writ compress runs<br/>(offline, after corpus changes)"]) --> NM
NEO[("Neo4j")]
NM["Non-mandatory rules<br/>for the project"]
EMB["Embed each rule's<br/>trigger + statement<br/>(offline only)"]
HD["HDBSCAN<br/>auto-discovers cluster count"]
KM["k-means<br/>k swept up to 15"]
PICK{"Better<br/>silhouette score?"}
GEN["One Abstraction per cluster:<br/>summary = centroid-nearest rule's statement<br/>no LLM, deterministic"]
WR["Replace the project's<br/>Abstraction nodes + ABSTRACTS edges"]
ART["bible/abstractions.json<br/>dependency-free replay artifact"]
NEO --> NM --> EMB
EMB --> HD --> PICK
EMB --> KM --> PICK
PICK --> GEN --> WR --> NEO
GEN --> ART
classDef entry fill:#0B5E55,stroke:#083F39,color:#FFFFFF,font-weight:bold
classDef finish fill:#F6EEDF,stroke:#B4560F,color:#6B3D14,font-weight:bold
classDef hub fill:#DCE9E5,stroke:#0B5E55,stroke-width:2px
class START entry
class WR finish
class ART finish
The JSON artifact matters operationally: the clustering dependencies (scikit-learn, sentence-transformers) are excluded from production installs, so a later ingest can rematerialize the abstraction layer from the artifact without re-clustering (abstractions.py:26-56).
4.4Integrity: 32 checks in 7 clusters
IntegrityChecker (writ/graph/integrity/__init__.py) composes seven check mixins. writ validate renders findings through a pure reporter (validate_report.py) and exits non-zero on any finding, so it can gate automation.
| Cluster | Checks | Guards against |
|---|---|---|
| Structural (7) | conflicts, orphans, orphans across all labels, dangling dispatched roles, stale, redundant, confidence defaults | Contradictory or disconnected knowledge; near-duplicate rules caught by cosine similarity at 0.95. |
| Frequency (6) | stranded mandatory, ranked-exclusion mismatch, always-on budget breach, unreviewed count, frequency stale, graduation flags | The mandatory/ranked invariant (chapter 3); injection budget creep; stalled graduations. |
| Parity (5) | category reachability, source parity, edge parity, property parity, methodology field drift | The graph diverging from the markdown corpus. |
| Edge contracts (4) | dispatch/invokes invariant, teaches source, counter nodes, dispatched-by parity | Edges violating their declared endpoint contracts. |
| Content (5) | example lint, domain enum, enforceable-severity coupling, forbidden-phrase overlap, shared code example | Taxonomy drift and broken worked examples. |
| Routing (4) | floor completeness, trigger-keyword invariant, push reachability, action vocabulary closure | Channel 3 routing data that would silently stop matching. |
| Artifact (1) | dangling rule ids in the abstractions artifact | The compression artifact referencing deleted rules. |
Every invariant that spans two mechanisms is encoded once and imported by both sides: the edge derivation (writer and reconcile oracle), the section headers (export and ingest), the mandatory predicates (pipeline and validator). Where one definition is impossible, a paired integrity check exists instead. Drift between representations is treated as a bug class with a named owner, not an operational surprise.