docs(research): parser/generator ladder — ZetaId bits → value-tree codecs → ZetaParse Grammar IR

docs/research/2026-07-02-parser-generator-foundation-ladder-zetaid-bits-to-value-tree-codecs-to-zetaparse-grammar-ir-antlr.md

@@ -0,0 +1,104 @@
+# The parser/generator ladder — ZetaId bits → value-tree codecs → ZetaParse Grammar IR (ANTLR ingest)
+
+**Date:** 2026-07-02
+**Author:** Otto (shadow*), synthesising Aaron's stream against the existing foundation
+**Status:** orientation / scope synthesis (no new code) — connects three existing bodies of work
+
+> Carved sentence: **Zeta already has a spec → generated-codec pipeline at three rungs — bit
+> (ZetaId), format (value-tree codecs), and (designed) CFG/text (ZetaParse). They are ONE
+> ladder: a grammar/spec is ingested into an IR and a parser/generator is emitted from it. The
+> parser-generator foundation is the CFG rung of that same ladder, built on the ZetaParse
+> design and consuming the open ANTLR grammars — not reinvented.**
+
+## What already exists (take it into consideration — Aaron 2026-07-02)
+
+Aaron: *"we have some small parser combinator foundation already … our zetaid is a tiny parser
+generator kind of thing … bits, a bit parser/generator … the ANTLR stuff is likely just some
+docs/research/persona conversation."* Confirmed by inspection:
+
+### Rung 1 — BUILT: ZetaId, a spec-driven **bit** parser/generator
+
+`src/Core.FSharp.ZetaId/` — `BitLayout.fs` reads/writes a 128-bit ZetaId into structured fields
+(version, timestamp, chromosome, category, firefly, authority, persona, momentum, location, …)
+with a `next width` / `skip bits` cursor; `GeneratedBitLayout.fs` **generates** those field
+widths from a layout spec (`docs/zeta-id-v1-layout.yaml`). This is already **spec → generated
+layout → parser/generator**, at the fixed-bit-layout level. The tiny foundation is not a toy —
+it is the bit-level instance of exactly the pattern the big scope wants.
+
+### Rung 2 — BUILT (this session): value-tree codecs
+
+`src/Core/ValueTreeCodec.fs` + `ValueTreeEnvelope` / `Asn1Der` / `EventEnvelope` / `Frontmatter`
+(#9185–#9196). Each codec is a **spec (a format) → a parser/generator** over the `DynamicValue`
+value tree, behind an owned hexagonal port, provenance-tracked, versioned/rollable. The
+format-level rung.
+
+### Rung 3 — DESIGNED (docs/research, not built): ZetaParse + ANTLR ingest
+
+- `docs/research/zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21.md` (Amara-in-Zeta):
+  **ZetaParse** — a compiler-owned F# grammar substrate. Ingest grammar assets (ANTLR `.g4`,
+  Yacc/Bison `.y`, Tree-sitter `grammar.js`, Zeta native `.zg`) into a shared **Zeta Grammar
+  IR**, normalise the compatible subset, detect conflicts, and generate F# parsers via a
+  Zeta-owned LR/GLR backend. Key stance: *"do not run ANTLR grammars directly … ANTLR remains
+  useful as a source of community grammars; it does not need to own the parser runtime."*
+- `docs/research/antlr-grammar-survey-{2026-05-21,2026-06-13}.md` (Lior): the licensing +
+  6-language target matrix. **`antlr/grammars-v4` is MIT/BSD** — zero compliance hazard; a ready
+  corpus of community grammars to ingest across F#/C#/TS/Rust/Go/Python.
+- Related: `docs/research/2026-06-14-zeta-language-and-canonical-ir-compiler-pipeline-design.md`,
+  `…-ir-compiler-v2-…-futamura.md`, `2026-06-07-the-zeta-idl-…`; `gen/` (generators read the
+  free interface).
+
+## The synthesis (what this doc adds)
+
+The three rungs are the **same discipline** — `spec → IR → generated parser/generator` — at
+increasing grammar power:
+
+| Rung | Spec | IR | Emitted | Status |
+|------|------|----|---------|--------|
+| bit | `zeta-id-v1-layout.yaml` | field/width table | `BitLayout` read/write | ✅ built |
+| format | a codec (JSON/CBOR/DER/…) | `DynamicValue` | `ValueTreeCodec` encode/decode | ✅ built |
+| CFG/text | ANTLR `.g4` / `.y` / `.zg` | **Zeta Grammar IR** | LR/GLR parser (6 langs) | 📐 designed (ZetaParse) |
+
+So the parser-generator foundation is **not a green field**: it is rung 3 of a ladder whose
+rungs 1–2 are already load-bearing, and whose rung-3 design (ZetaParse Grammar IR + ANTLR
+ingest) already exists. Two prior findings converge here and are **subsumed** by it:
+
+- *lenient YAML parser* (the frontmatter finding, #9196) — a rung-3 consumer (a YAML grammar
+  ingested to the Grammar IR yields the lenient parser).
+- *general KDL reader* (the delayed fork) — likewise a grammar (KDL `.g4` exists in grammars-v4)
+  ingested to the Grammar IR. Neither needs a bespoke hand-parser; both fall out of rung 3.
+
+## Scope (Aaron 2026-07-02) and how the foundation shapes it
+
+> *"it's a LARGE scope — we want to be able to compile to and from our IR and most other ANTLR
+> grammars using our parser generators and the open free ANTLR grammars; small changes to
+> existing ANTLR grammars to make it work with our stuff is fine."*
+
+This is exactly the ZetaParse stance, now the standing intent: **to/from** the Zeta Grammar IR
+and the grammars-v4 corpus; our LR/GLR generator owns the runtime; small `.g4` normalisations to
+fit the compatible subset are acceptable (ZetaParse's "normalise the compatible subset"). The
+anti-reinvention principle is the same one behind reusing TOSEC/MAME for game signatures
+(#9184): **grammars-v4 is the existing, license-clean corpus — build on it, don't re-author
+grammars.**
+
+## Next moves (recorded; large, so scope with Aaron)
+
+1. **Zeta Grammar IR as a `DynamicValue` schema** — the IR is a value tree, so it rides the
+   codecs already built (rung 2 serves rung 3: a grammar is data, byte-lockable, DST-replayable).
+2. **`.g4` → Grammar IR ingester** (the compatible subset first; log what is dropped — no silent
+   truncation), starting from one or two grammars-v4 assets (e.g. JSON, then KDL/YAML to retire
+   the two subsumed findings).
+3. **LR/GLR backend** emitting an F# parser from the Grammar IR (ZetaParse rung); the ZetaId
+   bit-generator is the proven-in-miniature precedent for spec-driven generation.
+
+Sizes 2–3 are large and design-heavy; do them under Aaron's scope steer, not unilaterally.
+
+## Anchors (Beacon)
+
+- **Existing Zeta work:** ZetaParse (`zetaparse-lr-glr-…-2026-05-21`, Amara); ANTLR survey
+  (`antlr-grammar-survey-{2026-05-21,2026-06-13}`, Lior); ZetaId `Core.FSharp.ZetaId`;
+  value-tree codecs (`docs/trajectories/value-tree-codecs/RESUME.md`).
+- **Parser theory:** Knuth (LR); Tomita (GLR); Parr (ANTLR / LL(*)); tree-sitter (incremental
+  GLR); Bison (LALR/IELR). **Corpus:** `antlr/grammars-v4` (MIT/BSD).
+- **Discipline:** `only-the-irreducible-is-primitive-generate-the-rest` (the generator IS the
+  ECC — a grammar generates and corrects its parsers); `anchor-to-human-prior-art` (reuse the
+  corpus); Futamura projections (the IR-compiler doc) — generation is specialization.

docs/trajectories/value-tree-codecs/RESUME.md

@@ -39,16 +39,36 @@ interop) is blocked on a **lenient YAML parser** — the same backlogged parser-
 layer (FParsec / GLR / ANTLR-shaped) noted in `RomDat`'s tokenizer seam. Frontmatter ships
 as a lossless verbatim split now; `tryMeta` is best-effort until the lenient parser lands.
 
-## NEXT PICKUP (delayed slices — Aaron 2026-07-02 "either both … whichever we delay")
-
-1. **KDL** — the clean text 2-ary codec (node children ⊕ properties). Our-own reader/writer.
-   **← resume here.** NOTE: its value-tree MAPPING is a convention decision (rigid
-   KDL-shaped encoding vs. general-KDL parser) — pick deliberately, likely with Aaron's input.
-2. **Lenient YAML parser** — unblocks structured frontmatter meta + human-YAML interop
-   (the canonical-only finding above). Part of the parser-combinator backlog.
-3. Parity categories needing a core `DynamicValue` shape first (Decimal / SoftValue / Kleene
-   tri-boolean) — each needs a DU decision, do NOT add unilaterally.
-4. HDF5 (starts `ThirdParty` — the case the port exists for); GraphViz DOT (graph, lossy).
+## THE LADDER (2026-07-02): codecs are rung 2 of a spec→generated-parser ladder
+
+Aaron: *"our zetaid is a tiny parser generator … bits, a bit parser/generator … the ANTLR
+stuff is likely just docs/research … LARGE scope: compile to/from our IR and most other ANTLR
+grammars using our parser generators and the open free ANTLR grammars; small changes to
+existing ANTLR grammars is fine."* The value-tree codecs are **rung 2** of one ladder:
+
+- **rung 1 (built):** `Core.FSharp.ZetaId` — spec (`zeta-id-v1-layout.yaml`) → `GeneratedBitLayout`
+  → bit parser/generator. The proven-in-miniature seed.
+- **rung 2 (built, this trajectory):** format codecs over `DynamicValue`.
+- **rung 3 (designed, not built):** **ZetaParse** — ingest ANTLR `.g4` / Yacc / Tree-sitter /
+  `.zg` → **Zeta Grammar IR** → LR/GLR parser (6 langs). Consume `antlr/grammars-v4` (MIT/BSD;
+  don't reinvent). Docs: `zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21` (Amara);
+  `antlr-grammar-survey-{2026-05-21,2026-06-13}` (Lior).
+
+**Full synthesis:** `docs/research/2026-07-02-parser-generator-foundation-ladder-zetaid-bits-to-value-tree-codecs-to-zetaparse-grammar-ir-antlr.md`.
+
+## NEXT PICKUP (subsumed by the ladder — scope rung 3 with Aaron; LARGE)
+
+1. **Zeta Grammar IR as a `DynamicValue` schema** — the grammar is data ⇒ rides rung-2 codecs
+   (byte-lockable, DST-replayable). The natural first, bounded piece.
+2. **`.g4` → Grammar IR ingester** (compatible subset; log drops — no silent truncation), from
+   one grammars-v4 asset. **KDL and lenient-YAML are subsumed here** — both are just grammars
+   ingested (KDL/YAML `.g4` exist), not bespoke hand-parsers.
+3. **LR/GLR backend** emitting an F# parser from the Grammar IR (the ZetaParse rung).
+4. Parity categories needing a core `DynamicValue` shape first (Decimal / SoftValue / Kleene) —
+   each needs a DU decision, do NOT add unilaterally. HDF5 / DOT remain on the codec ledger.
+
+> Rung 3 is LARGE + design-heavy and already has a design (ZetaParse). Do it under Aaron's
+> scope steer, building on the existing foundation — not unilaterally, not reinvented.
 
 ## Anchors