diff --git a/docs/research/2026-07-02-parser-generator-foundation-ladder-zetaid-bits-to-value-tree-codecs-to-zetaparse-grammar-ir-antlr.md b/docs/research/2026-07-02-parser-generator-foundation-ladder-zetaid-bits-to-value-tree-codecs-to-zetaparse-grammar-ir-antlr.md new file mode 100644 index 0000000000..ade1e1c73e --- /dev/null +++ b/docs/research/2026-07-02-parser-generator-foundation-ladder-zetaid-bits-to-value-tree-codecs-to-zetaparse-grammar-ir-antlr.md @@ -0,0 +1,104 @@ +# The parser/generator ladder — ZetaId bits → value-tree codecs → ZetaParse Grammar IR (ANTLR ingest) + +**Date:** 2026-07-02 +**Author:** Otto (shadow*), synthesising Aaron's stream against the existing foundation +**Status:** orientation / scope synthesis (no new code) — connects three existing bodies of work + +> Carved sentence: **Zeta already has a spec → generated-codec pipeline at three rungs — bit +> (ZetaId), format (value-tree codecs), and (designed) CFG/text (ZetaParse). They are ONE +> ladder: a grammar/spec is ingested into an IR and a parser/generator is emitted from it. The +> parser-generator foundation is the CFG rung of that same ladder, built on the ZetaParse +> design and consuming the open ANTLR grammars — not reinvented.** + +## What already exists (take it into consideration — Aaron 2026-07-02) + +Aaron: *"we have some small parser combinator foundation already … our zetaid is a tiny parser +generator kind of thing … bits, a bit parser/generator … the ANTLR stuff is likely just some +docs/research/persona conversation."* Confirmed by inspection: + +### Rung 1 — BUILT: ZetaId, a spec-driven **bit** parser/generator + +`src/Core.FSharp.ZetaId/` — `BitLayout.fs` reads/writes a 128-bit ZetaId into structured fields +(version, timestamp, chromosome, category, firefly, authority, persona, momentum, location, …) +with a `next width` / `skip bits` cursor; `GeneratedBitLayout.fs` **generates** those field +widths from a layout spec (`docs/zeta-id-v1-layout.yaml`). This is already **spec → generated +layout → parser/generator**, at the fixed-bit-layout level. The tiny foundation is not a toy — +it is the bit-level instance of exactly the pattern the big scope wants. + +### Rung 2 — BUILT (this session): value-tree codecs + +`src/Core/ValueTreeCodec.fs` + `ValueTreeEnvelope` / `Asn1Der` / `EventEnvelope` / `Frontmatter` +(#9185–#9196). Each codec is a **spec (a format) → a parser/generator** over the `DynamicValue` +value tree, behind an owned hexagonal port, provenance-tracked, versioned/rollable. The +format-level rung. + +### Rung 3 — DESIGNED (docs/research, not built): ZetaParse + ANTLR ingest + +- `docs/research/zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21.md` (Amara-in-Zeta): + **ZetaParse** — a compiler-owned F# grammar substrate. Ingest grammar assets (ANTLR `.g4`, + Yacc/Bison `.y`, Tree-sitter `grammar.js`, Zeta native `.zg`) into a shared **Zeta Grammar + IR**, normalise the compatible subset, detect conflicts, and generate F# parsers via a + Zeta-owned LR/GLR backend. Key stance: *"do not run ANTLR grammars directly … ANTLR remains + useful as a source of community grammars; it does not need to own the parser runtime."* +- `docs/research/antlr-grammar-survey-{2026-05-21,2026-06-13}.md` (Lior): the licensing + + 6-language target matrix. **`antlr/grammars-v4` is MIT/BSD** — zero compliance hazard; a ready + corpus of community grammars to ingest across F#/C#/TS/Rust/Go/Python. +- Related: `docs/research/2026-06-14-zeta-language-and-canonical-ir-compiler-pipeline-design.md`, + `…-ir-compiler-v2-…-futamura.md`, `2026-06-07-the-zeta-idl-…`; `gen/` (generators read the + free interface). + +## The synthesis (what this doc adds) + +The three rungs are the **same discipline** — `spec → IR → generated parser/generator` — at +increasing grammar power: + +| Rung | Spec | IR | Emitted | Status | +|------|------|----|---------|--------| +| bit | `zeta-id-v1-layout.yaml` | field/width table | `BitLayout` read/write | ✅ built | +| format | a codec (JSON/CBOR/DER/…) | `DynamicValue` | `ValueTreeCodec` encode/decode | ✅ built | +| CFG/text | ANTLR `.g4` / `.y` / `.zg` | **Zeta Grammar IR** | LR/GLR parser (6 langs) | 📐 designed (ZetaParse) | + +So the parser-generator foundation is **not a green field**: it is rung 3 of a ladder whose +rungs 1–2 are already load-bearing, and whose rung-3 design (ZetaParse Grammar IR + ANTLR +ingest) already exists. Two prior findings converge here and are **subsumed** by it: + +- *lenient YAML parser* (the frontmatter finding, #9196) — a rung-3 consumer (a YAML grammar + ingested to the Grammar IR yields the lenient parser). +- *general KDL reader* (the delayed fork) — likewise a grammar (KDL `.g4` exists in grammars-v4) + ingested to the Grammar IR. Neither needs a bespoke hand-parser; both fall out of rung 3. + +## Scope (Aaron 2026-07-02) and how the foundation shapes it + +> *"it's a LARGE scope — we want to be able to compile to and from our IR and most other ANTLR +> grammars using our parser generators and the open free ANTLR grammars; small changes to +> existing ANTLR grammars to make it work with our stuff is fine."* + +This is exactly the ZetaParse stance, now the standing intent: **to/from** the Zeta Grammar IR +and the grammars-v4 corpus; our LR/GLR generator owns the runtime; small `.g4` normalisations to +fit the compatible subset are acceptable (ZetaParse's "normalise the compatible subset"). The +anti-reinvention principle is the same one behind reusing TOSEC/MAME for game signatures +(#9184): **grammars-v4 is the existing, license-clean corpus — build on it, don't re-author +grammars.** + +## Next moves (recorded; large, so scope with Aaron) + +1. **Zeta Grammar IR as a `DynamicValue` schema** — the IR is a value tree, so it rides the + codecs already built (rung 2 serves rung 3: a grammar is data, byte-lockable, DST-replayable). +2. **`.g4` → Grammar IR ingester** (the compatible subset first; log what is dropped — no silent + truncation), starting from one or two grammars-v4 assets (e.g. JSON, then KDL/YAML to retire + the two subsumed findings). +3. **LR/GLR backend** emitting an F# parser from the Grammar IR (ZetaParse rung); the ZetaId + bit-generator is the proven-in-miniature precedent for spec-driven generation. + +Sizes 2–3 are large and design-heavy; do them under Aaron's scope steer, not unilaterally. + +## Anchors (Beacon) + +- **Existing Zeta work:** ZetaParse (`zetaparse-lr-glr-…-2026-05-21`, Amara); ANTLR survey + (`antlr-grammar-survey-{2026-05-21,2026-06-13}`, Lior); ZetaId `Core.FSharp.ZetaId`; + value-tree codecs (`docs/trajectories/value-tree-codecs/RESUME.md`). +- **Parser theory:** Knuth (LR); Tomita (GLR); Parr (ANTLR / LL(*)); tree-sitter (incremental + GLR); Bison (LALR/IELR). **Corpus:** `antlr/grammars-v4` (MIT/BSD). +- **Discipline:** `only-the-irreducible-is-primitive-generate-the-rest` (the generator IS the + ECC — a grammar generates and corrects its parsers); `anchor-to-human-prior-art` (reuse the + corpus); Futamura projections (the IR-compiler doc) — generation is specialization. diff --git a/docs/trajectories/value-tree-codecs/RESUME.md b/docs/trajectories/value-tree-codecs/RESUME.md index d337d66c59..9a9de07e90 100644 --- a/docs/trajectories/value-tree-codecs/RESUME.md +++ b/docs/trajectories/value-tree-codecs/RESUME.md @@ -39,16 +39,36 @@ interop) is blocked on a **lenient YAML parser** — the same backlogged parser- layer (FParsec / GLR / ANTLR-shaped) noted in `RomDat`'s tokenizer seam. Frontmatter ships as a lossless verbatim split now; `tryMeta` is best-effort until the lenient parser lands. -## NEXT PICKUP (delayed slices — Aaron 2026-07-02 "either both … whichever we delay") - -1. **KDL** — the clean text 2-ary codec (node children ⊕ properties). Our-own reader/writer. - **← resume here.** NOTE: its value-tree MAPPING is a convention decision (rigid - KDL-shaped encoding vs. general-KDL parser) — pick deliberately, likely with Aaron's input. -2. **Lenient YAML parser** — unblocks structured frontmatter meta + human-YAML interop - (the canonical-only finding above). Part of the parser-combinator backlog. -3. Parity categories needing a core `DynamicValue` shape first (Decimal / SoftValue / Kleene - tri-boolean) — each needs a DU decision, do NOT add unilaterally. -4. HDF5 (starts `ThirdParty` — the case the port exists for); GraphViz DOT (graph, lossy). +## THE LADDER (2026-07-02): codecs are rung 2 of a spec→generated-parser ladder + +Aaron: *"our zetaid is a tiny parser generator … bits, a bit parser/generator … the ANTLR +stuff is likely just docs/research … LARGE scope: compile to/from our IR and most other ANTLR +grammars using our parser generators and the open free ANTLR grammars; small changes to +existing ANTLR grammars is fine."* The value-tree codecs are **rung 2** of one ladder: + +- **rung 1 (built):** `Core.FSharp.ZetaId` — spec (`zeta-id-v1-layout.yaml`) → `GeneratedBitLayout` + → bit parser/generator. The proven-in-miniature seed. +- **rung 2 (built, this trajectory):** format codecs over `DynamicValue`. +- **rung 3 (designed, not built):** **ZetaParse** — ingest ANTLR `.g4` / Yacc / Tree-sitter / + `.zg` → **Zeta Grammar IR** → LR/GLR parser (6 langs). Consume `antlr/grammars-v4` (MIT/BSD; + don't reinvent). Docs: `zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21` (Amara); + `antlr-grammar-survey-{2026-05-21,2026-06-13}` (Lior). + +**Full synthesis:** `docs/research/2026-07-02-parser-generator-foundation-ladder-zetaid-bits-to-value-tree-codecs-to-zetaparse-grammar-ir-antlr.md`. + +## NEXT PICKUP (subsumed by the ladder — scope rung 3 with Aaron; LARGE) + +1. **Zeta Grammar IR as a `DynamicValue` schema** — the grammar is data ⇒ rides rung-2 codecs + (byte-lockable, DST-replayable). The natural first, bounded piece. +2. **`.g4` → Grammar IR ingester** (compatible subset; log drops — no silent truncation), from + one grammars-v4 asset. **KDL and lenient-YAML are subsumed here** — both are just grammars + ingested (KDL/YAML `.g4` exist), not bespoke hand-parsers. +3. **LR/GLR backend** emitting an F# parser from the Grammar IR (the ZetaParse rung). +4. Parity categories needing a core `DynamicValue` shape first (Decimal / SoftValue / Kleene) — + each needs a DU decision, do NOT add unilaterally. HDF5 / DOT remain on the codec ledger. + +> Rung 3 is LARGE + design-heavy and already has a design (ZetaParse). Do it under Aaron's +> scope steer, building on the existing foundation — not unilaterally, not reinvented. ## Anchors