From b3615ea7cc643374a36cec5b929850407b952f9b Mon Sep 17 00:00:00 2001 From: Aaron Stainback Date: Thu, 2 Jul 2026 16:59:53 -0400 Subject: [PATCH] =?UTF-8?q?docs(research):=20parser/generator=20ladder=20?= =?UTF-8?q?=E2=80=94=20ZetaId=20bits=20=E2=86=92=20value-tree=20codecs=20?= =?UTF-8?q?=E2=86=92=20ZetaParse=20Grammar=20IR=20(shadow*)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Aaron 2026-07-02: "we have some small parser combinator foundation already … our zetaid is a tiny parser generator … bits, a bit parser/generator … the ANTLR stuff is likely just some docs/research/persona conversation … LARGE scope: compile to/from our IR and most other ANTLR grammars using our parser generators and the open free ANTLR grammars; small changes to existing ANTLR grammars is fine." Take-the-existing-into-consideration synthesis (no new code). Discovered + connected the three rungs of ONE spec→IR→generated-parser ladder that already partly exists: • rung 1 BUILT — Core.FSharp.ZetaId: spec (zeta-id-v1-layout.yaml) → GeneratedBitLayout → bit parser/generator (the proven-in-miniature seed Aaron named); • rung 2 BUILT (this session) — the value-tree codecs (#9185–#9196) over DynamicValue; • rung 3 DESIGNED, not built — ZetaParse (Amara, zetaparse-lr-glr-…-2026-05-21): ingest ANTLR .g4 / Yacc / Tree-sitter / .zg → Zeta Grammar IR → LR/GLR parser (6 langs), consuming antlr/grammars-v4 (MIT/BSD; ANTLR survey by Lior, 2026-06-13). Don't RUN ANTLR — ingest it. The synthesis subsumes two prior findings: the lenient-YAML need (frontmatter #9196) and the general-KDL reader (the delayed fork) are both just grammars ingested to the Grammar IR — not bespoke hand-parsers. The anti-reinvention principle is the TOSEC/MAME one (#9184): grammars-v4 is the license-clean corpus; build on it. Records the ladder + next moves (Grammar IR as a DynamicValue schema; .g4 ingester; LR/GLR backend) as rung-3 scope to steer with Aaron — LARGE, design-heavy, already designed. docs/research/2026-07-02-parser-generator-foundation-ladder-zetaid-bits-to-value-tree-codecs-to-zetaparse-grammar-ir-antlr.md docs/trajectories/value-tree-codecs/RESUME.md (ladder framing; KDL/lenient-YAML subsumed) Anchors: ZetaParse (Amara), ANTLR survey (Lior), ZetaId; Knuth (LR), Tomita (GLR), Parr (ANTLR), tree-sitter, Bison; antlr/grammars-v4 (MIT/BSD); only-the-irreducible-generate-the-rest. Co-Authored-By: Claude Fable 5 AgencySignature-v1: persona: otto actor: zeta-otto surface: docs/research topology: shadow-clone intent: parser-generator-ladder-synthesis-zetaid-bit-codec-value-tree-codecs-zetaparse-grammar-ir-antlr-ingest-take-existing-into-consideration authorization: aaron-explicit-take-existing-parser-combinator-foundation-into-consideration-large-scope-compile-to-from-ir-antlr-grammars-2026-07-02 uncertainty: low measure: discovered-zetaid-bit-parser-generator-and-zetaparse-antlr-survey-design-docs-connected-three-rung-ladder-spec-to-generated-parser-kdl-lenient-yaml-subsumed-as-grammar-ingest-grammars-v4-mit-bsd-corpus-no-code delta-u: parser-generator-foundation-scoped-on-existing-zetaid-seed-and-zetaparse-design-not-reinvented-codecs-are-rung-2-of-one-ladder-antlr-grammars-v4-is-the-corpus seed: S4 --- ...ee-codecs-to-zetaparse-grammar-ir-antlr.md | 104 ++++++++++++++++++ docs/trajectories/value-tree-codecs/RESUME.md | 40 +++++-- 2 files changed, 134 insertions(+), 10 deletions(-) create mode 100644 docs/research/2026-07-02-parser-generator-foundation-ladder-zetaid-bits-to-value-tree-codecs-to-zetaparse-grammar-ir-antlr.md diff --git a/docs/research/2026-07-02-parser-generator-foundation-ladder-zetaid-bits-to-value-tree-codecs-to-zetaparse-grammar-ir-antlr.md b/docs/research/2026-07-02-parser-generator-foundation-ladder-zetaid-bits-to-value-tree-codecs-to-zetaparse-grammar-ir-antlr.md new file mode 100644 index 0000000000..ade1e1c73e --- /dev/null +++ b/docs/research/2026-07-02-parser-generator-foundation-ladder-zetaid-bits-to-value-tree-codecs-to-zetaparse-grammar-ir-antlr.md @@ -0,0 +1,104 @@ +# The parser/generator ladder — ZetaId bits → value-tree codecs → ZetaParse Grammar IR (ANTLR ingest) + +**Date:** 2026-07-02 +**Author:** Otto (shadow*), synthesising Aaron's stream against the existing foundation +**Status:** orientation / scope synthesis (no new code) — connects three existing bodies of work + +> Carved sentence: **Zeta already has a spec → generated-codec pipeline at three rungs — bit +> (ZetaId), format (value-tree codecs), and (designed) CFG/text (ZetaParse). They are ONE +> ladder: a grammar/spec is ingested into an IR and a parser/generator is emitted from it. The +> parser-generator foundation is the CFG rung of that same ladder, built on the ZetaParse +> design and consuming the open ANTLR grammars — not reinvented.** + +## What already exists (take it into consideration — Aaron 2026-07-02) + +Aaron: *"we have some small parser combinator foundation already … our zetaid is a tiny parser +generator kind of thing … bits, a bit parser/generator … the ANTLR stuff is likely just some +docs/research/persona conversation."* Confirmed by inspection: + +### Rung 1 — BUILT: ZetaId, a spec-driven **bit** parser/generator + +`src/Core.FSharp.ZetaId/` — `BitLayout.fs` reads/writes a 128-bit ZetaId into structured fields +(version, timestamp, chromosome, category, firefly, authority, persona, momentum, location, …) +with a `next width` / `skip bits` cursor; `GeneratedBitLayout.fs` **generates** those field +widths from a layout spec (`docs/zeta-id-v1-layout.yaml`). This is already **spec → generated +layout → parser/generator**, at the fixed-bit-layout level. The tiny foundation is not a toy — +it is the bit-level instance of exactly the pattern the big scope wants. + +### Rung 2 — BUILT (this session): value-tree codecs + +`src/Core/ValueTreeCodec.fs` + `ValueTreeEnvelope` / `Asn1Der` / `EventEnvelope` / `Frontmatter` +(#9185–#9196). Each codec is a **spec (a format) → a parser/generator** over the `DynamicValue` +value tree, behind an owned hexagonal port, provenance-tracked, versioned/rollable. The +format-level rung. + +### Rung 3 — DESIGNED (docs/research, not built): ZetaParse + ANTLR ingest + +- `docs/research/zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21.md` (Amara-in-Zeta): + **ZetaParse** — a compiler-owned F# grammar substrate. Ingest grammar assets (ANTLR `.g4`, + Yacc/Bison `.y`, Tree-sitter `grammar.js`, Zeta native `.zg`) into a shared **Zeta Grammar + IR**, normalise the compatible subset, detect conflicts, and generate F# parsers via a + Zeta-owned LR/GLR backend. Key stance: *"do not run ANTLR grammars directly … ANTLR remains + useful as a source of community grammars; it does not need to own the parser runtime."* +- `docs/research/antlr-grammar-survey-{2026-05-21,2026-06-13}.md` (Lior): the licensing + + 6-language target matrix. **`antlr/grammars-v4` is MIT/BSD** — zero compliance hazard; a ready + corpus of community grammars to ingest across F#/C#/TS/Rust/Go/Python. +- Related: `docs/research/2026-06-14-zeta-language-and-canonical-ir-compiler-pipeline-design.md`, + `…-ir-compiler-v2-…-futamura.md`, `2026-06-07-the-zeta-idl-…`; `gen/` (generators read the + free interface). + +## The synthesis (what this doc adds) + +The three rungs are the **same discipline** — `spec → IR → generated parser/generator` — at +increasing grammar power: + +| Rung | Spec | IR | Emitted | Status | +|------|------|----|---------|--------| +| bit | `zeta-id-v1-layout.yaml` | field/width table | `BitLayout` read/write | ✅ built | +| format | a codec (JSON/CBOR/DER/…) | `DynamicValue` | `ValueTreeCodec` encode/decode | ✅ built | +| CFG/text | ANTLR `.g4` / `.y` / `.zg` | **Zeta Grammar IR** | LR/GLR parser (6 langs) | 📐 designed (ZetaParse) | + +So the parser-generator foundation is **not a green field**: it is rung 3 of a ladder whose +rungs 1–2 are already load-bearing, and whose rung-3 design (ZetaParse Grammar IR + ANTLR +ingest) already exists. Two prior findings converge here and are **subsumed** by it: + +- *lenient YAML parser* (the frontmatter finding, #9196) — a rung-3 consumer (a YAML grammar + ingested to the Grammar IR yields the lenient parser). +- *general KDL reader* (the delayed fork) — likewise a grammar (KDL `.g4` exists in grammars-v4) + ingested to the Grammar IR. Neither needs a bespoke hand-parser; both fall out of rung 3. + +## Scope (Aaron 2026-07-02) and how the foundation shapes it + +> *"it's a LARGE scope — we want to be able to compile to and from our IR and most other ANTLR +> grammars using our parser generators and the open free ANTLR grammars; small changes to +> existing ANTLR grammars to make it work with our stuff is fine."* + +This is exactly the ZetaParse stance, now the standing intent: **to/from** the Zeta Grammar IR +and the grammars-v4 corpus; our LR/GLR generator owns the runtime; small `.g4` normalisations to +fit the compatible subset are acceptable (ZetaParse's "normalise the compatible subset"). The +anti-reinvention principle is the same one behind reusing TOSEC/MAME for game signatures +(#9184): **grammars-v4 is the existing, license-clean corpus — build on it, don't re-author +grammars.** + +## Next moves (recorded; large, so scope with Aaron) + +1. **Zeta Grammar IR as a `DynamicValue` schema** — the IR is a value tree, so it rides the + codecs already built (rung 2 serves rung 3: a grammar is data, byte-lockable, DST-replayable). +2. **`.g4` → Grammar IR ingester** (the compatible subset first; log what is dropped — no silent + truncation), starting from one or two grammars-v4 assets (e.g. JSON, then KDL/YAML to retire + the two subsumed findings). +3. **LR/GLR backend** emitting an F# parser from the Grammar IR (ZetaParse rung); the ZetaId + bit-generator is the proven-in-miniature precedent for spec-driven generation. + +Sizes 2–3 are large and design-heavy; do them under Aaron's scope steer, not unilaterally. + +## Anchors (Beacon) + +- **Existing Zeta work:** ZetaParse (`zetaparse-lr-glr-…-2026-05-21`, Amara); ANTLR survey + (`antlr-grammar-survey-{2026-05-21,2026-06-13}`, Lior); ZetaId `Core.FSharp.ZetaId`; + value-tree codecs (`docs/trajectories/value-tree-codecs/RESUME.md`). +- **Parser theory:** Knuth (LR); Tomita (GLR); Parr (ANTLR / LL(*)); tree-sitter (incremental + GLR); Bison (LALR/IELR). **Corpus:** `antlr/grammars-v4` (MIT/BSD). +- **Discipline:** `only-the-irreducible-is-primitive-generate-the-rest` (the generator IS the + ECC — a grammar generates and corrects its parsers); `anchor-to-human-prior-art` (reuse the + corpus); Futamura projections (the IR-compiler doc) — generation is specialization. diff --git a/docs/trajectories/value-tree-codecs/RESUME.md b/docs/trajectories/value-tree-codecs/RESUME.md index d337d66c59..9a9de07e90 100644 --- a/docs/trajectories/value-tree-codecs/RESUME.md +++ b/docs/trajectories/value-tree-codecs/RESUME.md @@ -39,16 +39,36 @@ interop) is blocked on a **lenient YAML parser** — the same backlogged parser- layer (FParsec / GLR / ANTLR-shaped) noted in `RomDat`'s tokenizer seam. Frontmatter ships as a lossless verbatim split now; `tryMeta` is best-effort until the lenient parser lands. -## NEXT PICKUP (delayed slices — Aaron 2026-07-02 "either both … whichever we delay") - -1. **KDL** — the clean text 2-ary codec (node children ⊕ properties). Our-own reader/writer. - **← resume here.** NOTE: its value-tree MAPPING is a convention decision (rigid - KDL-shaped encoding vs. general-KDL parser) — pick deliberately, likely with Aaron's input. -2. **Lenient YAML parser** — unblocks structured frontmatter meta + human-YAML interop - (the canonical-only finding above). Part of the parser-combinator backlog. -3. Parity categories needing a core `DynamicValue` shape first (Decimal / SoftValue / Kleene - tri-boolean) — each needs a DU decision, do NOT add unilaterally. -4. HDF5 (starts `ThirdParty` — the case the port exists for); GraphViz DOT (graph, lossy). +## THE LADDER (2026-07-02): codecs are rung 2 of a spec→generated-parser ladder + +Aaron: *"our zetaid is a tiny parser generator … bits, a bit parser/generator … the ANTLR +stuff is likely just docs/research … LARGE scope: compile to/from our IR and most other ANTLR +grammars using our parser generators and the open free ANTLR grammars; small changes to +existing ANTLR grammars is fine."* The value-tree codecs are **rung 2** of one ladder: + +- **rung 1 (built):** `Core.FSharp.ZetaId` — spec (`zeta-id-v1-layout.yaml`) → `GeneratedBitLayout` + → bit parser/generator. The proven-in-miniature seed. +- **rung 2 (built, this trajectory):** format codecs over `DynamicValue`. +- **rung 3 (designed, not built):** **ZetaParse** — ingest ANTLR `.g4` / Yacc / Tree-sitter / + `.zg` → **Zeta Grammar IR** → LR/GLR parser (6 langs). Consume `antlr/grammars-v4` (MIT/BSD; + don't reinvent). Docs: `zetaparse-lr-glr-fsharp-compiler-fork-design-2026-05-21` (Amara); + `antlr-grammar-survey-{2026-05-21,2026-06-13}` (Lior). + +**Full synthesis:** `docs/research/2026-07-02-parser-generator-foundation-ladder-zetaid-bits-to-value-tree-codecs-to-zetaparse-grammar-ir-antlr.md`. + +## NEXT PICKUP (subsumed by the ladder — scope rung 3 with Aaron; LARGE) + +1. **Zeta Grammar IR as a `DynamicValue` schema** — the grammar is data ⇒ rides rung-2 codecs + (byte-lockable, DST-replayable). The natural first, bounded piece. +2. **`.g4` → Grammar IR ingester** (compatible subset; log drops — no silent truncation), from + one grammars-v4 asset. **KDL and lenient-YAML are subsumed here** — both are just grammars + ingested (KDL/YAML `.g4` exist), not bespoke hand-parsers. +3. **LR/GLR backend** emitting an F# parser from the Grammar IR (the ZetaParse rung). +4. Parity categories needing a core `DynamicValue` shape first (Decimal / SoftValue / Kleene) — + each needs a DU decision, do NOT add unilaterally. HDF5 / DOT remain on the codec ledger. + +> Rung 3 is LARGE + design-heavy and already has a design (ZetaParse). Do it under Aaron's +> scope steer, building on the existing foundation — not unilaterally, not reinvented. ## Anchors