feat(project): schemas for prepared questions and the web environment (G-K1)

[trakhimenok]

What

Task G-K1, the two file formats, of demo-as-github-project.md (backstage design, 5.2a, 4.8, 3.6, 5.4, 6.4): JSON Schemas, TypeScript types and validators for ai/prepared-questions.json and the https-json catalog file, and a recorded fixture of the demo project in the new layout. New files only, under libs/datatug/main/src/lib/project-files/. Nothing is imported by shipped code or exported from the barrel; the app's routes, start-up file, chat page and IndexedDB are untouched.

Decision (orchestrator, following the design's own fallback in 5.2a)

The first step of the task found that the project validation and the CLI loader reject the web environment as 5.2a writes it. Per the design, the three files live under web/ at the project root (web/web.env.json, web/catalogs/chinook/chinook.db.json, web/catalogs/geo/geo.db.json) until the CLI accepts the new drivers; the app loader will look there. datatug validate stays green on the released CLI. Not chosen: writing the server entry as sqlite3 (a server entry must not name a driver it is not).

Measured 2026-10-02, CLI v0.52.0 (built from the commit the tag points to, 3751870) and the installed release 0.51.0, which is what datatug/datatug-action runs (datatug validate -d=<dir>):

Sample (datatug/chinook-demo at 0aca653 plus the new files)	Result
unchanged project	DataTug project is valid., exit 0
environments/web/web.env.json with servers https-json and ingitdb (the design's shape)	exit 1: validation failed for environment at index=5, id=web: invalid env db server at index 0: bad value for field [driver]: unexpected value: https-json. (with ingitdb alone: unexpected value: ingitdb.). Listing web in datatug-project.json changes nothing: the loader finds environments by directory.
the same three files under web/ at the root, plus ai/prepared-questions.json	DataTug project is valid., exit 0

The later CLI task (moves the files to environments/web/)

Change datatug-core pkg/datatug/server.go:86, the default branch of ServerRef.Validate: accept https-json and ingitdb as file-like drivers with no host and no port, exactly as case "sqlite3" does (reject a host or a port, return nil). Then release datatug-core, bump and release the CLI (the action uses releases/latest). Catalog files need no change: DbCatalogBase.Validate already accepts any non-empty driver and the loader ignores unknown fields. After that release the three files move to environments/web/ as a path change only: the schemas and validators describe documents, not locations, and web.env.json already carries the design's drivers. Written up in fixtures/chinook-demo.README.md.

Files

	Path
Schema, prepared questions	libs/datatug/main/src/lib/project-files/schemas/prepared-questions.schema.json
Schema, https-json catalog	libs/datatug/main/src/lib/project-files/schemas/https-json-catalog.schema.json
Types + validator	libs/datatug/main/src/lib/project-files/prepared-questions.ts, libs/datatug/main/src/lib/project-files/https-json-catalog.ts
Address rules (3.6)	libs/datatug/main/src/lib/project-files/project-address-rules.ts
Limits and patterns (mirrored in the schemas)	libs/datatug/main/src/lib/project-files/project-file-limits.ts
Fixture	libs/datatug/main/src/lib/project-files/fixtures/chinook-demo/, fixtures/chinook-demo.manifest.json, fixtures/chinook-demo.README.md

Pinning: schemas are referenced by commit, not tag: https://raw.githubusercontent.com/datatug/datatug-apps/<commit>/libs/datatug/main/src/lib/project-files/schemas/<name>.schema.json. Pin the commit on main that lands this change (a squash changes the SHA, so take it after the merge). The schema blobs are 1dd111042022a34df979273cc5e7b82e5c323129 (prepared questions) and 5cdb33bb93b05225cde7618e827953aa8cc7ac1c (catalog) at head 58b964f; git rev-parse <commit>:<path> must give the same.

Acceptance (task text and the security list of 3.6)

Item	Where	Proven by
JSON Schemas for both files, 5.2a	schemas/*.schema.json	project-file-schemas.spec.ts:295,302 (schema and validator agree on 60+ documents; ajv strict mode compiles both)
Types and validators	prepared-questions.ts:69, https-json-catalog.ts:89	prepared-questions.spec.ts, https-json-catalog.spec.ts
HTTPS only; no loopback or private range, by name or number; no credentials in URLs	project-address-rules.ts:46 (v4 ranges :125, v6 :163)	project-address-rules.spec.ts:13 (http to localhost/127.0.0.1/[::1], decimal/hex/octal spellings, mapped, NAT64, 6to4, link-local, user:pass@, trusted@evil), https-json-catalog.spec.ts:133
Addresses of a trusted project begin with an allowed prefix (a prefix of the full address, not a host)	project-address-rules.ts:17,190,203; https-json-catalog.ts:199	project-address-rules.spec.ts:145, https-json-catalog.spec.ts:267 (another path on chinookdb.com, another repo on jsDelivr, the right repo at a branch or short commit, .. out of the prefix, look-alike hosts); trust is a required option, never defaulted
Size limits declared in the schema	schemas/* (maxItems, maxLength, maxProperties); 256 KB file cap project-file-limits.ts:6 enforced by parse* on bytes	project-file-schemas.spec.ts:358 (schema numbers equal project-file-limits.ts; every string bounded), prepared-questions.spec.ts:472, https-json-catalog.spec.ts:481 (multi-byte text over the cap)
Strings rendered as text (no HTML fields)	no markup field in either schema; validators carry markup unchanged	prepared-questions.spec.ts:313, https-json-catalog.spec.ts:415, project-file-schemas.spec.ts ("no markup-bearing field")
Unknown keys rejected	additionalProperties: false everywhere; ProblemCollector.onlyKeys (project-file-problems.ts)	every spec above; __proto__ is an unknown key; the schema walk in project-file-schemas.spec.ts fails on any object without it
Fallback address needs a checksum (4.8)	https-json-catalog.ts:142-154	https-json-catalog.spec.ts:334
Recorded fixture of the demo project in the new layout, from datatug-demo-projects 0e5b98f plus the new files	fixtures/chinook-demo/ (22 files = rows 1 to 22 of design 4.3)	chinook-demo-fixture.spec.ts:49 (manifest of sizes and SHA-256), :82,:166 (new files pass validators and schemas), :208 (the golden result recomputed from the fixture alone: 24 countries, 412 invoices, Ireland 8.32 first, Czech Republic 8.29, USA 1.53 at rank 17)
Report 5.2a asks for, fallback location chosen	this description; fixtures/chinook-demo.README.md	measured above

Interpretations the design did not spell out: urlTemplate, keys, sha256 and driver are required, the rest optional; with a fallback address every keyed table needs a checksum; addresses carry no port and no IP-literal host shape in the schema, and loopback and private numbers are refused by the validator; a text has no control characters; a table name has no leading underscore. Invoice rows are the chinookdb.com file (a local clone of datatug/chinookdb at its mirror commit 0b6bb6b, SHA-256 88eb7fae…c373c).

ajv 8.20.0 (already in the lockfile transitively) is added as a dev dependency for the schema-versus-validator spec only; the validators themselves have no dependency and no eval.

Review round 1 (independent review at aff4a7b: 1 blocker, 6 minors), fixed at 58b964f

• B1, the allow-list was checked against a stand-in. A template now has {table} exactly once, in a path segment that starts with a letter or digit; no %, ? or # anywhere; no empty segment and no segment starting with a dot (checkUrlTemplate, project-address-rules.ts). expandUrlTemplate(template, table, trust) validates the table name, expands, parses, and re-checks the parsed URL (general rules, prefix list, dot segments); it returns a CheckedDataUrl, the only way to a fetchable URL (tableUrls(catalog, table, trust) for both templates). Property test over 20,000 generated templates x 10 table names; the reviewer's cases ported (project-address-rules.spec.ts, https-json-catalog.spec.ts).
• Schema laxer than validator: $schema needs a character; lengths counted in code points in the validator; homepage and upstream.repository allow no brace; a mutation test over about 2,500 documents (project-file-schemas.spec.ts) fails if either side accepts what only the other should refuse (the only allowed gap: rules a schema cannot state).
• checkProjectAddress direct calls: IPv6 expanded and range-tested (all of ::/8 in every spelling, site-local, Teredo, NAT64 well-known and local-use, 6to4, ORCHID, discard, documentation); the host is read as the URL parser reads it; localhost.. and empty labels refused.
• Texts: bidi overrides and isolates, line and paragraph separators, zero-width and tag characters refused; a leading byte order mark is dropped (and counted toward the cap).
• Licence: fixtures/chinook-demo/NOTICE.md (Chinook MIT text copied from upstream LICENSE.md at 7f67772, trailing spaces removed because the repo's whitespace hook refuses them) and data/geo/DATA-LICENSE.md, both in the manifest.
• Documented: the byte cap must be enforced by the caller on the response stream (G-A2); a jsDelivr 40-hex pin is only as trustworthy as the trusted project's own file.

Production bundle

Before: a clean copy of origin/main (453423a); re-proved after review round 1; after: this branch. Same build-info stamp, pnpm nx build datatug-app --configuration=production --excludeTaskDependencies --skip-nx-cache. diff -rq of the two dist/apps/datatug-app trees: no differences (1,797 files each; the sorted listing of per-file SHA-256 hashes hashes to 9216987ae99ff52bca5e6b378d1441b7d2a685e980d509a185bc9828ce241235 for both). Byte-identical, hashes included, as no shipped code imports the new files.

Verification run locally

• wb run -- pnpm exec vitest run src/lib/project-files in libs/datatug/main: 5 files, 534 tests pass; statement coverage of the new modules 96.7%, lines 98.4%
• pnpm nx lint datatug-main: clean; pnpm run check:zoneless: OK; tsc over the library and spec configs: no error in project-files
• production build as above

🤖 Generated with Claude Code

[trakhimenok]

[review r2 #178]

Reviewed-Head: 58b964f

Two rounds by an independent Opus reviewer with executable probes. Round 1 found the allow-list checked a stand-in address rather than the expanded one; closed. Re-run on this head: 1.77 million template expansions issued 7,436 URLs with zero escapes from the allowed prefixes; the golden result recomputes from the fixture (24 countries, 412 invoices, Ireland 8.32); nothing shipped imports the directory. All four checks pass.

Seven minors are carried to #180, each assigned to the task that wires these functions.

VERDICT: blockers=0 majors=0 minors=7 land=yes

🤖 Generated with Claude Code