feat(project): GitHub reader pins a commit, caches, fails over and enforces limits (G-A2)

[trakhimenok]

What

G-A2 of docs/design/demo-as-github-project.md (datatug/backstage): the GitHub reader pins a commit, caches what it read, fails over to jsDelivr and enforces the limits of 3.6. Public method signatures are unchanged (new optional options argument on getRawJson/getRawText), so no project page changes. Not part of this PR: routes, index.html, main.ts, the demo hand-off, the chat page and its IndexedDB (datatug-chat-sessions stays at version 2, untouched).

How a project is read now

1. One call resolves the commit (api.github.com/repos/<o>/<r>/commits/<ref or HEAD>, Accept: application/vnd.github.sha), remembered 5 minutes across page loads. A full SHA in the id skips it, never for a trusted project (3.6 point 3: a trusted run reads what GitHub says the default branch is now; trust is isTrustedProjectAddress, so a trusted id never has a ref).
2. The listing (git/trees/<sha>?recursive=1) and every file (raw.githubusercontent.com/<o>/<r>/<sha>/<path>) come from that one commit. The project summary now goes through the reader too (it used its own HTTP read, which could come from another commit).
3. A new database datatug-github-files v1 keeps files, absent files and the listing by owner/repo@sha/path; bounded 20 MB, 20 repos, two commits per repo, least recently used out first. It is opened from a lazy chunk. Blocked, full or missing storage means no cache, never a failed read.
4. Raw refused (403/429), down (5xx), unreachable or timed out: the same path from cdn.jsdelivr.net at the same commit.
5. Resolve call fails: remembered commit, else HEAD (memory only), else unversioned jsDelivr (memory only), else a failed step naming the hosts. readInfo(projectId) reports state (given | resolved | remembered | unresolved | missing | moved), commit, fromMirror, mayBeStale for the later "sources" line.
6. main is HEAD; four-part ids (branch, tag, commit) read instead of being refused.

Limits and rules (3.6)

• every request: credentials: 'omit', referrerPolicy: 'no-referrer', redirect: 'error', https on exactly three hosts (api.github.com, raw.githubusercontent.com, cdn.jsdelivr.net), 20 s timeout including the body
• 256 KB per project file, counted on the bytes received; the transfer is cancelled past the cap (no mirror retry); the listing is capped at 8 MB
• 40 distinct files per run through an optional GithubReadBudget (cache hits count; a shell read passes no budget, so existing pages are not limited by it)
• a renamed or moved repo: GitHub's API answers a redirect; it is never followed and reads as "No DataTug project here" (GithubProjectNotFoundError, reason moved)

What an existing user can observe (datatug-demo-projects@datatug@demo-project-1)

Measured against real GitHub, Chromium, full side-menu walk (github-store.spec.ts, first test):

	main	this PR, cold	this PR, same browser within 5 min	later, project unchanged
api.github.com	1 (listing)	2 (commit + listing)	0	1 (commit)
raw.githubusercontent.com	20	20	0	0

So: one more rate-limited call on a first visit (30 first visits an hour per address instead of 60), and none on return visits; files and listing are one commit; the project summary is no longer a separate read. Files are served from the local cache on return visits. Error text for an exhausted file host now names the hosts (it used to say "GitHub API rate limit"). A project at the repo root now lists (it listed nothing). Nothing else visible: the e2e suite passes unchanged.

datatug/chinook-demo (project at the root), measured: cold shell walk (project, entities, environments, queries) 2 API + 13 raw; warm 0.

Acceptance, item by item

Item	Where	Proof
resolved commit for tree AND files, never mixed	github-project-reader.service.ts resolveCommit, loadTree, loadText	commit.spec "a push in the middle of a visit…", e2e budget test (one sha in every raw URL and the listing)
trusted run never reads a commit from the address	resolveCommit	commit.spec "the commit of an address (3.6, point 3)"
every row of the degrade table	loadText, resolveCommit	commit.spec "the degrade table, row by row" (rows 1 to 6 plus the cached listing)
file cache, own database, bounded 20 MB / 20 repos / 2 commits, LRU	github-file-store.ts	github-file-store.spec (eviction plan table; over IndexedDB)
rollback-safe	github-file-store.ts header, openDatabase	github-file-store.spec "rollback-safe": the chat database made by the chat service is untouched at v2 and still lists its chat after the cache ran; a later-version cache database is still used; a later build's upgrade is not blocked
256 KB on the stream, 40 files	github-http.ts readBodyLimited, GithubReadBudget	github-http.spec (cancelled after the cap whatever the headers say; exactly cap passes; bytes not characters), commit.spec "limits"
redirects refused; renamed repo is "No DataTug project here"	githubGet (redirect), resolveCommit	github-http.spec, commit.spec "requests"
four-part ids	assertReadableGithubProjectId	reader.spec "a project id the reader reads at a ref"
root project listDirectory prefix (issue 180)	listDirectory, getQuery	reader.spec "lists a project at the repo root"; real chinook-demo lists its queries
request budget cold and warm		commit.spec "the cache and the request budget" (demo project and the 4.3 file set), e2e "request budget (G-A2)"

Bundle (production build, nx run datatug-app:build)

Initial total 470.51 kB to 470.10 kB estimated transfer (2.09 MB raw, unchanged). The reader's lazy chunk: 13,865 to 19,509 bytes raw (+5.6 kB, +2.1 kB gzip). New lazy chunk with the IndexedDB code: 3,220 bytes raw (1.5 kB gzip), loaded on the first GitHub read only. All JS: 6,085,197 to 6,094,788 bytes raw.

Verification

• nx test datatug-main datatug-app (full, with coverage): 139 files / 1910 tests and 13 files / 352 tests pass. Two small test-only follow-up commits: the GitHub-store root-folder spec now serves the project file through the reader's fake fetch, and the exhaustive project-url round-trip spec (3.5 s of its 5 s timeout on main CI) gets a 60 s timeout.
• nx run-many -t lint -p datatug-main datatug-app: clean; pnpm run check:zoneless: ok; production build: ok
• Playwright against real GitHub (github-store.spec.ts incl. the new budget test, label-spacing.spec.ts), Chromium: 10 of 10 pass

Not in this PR (issue 180)

The CheckedDataUrl / tableUrls trust items and the http://localhost allowance of the federated executor belong to G-A3b: there is no fetch layer for outside data yet to wire them into. Closed here: resolved commit, listDirectory root prefix, 256 KB cap on the stream, redirect refused, four-part ids.

Design versus reality

• raw.githubusercontent.com and jsDelivr answer 200 for a renamed repo (measured); only the API redirects. So the refusal is enforced on the API calls, where redirect: 'manual' is used to tell a redirect from an outage (it is still never followed). With redirect: 'error' a rename would look like an outage and degrade into reading the renamed repo. The file hosts keep redirect: 'error'.
• Absent files are cached too (immutable at a commit), otherwise a warm load could not be free.
• The listing has no jsDelivr failover (4.3 leaves it to measurement); a refused listing keeps today's friendly rate-limit message, and a listing already cached for the remembered commit still works.

🤖 Generated with Claude Code

[trakhimenok]

[review r1 #181] Adversarial review (Opus): code read, 385 targeted tests, 47 probe cases and four mutations run on an export of this head.

Reviewed-Head: fa6fe54

Held up under attack: commit pinning and trust (a fork commit never reaches the trusted read), absent-caching (403/429/5xx/network/redirect store nothing), byte caps on the stream, the host allow-list and path encoding, cache accounting across two tabs, rollback of the cache database. Paths below are under libs/datatug/main/src/lib/.

Majors

• S1. A project created in a repo this browser has already read does not open. services/repo/github/github-project-reader.service.ts:370-386 (commit resolved once per page), :401-407 (5-minute memo), :644-651 (absence cached under that commit); nothing invalidates them after project/new-project/new-project-form.component.ts:270-281 commits and navigates. Result: "No DataTug project here (missing)" until the memo expires plus a reload. Fix: a forget(org, repo) on the reader dropping the session and the resolved memo, called (with summaryCache cleared) after createProject succeeds; do not persist absence for datatug-project.json. Related: github-project-create.service.ts:29 still commits to main while the reader reads HEAD; use the repo's default branch (github-api.ts:50 exposes it).
• S2. A transient failure of the project file sticks until a full reload. services/repo/datatug-store.service.github.ts:51-82: getRawJson is called once when the cached observable is built, so shareReplay's reset re-subscribes to the same rejected promise. Fix: defer(() => this.githubReader.getRawJson(...)) or delete the cache entry on error.
• S3. The cache can hang every read. github-project-reader.service.ts:401, :493, :610 await the store before any request; github-file-store.ts:136-154 has no timeout on open, and a blocked open that later succeeds leaves an unclosed connection with no onversionchange. Fix: race store calls against a 1-2 s timeout falling back to NO_GITHUB_FILE_STORE; close a late connection.

Minors

• M1. Mutation :647 to if (commitKey) (a mirror 404 after a refused raw stored as absent) fails no test.
• M2. Degrade rows 4 and 5 ("in memory for this visit only") are not proven: the second visit has the API answering. Keep it refused.
• M3. The !isTrustedProjectAddress clause at :393-397 is dead code (a trusted id cannot carry a ref); its test does not exercise it.
• M4. Test "a trusted project never uses a remembered answer older than 5 minutes" asserts only that the API is asked; with the API refused the remembered commit is used (row 3). Rename or assert the rule.
• M5. Raw refused plus mirror 404 reports an existing file as absent for the whole visit.
• M6. A failed first resolve leaves the whole visit unresolved with no retry; missing and moved are equally permanent per page.
• M7. A remembered commit that no longer exists reads as "missing" and is cached absent instead of degrading to HEAD.
• M8. A cached listing is trusted blindly at :494-496; check its shape and fall through to the network.
• M9. An empty or whitespace-only .json file now errors; HttpClient gave null and callers used defaults.
• M10. The reader's observables are eager and single-shot; retry() never re-reads.
• M11. The 20 s timeout applies to the listing too; a hanging api.github.com delays every page load by 20 s.
• M12. Four-part ids are readable before the "not published by DataTug" notice (G-A4b) exists.
• M13. Each cold e2e test spends 2 anonymous API calls on the CI runner's shared address.
• M14. Nits: githubGet hands fetch the original string, not parsed.href; stale comment at github-project-create.service.ts:164-168; MAX_DATA_FILE_BYTES unused.

Not verified: a real browser (gzip bodies, real opaqueredirect, private mode, real quota), jsDelivr's answer when GitHub refuses, main's behaviour for S1/S2 (read, not run).

VERDICT: blockers=0 majors=3 minors=14 land=no

🤖 Generated with Claude Code

[trakhimenok]

[fix r1 #181]

Head: 6a8376c (two commits on top of fa6fe54: a lint commit and the fix commit). CI on this head: build, test, e2e and journey e2e all green. Paths are under libs/datatug/main/src/lib/services/repo/ unless stated. Every fix has a test written first; the new tests are in github/github-project-reader.resilience.spec.ts (reader, store service), github/github-file-store-api.spec.ts (the guard), and the existing specs of the touched files.

Majors

• S1 done. github/github-project-reader.service.ts:432 forget(org, repo) drops the repo's sessions (every ref) and calls the new IGithubFileStore.forgetResolved (github/github-file-store.ts:311, drops the memos of that repo at every ref, none of another repo whose name begins the same). datatug-store.service.github.ts:47 forget(org, repo) drops the summaries of that repo. github/github-project-create.service.ts:148-150 calls both after both files are committed, before the registration and before the project ref is emitted (so before the form navigates); the form itself is unchanged. Absence of datatug-project.json is no longer persisted (github-project-reader.service.ts:792, in memory for the visit only). The files are committed to the repository's default branch, asked of GitHub (GET /repos/{org}/{repo}, github-project-create.service.ts:173, read with toGithubRepo), the GITHUB_PROJECT_BRANCH = 'main' constant is gone, and the stale HttpClient comments are fixed. Tests: project A open then B created in the same repo; the address opened before the project exists then created; a repo whose default branch is trunk (GET ref and PUT branch are trunk); forget keeps other repos; the create service forgets in order before registering.
• S2 done. datatug-store.service.github.ts:72 builds the summary with defer, and the reader is lazy now (M10), so a failed read is read again. Test: both hosts fail once, then recover; the second getProjectSummary succeeds.
• S3 done. github/github-file-store-api.ts:93 guardGithubFileStore: every store call the reader makes (the reader wraps its store at github-project-reader.service.ts:414) fails or times out into the empty answer; the timeout is GITHUB_STORE_TIMEOUT_MS = 1500 and the first timeout turns the cache off for the rest of the visit, so a hanging database costs one wait, not one per file. The timer is injectable (GITHUB_TIMER, GithubSetTimer; the default is setTimeout), tests fire it by hand (ManualTimers). A late answer is ignored. github/github-file-store.ts:136 (open): a blocked (or failed) open that later succeeds is closed, and every connection gets onversionchange the moment it opens. Tests: a store that never settles and an IDBFactory.open that never settles, each with the read completing from the network and no further wait; the late connection is closed; the guard's own cases.

Minors

• M1 done: github-project-reader.resilience.spec.ts "the file host … and the mirror has not the file" (nothing is stored, a retry works); with M5 the old mutation (if (commitKey)) has no path left, the persisted-absent case for raw's own 404 is asserted too.
• M2 done: rows 4 and 5 (github-project-reader.commit.spec.ts) run the second visit with the API still refused, and assert the exact requests and that nothing is in the store (file or resolved memo).
• M3 done: the clause is removed (github-project-reader.service.ts:466-474). It was unreachable: isTrustedGithubProject (nav/github-project-address.ts:217) refuses every ref but HEAD, and a 40-hex ref is never HEAD. The comment where the code was says so; the commit spec asserts both sides (isTrustedProjectAddress false for the demo repo at a SHA, true for it with no ref) next to the existing test, and nav/github-project-address.spec.ts already tests every SHA form.
• M4 done: renamed to "a trusted project asks GitHub again once the remembered answer is older than 5 minutes" (it asserts exactly that). I did not add an assertion on what a refused call does for a trusted project: today it uses the remembered commit like any project (row 3); if that should differ for the demo project it is a design call, not a fix.
• M5 done: github-project-reader.service.ts:773-780: raw refused or down and the mirror 404 throws GithubReadError naming both hosts; only raw's own 404 means absent. One existing test (readLikeThePages with the file host down) read absent files through the mirror, it now skips those, with a comment.
• M7 done: github-project-reader.service.ts:520 replaceRememberedCommit. When the commit came from the memory (the 5-minute memo, or the remembered fallback) and GitHub says it does not know it (the listing 404/422, or a raw 404 for any file, the project file included), the memo is dropped and the commit resolved again once, with every read of the session (including those waiting) redone at the new commit; nothing is cached as absent under the gone commit. If the re-resolve is refused the reads degrade to HEAD (state unresolved). A commit that is alive and a file that really is missing costs one extra commits/ call, once per visit, then the file is cached absent (the project file is never cached absent). Tests: the listing, the project file, parallel reads, refused re-resolve, alive commit (project file and another file), fresh commit not doubted.
• M8 done: github-project-reader.service.ts:351 readCachedTree checks array and entry shape; wrong shapes (not JSON, object, no path, unknown type, non-objects, empty) read the network. Note: a bad cached listing is not overwritten (the store keeps the first write of a key), so it costs one listing call per visit for that commit; the store never writes one, so I left it.
• M9 done: getRawJson returns null for an empty or whitespace-only file (type is now T | null | undefined); getEnvironmentSummary, getEntity, getBoard keep their defaults (tested, cold and from the cache). Invalid JSON still errors.
• M10 done: getRawJson, getRawText, readInfo and the listing are deferred (github-project-reader.service.ts:536, 558, 831, 868); invalid ids now error on subscription instead of throwing when the observable is made. retry() re-reads a failed file and a failed listing; re-subscribing reads what failed only; the budget is a set, so it counts distinct files, and now it counts at subscription (tested: used is 0 before, 1 after three subscriptions).
• M11 done, with its degrade path: the resolve call has its own limit, GITHUB_RESOLVE_TIMEOUT_MS = 5_000 (github/github-read-limits.ts:27, used at github-project-reader.service.ts:493); after it the read goes on at HEAD (state unresolved, tested with fake timers) and the listing keeps 20 s (tested: not aborted at 5 s, aborted at 20 s).
• M14 done: github/github-http.ts:126 passes parsed.href; tested with a mixed-case host, an explicit default port, a space and a fragment.

Left, as asked

• M6 (a failed first resolve leaves the visit unresolved; missing and moved are per page): not changed. A retry that re-resolves needs a policy (when, how often, what the sources line says), not two lines; the "Run again" step and the sources line are G-A4b's (chat page), measured in G-A6. Note M10 now lets a caller re-subscribe, but a resolved session is still kept for the page.
• M12 (four-part ids readable before the notice): left; the "not published by DataTug" notice and the trusted-list rule are G-A4b's acceptance (design 8.2). Nothing in the reader trusts a ref.
• M13 (2 anonymous API calls per cold e2e): left; the budget is the acceptance of G-A2/G-A4c (the e2e asserts at most 2 and still passes); how the CI runner's shared address should be spent (a fake, or a token) is G-A6's decision. This change adds none on a cold visit.
• MAX_DATA_FILE_BYTES unused: left (G-A3b).

Checks run locally: targeted specs for the reader, store, file store, http, create service, new-project form, nav, folders and services (740 passed); nx run-many -t lint -p datatug-main datatug-app clean (the one warning that was already in the tree, an unused of import in folders/core/datatug-folders.service.spec.ts, is fixed in its own commit); pnpm run check:zoneless OK. Not verified: a real browser (real IndexedDB blocked, real quota, gzip), jsDelivr's real 404 timing after a push. One thing I noticed and did not change: createProject commits README.md and datatug-project.json with two parallel contents-API PUTs to the same branch, which GitHub can answer with a 409; it predates this PR and is not in the review.

🤖 Generated with Claude Code

[trakhimenok]

[review r2 #181] Second adversarial review (Opus): 376 targeted tests, 35 mutations and 15 probe cases on an export of this head.

Reviewed-Head: 6a8376c

S1, S2, S3 and every minor claimed fixed are confirmed, each by a mutation that fails tests. The trust property holds (after a re-resolve the trusted project's commit comes only from commits/HEAD). In no case found is an existing user worse off than on main. Paths are under libs/datatug/main/src/lib/services/repo/.

Major, accepted as a tracked follow-up (must close before the demo is switched on; added to #180)

• A1. Any raw 404 at a remembered commit is treated as proof the commit is gone (github/github-project-reader.service.ts:780-789, :520-530). (a) With a memo under 5 minutes old at a live commit and a push since, one absent-file probe moves later reads to the new commit while the project file, session.tree, files and json stay at the old one, and readInfo names the new commit: two commits in one visit, against design 4.5. (b) When the API is refusing, the memo is deleted before the re-resolve is refused, so the visit drops to unresolved and cached listings stop being used on the next load. Fix: doubt a remembered commit only on a listing 404/422, or a project-file 404 with nothing read at that commit; delete the memo only after a fresh resolve answered; on a commit change reset the whole session. Main reads the branch unpinned, so this is not a regression.

Minors (follow-up, same issue)

1. The cache guard's first 1500 ms includes loading the chunk and opening the database, and a late success never turns the cache back on (github/github-file-store-api.ts:98-107); a slow phone never gets a cache.
2. forget races with a resolve in flight, which writes the old commit back (github-project-reader.service.ts:432-440 against :499); self-heals at one API call.
3. Raw refused, mirror working, optional file absent now errors for that file (:773-779); a jsDelivr 404 at a pinned SHA could be taken as absent for the visit.
4. Re-resolve drops the memos of every ref of the repo, untested (:524-526); three mutations survive.
5. A repo with no project file costs one API call on every page load.
6. github/github-api.ts:50 still falls back to 'main' when the payload has no default_branch.

Not verified: a real browser (IndexedDB open time, blocked), the create flow against real GitHub, jsDelivr's lag for a just-pushed SHA.

VERDICT: blockers=0 majors=1 minors=6 land=yes

🤖 Generated with Claude Code