Skip to content

Changelog CDN fetchers: skip unchanged folders via shallow registry maps - #3801

Open
cotti wants to merge 1 commit into
mainfrom
changelog-shallow-map-consumer
Open

Changelog CDN fetchers: skip unchanged folders via shallow registry maps#3801
cotti wants to merge 1 commit into
mainfrom
changelog-shallow-map-consumer

Conversation

@cotti

@cotti cotti commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Implements the consumer half of the shallow per-tree registry maps introduced in #3738 (stacked on that PR's branch): the bundle CDN fetcher now consults bundle/registry.json before fetching per-product registries, and skips folders whose content demonstrably hasn't changed.

Refs: elastic/docs-eng-team#737, elastic/docs-eng-team#688, #3738.

What it does

CdnChangelogFetcher fetches the tree's shallow map (bundle/registry.json, shape {"<product>": "<token>"}) once per fetcher run, memoized per base URI. For each product:

  • Token match + warm cache → the per-product bundle/{product}/registry.json fetch is skipped and the locally cached registry is reused. Bundle content then resolves through the existing ETag-keyed cache, so an unchanged folder with a warm cache produces zero per-folder requests (asserted in tests).
  • Token mismatch, product missing from the map, or no cached state → the flow is byte-identical to today's: fetch the registry, download bundles, and record the fresh registry under the folder's current token for the next run.

Graceful degradation (non-negotiable, preserved)

A shallow map that is absent (404), unparseable, or fails to fetch degrades to null inside FetchShallowMapAsync — a single debug log, no errors, no warnings — and every per-product registry is fetched exactly as before the map existed. Pre-cutover CDNs and buckets without maps keep working unchanged. Tokens are treated as fully opaque: compared with ordinal string equality only, never parsed.

Cache/token bookkeeping design

  • The registry text is cached under key registry-{product}-{token} in the fetcher's existing memory + disk cache ({ApplicationData}/changelog-bundles/), the same store and conventions the ETag-keyed bundle cache (changelog-{product}-{file}-{etag}) already uses. Embedding the token in the key makes a token mismatch a plain cache miss under the new key — no separate "last-seen token" state to keep consistent.
  • This keying is sound because the producer (ShallowRegistryReconciler) excludes group manifests from the token digest precisely so rewriting a manifest cannot invalidate consumer caches: same token ⇒ same folder listing ⇒ same derived registry.
  • The shallow map itself is deliberately never cached to disk — it is the freshness signal — and is fetched at most once per fetcher run regardless of product count.
  • Defense-in-depth: a token that is not a safe single path segment is ignored (treated as "no token") before it can be joined into a cache file path, mirroring the fetchers' existing file-name validation.

Deliberately not covered: CdnChangelogEntryFetcher

The changelog-pool entry fetcher (changelog/{org}/{repo}/{branch}) has no local cache of any kind — it downloads entries and returns them directly to the bundle command. A matching pool token could therefore never skip anything: there is no cached state a skip could reuse, and honoring the issue's "no behavior change" constraint would require inventing a parallel content store for entry YAML. Per the review direction ("scope the opt-out to whatever cache genuinely exists; correctness over cleverness"), that fetcher is left unchanged; fetching changelog/registry.json there would only add a wasted request per run. If a pool-level entry cache is ever introduced, the same token-keyed pattern applies directly.

Test plan

All in CdnChangelogFetcherTests (fake HttpMessageHandler + MockFileSystem, per the existing pattern):

  • Map absent (404) → identical behavior to today (registry + bundles fetched, no diagnostics).
  • Map unparseable → identical behavior, no crash.
  • Token match with warm cache → the fake handler sees only the single map request; results served entirely from cache.
  • Token mismatch → normal fetch, fresh registry recorded under the new token.
  • Token present but cold cache → normal fetch on run one, full skip on a second run sharing the disk cache.
  • Partial run across two products (one matching, one not) → only the changed product's folder is hit; the map is fetched exactly once.
  • Existing ETag-cache tests updated for the one extra map probe per run.

Checks run: dotnet format (clean), ./build.sh build --skip-dirty-check (pass), dotnet publish src/tooling/docs-builder -c Release AOT publish (zero trim/AOT warnings; the shallow map type is registered on the source-generated ChangelogRegistryJsonContext), dotnet test for Elastic.Documentation.Configuration.Tests (608 passed) and Elastic.Changelog.Tests (865 passed). No CLI surface changes, so no docs/cli-schema.json regen.

@cotti
cotti requested a review from a team as a code owner August 6, 2026 15:54
@cotti
cotti requested a review from technige August 6, 2026 15:54
@cotti
cotti force-pushed the changelog-registry-scrubber-ownership branch from 8f92f56 to 7f4181c Compare August 10, 2026 16:21
Base automatically changed from changelog-registry-scrubber-ownership to main August 11, 2026 12:54
@cotti
cotti force-pushed the changelog-shallow-map-consumer branch from d460310 to 08cd980 Compare August 11, 2026 13:01
@github-actions

Copy link
Copy Markdown

Label error. Requires exactly 1 of: automation, breaking, bug, changelog:skip, chore, ci, dependencies, documentation, enhancement, feature, fix, redesign. Found:

cotti added a commit that referenced this pull request Aug 13, 2026
The draft predated the #3738 review rework: RegistryReconciler is now BundleRegistryReconciler and reconciles the bundle/{product}/ tree only, the changelog/ pool manifests remain legacy client-authored pass-through (changelog bundle still enumerates pools through them, RFC #698 replaces that), the registry reconcile/verify operator CLI was dropped with #3741, and the scrubber now also maintains the shallow per-tree folder-to-token maps (consumer side: #3801). Infra bullets match the applied docs-infra#360 IAM (no registry-operator grant, no private ListBucket) and observability as it exists (metric stream to docs-o11y; alerts and runbook tracked in docs-eng-team#692).
cotti added a commit that referenced this pull request Aug 13, 2026
* Scrubber Lambda owns the public changelog registry via state reconcile

Phase 1 of elastic/docs-eng-team#688. The public registry.json was a log of
upload operations (client-written, pass-through copied); every known
consistency gap followed from that. The scrubber Lambda now derives it from
the public bucket's actual state: registry = f(state), never f(event).

- Extract the Lambda's top-level handler logic into testable classes in
  Elastic.Changelog: ScrubberProcessor (batch coalescing by key and group,
  object-level reconcile with post-write source validation) and
  RegistryReconciler (delimited/paginated group listing, ETag reuse with
  amends always recomputed, semantic idempotence, conditional PUT/DELETE
  with bounded jittered retries on 412/409, newer-schema refusal).
  Program.cs is now a thin adapter.
- Retire the registry pass-through in the same deploy: registry-key events
  only schedule a group reconcile, so client-authored JSON no longer
  reaches the public bucket uninspected.
- Add a producer (algorithm version) field to the manifest; a mismatch —
  including legacy pass-through manifests — forces a full metadata
  recompute and a write even when entries are identical.
- Emit per-invocation reconcile metrics as CloudWatch EMF (the Phase 0
  observability item that could only land with the reconciler).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add changelog registry reconcile and verify commands

Phase 2 of elastic/docs-eng-team#688. The cutover/heal tooling for the
Lambda-owned public registry:

- `changelog registry reconcile` plans groups (one scope, or the union of
  both buckets so orphan public groups are covered) and sends one versioned,
  discriminated reconcile message per group to the scrubber queue —
  {kind, version, scope, group, correlation_id}, validated through
  ChangelogKeys on both ends. The CLI never mutates S3; the Lambda stays the
  public bucket's single writer. --dry-run prints the plan; the non-dry-run
  path asks for confirmation (--yes for CI). Each run stamps one correlation
  id and prints a ledger line per group.
- On a reconcile message the Lambda performs a full group heal:
  object-level reconcile over the union of both buckets' listings (copy
  what's live, delete what isn't), then the group reconcile — recovering
  lost/DLQ-expired scrub events. Requires the new optional
  PRIVATE_BUCKET_NAME Lambda env var; malformed messages are rejected to
  the DLQ where the Phase 0 alarm surfaces them.
- `changelog registry verify` is the read-only sibling and cutover gate:
  compares each public manifest against what a reconcile would write (same
  listing spec and entry rules by construction) and reports divergence as
  missing/stale/corrupt/object-divergent, with unsupported schemas reported
  distinctly.
- Fix the manifest ETag wire format: the snake_case policy serialized the
  producer-side field as "e_tag" while consumers and the documented format
  read "etag" — recorded ETags were invisible to every consumer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Suppress CA1001 on ChangelogRegistryServiceTests

Same suppression RegistryBuilderTests carries: xUnit owns the test class
lifetime and TestDiagnosticsCollector needs no disposal in these tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Retire the client-side registry refresh from changelog upload

The scrubber Lambda is the sole producer of the public registry.json,
reconciled from public bucket state on the S3 events every upload already
emits (elastic/docs-eng-team#688 Phase 3). Uploads now write YAML objects
only; RegistryBuilder and the private-manifest write path are removed, and
the amend end-to-end test exercises RegistryReconciler instead.

* Rewrite changelog registry docs for scrubber Lambda ownership

The registry docs still described the retired model: client-side refresh,
registry pass-through, pre-scrub ETags, a 1 h CloudFront TTL (caching is
disabled), and a refresh "skipped for --artifact-type changelog". Documents
the reconciler as sole producer, the public-object ETag, convergence
semantics, absent-vs-empty manifests, the reconcile message contract, and
the registry reconcile/verify operator commands (docs-eng-team#688 Phase 4).

* Align registry docs with the merged #3738 rework and #3760

The draft predated the #3738 review rework: RegistryReconciler is now BundleRegistryReconciler and reconciles the bundle/{product}/ tree only, the changelog/ pool manifests remain legacy client-authored pass-through (changelog bundle still enumerates pools through them, RFC #698 replaces that), the registry reconcile/verify operator CLI was dropped with #3741, and the scrubber now also maintains the shallow per-tree folder-to-token maps (consumer side: #3801). Infra bullets match the applied docs-infra#360 IAM (no registry-operator grant, no private ListBucket) and observability as it exists (metric stream to docs-o11y; alerts and runbook tracked in docs-eng-team#692).

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Mpdreamz pushed a commit that referenced this pull request Aug 14, 2026
* Scrubber Lambda owns the public changelog registry via state reconcile

Phase 1 of elastic/docs-eng-team#688. The public registry.json was a log of
upload operations (client-written, pass-through copied); every known
consistency gap followed from that. The scrubber Lambda now derives it from
the public bucket's actual state: registry = f(state), never f(event).

- Extract the Lambda's top-level handler logic into testable classes in
  Elastic.Changelog: ScrubberProcessor (batch coalescing by key and group,
  object-level reconcile with post-write source validation) and
  RegistryReconciler (delimited/paginated group listing, ETag reuse with
  amends always recomputed, semantic idempotence, conditional PUT/DELETE
  with bounded jittered retries on 412/409, newer-schema refusal).
  Program.cs is now a thin adapter.
- Retire the registry pass-through in the same deploy: registry-key events
  only schedule a group reconcile, so client-authored JSON no longer
  reaches the public bucket uninspected.
- Add a producer (algorithm version) field to the manifest; a mismatch —
  including legacy pass-through manifests — forces a full metadata
  recompute and a write even when entries are identical.
- Emit per-invocation reconcile metrics as CloudWatch EMF (the Phase 0
  observability item that could only land with the reconciler).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add changelog registry reconcile and verify commands

Phase 2 of elastic/docs-eng-team#688. The cutover/heal tooling for the
Lambda-owned public registry:

- `changelog registry reconcile` plans groups (one scope, or the union of
  both buckets so orphan public groups are covered) and sends one versioned,
  discriminated reconcile message per group to the scrubber queue —
  {kind, version, scope, group, correlation_id}, validated through
  ChangelogKeys on both ends. The CLI never mutates S3; the Lambda stays the
  public bucket's single writer. --dry-run prints the plan; the non-dry-run
  path asks for confirmation (--yes for CI). Each run stamps one correlation
  id and prints a ledger line per group.
- On a reconcile message the Lambda performs a full group heal:
  object-level reconcile over the union of both buckets' listings (copy
  what's live, delete what isn't), then the group reconcile — recovering
  lost/DLQ-expired scrub events. Requires the new optional
  PRIVATE_BUCKET_NAME Lambda env var; malformed messages are rejected to
  the DLQ where the Phase 0 alarm surfaces them.
- `changelog registry verify` is the read-only sibling and cutover gate:
  compares each public manifest against what a reconcile would write (same
  listing spec and entry rules by construction) and reports divergence as
  missing/stale/corrupt/object-divergent, with unsupported schemas reported
  distinctly.
- Fix the manifest ETag wire format: the snake_case policy serialized the
  producer-side field as "e_tag" while consumers and the documented format
  read "etag" — recorded ETags were invisible to every consumer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Suppress CA1001 on ChangelogRegistryServiceTests

Same suppression RegistryBuilderTests carries: xUnit owns the test class
lifetime and TestDiagnosticsCollector needs no disposal in these tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Retire the client-side registry refresh from changelog upload

The scrubber Lambda is the sole producer of the public registry.json,
reconciled from public bucket state on the S3 events every upload already
emits (elastic/docs-eng-team#688 Phase 3). Uploads now write YAML objects
only; RegistryBuilder and the private-manifest write path are removed, and
the amend end-to-end test exercises RegistryReconciler instead.

* Rewrite changelog registry docs for scrubber Lambda ownership

The registry docs still described the retired model: client-side refresh,
registry pass-through, pre-scrub ETags, a 1 h CloudFront TTL (caching is
disabled), and a refresh "skipped for --artifact-type changelog". Documents
the reconciler as sole producer, the public-object ETag, convergence
semantics, absent-vs-empty manifests, the reconcile message contract, and
the registry reconcile/verify operator commands (docs-eng-team#688 Phase 4).

* Align registry docs with the merged #3738 rework and #3760

The draft predated the #3738 review rework: RegistryReconciler is now BundleRegistryReconciler and reconciles the bundle/{product}/ tree only, the changelog/ pool manifests remain legacy client-authored pass-through (changelog bundle still enumerates pools through them, RFC #698 replaces that), the registry reconcile/verify operator CLI was dropped with #3741, and the scrubber now also maintains the shallow per-tree folder-to-token maps (consumer side: #3801). Infra bullets match the applied docs-infra#360 IAM (no registry-operator grant, no private ListBucket) and observability as it exists (metric stream to docs-o11y; alerts and runbook tracked in docs-eng-team#692).

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant