Skip to content

Rewrite changelog registry docs for scrubber Lambda ownership - #3761

Merged
cotti merged 7 commits into
mainfrom
changelog-registry-docs
Aug 13, 2026
Merged

Rewrite changelog registry docs for scrubber Lambda ownership#3761
cotti merged 7 commits into
mainfrom
changelog-registry-docs

Conversation

@cotti

@cotti cotti commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Why

Phase 4 of elastic/docs-eng-team#688. After Phases 1–3 the registry docs described a retired system: client-side refresh, registry pass-through everywhere, an ETag "documented as useless", a CloudFront "1 h TTL" that hasn't existed since caching was disabled, and a claim that the refresh was "skipped for --artifact-type changelog".

Note

Un-drafted 2026-08-13: the original draft predated the #3738 review rework, so it documented the dropped per-product operator tooling. It now documents the merged end state (#3738, #3760, docs-infra#360). The branch was brought up to date by merging main; everything outside docs/ resolves to main, so the diff is docs-only.

What

docs/development/changelog-bundle-registry.md now documents the end state:

  • Ownership split per tree: bundle/{product}/registry.json is produced solely by the scrubber Lambda's BundleRegistryReconciler (registry = f(state)); the changelog/{org}/{repo}/{branch}/ pool manifests remain legacy client-authored pass-through — changelog bundle still enumerates a pool through its manifest — until PR-list-driven discovery (RFC elastic/docs-eng-team#698) replaces them.
  • The producer field (changelog-scrubber-reconcile/1, null on legacy manifests) and bundles[].etag as the public object's ETag, finally usable for CDN cache revalidation.
  • The shallow per-tree change maps (bundle/registry.json, changelog/registry.json): folder → opaque digest token, cache opt-out only; consumer side tracked in Changelog CDN fetchers: skip unchanged folders via shallow registry maps #3801.
  • Consistency stated as convergence (with the consumer's listed-but-missing tolerance kept) and the deliberate absent ≠ empty manifest semantics.
  • No operator CLI: the registry reconcile/verify command pages were removed (Add changelog registry reconcile and verify commands #3741 closed unmerged); out-of-band drift repair rides the normal event path (--skip-etag-check re-upload).
  • Infrastructure bullets match the applied docs-infra#360 IAM (no registry-operator grant, no private ListBucket for the Lambda) and observability as it exists today (SQS metric stream to docs-o11y; alert rules + runbook tracked in elastic/docs-eng-team#692).

The scrubber README's event-handling section and cmd-upload.md's registry paragraph are aligned with the same model.

Part of elastic/docs-eng-team#688 (Phase 4).

@cotti cotti added the documentation Improvements or additions to documentation label Aug 4, 2026
@cotti
cotti requested review from a team as code owners August 4, 2026 01:36
@cotti cotti added the documentation Improvements or additions to documentation label Aug 4, 2026
@cotti
cotti requested a review from reakaleek August 4, 2026 01:36
cotti and others added 4 commits August 3, 2026 22:50
@cotti @claude
Scrubber Lambda owns the public changelog registry via state reconcile
b0042e1 
Phase 1 of elastic/docs-eng-team#688. The public registry.json was a log of
upload operations (client-written, pass-through copied); every known
consistency gap followed from that. The scrubber Lambda now derives it from
the public bucket's actual state: registry = f(state), never f(event).

- Extract the Lambda's top-level handler logic into testable classes in
  Elastic.Changelog: ScrubberProcessor (batch coalescing by key and group,
  object-level reconcile with post-write source validation) and
  RegistryReconciler (delimited/paginated group listing, ETag reuse with
  amends always recomputed, semantic idempotence, conditional PUT/DELETE
  with bounded jittered retries on 412/409, newer-schema refusal).
  Program.cs is now a thin adapter.
- Retire the registry pass-through in the same deploy: registry-key events
  only schedule a group reconcile, so client-authored JSON no longer
  reaches the public bucket uninspected.
- Add a producer (algorithm version) field to the manifest; a mismatch —
  including legacy pass-through manifests — forces a full metadata
  recompute and a write even when entries are identical.
- Emit per-invocation reconcile metrics as CloudWatch EMF (the Phase 0
  observability item that could only land with the reconciler).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cotti @claude
Add changelog registry reconcile and verify commands
8a53106 
Phase 2 of elastic/docs-eng-team#688. The cutover/heal tooling for the
Lambda-owned public registry:

- `changelog registry reconcile` plans groups (one scope, or the union of
  both buckets so orphan public groups are covered) and sends one versioned,
  discriminated reconcile message per group to the scrubber queue —
  {kind, version, scope, group, correlation_id}, validated through
  ChangelogKeys on both ends. The CLI never mutates S3; the Lambda stays the
  public bucket's single writer. --dry-run prints the plan; the non-dry-run
  path asks for confirmation (--yes for CI). Each run stamps one correlation
  id and prints a ledger line per group.
- On a reconcile message the Lambda performs a full group heal:
  object-level reconcile over the union of both buckets' listings (copy
  what's live, delete what isn't), then the group reconcile — recovering
  lost/DLQ-expired scrub events. Requires the new optional
  PRIVATE_BUCKET_NAME Lambda env var; malformed messages are rejected to
  the DLQ where the Phase 0 alarm surfaces them.
- `changelog registry verify` is the read-only sibling and cutover gate:
  compares each public manifest against what a reconcile would write (same
  listing spec and entry rules by construction) and reports divergence as
  missing/stale/corrupt/object-divergent, with unsupported schemas reported
  distinctly.
- Fix the manifest ETag wire format: the snake_case policy serialized the
  producer-side field as "e_tag" while consumers and the documented format
  read "etag" — recorded ETags were invisible to every consumer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cotti @claude
Suppress CA1001 on ChangelogRegistryServiceTests
d5a2c93 
Same suppression RegistryBuilderTests carries: xUnit owns the test class
lifetime and TestDiagnosticsCollector needs no disposal in these tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cotti
Retire the client-side registry refresh from changelog upload
cc65ecf 
The scrubber Lambda is the sole producer of the public registry.json,
reconciled from public bucket state on the S3 events every upload already
emits (elastic/docs-eng-team#688 Phase 3). Uploads now write YAML objects
only; RegistryBuilder and the private-manifest write path are removed, and
the amend end-to-end test exercises RegistryReconciler instead.
@cotti
cotti force-pushed the changelog-retire-client-registry-refresh branch from 4cefbe5 to cc65ecf Compare August 4, 2026 01:50
@cotti
Rewrite changelog registry docs for scrubber Lambda ownership
9bebf38 
The registry docs still described the retired model: client-side refresh,
registry pass-through, pre-scrub ETags, a 1 h CloudFront TTL (caching is
disabled), and a refresh "skipped for --artifact-type changelog". Documents
the reconciler as sole producer, the public-object ETag, convergence
semantics, absent-vs-empty manifests, the reconcile message contract, and
the registry reconcile/verify operator commands (docs-eng-team#688 Phase 4).
@cotti
cotti force-pushed the changelog-registry-docs branch from 7af75b0 to 9bebf38 Compare August 4, 2026 01:50
@cotti cotti mentioned this pull request Aug 4, 2026
Merged
@cotti

cotti commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Converting to draft: these docs describe the per-product registry model that's being dropped per the review on #3738. I'll rewrite them once the thin folder→ETag registry shape lands there.

@cotti
cotti marked this pull request as draft August 6, 2026 13:07
@cotti cotti mentioned this pull request Aug 6, 2026
Merged
@cotti
cotti force-pushed the changelog-retire-client-registry-refresh branch 2 times, most recently from 95b64ed to 5695678 Compare August 10, 2026 16:21
@cotti
cotti force-pushed the changelog-retire-client-registry-refresh branch 2 times, most recently from ba57fc6 to 31d19e6 Compare August 11, 2026 12:54
Base automatically changed from changelog-retire-client-registry-refresh to main August 11, 2026 14:39
cotti added 2 commits August 13, 2026 10:31
@cotti
Merge origin/main: code superseded by merged #3738/#3760, keep docs
edc9575 
Everything outside docs/ resolves to main (the stacked code merged via #3738/#3760); docs/cli-schema.json resolves to main (the registry commands were dropped with #3741).
@cotti
Align registry docs with the merged #3738 rework and #3760
7589980 
The draft predated the #3738 review rework: RegistryReconciler is now BundleRegistryReconciler and reconciles the bundle/{product}/ tree only, the changelog/ pool manifests remain legacy client-authored pass-through (changelog bundle still enumerates pools through them, RFC #698 replaces that), the registry reconcile/verify operator CLI was dropped with #3741, and the scrubber now also maintains the shallow per-tree folder-to-token maps (consumer side: #3801). Infra bullets match the applied docs-infra#360 IAM (no registry-operator grant, no private ListBucket) and observability as it exists (metric stream to docs-o11y; alerts and runbook tracked in docs-eng-team#692).
@cotti
cotti marked this pull request as ready for review August 13, 2026 13:34
@cotti

cotti commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

Un-drafted and caught up with main (merge, no rewrite — everything outside docs/ resolves to main, so the diff is docs-only again). Since the approved 9bebf38, the docs were realigned with the merged rework in 7589980: BundleRegistryReconciler name and bundle-tree-only scope, the pool manifests documented as legacy client-authored pass-through (still enumerated by changelog bundle until RFC #698 discovery lands), a new section for the shallow per-tree maps (consumer side: #3801), the registry reconcile/verify pages removed (#3741 closed unmerged) with drift repair via --skip-etag-check documented instead, and infra bullets matching the applied docs-infra#360 IAM and the metric-stream-only observability (alerts/runbook: docs-eng-team#692). Strict docs build passes locally.

@cotti
cotti merged commit 60feab0 into main Aug 13, 2026
25 checks passed
@cotti
cotti deleted the changelog-registry-docs branch August 13, 2026 13:47
Mpdreamz pushed a commit that referenced this pull request Aug 14, 2026
@cotti @claude @Mpdreamz
Rewrite changelog registry docs for scrubber Lambda ownership (#3761)
cd950a3 
* Scrubber Lambda owns the public changelog registry via state reconcile

Phase 1 of elastic/docs-eng-team#688. The public registry.json was a log of
upload operations (client-written, pass-through copied); every known
consistency gap followed from that. The scrubber Lambda now derives it from
the public bucket's actual state: registry = f(state), never f(event).

- Extract the Lambda's top-level handler logic into testable classes in
  Elastic.Changelog: ScrubberProcessor (batch coalescing by key and group,
  object-level reconcile with post-write source validation) and
  RegistryReconciler (delimited/paginated group listing, ETag reuse with
  amends always recomputed, semantic idempotence, conditional PUT/DELETE
  with bounded jittered retries on 412/409, newer-schema refusal).
  Program.cs is now a thin adapter.
- Retire the registry pass-through in the same deploy: registry-key events
  only schedule a group reconcile, so client-authored JSON no longer
  reaches the public bucket uninspected.
- Add a producer (algorithm version) field to the manifest; a mismatch —
  including legacy pass-through manifests — forces a full metadata
  recompute and a write even when entries are identical.
- Emit per-invocation reconcile metrics as CloudWatch EMF (the Phase 0
  observability item that could only land with the reconciler).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add changelog registry reconcile and verify commands

Phase 2 of elastic/docs-eng-team#688. The cutover/heal tooling for the
Lambda-owned public registry:

- `changelog registry reconcile` plans groups (one scope, or the union of
  both buckets so orphan public groups are covered) and sends one versioned,
  discriminated reconcile message per group to the scrubber queue —
  {kind, version, scope, group, correlation_id}, validated through
  ChangelogKeys on both ends. The CLI never mutates S3; the Lambda stays the
  public bucket's single writer. --dry-run prints the plan; the non-dry-run
  path asks for confirmation (--yes for CI). Each run stamps one correlation
  id and prints a ledger line per group.
- On a reconcile message the Lambda performs a full group heal:
  object-level reconcile over the union of both buckets' listings (copy
  what's live, delete what isn't), then the group reconcile — recovering
  lost/DLQ-expired scrub events. Requires the new optional
  PRIVATE_BUCKET_NAME Lambda env var; malformed messages are rejected to
  the DLQ where the Phase 0 alarm surfaces them.
- `changelog registry verify` is the read-only sibling and cutover gate:
  compares each public manifest against what a reconcile would write (same
  listing spec and entry rules by construction) and reports divergence as
  missing/stale/corrupt/object-divergent, with unsupported schemas reported
  distinctly.
- Fix the manifest ETag wire format: the snake_case policy serialized the
  producer-side field as "e_tag" while consumers and the documented format
  read "etag" — recorded ETags were invisible to every consumer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Suppress CA1001 on ChangelogRegistryServiceTests

Same suppression RegistryBuilderTests carries: xUnit owns the test class
lifetime and TestDiagnosticsCollector needs no disposal in these tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Retire the client-side registry refresh from changelog upload

The scrubber Lambda is the sole producer of the public registry.json,
reconciled from public bucket state on the S3 events every upload already
emits (elastic/docs-eng-team#688 Phase 3). Uploads now write YAML objects
only; RegistryBuilder and the private-manifest write path are removed, and
the amend end-to-end test exercises RegistryReconciler instead.

* Rewrite changelog registry docs for scrubber Lambda ownership

The registry docs still described the retired model: client-side refresh,
registry pass-through, pre-scrub ETags, a 1 h CloudFront TTL (caching is
disabled), and a refresh "skipped for --artifact-type changelog". Documents
the reconciler as sole producer, the public-object ETag, convergence
semantics, absent-vs-empty manifests, the reconcile message contract, and
the registry reconcile/verify operator commands (docs-eng-team#688 Phase 4).

* Align registry docs with the merged #3738 rework and #3760

The draft predated the #3738 review rework: RegistryReconciler is now BundleRegistryReconciler and reconciles the bundle/{product}/ tree only, the changelog/ pool manifests remain legacy client-authored pass-through (changelog bundle still enumerates pools through them, RFC #698 replaces that), the registry reconcile/verify operator CLI was dropped with #3741, and the scrubber now also maintains the shallow per-tree folder-to-token maps (consumer side: #3801). Infra bullets match the applied docs-infra#360 IAM (no registry-operator grant, no private ListBucket) and observability as it exists (metric stream to docs-o11y; alerts and runbook tracked in docs-eng-team#692).

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Mpdreamz Mpdreamz Mpdreamz approved these changes
@reakaleek reakaleek Awaiting requested review from reakaleek reakaleek is a code owner automatically assigned from elastic/docs-engineering

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cotti @Mpdreamz