elastic · cotti · Jul 23, 2026
@@ -0,0 +1,39 @@
+## Description
+
+Compares a scope's private `registry.json` manifest against the objects actually stored under the scope's key prefix in the private bucket, and classifies every divergence:
+
+| Class | Meaning |
+| ----- | ------- |
+| `missing` | An object exists in the scope but the registry has no entry for it. |
+| `stale` | The registry lists a file whose object no longer exists. |
+| `corrupt` | The manifest itself is unparseable or contains invalid (unsafe or duplicate) entries. |
+| `object-divergent` | A registry entry's recorded metadata (ETag or target) disagrees with the actual object. |
+
+The command is strictly **read-only** — it never writes to any bucket. It exits non-zero when the scope diverged; use [`changelog registry repair`](/cli/changelog/registry/repair.md) to reconcile.
+
+A manifest that declares a `schema_version` newer than this docs-builder understands is reported as `UnsupportedSchema`: its entries cannot be judged and repair refuses to touch it.
+
+For bundle scopes, the expected `target` of each entry is derived by reading the bundle YAML from S3 (for legacy amend sidecars without `products`, the parent bundle's target is used, matching the upload-time registry builder). Changelog scopes enumerate files only and never record a target.
+
+## State snapshot
+
+`--out <path>` writes a machine-readable JSON snapshot of the scope: registry health, the actual objects, the registry's current entries, the entries the registry *should* contain, and every divergence. The snapshot is the trustworthy current-state input for backfill planning, which cannot rely on the additive registry for discovery or removals.
+
+## Examples
+
+Inspect a product bundle scope:
+
+```sh
+docs-builder changelog registry inspect \
+  --s3-bucket-name elastic-docs-v3-changelog-bundles-private \
+  --product elasticsearch
+```
+
+Inspect an authoring pool and write the snapshot:
+
+```sh
+docs-builder changelog registry inspect \
+  --s3-bucket-name elastic-docs-v3-changelog-bundles-private \
+  --owner elastic --repo elasticsearch --branch main \
+  --out ./state-snapshot.json
+```
@@ -0,0 +1,42 @@
+## Description
+
+Reconciles a scope's **private** `registry.json` from the objects actually stored in the private bucket: missing entries are added, stale entries removed, and object-divergent metadata (ETag, target) corrected. A corrupt manifest is rebuilt from scratch.
+
+Repair is a separate, explicit operation — nothing runs it implicitly — and it is **idempotent**: a clean scope writes nothing, and running repair twice yields no further changes.
+
+## Concurrency safety
+
+The write uses the same optimistic-concurrency conditional PUT as the live upload path:
+
+- **update**: `If-Match: <etag-from-read>` — only succeeds if the manifest hasn't changed since the repair read it;
+- **create**: `If-None-Match: *` — only succeeds if the manifest still doesn't exist.
+
+A `412 Precondition Failed` means a concurrent live upload refreshed the manifest; the repair then re-inspects — a fresh registry read **and** a fresh object listing — and retries (bounded attempts). The registry is always read before the objects are listed, so an object uploaded concurrently either appears in the re-listing or its registry refresh invalidates the precondition; either way it survives the repair.
+
+## Safety rails
+
+- A repair that would produce an **empty** manifest aborts unless `--allow-empty` is passed.
+- A manifest with a `schema_version` newer than this docs-builder is never rewritten (that would silently downgrade it).
+- `--dry-run` reports the full audit (what would be added, removed, and corrected) without writing.
+- Every applied change is logged entry by entry with before/after values for audit.
+
+Only the private registry is written. The public copy is scrubber-owned: the repaired manifest reaches the public bucket through the scrubber's verbatim pass-through, triggered by this write's own `ObjectCreated` event.
+
+## Examples
+
+Preview a repair:
+
+```sh
+docs-builder changelog registry repair \
+  --s3-bucket-name elastic-docs-v3-changelog-bundles-private \
+  --product elasticsearch \
+  --dry-run
+```
+
+Repair an authoring pool:
+
+```sh
+docs-builder changelog registry repair \
+  --s3-bucket-name elastic-docs-v3-changelog-bundles-private \
+  --owner elastic --repo elasticsearch --branch main
+```
@@ -0,0 +1,38 @@
+## Description
+
+Re-emits the private-bucket `s3:ObjectCreated` event for selected objects in a scope so the changelog scrubber Lambda re-processes them. This is the explicit recovery path when [`changelog registry verify-public`](/cli/changelog/registry/verify-public.md) shows a persistently missing public object — typically because a scrub event was lost or ended up in the dead-letter queue.
+
+The re-emission is a **metadata-preserving S3 self-copy**: each selected object is copied onto its own key with `MetadataDirective: REPLACE`, re-supplying its original content type and user metadata. Content and ETag are unchanged; the copy produces the `ObjectCreated` notification the scrubber listens for, and the scrubber then re-scrubs and re-publishes the object to the public bucket itself.
+
+Republishing never writes to the public bucket — the scrubber remains the sole public-side writer — and it only ever happens through this explicit command.
+
+## Selection
+
+Exactly one selection is required:
+
+- `--files <name>[,<name>…]` — specific file names within the scope (for example `9.3.0.yaml`, or `registry.json` to re-trigger the manifest pass-through);
+- `--all` — every object in the scope, including its `registry.json`.
+
+## Examples
+
+Re-emit one lost bundle scrub event:
+
+```sh
+docs-builder changelog registry republish \
+  --s3-bucket-name elastic-docs-v3-changelog-bundles-private \
+  --product elasticsearch \
+  --files 9.3.0.yaml
+```
+
+Re-emit everything in an authoring pool:
+
+```sh
+docs-builder changelog registry republish \
+  --s3-bucket-name elastic-docs-v3-changelog-bundles-private \
+  --owner elastic --repo elasticsearch --branch main \
+  --all
+```
+
+:::{note}
+If the objects also exist locally, `changelog upload --skip-etag-check` achieves a similar re-trigger by re-uploading unchanged files. `republish` works purely from bucket state and needs no local checkout.
+:::
@@ -0,0 +1,36 @@
+## Description
+
+Verifies that the scrubber-owned **public** bucket has converged to the state expected from the **private** bucket for one scope:
+
+- the public `registry.json` must equal the private one (the scrubber passes manifests through verbatim);
+- every private YAML object must have a public counterpart at the same key;
+- no public object may outlive its private source.
+
+Registry and YAML scrub events propagate independently, so transient divergence is normal. The command therefore re-checks under a **bounded retry policy** — up to `--max-attempts` comparisons, `--poll-interval-seconds` apart (defaults: 12 × 10 s, a two-minute budget) — and succeeds as soon as the state converges.
+
+## Read-only by construction
+
+The public bucket is a hard write boundary: this command never writes to it. Internally the comparison runs against a reader interface that exposes no write operations, so no code path can mutate either bucket.
+
+If divergence persists after the retry budget — typically a lost or dead-lettered scrub event, or a bundle the scrubber refused to publish (unallowlisted private references) — the command reports each finding and exits non-zero. Recover with the explicit [`changelog registry republish`](/cli/changelog/registry/republish.md) operation on the private side.
+
+## Divergence classes
+
+| Finding | Meaning |
+| ------- | ------- |
+| `MissingPublicRegistry` | The private registry exists but its public pass-through copy does not. |
+| `CorruptPublicRegistry` | The public registry cannot be parsed. |
+| `RegistryMismatch` | Public registry entries differ from the private registry. |
+| `MissingPublicObject` | A private object has no public counterpart. |
+| `StalePublicObject` | A public object has no private counterpart. |
+
+## Examples
+
+```sh
+docs-builder changelog registry verify-public \
+  --s3-bucket-name elastic-docs-v3-changelog-bundles-private \
+  --public-s3-bucket-name elastic-docs-v3-changelog-bundles \
+  --product elasticsearch \
+  --max-attempts 12 \
+  --poll-interval-seconds 10
+```
@@ -0,0 +1,19 @@
+The `changelog registry` commands inspect, repair, and verify the per-scope `registry.json` manifests that index published changelog artifacts in S3 — `bundle/{product}/registry.json` for bundle scopes and `changelog/{org}/{repo}/{branch}/registry.json` for authoring pools.
+
+Registries merge additively under optimistic concurrency and are not authoritative for removals or discovery, so they can drift from the objects actually in the bucket. These commands detect that drift, reconcile the **private** registry from the actual private objects, and verify (without ever writing) that the scrubber-owned **public** bucket has converged.
+
+## Scope selection
+
+Every command addresses exactly one scope:
+
+- `--product <id>` — a bundle scope (`bundle/{product}/`)
+- `--owner <org> --repo <repo> --branch <branch>` — a changelog authoring pool (`changelog/{org}/{repo}/{branch}/`)
+
+## Typical workflow
+
+1. **Inspect** — `changelog registry inspect` reports every divergence between a scope's private registry and its actual objects, and can emit a machine-readable state snapshot.
+2. **Repair** — `changelog registry repair` reconciles the private registry from the actual objects (explicit, never implicit; idempotent).
+3. **Verify** — `changelog registry verify-public` waits with a bounded retry policy for the scrubber to propagate state to the public bucket and diagnoses divergence, strictly read-only.
+4. **Republish** — `changelog registry republish` re-emits the private-bucket `ObjectCreated` event for selected objects when a scrub event was lost or dead-lettered.
+
+See [Changelog bundle registry and CDN delivery](/development/changelog-bundle-registry.md) for the underlying architecture and reconciliation semantics.
@@ -75,8 +75,8 @@ Stored at `bundle/{product}/registry.json` (bundle index) or `changelog/{org}/{r
   "product": "elasticsearch",
   "generated_at": "2026-05-06T12:00:00+00:00",
   "bundles": [
-    { "file": "9.4.0.yaml", "target": "9.4.0", "etag": "…" },
-    { "file": "9.3.0.yaml", "target": "9.3.0", "etag": "…" }
+    { "file": "9.4.0.yaml", "target": "9.4.0", "e_tag": "…" },
+    { "file": "9.3.0.yaml", "target": "9.3.0", "e_tag": "…" }
   ]
 }
 ```
@@ -88,14 +88,14 @@ Stored at `bundle/{product}/registry.json` (bundle index) or `changelog/{org}/{r
 | `generated_at` | UTC timestamp of the last regeneration. |
 | `bundles[].file` | Bundle file name, resolved at `bundle/{product}/{file}` (or entry file at `changelog/{org}/{repo}/{branch}/{file}` for the entry index). |
 | `bundles[].target` | Target version/date from the bundle's declaration of **this** product (may be null). For an amend sidecar (`{name}.amend-{N}.yaml`) that declares no products itself (created by older docs-builder versions), the parent bundle's target is recorded when the parent file is available in the same upload run. |
-| `bundles[].etag` | See the ETag caveat below. |
+| `bundles[].e_tag` | See the ETag caveat below. |
 
 Bundles are sorted by `target` descending (newest first) with a deterministic tiebreak on
 `file`, so the JSON is stable across reruns.
 
 ### ETag caveat
 
-`bundles[].etag` is the ETag of the bundle object **as uploaded to the private bucket**
+`bundles[].e_tag` is the ETag of the bundle object **as uploaded to the private bucket**
 (pre-scrub). The scrubber rewrites any bundle that contains private references, so for
 scrubbed bundles this value **will not match** the public (CDN) object's ETag.
 
@@ -174,6 +174,68 @@ build that includes this feature.
 
 Consumers must therefore treat a missing bundle as non-fatal (skip + warn), not an error.
 
+## State discovery and reconciliation
+
+The merge-by-filename refresh above is **additive**: a manifest never has entries removed by
+the live path, it is best-effort (a failed refresh leaves it stale), and nothing in the live
+path ever compares it against the objects actually in the bucket. Registries are therefore not
+authoritative for removals or discovery, and they can drift — which matters as soon as
+something (an operator, or backfill planning) needs a trustworthy view of a scope's current
+state.
+
+The `changelog registry` command group (`src/services/Elastic.Changelog/Reconciliation/`)
+closes that gap per scope (`bundle/{product}/` or `changelog/{org}/{repo}/{branch}/`):
+
+### Inspection (read-only)
+
+`changelog registry inspect` lists the actual private objects under the scope prefix, reads the
+private manifest, and classifies every divergence into a four-class taxonomy:
+
+| Class | Meaning |
+|---|---|
+| **missing** | Object exists in the scope, the registry has no entry for it. |
+| **stale** | Registry entry whose object no longer exists. |
+| **corrupt** | The manifest is unparseable, or contains unsafe/duplicate entries. |
+| **object-divergent** | Entry metadata (ETag or target) disagrees with the actual object. |
+
+For bundle scopes the expected `target` is re-derived from the bundle YAML in S3 (including the
+parent-bundle fallback for legacy amends without `products`); changelog scopes enumerate files
+only. `--out` writes a machine-readable `RegistryStateSnapshot` (registry health, actual
+objects, current entries, expected entries, divergences) — the current-state input the backfill
+planner consumes. A manifest with a newer `schema_version` is reported as *unsupported*, not
+corrupt: its entries cannot be judged by an older tool.
+
+### Repair (explicit, private-side only)
+
+`changelog registry repair` converges the **private** manifest to the actual objects: it is a
+separate, explicit operation (nothing repairs implicitly), idempotent (a clean scope writes
+nothing; a second run is a no-op), and audited (every added/removed/corrected entry is logged
+with before/after values; `--dry-run` prints the plan). Writes go through the same conditional
+PUT as the live refresh — `If-Match` on update, `If-None-Match: *` on create — and a `412`
+triggers a full re-inspection (fresh manifest read **and** fresh object listing) before the
+bounded retry. Because the manifest is read *before* the objects are listed, a concurrent
+upload either shows up in the re-listing or invalidates the precondition; it cannot be dropped.
+Two rails guard against operator error: an empty result requires `--allow-empty`, and a
+newer-schema manifest is never rewritten (that would silently downgrade it).
+
+### Public-side verification (strictly read-only) and republish
+
+The public bucket is scrubber-owned: the scrubber Lambda is the **sole** writer there. So
+`changelog registry verify-public` only ever compares — the public registry must equal the
+private one (verbatim pass-through) and every private YAML object must have a public
+counterpart (and vice versa) — under a bounded retry policy
+(`--max-attempts` × `--poll-interval-seconds`), because registry and YAML scrub events
+propagate independently and transient divergence is normal. The boundary is structural, not
+conventional: the comparison code operates on a read-only S3 reader interface
+(`IS3ScopeReader`) that exposes no write operations.
+
+Recovery from a lost or dead-lettered scrub event is the explicit
+`changelog registry republish` operation on the **private** side: a metadata-preserving S3
+self-copy (`CopyObject` onto the same key with `MetadataDirective: REPLACE`, re-supplying the
+original content type and user metadata) that leaves content and ETag untouched while emitting
+the `ObjectCreated` event the scrubber reacts to. The scrubber then re-scrubs and re-publishes
+on its own.
+
 ## `changelog bundle` entry sourcing (org/repo/branch gate)
 
 The `changelog bundle` command aggregates individual changelog **entries**. It can read those