Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 90 additions & 1 deletion docs/cli-schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -4433,7 +4433,96 @@
]
}
],
"namespaces": []
"namespaces": [
{
"segment": "backfill",
"summary": "Backfill historical release-note bundles (docs-eng-team#656): census, planning, and guarded publication.",
"options": [],
"commands": [
{
"path": [
"changelog",
"backfill"
],
"name": "inventory",
"summary": "Build the backfill census: an inventory document covering every release-notes product.",
"notes": "Enumerates every product in products.yml that participates in release notes (the\nrelease-notes feature defaults to enabled, so products.yml alone cannot say which\nproducts have release-note surfaces), merges in the hand-maintained census seed, and writes the\nversioned inventory document that backfill planning consumes. Products the seed does not cover\nstay visible as source-unresolved entries with a warning \u2014 an unresolved scope must never\nsilently produce empty bundles. Attributed repositories are checked against the link allowlist in\nthe local assembler.yml; planning re-validates against the deployed scrubber allowlist\nbefore any upload. This command only reads configuration and writes a local file: no S3 access,\nno writes to any remote system.",
"usage": "docs-builder changelog backfill inventory [options]",
"examples": [],
"parameters": [
{
"role": "flag",
"name": "sources",
"type": "string",
"required": false,
"summary": "Path to the census seed YAML mapping products to their release-note sources. Without it, every release-notes product is reported as source-unresolved.",
"validations": [
{
"kind": "rejectSymbolicLinks"
},
{
"kind": "existing"
},
{
"kind": "fileExtensions",
"values": [
"yml",
"yaml"
]
}
]
},
{
"role": "flag",
"name": "output",
"type": "string",
"required": false,
"summary": "Where to write the inventory document JSON.",
"defaultValue": "backfill-inventory.json"
},
{
"role": "flag",
"name": "log-level",
"shortName": "l",
"type": "enum",
"required": false,
"summary": "Minimum log level. Default: information",
"enumValues": [
"trace",
"debug",
"information",
"warning",
"error",
"critical",
"none"
]
},
{
"role": "flag",
"name": "config-source",
"shortName": "c",
"type": "enum",
"required": false,
"summary": "Override the configuration source: local, remote",
"enumValues": [
"local",
"remote",
"embedded"
]
},
{
"role": "flag",
"name": "skip-private-repositories",
"type": "boolean",
"required": false,
"summary": "Skip cloning private repositories"
}
]
}
],
"namespaces": []
}
]
},
{
"segment": "codex",
Expand Down
59 changes: 59 additions & 0 deletions docs/cli/changelog/backfill/cmd-inventory.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
## Description

Build the backfill census: an inventory document covering every release-notes product.

`config/products.yml` cannot say which products have release-note surfaces on its own, because the `release-notes` feature defaults to enabled. This command enumerates every product that participates in release notes, merges in a hand-maintained census seed mapping products to their sources, and writes the versioned inventory document that backfill planning consumes.

Products the seed does not cover stay visible as `source-unresolved` entries and produce a warning — "we looked and decided no" must always be distinguishable from "we never looked", and an unresolved scope can never silently produce empty bundles. Products can be deliberately deferred in the seed's `unmapped` list, each with a reason, which records the deferral without a warning.

The command is read-only apart from the local output file: it reads configuration, no S3 access, no remote writes.

### Defaults applied by the census

- Sources whose products are all stack-versioned and whose scheme is `semver` get the epic's default cutoff of `9.0.0` when the seed does not specify one.
- Unresolved products get a target scheme derived from their versioning system (`serverless`/project versioning → `date`, `ech` → `monthly`, everything else → `semver`), always paired with an unresolved note so a guess never reads as a confirmed fact.
- Attributed repositories are checked against the link allowlist in the local `assembler.yml`. Planning re-validates against the **deployed** scrubber allowlist identity before any upload; the census status is advisory.

## Seed format

```yaml
sources:
- repository: elastic/docs-content # where the release-note content lives
git_ref: main
docset: docs-content # optional
paths:
- release-notes/elasticsearch
products: [elasticsearch] # products.yml ids
target_scheme: semver # semver | date | monthly
cutoff: # optional; stack semver defaults to 9.0.0
kind: version # version | date
value: 9.0.0
notes: optional free text
substitutions: {} # docset variable expansions
link_mappings: {} # source link -> canonical destination
attributed_repositories: # repos entries attribute changes to
- elastic/elasticsearch
default_repository: elastic/elasticsearch
bundle_filename_convention: "{repo}-{target}.yaml"
adoption: not-adopted # not-adopted | partially-adopted | fully-adopted
classification: published-history-found
unresolved: [] # open questions for a human
unmapped:
- product: kibana
reason: Deferred to the stack family pass.
```

Valid classifications: `published-history-found`, `native-artifacts-found`, `hybrid-page`, `declared-no-history`, `outside-cutoff`, `already-live`. `source-unresolved` is deliberately not seedable — it is the census's own conclusion for products nobody mapped, never something an operator writes by hand.

## Examples

```sh
# Census with a seed; writes the inventory document
docs-builder changelog backfill inventory \
--sources config/backfill/inventory-sources.yml \
--output .artifacts/backfill-inventory.json

# Without a seed: every release-notes product is reported source-unresolved,
# useful to see the full census surface before mapping begins
docs-builder changelog backfill inventory --output .artifacts/backfill-inventory.json
```
12 changes: 12 additions & 0 deletions docs/cli/changelog/backfill/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
## Description

Backfill historical release-note bundles ([docs-eng-team#656](https://github.com/elastic/docs-eng-team/issues/656)).

The changelog/bundle pipeline only contains data produced since each repository adopted the live workflows. The backfill commands make the public data look as if the current system had been in use throughout the docs-builder era: they census the products with published release notes, plan exactly which resolved bundles to create, and publish them with create-only writes.

The pipeline is staged, and every stage exchanges versioned, content-addressed JSON documents (see `src/services/Elastic.Changelog/Backfill/README.md`):

1. `inventory` — the census: which products and release-note sources exist and what was decided about each.
2. Planning, materialization, apply, and verification stages follow as they are implemented.

No stage writes to S3 except the guarded apply stage; everything before it is read-only and reviewable.
Loading
Loading