diff --git a/.github/workflows/sync-plugins.yml b/.github/workflows/sync-plugins.yml index e3ea1e058a..76b5bce0c1 100644 --- a/.github/workflows/sync-plugins.yml +++ b/.github/workflows/sync-plugins.yml @@ -119,7 +119,7 @@ jobs: if [ "$NEEDS_ATTENTION" = "true" ]; then cat <<'WARNING' > [!WARNING] - > One or more plugins were skipped, newly scaffolded, or no longer + > One or more plugins were skipped, pruned, newly scaffolded, or no longer > appear in the upstream registry index. Check the Status column > before merging. @@ -150,7 +150,9 @@ jobs: - `updated` / `unchanged` — the generated region of the shared page. - `scaffolded` — a new product stub. Review its title, menu name, and tags; the generator cannot derive editorial tags from the registry. - - `skipped` — no README the transform could read. Fix it upstream. + - `skipped` — the upstream checkout or a source README was unavailable. + - `pruned` — a plugin in the registry has no source README; its shared + page and product stubs were removed. Check for an upstream rename. - `removed` — a shared page whose plugin left the registry index. Resolve by hand: a rename and a delete are indistinguishable here, so the sync never deletes a published page. @@ -164,7 +166,8 @@ jobs: regenerated every run; anything outside those markers is hand-owned and preserved. Product stubs under `content/influxdb3/{core,enterprise}/plugins/library/official/` are - created once and never rewritten. + created once and never rewritten. A missing source README causes the + sync to remove the shared page and both stubs. LEGEND } > "$BODY_FILE" echo "path=$BODY_FILE" >> "$GITHUB_OUTPUT" diff --git a/helper-scripts/influxdb3-plugins/README.md b/helper-scripts/influxdb3-plugins/README.md index ae33862fd9..f800a4eb7a 100644 --- a/helper-scripts/influxdb3-plugins/README.md +++ b/helper-scripts/influxdb3-plugins/README.md @@ -5,8 +5,9 @@ This directory holds the generator that turns official InfluxDB 3 plugins in into documentation in docs-v2. The sync runs from `.github/workflows/sync-plugins.yml` and opens one aggregate -pull request per run. It never pushes to a documentation branch directly, and -it never deletes a published page. +pull request per run. It never pushes to a documentation branch directly. +When a registry plugin loses its source README, the sync prunes its published +pages for review in that pull request. Architecture decisions are recorded in [docs/adr/0004-plugin-sync-ownership-seam.md](../../docs/adr/0004-plugin-sync-ownership-seam.md). @@ -55,12 +56,12 @@ Documentation uses the following, mapped from the registry index. Ownership splits three ways by file. This is the seam that lets a generator and a human writer share the same plugin page. -| Artifact | Owner | Rewritten | -| ------------------------------------------------------------- | -------- | ----------------------------- | -| `data/influxdb3_plugins.yml` | The sync | Fully, every run | -| Generated region of a shared page | The sync | Every run | -| Everything outside that region | A human | Never | -| Product stubs under `content/influxdb3/{core,enterprise}/...` | A human | Created once, never rewritten | +| Artifact | Owner | Rewritten | +| ------------------------------------------------------------- | -------- | ---------------------------------------------------- | +| `data/influxdb3_plugins.yml` | The sync | Fully, every run | +| Generated region of a shared page | The sync | Every run | +| Everything outside that region | A human | Never | +| Product stubs under `content/influxdb3/{core,enterprise}/...` | A human | Created once; pruned if the source README disappears | A shared page marks its generated region with HTML comments: @@ -93,19 +94,22 @@ is the worked example. list; there is no hand-maintained roster of plugins to keep current. A plugin merged upstream but not yet published to the registry does not appear until it is published. -2. Write `data/influxdb3_plugins.yml` from the registry entries. Output is +2. Check each discovered plugin for a source README. When the upstream checkout + is present, omit plugins without one from generated data and prune their + shared page and product stubs. When the checkout is absent, skip pruning. +3. Write `data/influxdb3_plugins.yml` from the available registry entries. Output is deterministic, so an unchanged registry produces a byte-identical file and no pull request. -3. Scaffold Core and Enterprise stubs for any plugin that lacks them. An +4. Scaffold Core and Enterprise stubs for any available plugin that lacks them. An existing stub is never opened for writing. -4. Transform the README of every discovered plugin and merge it into the +5. Transform the README of every available plugin and merge it into the generated region of its shared page. `docs_mapping.yaml` supplies only exceptions to the standard upstream README and shared-page paths. -5. Report one row per plugin to the step summary, and set the +6. Report one row per plugin to the step summary, and set the `needs_attention` output. -Only step 4 depends on `docs_mapping.yaml`. Steps 1 through 3 cover every -official plugin in the registry. +The README check and transform use `docs_mapping.yaml` for nonstandard paths. +Discovery still covers every official plugin in the registry. ## Statuses @@ -118,11 +122,12 @@ run fails and whether the pull request body carries a warning. | `updated` | The generated region was rewritten. | No | No | | `scaffolded` | A new product stub was created. | No | Yes | | `skipped` | No README the transform could read, or a registry fetch failure. | No | Yes | +| `pruned` | A registry plugin lost its README; published pages were removed. | No | Yes | | `removed` | A shared page whose plugin is no longer in the registry. | No | Yes | | `error` | A write failed, or a generated region was malformed. | Yes | Yes | The split between `skipped` and `error` is deliberate. A flaky network or a -malformed upstream README must not fail a scheduled run, because a red nightly +missing upstream README must not fail a scheduled run, because a red nightly that nobody can fix locally gets ignored. A failed write must fail the run, because a sync that reports success while publishing nothing is the failure mode this pipeline was rebuilt to end. @@ -172,6 +177,7 @@ git clone --depth 1 https://github.com/influxdata/influxdb3_plugins.git \ Without that checkout, discovery, the data file, and stub scaffolding still work; every mapped plugin reports `skipped` because its README is missing. +The sync does not prune pages when the entire checkout is absent. | Command | Effect | | ----------------------------- | -------------------------------------------------- | @@ -183,11 +189,13 @@ work; every mapped plugin reports `skipped` because its README is missing. ## Coverage -`yarn verify-plugin-coverage` reconciles the official plugins in the registry -against what docs-v2 actually publishes, on four axes: a `data/influxdb3_plugins.yml` -entry, a shared page, a Core stub, and an Enterprise stub. It names the missing -plugins per axis rather than printing one total, because a plugin can have a -shared page and no Enterprise stub. +`yarn verify-plugin-coverage` reconciles official plugins with available +READMEs against what docs-v2 publishes. It checks four artifacts: a +`data/influxdb3_plugins.yml` entry, a shared page, a Core stub, and an +Enterprise stub. It names the missing plugins per artifact rather than printing +one total, because a plugin can have a shared page and no Enterprise stub. +If the upstream checkout is absent or incomplete, the check skips measurement +instead of treating every plugin as missing. `coverage-baseline.json` records the gap the repository has accepted. The check fails when the gap grows past that baseline and names what grew; a gap that diff --git a/helper-scripts/influxdb3-plugins/coverage.js b/helper-scripts/influxdb3-plugins/coverage.js index a60b105619..7e19128c73 100644 --- a/helper-scripts/influxdb3-plugins/coverage.js +++ b/helper-scripts/influxdb3-plugins/coverage.js @@ -21,6 +21,7 @@ const AXES = ['data', 'shared', 'core', 'enterprise']; export function computeCoverage({ plugins, excluded = [], + unavailable = [], dataFileIds, sharedPages, coreStubs, @@ -37,6 +38,7 @@ export function computeCoverage({ return { total: plugins.length, excluded, + unavailable, axes: { data: axis(dataFileIds, 'stubSlug'), shared: axis(sharedPages, 'slug'), @@ -107,7 +109,12 @@ export function formatCoverageTable(coverage) { ].join('\n'); const excluded = coverage.excluded ?? []; - if (excluded.length === 0) return table; - - return `${table}\n\nExcluded by \`docs_mapping.yaml\`: ${excluded.join(', ')}`; + const notes = []; + if (excluded.length > 0) { + notes.push(`Excluded by \`docs_mapping.yaml\`: ${excluded.join(', ')}`); + } + if (coverage.unavailable?.length > 0) { + notes.push(`Source README missing: ${coverage.unavailable.join(', ')}`); + } + return notes.length ? `${table}\n\n${notes.join('\n\n')}` : table; } diff --git a/helper-scripts/influxdb3-plugins/port_to_docs.js b/helper-scripts/influxdb3-plugins/port_to_docs.js index 3a84475f69..7ddf805641 100644 --- a/helper-scripts/influxdb3-plugins/port_to_docs.js +++ b/helper-scripts/influxdb3-plugins/port_to_docs.js @@ -428,7 +428,8 @@ async function scaffoldMissingStubs(discoveredPlugins, dryRun = false) { * would leave only the last plugin's outcome visible to the workflow. */ function selectPlugins(configPlugins, pluginArg) { - const normalized = typeof pluginArg === 'string' ? pluginArg.trim() : pluginArg; + const normalized = + typeof pluginArg === 'string' ? pluginArg.trim() : pluginArg; const entries = Object.entries(configPlugins); if (!normalized || normalized === 'all') { @@ -496,6 +497,61 @@ async function findRemovedPlugins(discoveredPlugins) { return detectRemovedPlugins(discoveredPlugins, filenames); } +/** + * A full sync can prune missing READMEs only when the upstream checkout exists. + * Otherwise every source would appear missing on a local run without `.ext`. + */ +async function partitionPluginsByReadme( + discoveredPlugins, + configPlugins, + upstreamDir = UPSTREAM_OFFICIAL_DIR +) { + try { + await fs.access(upstreamDir); + } catch (error) { + if (error.code === 'ENOENT') return null; + throw error; + } + + const available = []; + const missing = []; + for (const plugin of discoveredPlugins) { + const mapping = mappingForDiscoveredPlugin(plugin, configPlugins); + try { + const source = await fs.stat(mapping.source); + (source.isFile() ? available : missing).push(plugin); + } catch (error) { + if (error.code !== 'ENOENT') throw error; + missing.push(plugin); + } + } + // An empty sparse checkout looks like every README disappeared. Preserve + // published pages until at least one plugin source confirms the checkout. + if (discoveredPlugins.length > 0 && available.length === 0) return null; + return { available, missing }; +} + +/** Remove only the known shared page and product stubs for one plugin. */ +async function prunePlugin(pluginName, paths, dryRun = false) { + const removed = []; + for (const targetPath of paths) { + try { + await fs.stat(targetPath); + if (!dryRun) await fs.unlink(targetPath); + removed.push(targetPath); + } catch (error) { + if (error.code !== 'ENOENT') throw error; + } + } + return { + plugin: pluginName, + status: removed.length ? 'pruned' : 'skipped', + detail: removed.length + ? `${dryRun ? 'would remove' : 'removed'} ${removed.join(', ')}` + : 'source README missing; no published pages to remove', + }; +} + /** * Process a single plugin README. * @@ -704,6 +760,8 @@ async function main() { // entry. `main` collapses them to one row per plugin before reporting. const artifactResults = []; let discovered = null; + let pluginsWithReadmes = null; + let pluginsWithoutReadmes = []; if (shouldRunDiscovery(options.plugin)) { console.log('Discovering official plugins from the registry index...'); @@ -717,6 +775,12 @@ async function main() { exclude: config.exclude ?? [], }); discovered = parsed.plugins; + const readmes = await partitionPluginsByReadme( + discovered, + config.plugins + ); + pluginsWithReadmes = readmes?.available ?? discovered; + pluginsWithoutReadmes = readmes?.missing ?? []; console.log( `Discovered ${discovered.length} official plugin(s) in the registry.` @@ -745,7 +809,7 @@ async function main() { // the sync reported success while publishing nothing, which is the // failure this pipeline is being rebuilt to stop having. try { - const dataYaml = renderPluginDataYaml(discovered.map(mapEntry)); + const dataYaml = renderPluginDataYaml(pluginsWithReadmes.map(mapEntry)); const dataFilePath = '../../data/influxdb3_plugins.yml'; if (options.dryRun) { console.log(`DRY RUN: would write ${dataFilePath}`); @@ -755,7 +819,7 @@ async function main() { } const scaffoldResults = await scaffoldMissingStubs( - discovered, + pluginsWithReadmes, options.dryRun ); console.log( @@ -779,6 +843,29 @@ async function main() { detail: 'shared page has no plugin in the registry index', })) ); + + for (const plugin of pluginsWithoutReadmes) { + const mapping = mappingForDiscoveredPlugin(plugin, config.plugins); + try { + artifactResults.push( + await prunePlugin( + plugin.name, + [ + mapping.target, + stubPath(plugin, 'core'), + stubPath(plugin, 'enterprise'), + ], + options.dryRun + ) + ); + } catch (error) { + artifactResults.push({ + plugin: plugin.name, + status: 'error', + detail: `could not prune plugin pages: ${error.message}`, + }); + } + } } console.log(''); } @@ -798,7 +885,7 @@ async function main() { } const pluginsToProcess = discovered - ? discovered.map((plugin) => [ + ? pluginsWithReadmes.map((plugin) => [ plugin.name, mappingForDiscoveredPlugin(plugin, config.plugins), ]) @@ -853,4 +940,6 @@ export { selectPlugins, shouldRunDiscovery, mappingForDiscoveredPlugin, + partitionPluginsByReadme, + prunePlugin, }; diff --git a/helper-scripts/influxdb3-plugins/reporting.js b/helper-scripts/influxdb3-plugins/reporting.js index d8683829fa..d9660b3cd3 100644 --- a/helper-scripts/influxdb3-plugins/reporting.js +++ b/helper-scripts/influxdb3-plugins/reporting.js @@ -14,13 +14,14 @@ import { randomBytes } from 'node:crypto'; // alternative is a green build that silently stopped syncing -- the failure // mode this pipeline already had for eight months. // -// `skipped`, `scaffolded`, and `removed` are all "a human should look at this +// `skipped`, `scaffolded`, `pruned`, and `removed` are all "a human should look at this // pull request", not "the sync is broken". A plugin published without a README // the transform can read, a plugin that just gained its first stub, and a // plugin that vanished upstream are each resolved by review, not by a red X. const FATAL_STATUSES = new Set(['error']); const ATTENTION_STATUSES = new Set([ 'scaffolded', + 'pruned', 'skipped', 'removed', 'error', @@ -32,6 +33,7 @@ const ATTENTION_STATUSES = new Set([ const STATUS_SEVERITY = [ 'error', 'removed', + 'pruned', 'skipped', 'scaffolded', 'updated', diff --git a/helper-scripts/influxdb3-plugins/test/coverage.test.js b/helper-scripts/influxdb3-plugins/test/coverage.test.js index 2624bb0e9c..4d72c9248c 100644 --- a/helper-scripts/influxdb3-plugins/test/coverage.test.js +++ b/helper-scripts/influxdb3-plugins/test/coverage.test.js @@ -29,6 +29,24 @@ test('names excluded plugins instead of counting them as a gap', () => { assert.equal(coverage.axes.data.missing.length, 0); }); +test('names plugins without a source README outside coverage totals', () => { + const coverage = computeCoverage({ + plugins: [PLUGINS[0]], + unavailable: ['nori_regression'], + dataFileIds: ['notifier'], + sharedPages: ['notifier'], + coreStubs: ['notifier'], + enterpriseStubs: ['notifier'], + }); + + assert.equal(coverage.total, 1); + assert.deepEqual(coverage.axes.shared.missing, []); + assert.match( + formatCoverageTable(coverage), + /Source README missing: nori_regression/ + ); +}); + test('names what is missing on each axis, not just a total', () => { const coverage = computeCoverage({ plugins: PLUGINS, diff --git a/helper-scripts/influxdb3-plugins/test/reporting.test.js b/helper-scripts/influxdb3-plugins/test/reporting.test.js index 613024fc1a..1e80ed8ffd 100644 --- a/helper-scripts/influxdb3-plugins/test/reporting.test.js +++ b/helper-scripts/influxdb3-plugins/test/reporting.test.js @@ -70,7 +70,7 @@ test('a routine run needs no attention and is not fatal', () => { }); test('a new stub, a skip, or a removal needs attention but does not fail', () => { - for (const status of ['scaffolded', 'skipped', 'removed']) { + for (const status of ['scaffolded', 'skipped', 'pruned', 'removed']) { const results = [row('unchanged'), row(status)]; assert.equal(needsAttention(results), true, `${status} needs attention`); diff --git a/helper-scripts/influxdb3-plugins/test/sync-results.test.js b/helper-scripts/influxdb3-plugins/test/sync-results.test.js index d3830c6b11..1b0630514f 100644 --- a/helper-scripts/influxdb3-plugins/test/sync-results.test.js +++ b/helper-scripts/influxdb3-plugins/test/sync-results.test.js @@ -1,6 +1,12 @@ import { test } from 'node:test'; import assert from 'node:assert/strict'; -import { mkdtempSync, readFileSync, writeFileSync, existsSync } from 'node:fs'; +import { + mkdtempSync, + mkdirSync, + readFileSync, + writeFileSync, + existsSync, +} from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { @@ -8,6 +14,8 @@ import { selectPlugins, shouldRunDiscovery, mappingForDiscoveredPlugin, + partitionPluginsByReadme, + prunePlugin, } from '../port_to_docs.js'; const CONFIG_PLUGINS = { @@ -90,6 +98,54 @@ test('uses an explicit mapping when a plugin needs nonstandard paths', () => { assert.equal(mapping, customMapping); }); +test('separates missing plugin READMEs when the upstream checkout exists', async () => { + const dir = mkdtempSync(join(tmpdir(), 'plugin-sources-')); + const present = join(dir, 'present.md'); + writeFileSync(present, README, 'utf8'); + const plugins = [{ name: 'present' }, { name: 'missing' }]; + const mappings = { + present: { source: present }, + missing: { source: join(dir, 'missing.md') }, + }; + + assert.deepEqual(await partitionPluginsByReadme(plugins, mappings, dir), { + available: [plugins[0]], + missing: [plugins[1]], + }); + assert.equal( + await partitionPluginsByReadme(plugins, mappings, join(dir, 'absent')), + null, + 'an absent checkout must not classify every plugin as missing' + ); + const emptyCheckout = join(dir, 'empty-checkout'); + mkdirSync(emptyCheckout); + assert.equal( + await partitionPluginsByReadme([plugins[1]], mappings, emptyCheckout), + null, + 'an incomplete checkout must not trigger mass pruning' + ); +}); + +test("prunes only a missing plugin's known pages and previews a dry run", async () => { + const dir = mkdtempSync(join(tmpdir(), 'plugin-prune-')); + const paths = ['shared.md', 'core.md', 'enterprise.md'].map((name) => + join(dir, name) + ); + for (const file of paths) writeFileSync(file, 'old docs', 'utf8'); + const unrelated = join(dir, 'another-plugin.md'); + writeFileSync(unrelated, 'keep', 'utf8'); + + const preview = await prunePlugin('nori_regression', paths, true); + assert.equal(preview.status, 'pruned'); + assert.match(preview.detail, /would remove/); + assert.equal(paths.every(existsSync), true); + + const result = await prunePlugin('nori_regression', paths); + assert.equal(result.status, 'pruned'); + assert.equal(paths.some(existsSync), false); + assert.equal(readFileSync(unrelated, 'utf8'), 'keep'); +}); + test('ignores empty entries from a trailing or doubled comma', () => { const { selected, unknown } = selectPlugins(CONFIG_PLUGINS, 'notifier,,'); diff --git a/helper-scripts/influxdb3-plugins/verify-coverage.js b/helper-scripts/influxdb3-plugins/verify-coverage.js index bcbb3a12bb..ab77d9d8e5 100644 --- a/helper-scripts/influxdb3-plugins/verify-coverage.js +++ b/helper-scripts/influxdb3-plugins/verify-coverage.js @@ -19,6 +19,7 @@ import { formatCoverageTable, } from './coverage.js'; import { writeStepOutputs, writeStepSummary } from './reporting.js'; +import { partitionPluginsByReadme } from './port_to_docs.js'; const MAPPING_CONFIG = 'docs_mapping.yaml'; const BASELINE_FILE = 'coverage-baseline.json'; @@ -61,10 +62,20 @@ async function main() { } const dataFile = yaml.load(await fs.readFile(DATA_FILE, 'utf8')) ?? []; + const readmes = await partitionPluginsByReadme( + parsed.plugins, + config.plugins + ); + if (!readmes) { + console.warn('⚠️ Upstream plugin READMEs are unavailable.'); + console.warn(' Coverage was not measured on this run.'); + process.exit(0); + } const coverage = computeCoverage({ - plugins: parsed.plugins, + plugins: readmes.available, excluded: parsed.excluded, + unavailable: readmes.missing.map((plugin) => plugin.name), dataFileIds: dataFile.map((entry) => entry.id), sharedPages: await pageNames(SHARED_DIR), coreStubs: await pageNames(stubDir('core')),