feat: documentation build-test proof of concept (dasharo-issues#1153)

scripts/ci/doc-build-test/README.md

@@ -0,0 +1,183 @@
+<!--
+SPDX-FileCopyrightText: 2026 Amey Pawar <ameyap007aaa@gmail.com>
+
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Documentation build test
+
+A proof-of-concept harness that reads a Dasharo build manual from
+`docs.dasharo.com` and resolves the exact commands a reader would run to build
+one firmware variant, so a resulting binary can be compared against a published
+release.
+
+It is a first step towards
+[dasharo-issues#1153](https://github.com/Dasharo/dasharo-issues/issues/1153):
+_"Create automatic tests of Dasharo build documentation"_. The goal of that
+issue is to verify that following the documentation - as a person would,
+not as a hand-maintained CI script would - reproduces every historic release
+for every supported device.
+
+## The problem: build manuals are decision trees
+
+A `building-manual.md` page is authored for MkDocs Material. It is not a
+linear script; it is a decision tree built out of `pymdownx.tabbed` content
+tabs, admonitions and fenced code blocks. A single page routinely encodes
+several mutually exclusive branches at once:
+
+```text
+=== "Dasharo (UEFI)"
+    === "PRO Z690-A (WIFI) DDR4"
+        ./build.sh z690a_ddr4
+    === "PRO Z690-A (WIFI)"
+        ./build.sh z690a_ddr5
+=== "Dasharo (coreboot + Heads)"
+    ...
+```
+
+Off-the-shelf "runnable docs" tools such as `codedown`, `doc-detective` and
+`tuttest` extract _every_ fenced block on the page and run them in order. On
+the page above that means running the DDR4 build, the DDR5 build and the Heads
+build back to back - incompatible branches concatenated into one broken
+script. Branch concatenation is one failure mode. A separate one the
+maintainers hit when trying doc-detective - its container lacked host
+dependencies the docs assume, such as `sudo` - is about capturing what a fresh
+OS needs, and is out of scope here (see Scope below).
+
+This harness instead resolves a _single path_ through the tree. Tab
+resolution follows MkDocs' own `content.tabs.link` semantics: tabs that share
+a title are the same choice, so a device selected once applies to every tab
+group that offers it.
+
+## What the harness does
+
+The parser (`mkdocs_build_extractor.py`) turns a manual into build recipes.
+The CLI (`doc_build_test.py`) exposes five subcommands:
+
+- `list` - enumerate every build target (leaf path) the page describes.
+- `extract` - print the ordered commands, expected artifact and caveats for
+  one selection, with version placeholders substituted.
+- `script` - emit a standalone, fail-fast shell script for one selection.
+- `verify` - compare a locally built binary against a published release.
+- `diagnose` - report documentation issues that block automated testing.
+
+The parser and its tests use only the Python standard library, so they run in
+CI with no extra dependencies and never touch the network.
+
+## Reproducibility check: sha256, then romscope
+
+The issue asks whether a build is "identical to the ones we publish". A naive
+`sha256` equality check is not enough, and using it alone would report a
+failure on almost every real build. Dasharo release binaries are signed with
+the 3mdeb Vboot key while a local build is not, so the `VBLOCK` and `GBB`
+regions legitimately differ. This is documented in the
+[reproducible build verification guide](https://docs.dasharo.com/guides/reproducible-build-verification/).
+
+`verify` returns `IDENTICAL` (sha256 match) or `DIFFERS`. A `DIFFERS` result is
+_not_ by itself a failure: a legitimately reproducible Dasharo build is not
+byte-identical to the release, because the release is Vboot-signed and both
+carry version strings and build metadata a local build will not match. Deciding
+whether a `DIFFERS` result is functionally reproducible needs
+[romscope](https://github.com/Dasharo/romscope) `compare` and a human reading
+of its report (string / compression / program-data differences). `verify`
+therefore surfaces romscope's raw output for a person to interpret rather than
+inventing a pass/fail verdict from it. The romscope call is an injected runner,
+so the logic stays unit-testable without romscope or Docker present.
+
+## What it reveals in the current docs
+
+Run against the live MSI building manual, the harness already surfaces three
+classes of documentation issue - exactly the kind of human-error faults the
+issue is about - without any change to the docs:
+
+- _Version-conditional prose._ The UEFI build tab hides a
+  "For v1.1.1 and older / For v1.1.2 and newer" choice in prose rather than in
+  a tab, so the resolved recipe contains two mutually exclusive `build.sh`
+  commands. A machine cannot pick one without parsing the prose.
+- _Unlinkable tabs._ In the Heads branch the checkout step labels a device
+  `PRO Z690-A` while the build step labels it `PRO Z690-A (WIFI) DDR4`.
+  Because the labels differ, the two tab groups cannot be linked, and
+  enumeration produces device combinations that make no sense.
+- _Unresolvable choices._ Selecting a firmware type but omitting a required
+  device choice is reported as an ambiguous path, listing the options that
+  still need a decision.
+
+The `diagnose` subcommand reports these for a page and exits non-zero when any
+are found, so it can gate CI. The committed self-tests run against small
+inlined fixtures; the eight findings above were observed by running `diagnose`
+against the live `unified/msi/building-manual.md`, not asserted in the suite.
+
+## Usage
+
+```bash
+# List every build target in a manual
+./doc_build_test.py list building-manual.md
+
+# Resolve one build and print its commands
+./doc_build_test.py extract building-manual.md \
+    --select "Dasharo (UEFI)" \
+    --select "PRO Z690-A (WIFI) DDR4" \
+    --version 1.1.3 --revision msi_ms7d25_v1.1.3
+
+# Emit a runnable build script for one selection
+./doc_build_test.py script building-manual.md \
+    --select "Dasharo (UEFI)" --select "PRO Z690-A (WIFI) DDR4" \
+    --version 1.1.3 --revision msi_ms7d25_v1.1.3 -o build.sh
+
+# Compare a locally built binary against a published release
+./doc_build_test.py verify --built out.rom --published release.rom \
+    --romscope ./romscope
+
+# Report documentation issues that block automated testing
+./doc_build_test.py diagnose building-manual.md
+```
+
+## Scope and limitations
+
+This is a proof of concept focused on the deterministic, testable core:
+turning a tabbed manual into a single correct build recipe and deciding a
+reproducibility verdict. Deliberately out of scope for now:
+
+- _Running the build._ Building firmware needs Docker and takes minutes per
+  target; the issue itself flags CI time as a concern. `script` produces a
+  runnable recipe, but actually executing it and downloading release binaries
+  is left to a follow-up, run outside pull-request CI.
+- _Release discovery._ Mapping a device to its published releases and hashes
+  (from the per-device `releases.md` pages) is a follow-up.
+- _Fresh-OS dependency capture._ Verifying the documented steps on a clean OS
+  with nothing missing - the `sudo`/toolchain gap the maintainers hit - is the
+  harder half of the issue and is not attempted here.
+- _Parsing assumptions._ Only fences tagged with a shell language are treated
+  as commands; heredocs and unusual tab-label conventions are not handled.
+  Placeholder substitution covers the common `X.Y.Z`, `VERSION` and `REVISION`
+  tokens only.
+
+## Relation to existing work
+
+- Test specifications `[BNO] Build on a fresh OS Installation` and
+  `[FLB] Firmware locally building and flashing` describe the manual
+  procedure this harness is meant to automate.
+- This "parse the docs, then hash-compare" approach follows the plan macpijan
+  sketched in
+  [osfv#545](https://github.com/Dasharo/open-source-firmware-validation/pull/545).
+  An alternative discussed in that thread is to standardise the build (a
+  universal script, as trialled in
+  [coreboot#579](https://github.com/Dasharo/coreboot/pull/579), or
+  Jinja-generated docs) so that parsing becomes unnecessary; this PoC does not
+  preclude that direction.
+- [docs#1240](https://github.com/Dasharo/docs/pull/1240) is a parallel,
+  non-executing build-docs command checker in the docs repo. This harness is
+  complementary - single-path resolution, build-script generation and a
+  reproducibility comparison - and the two should be reconciled rather than
+  duplicated.
+
+## Running the self-tests
+
+```bash
+python3 -m unittest discover -s scripts/ci/doc-build-test \
+    -t scripts/ci/doc-build-test -p "*_selftests.py"
+```
+
+The suite uses inlined markdown fixtures that reproduce the real nesting of
+the Dasharo manuals, so it needs no network access and no repository
+checkout of the docs.

scripts/ci/doc-build-test/doc_build_test.py

@@ -0,0 +1,172 @@
+#!/usr/bin/env python
+
+# SPDX-FileCopyrightText: 2026 Amey Pawar <ameyap007aaa@gmail.com>
+#
+# SPDX-License-Identifier: Apache-2.0
+
+"""Command-line front end for the Dasharo documentation build test.
+
+Turns a MkDocs building manual into concrete, single-path build recipes and
+verifies a locally built binary against a published release.  See the module
+docstring in ``mkdocs_build_extractor`` and the directory ``README.md`` for the
+design rationale.
+
+Subcommands
+-----------
+list     Enumerate every build target (leaf path) a document describes.
+extract  Print the resolved commands, artifacts and caveats for one selection.
+script   Emit a standalone, fail-fast shell script for one selection.
+verify   Compare a built binary against a published release (sha256 + romscope).
+"""
+
+import argparse
+import subprocess
+import sys
+
+import mkdocs_build_extractor as ext
+
+
+def _read(path: str) -> str:
+    with open(path, "r", encoding="utf-8") as handle:
+        return handle.read()
+
+
+def _cmd_list(args: argparse.Namespace) -> int:
+    targets = ext.list_targets(_read(args.doc))
+    if not targets:
+        print("no build targets found", file=sys.stderr)
+        return 1
+    for target in targets:
+        print(" / ".join(target.selections) or "(single path)")
+        if target.artifacts:
+            print("    artifacts: " + ", ".join(target.artifacts))
+    print(f"\n{len(targets)} target(s)")
+    return 0
+
+
+def _resolve(args: argparse.Namespace) -> ext.Recipe:
+    return ext.resolve(
+        _read(args.doc),
+        select=args.select or [],
+        version=args.version,
+        revision=args.revision,
+    )
+
+
+def _cmd_extract(args: argparse.Namespace) -> int:
+    try:
+        recipe = _resolve(args)
+    except ext.AmbiguousSelection as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return 2
+    print("# selection: " + " / ".join(recipe.selections))
+    if recipe.caveats:
+        print("# caveats: " + ", ".join(recipe.caveats))
+    if recipe.artifacts:
+        print("# artifacts: " + ", ".join(recipe.artifacts))
+    print()
+    for command in recipe.commands:
+        print(command)
+    return 0
+
+
+def _cmd_script(args: argparse.Namespace) -> int:
+    try:
+        recipe = _resolve(args)
+    except ext.AmbiguousSelection as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return 2
+    script = ext.to_script(recipe)
+    if args.output:
+        with open(args.output, "w", encoding="utf-8") as handle:
+            handle.write(script)
+    else:
+        sys.stdout.write(script)
+    return 0
+
+
+def _cmd_verify(args: argparse.Namespace) -> int:
+    runner = None
+    if args.romscope:
+
+        def runner(published: str, built: str) -> str:
+            result = subprocess.run(
+                [args.romscope, "compare", published, built],
+                capture_output=True,
+                text=True,
+                check=False,
+            )
+            return result.stdout + result.stderr
+
+    result = ext.verify(args.built, args.published, romscope_runner=runner)
+    print(result.verdict)
+    if result.romscope_report:
+        print("\nromscope report (interpret per romscope's 'Interpreting results'):")
+        print(result.romscope_report)
+    return 0 if result.verdict == ext.IDENTICAL else 1
+
+
+def _cmd_diagnose(args: argparse.Namespace) -> int:
+    diagnostics = ext.diagnose(_read(args.doc))
+    if not diagnostics:
+        print("no documentation issues detected")
+        return 0
+    for diagnostic in diagnostics:
+        print(f"{diagnostic.kind}: {diagnostic.detail}")
+    print(f"\n{len(diagnostics)} issue(s)")
+    return 1
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(prog="doc_build_test", description=__doc__)
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    p_list = sub.add_parser("list", help="enumerate build targets")
+    p_list.add_argument("doc", help="path to a building-manual.md")
+    p_list.set_defaults(func=_cmd_list)
+
+    def add_select(p: argparse.ArgumentParser) -> None:
+        p.add_argument("doc", help="path to a building-manual.md")
+        p.add_argument(
+            "--select",
+            action="append",
+            metavar="LABEL",
+            help="tab label to choose (repeatable)",
+        )
+        p.add_argument("--version", help="value for X.Y.Z / VERSION placeholders")
+        p.add_argument("--revision", help="value for the REVISION placeholder")
+
+    p_extract = sub.add_parser("extract", help="print a resolved recipe")
+    add_select(p_extract)
+    p_extract.set_defaults(func=_cmd_extract)
+
+    p_script = sub.add_parser("script", help="emit a runnable build script")
+    add_select(p_script)
+    p_script.add_argument("-o", "--output", help="write script to this file")
+    p_script.set_defaults(func=_cmd_script)
+
+    p_verify = sub.add_parser("verify", help="compare built vs published binary")
+    p_verify.add_argument("--built", required=True, help="locally built .rom")
+    p_verify.add_argument("--published", required=True, help="published .rom")
+    p_verify.add_argument(
+        "--romscope",
+        help="path to a romscope binary for signature-aware comparison",
+    )
+    p_verify.set_defaults(func=_cmd_verify)
+
+    p_diagnose = sub.add_parser(
+        "diagnose", help="report documentation issues that block automated testing"
+    )
+    p_diagnose.add_argument("doc", help="path to a building-manual.md")
+    p_diagnose.set_defaults(func=_cmd_diagnose)
+    return parser
+
+
+def main(argv=None) -> int:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())