build: byte-reproducible releases, enforced in CI by zvonkok · Pull Request #170 · NVIDIA/nvrc

zvonkok · 2026-07-06T19:55:01Z

Codegen policy now lives in a single place: .cargo/config.toml keeps only target mechanics (linker, +crt-static) while [profile.release] owns the codegen flags. Previously the two fought each other, with rustflags silently overriding the profile. As a side effect, cargo test --target *-musl builds again.

The build inputs are pinned. rust-toolchain.toml fixes the compiler at 1.94.0, which is what "stable minus 2 releases" resolves to today, so nothing changes behaviorally. --locked makes Cargo.lock an enforced input instead of a suggestion.

Release bytes no longer depend on where the build ran. scripts/build-release.sh remaps $CARGO_HOME and the rust-src sysroot paths out of the binary; workspace paths were verified not to leak. VERIFY.md gains a "Reproduce the build" section with the exact procedure.

A new workflow enforces all of this on every PR: it builds the tree twice from different directories with different CARGO_HOMEs and fails unless the hashes are identical. It calls the same script the release workflow uses, so the check cannot drift from what ships.

Verified end-to-end: two builds from different locations under the pinned toolchain produce identical sha256, and the binary remains a fully static PIE.

Copilot

Pull request overview

This PR hardens NVRC’s release build process to be byte-reproducible and enforces that property in CI, aligning the shipped NVRC boot-line sha256 with verifiable public-source rebuilds.

Changes:

Moves release codegen policy into Cargo.toml’s [profile.release] and leaves .cargo/config.toml with only target mechanics (linker, +crt-static).
Pins the Rust toolchain via rust-toolchain.toml and enforces locked dependency inputs in builds.
Adds a reproducible release build script with path remapping, documents the rebuild procedure, and introduces a CI workflow that double-builds from different locations and compares hashes.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`VERIFY.md`	Documents how to reproduce a release build and compare sha256 digests.
`scripts/build-release.sh`	Adds a canonical release build entrypoint with remap-path-prefix for reproducible bytes.
`rust-toolchain.toml`	Pins the Rust compiler/toolchain and targets as explicit build inputs.
`Cargo.toml`	Centralizes release codegen settings in `[profile.release]`.
`.cargo/config.toml`	Simplifies target config to linker and `+crt-static` only.
`.github/workflows/reproducible.yaml`	New workflow that builds twice from different directories/CARGO_HOMEs and requires identical hashes.
`.github/workflows/release.yaml`	Uses the pinned toolchain and the shared release build script.
`.github/workflows/ci.yaml`	Uses the pinned toolchain and enforces `--locked` for release builds used in CI.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/lockdown.rs:75

panic::set_hook(saved_hook) only runs on the success path. If this test panics (e.g., assertion failure), the custom panic hook remains installed for the rest of the suite and can reintroduce the cross-test side effects this test is trying to prevent. Use a small Drop guard to restore the previous hook even during unwinding.

        let saved_hook = panic::take_hook();
        let called = Arc::new(AtomicBool::new(false));
        let called_clone = called.clone();

        // Install hook with test closure

Copilot

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 1 comment.

Rustflags in .cargo/config.toml override [profile.release] silently and apply to every profile: the manifest claimed opt-level "s" while musl builds shipped "z", and forced panic=abort broke cargo test for musl targets. The cargo config keeps target mechanics (linker, +crt-static); the release profile owns codegen. opt-level stays "z", codegen-units=1 keeps fat LTO effective. Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com> Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rustc embeds /rustc/<commit-hash> stdlib paths in the binary, so the compiler version is part of the release bytes. A floating "stable minus 2 releases" cannot reproduce a tag after the release calendar moves; 1.94.0 is what it resolves to today. rustup applies the pin, musl targets included, to every cargo invocation in the repo. Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com> Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

dtolnay/rust-toolchain only sets the rustup default, which rust-toolchain.toml outranks: stable-labeled jobs ran the pin via an implicit download and nightly tools broke against it. A composite action parses the pin, fails fast when unreadable, and installs what cargo will actually run; nightly-only tools (udeps, miri, fuzz) select their channel per invocation with cargo +nightly. All dtolnay refs are pinned by commit SHA like the other actions. Every job that resolves the workspace runs --locked. The miri job drops || true so a finding fails CI, capped at 30 minutes. Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com> Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Panic Location strings embed $CARGO_HOME/registry paths and, with the rust-src component installed, the sysroot source tree; both vary per machine and change the sha256, breaking the boot-line to Rekor correlation from NVIDIA#166. scripts/build-release.sh remaps both onto stable forms, rejects unparseable rustc commit hashes, unsets GIT_REV, owns SOURCE_DATE_EPOCH, and builds --locked. Workspace paths are compiled relative and do not leak. Both the release workflow and the reproducibility check call this script. VERIFY.md documents the rebuild procedure; README points at it. No apt-get update: the image index suffices and third-party repo breakage fails jobs. Verified: builds from different directories with different CARGO_HOMEs produce identical sha256; the binary remains static-PIE. Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com> Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Build the tree twice, varying the inputs a verifier cannot copy from CI (workspace directory, CARGO_HOME), and fail on hash mismatch with a strings diff. Runs the same script as the release workflow. Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com> Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

miri aborts at the first foreign operation it cannot interpret: process spawn and exec, fork, unix sockets, mount, O_NONBLOCK opens, raw syscall(SYS_getpid), the sudo re-exec in require_root, and hashing /proc/self/exe, which under miri is the interpreter binary. Each such test carries cfg_attr(miri, ignore) with its reason; the pure logic keeps running under full UB interpretation. #[serial] applies only outside miri: serial_test's lock map sits behind tagged pointers that defeat the leak checker, and miri runs single-threaded. Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com> Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The hook installer test leaked the process-global hook, so every later caught panic wrote /dev/kmsg and called sync(); the power_off variant outlived its test and powered off the machine on the next caught panic. Both hook tests now restore the previous hook, and a new test asserts a panic reaches the shutdown action, which was previously covered only by accidental firings of the leaked hook. power_off itself stays uncovered: executing it powers off the test machine. Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com> Assisted-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 1 comment.

    #[test]
+    #[cfg_attr(miri, ignore = "spawns a process, which miri cannot emulate")]
    fn test_ctk_fails_without_binary() {


manuelh-dev

LGTM!

Copilot AI review requested due to automatic review settings July 6, 2026 19:55

Copilot started reviewing on behalf of zvonkok July 6, 2026 19:55 View session

zvonkok added the ok-to-test Ok to test label Jul 6, 2026

Copilot AI reviewed Jul 6, 2026

View reviewed changes

Comment thread .github/workflows/reproducible.yaml Outdated

Comment thread .github/workflows/release.yaml Outdated

Comment thread .github/workflows/ci.yaml Outdated

manuelh-dev reviewed Jul 6, 2026

View reviewed changes

Comment thread .github/workflows/ci.yaml

github-actions Bot removed the ok-to-test Ok to test label Jul 6, 2026

Copilot AI review requested due to automatic review settings July 6, 2026 21:55

zvonkok force-pushed the reproducible-builds branch from 0204acd to 190dac6 Compare July 6, 2026 21:55

Copilot started reviewing on behalf of zvonkok July 6, 2026 21:57 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

Comment thread scripts/build-release.sh Outdated

zvonkok added the ok-to-test Ok to test label Jul 6, 2026

Copilot AI review requested due to automatic review settings July 6, 2026 22:49

github-actions Bot removed the ok-to-test Ok to test label Jul 6, 2026

Copilot started reviewing on behalf of zvonkok July 6, 2026 22:50 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

Comment thread .github/workflows/reproducible.yaml

Comment thread .github/workflows/reproducible.yaml Outdated

Comment thread .github/workflows/release.yaml Outdated

Comment thread .github/workflows/ci.yaml Outdated

Comment thread VERIFY.md

manuelh-dev previously approved these changes Jul 6, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 7, 2026 00:05

zvonkok dismissed manuelh-dev’s stale review via 2ca42f7 July 7, 2026 00:05

zvonkok force-pushed the reproducible-builds branch from 12382f3 to 2ca42f7 Compare July 7, 2026 00:05

Copilot started reviewing on behalf of zvonkok July 7, 2026 00:05 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread scripts/build-release.sh

Comment thread scripts/build-release.sh Outdated

Copilot AI review requested due to automatic review settings July 7, 2026 00:37

Copilot started reviewing on behalf of zvonkok July 7, 2026 00:37 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread .github/workflows/static-checks.yaml Outdated

Comment thread .github/workflows/static-checks.yaml Outdated

Comment thread .github/workflows/coverage.yaml Outdated

zvonkok added the ok-to-test Ok to test label Jul 7, 2026

Copilot AI review requested due to automatic review settings July 7, 2026 17:55

github-actions Bot removed the ok-to-test Ok to test label Jul 7, 2026

Copilot started reviewing on behalf of zvonkok July 7, 2026 17:56 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread .github/workflows/static-checks.yaml Outdated

Comment thread .github/workflows/static-checks.yaml Outdated

zvonkok added the ok-to-test Ok to test label Jul 7, 2026

Copilot AI review requested due to automatic review settings July 7, 2026 21:03

github-actions Bot removed the ok-to-test Ok to test label Jul 7, 2026

Copilot started reviewing on behalf of zvonkok July 7, 2026 21:04 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 7, 2026 22:07

Copilot started reviewing on behalf of zvonkok July 7, 2026 22:07 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread .github/workflows/static-checks.yaml Outdated

zvonkok added the ok-to-test Ok to test label Jul 7, 2026

zvonkok added 7 commits July 7, 2026 23:10

Copilot AI review requested due to automatic review settings July 7, 2026 23:10

zvonkok force-pushed the reproducible-builds branch from 9edf839 to bcf756e Compare July 7, 2026 23:10

github-actions Bot removed the ok-to-test Ok to test label Jul 7, 2026

Copilot started reviewing on behalf of zvonkok July 7, 2026 23:11 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread src/toolkit.rs

Comment on lines 31 to 33

#[test]

#[cfg_attr(miri, ignore = "spawns a process, which miri cannot emulate")]

fn test_ctk_fails_without_binary() {

manuelh-dev approved these changes Jul 7, 2026

View reviewed changes

Uh oh!

Conversation

zvonkok commented Jul 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

manuelh-dev left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants