Add configurable upstream cache filtering#30
Open
garbas wants to merge 5 commits into
Open
Conversation
Add support for checking multiple upstream Nix caches before pushing store paths to avoid redundant uploads. Paths that already exist in any configured upstream cache are automatically skipped. - Add 'upstream-caches' input parameter accepting comma or newline-separated cache URLs - Default to checking both cache.nixos.org and cache.flox.dev - Update push script to query each upstream cache before pushing paths - Only push paths that don't exist in any configured upstream cache This significantly reduces unnecessary uploads and bandwidth usage by avoiding duplication of paths already available in public caches.
Automatically add the configured substituter to the list of upstream caches to check. This prevents re-pushing paths that already exist in the target substituter, avoiding unnecessary uploads and potential errors. The base URL is extracted from CONFIGURE_NIX_SUBSTITUTER (removing query parameters) before checking.
Replace sequential per-path cache checking with parallel batch queries for dramatic performance improvement. Changes: - Batch all paths into a single nix path-info query per cache - Run cache queries in parallel using background jobs - Use comm for efficient set difference operation Performance impact (500 paths, 2 caches): - Before: 134 seconds (1000 sequential network calls) - After: 10 seconds (2 parallel batch calls) - Speedup: 13x faster This significantly reduces CI/CD pipeline time when pushing to custom substituters.
garbas
force-pushed
the
configure-upstream-caches
branch
from
December 24, 2025 09:41
b7420ab to
721abdc
Compare
Migrate the development environment from Nix flakes to Flox for better reproducibility and ease of use. Add comprehensive developer documentation and performance benchmarking tools. Added: - .flox/ directory with Flox environment manifest (provides Node.js 20) - .claude/CLAUDE.md with project architecture and development guidance - scripts/benchmark-cache-check.sh for performance testing (14x speedup verification) - .github/workflows/auto-label.yml for automatic PR labeling - .github/workflows/update.yml for Flox environment updates - Development setup documentation in README.md with Flox instructions Removed: - flake.nix and flake.lock (replaced by Flox) - .envrc (Flox handles environment activation) - .github/workflows/update-flake-lock.yml (no longer needed) Modified: - README.md: Added "Development" section with Flox setup, build commands, testing instructions, and performance benchmarking guide - package-lock.json: Updated dependencies The benchmark script validates the 14x performance improvement from parallel batch cache checking (134s -> 10s for 500 paths against 2 caches).
Replace Nix flake development environment with Flox in CI workflows. Changes: - ci.yml: Update test-javascript job to use Flox - update-dist.yml: Update dist/ check job to use Flox - Replace 'cachix/install-nix-action' with 'flox/activate-flox-action@v1.1.0' - Update all 'nix develop --command' calls to 'flox activate --' Note: Test jobs that verify the configure-nix-action itself (test-minimal-action, test-all-action, test-nix-action) intentionally still use Nix as they are testing the action's Nix configuration functionality.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds intelligent upstream cache filtering and performance optimizations to avoid redundant uploads when pushing Nix store paths to custom substituters. It also migrates the development environment to Flox and adds comprehensive developer tooling.
Key Features
🎯 Configurable Upstream Cache Filtering
upstream-cachesinput parameter that accepts comma or newline-separated cache URLscache.nixos.organdcache.flox.dev⚡ Performance Optimization (14x Faster)
Replaced sequential per-path cache checking with parallel batch queries:
Implementation:
nix path-infoquery per cachecommfor efficient set difference operations🛠️ Developer Tooling
.claude/CLAUDE.mdwith project architecture, development commands, and implementation detailsscripts/benchmark-cache-check.shto validate and measure cache checking performanceBenefits
Example Configuration
Benchmark Results
Testing with 500 paths against 2 caches:
Both approaches verified against identical path sets (checksum validation included).
Test Plan