[pull] main from jdx:main - #656
Merged
Merged
Conversation
Fresh `mise bootstrap packages apply` runs previously fetched Homebrew formula metadata one request at a time while discovering a dependency closure. Formulae in the same dependency frontier are independent, so mise now fetches that metadata concurrently using the existing `jobs` limit, then restores deterministic processing order before resolving the next frontier. This does not change the default job count or add a new concurrency setting. Tap fallback behavior, alias handling, and dependency-first installation order remain unchanged. ## Performance On an Apple M5 Max, 15 alternating fresh-cache runs of: ```sh mise bootstrap packages apply --dry-run brew:ffmpeg ``` reduced median dependency-resolution time from 288 ms to 112 ms (2.57x faster). Baseline and changed output were byte-for-byte identical. Network outliers remain possible; this comparison alternated binaries and isolated each run's cache and Homebrew prefix. ## Validation - 360 Brew-package unit tests passed - Full repository lint/fix pipeline passed - Optimized release build passed *AI-assisted — Tool: Codex; model: unavailable/unavailable; version: unavailable.*
`mise bootstrap packages apply` currently downloads bottles concurrently but then extracts, relocates, signs, and writes every bottle receipt one at a time. This change prepares independent bottles concurrently using the existing `jobs` limit, while preserving dependency-ordered Cellar commits and prefix linking. It does not change the default job count or add a new setting. For a warm-cache install of 14 bottles from: ```console mise bootstrap packages apply brew:jq brew:tree brew:wget brew:just brew:shellcheck ``` 8 alternating isolated-prefix runs on Apple silicon improved from a 4.097s median on the metadata-only base branch to 2.893s with concurrent staging: 29.4% less time, or 1.42x faster. The comparison reused downloaded bottles and metadata to isolate extraction, relocation, signing, receipt generation, and linking. Preparation runs in unique, lock-protected staging directories. On preparation failure, queued work is discarded, active work is drained for cleanup, and successfully prepared earlier dependencies still commit in order before the error returns. A later pour reclaims only unlocked staging created with the current lock protocol, leaving lockless directories from older mise processes untouched. Final Cellar and shared-prefix mutations use a prefix-local commit lock even when operation history is disabled. Bottle preparation remains concurrent outside it; source builds retain it while writing directly into the final Cellar. The installer rechecks completed state under the lock, and creates `opt/<name>` last so a partial public-link operation cannot look complete after a process is killed. Validation: - `mise run lint-fix` - `mise x -- cargo test --locked --bin mise system::packages::brew` (365 passed) - Installed the benchmark package set into an isolated prefix and ran `jq`, `tree`, `wget`, `just`, and `shellcheck` from that prefix Stacked on #13151. *AI-assisted — Tool: Codex; model: unavailable/unavailable; version: unavailable.* <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes concurrent filesystem staging, cross-process locking, and the order of prefix linking—core bootstrap install paths—with added tests but real multi-process and partial-failure edge cases. > > **Overview** > **`mise bootstrap packages apply`** still downloads bottles in parallel, but it now also **extracts, relocates, codesigns, and writes receipts concurrently** (bounded by the existing `jobs` setting). Only the final **Cellar move and prefix linking** stay sequential and dependency-ordered. > > Bottle work is split into **`prepare_bottle`** (lock-protected per-formula staging under the rack) and **`install_prepared`** (shared **`commit_lock`** before mutating Cellar/links). A new **`ConcurrentJobs`** helper can **drop queued prep jobs** after the first failure while **draining in-flight work** so staging locks clean up. Staging adds rack locks, `.mise-lock` files, and reclaim of abandoned staging dirs (without touching legacy lockless dirs). **`opt/<name>` is linked last** so partial linking cannot look “installed,” and source builds **hold the commit lock** for the whole build via **`run_blocking`** with a post-lock **already-installed** recheck. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 0151284. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
`mise bootstrap packages apply` currently waits for every bottle download to finish before starting any extraction, then waits for every bottle preparation job before linking the first dependency. This change makes each bounded job download and immediately prepare its own bottle, and commits prepared bottles as soon as dependency order permits. Downloads, extraction, relocation, codesigning, and sequential linking can therefore overlap without increasing the existing `jobs` limit. For 12 alternating fresh-cache, isolated-prefix installs (6 per implementation) of: ```console mise bootstrap packages apply brew:jq brew:tree brew:wget brew:just brew:shellcheck ``` the median fell from 6.57s on #13152 to 4.04s with the pipeline: 38.6% less wall time, or 1.63x faster. Each run used a fresh bottle and metadata cache, fresh data directory, and fresh Homebrew prefix on Apple silicon; the default remained 8 jobs. All 12 installs succeeded, and the installed `jq`, `tree`, `wget`, `just`, and `shellcheck` binaries were exercised after the initial sanity run. Source builds remain dependency ordered while concurrently advancing safe bottle download/preparation work. On failure, queued jobs are discarded and only already-active jobs are drained, so staging cleanup completes without starting additional downloads or preparations. The first failure remains authoritative, successfully installed rows remain visible, and Linux runtime setup still runs after a partial install. Validation: - `mise run lint-fix` - `mise x -- cargo test --locked --bin mise system::packages::brew` (366 passed) - Optimized release build - Cold-cache benchmark and installed-binary smoke test described above Stacked on #13152. *AI-assisted — Tool: Codex; model: unavailable/unavailable; version: unavailable.* <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes core brew install orchestration (concurrency, dependency-ordered linking, and failure paths) where incorrect ordering or cleanup could leave a partial prefix or mask errors. > > **Overview** > **Pipelines Homebrew bottle installs** so each bounded job downloads and prepares its own bottle instead of finishing all downloads before any extraction starts. Prepared bottles are linked as soon as dependency order allows, while source builds can overlap bottle work via `tokio::select!` without breaking prefix mutation ordering. > > **Refactors brew concurrency helpers** in `fetch.rs`: `concurrently` now returns a `ConcurrentJobs` handle that streams completions (replacing the old collect-all-at-once helper and the separate `concurrent_jobs` entry point). Tests cover incremental yields and cancel-pending behavior. > > **Tightens failure cleanup**: on error, pending bottle jobs are cancelled, active jobs are drained for staging cleanup, incomplete progress rows are abandoned while keeping successful installs visible, and Linux runtime setup still runs after a partial install before returning the first error. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit a390a07. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Brew installations now process downloads, bottle preparation, and source builds concurrently. * Installation progress is handled incrementally, improving responsiveness and reducing wait times before work begins. * Concurrent job results are delivered as they complete, while respecting configured limits. * **Bug Fixes** * Improved failure handling ensures installation errors are reported reliably and in-progress work is cleaned up safely. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )