Add caching for dependencies and pip to CI#4003
Open
johvincau wants to merge 9 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test suite dependencies can be cached to shorten install times. This PR targets both built dependencies (njoy, moab + dagmc, libmesh) as well as all unique configurations of Python called by the job matrix, so no time is spent either compiling dependencies or building new wheels on each runner.
This PR resolves #2140.
When all relevant caches can be restored, this feature brings the most expensive job install times in line with pure OpenMC jobs while providing modest install time improvements to all others.

The baseline install times come from a merge from openmc/develop into my fork, so there was no caching beyond the xs cache. The cached install times come from a test run on my branch. The time to restore from cache is around several seconds, which is not included in the above data.
Dependencies
For dependencies built from source,
gha-cache-keys.shchecks the following tags for the corresponding commit hash, which is then used as the cache key.Each one of these dependencies has its own separate cache, and in the case of libmesh, two caches for both an mpi and non-mpi build. A new binary will be built and cached if the commit hash changes.
Python Caching
The
~/.cache/pipdirectory is cached with two hashes in its key. The first is the hash ofrequirements-mpi-X.txt, where X is either 'y' or 'n' based on the job. These files are used purely to generate the hash, and not as a reference for installing dependencies on the runner (this has been handled by . The second is the hash ofmpi-deps.txtwhich records the versions of mpi apt dependenciesdpkg -W libmpich-dev libhdf5-mpich-dev. This file, also generated bygha-cache-keys.sh, is empty if the job does not need mpi, so the resulting hash is just the hash of nothing. The second hash is included so that in the event of an update to the mpi apt dependencies, a new pip cache built against it will be stored. The pip dependencies for MPI installs, mpi4py and h5py, should be built against these dependencies.Overall, 5 separate configurations of Python are stored:
Python 3.12 with and without MPI, 3.13, 3.14, and 3.14t, at around 250 MB each.
Checklist