Skip to content

Add caching for dependencies and pip to CI#4003

Open
johvincau wants to merge 9 commits into
openmc-dev:developfrom
johvincau:ci_cache
Open

Add caching for dependencies and pip to CI#4003
johvincau wants to merge 9 commits into
openmc-dev:developfrom
johvincau:ci_cache

Conversation

@johvincau

Copy link
Copy Markdown
Contributor

Summary

Test suite dependencies can be cached to shorten install times. This PR targets both built dependencies (njoy, moab + dagmc, libmesh) as well as all unique configurations of Python called by the job matrix, so no time is spent either compiling dependencies or building new wheels on each runner.

This PR resolves #2140.

When all relevant caches can be restored, this feature brings the most expensive job install times in line with pure OpenMC jobs while providing modest install time improvements to all others.
install_fig
The baseline install times come from a merge from openmc/develop into my fork, so there was no caching beyond the xs cache. The cached install times come from a test run on my branch. The time to restore from cache is around several seconds, which is not included in the above data.

Dependencies

For dependencies built from source, gha-cache-keys.sh checks the following tags for the corresponding commit hash, which is then used as the cache key.

  • NJOY = 2016.78
  • MOAB = 5.5.1
  • LIBMESH = v1.7.1
  • DAGMC has no specified version; the key references HEAD on its develop branch.

Each one of these dependencies has its own separate cache, and in the case of libmesh, two caches for both an mpi and non-mpi build. A new binary will be built and cached if the commit hash changes.

Python Caching

The ~/.cache/pip directory is cached with two hashes in its key. The first is the hash of requirements-mpi-X.txt, where X is either 'y' or 'n' based on the job. These files are used purely to generate the hash, and not as a reference for installing dependencies on the runner (this has been handled by . The second is the hash of mpi-deps.txt which records the versions of mpi apt dependencies dpkg -W libmpich-dev libhdf5-mpich-dev. This file, also generated by gha-cache-keys.sh, is empty if the job does not need mpi, so the resulting hash is just the hash of nothing. The second hash is included so that in the event of an update to the mpi apt dependencies, a new pip cache built against it will be stored. The pip dependencies for MPI installs, mpi4py and h5py, should be built against these dependencies.

Overall, 5 separate configurations of Python are stored:
Python 3.12 with and without MPI, 3.13, 3.14, and 3.14t, at around 250 MB each.

Checklist

  • I have performed a self-review of my own code
  • I have run clang-format (version 18) on any C++ source files (if applicable)
  • I have followed the style guidelines for Python source files (if applicable)
  • I have made corresponding changes to the documentation (if applicable)
  • I have added tests that prove my fix is effective or that my feature works (if applicable)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Github Actions: Cache source-build executables and libraries to speed up testing

1 participant