Uniquely keyed run artifacts#11
Open
MJohnson459 wants to merge 3 commits into
Open
Conversation
Each ./run invocation now writes everything under its own results/runs/<odm revision>/<image key>/<timestamp>/ directory: the ODM output per test, a JUnit XML report per dataset, a TSV manifest and a run_manifest.json with the run key, docker image id/digest, the ODM source revision (from the image's OCI label when present), host info and per-test wall time and output size. The revision and image key fall back to unknown when the image does not carry them, so runs of the same ODM version group together and can be compared. run prints a PASS/FAIL summary per dataset with wall time, lists failures with the ODM exit status (and signal name if it was killed) and exits non-zero if anything failed, so it can gate CI. Every run starts from a fresh directory, so --clear is gone and --tags is replaced by --tag: a run tests a single image, run once per version to compare. harvest now looks under results/runs/.
The manifest is meant to make a run directory self-contained, so the per-test output_dir has to survive the directory being archived or moved out of the checkout. Keep the repo-relative path in the TSV, which the terminal summary prints for the current checkout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OATS can run against the new pixi docker images but the process for comparing them is ad-hoc. This PR is the first of a series that will give us historical context to changes and allow comparison between values.
The main change is that we now store the results in a uniquely keyed results directory:
Where the ODM revision is the exact git sha followed by the image id and finally the timestamp. This follows the logic of an image has exactly one revision, and a timestamp (run) is from exactly one image. This means repeat runs get grouped together, and different images from the same sha are also grouped.
At the top level there is a manifest that records exactly what was run - it's fairly simple right now but we could expand it (i.e. max memory usage).
In future work I will:
While this is all going on, I'm primarily testing this change with the master and pixi branches.