"Pass, then timeout" should not be considered a spurious regression

[workingjubilee]

Issue is visible in this crater report:

• stabilize -Znext-solver=coherence rust#121848 (comment)
• https://crater-reports.s3.amazonaws.com/pr-121848-3/index.html

Crater was being used to diagnose usage of the new trait solver, which has hang issues in some cases.

This caused rust-lang/rust#130056 to be opened after landing rust-lang/rust#121848 even though we did notice essentially all the nalgebra dependents failed this way:

[workingjubilee]

other possible solutions might be to find some way to introduce retrying or extended second-pass build times that make this work.

[emilyalbini]

The reason why timeouts regressions are marked as spurious is because the amount of time it takes to build a crate varies wildly between builds, depending on whether the dependencies were cached or not in the target directory. If we didn't mark those as spurious every crater run would contain spurious regressions in the regressed crates list.

[Skgland]

other possible solutions might be to find some way to introduce retrying or extended second-pass build times that make this work.

I don't think this is what you meant but since afc6496 spurious regressed crates are included in the retry-regressed-list.txt, though it looks like the crater run mentioned was from before that change.

Maybe it would help if the crater bot would mention an unusually high amount of spuriously regressed crates in addition to regressed and fixed crates, or spuriously regressed could be a sub-category under regressed?

i.e.

🎉 Experiment pr-121848-3 is completed! 📊 16 regressed and 3 fixed (501375 total) 📰 Open the full report.

🔥 Unusual high amount of spurious regressed crates 2489 (instead of ~500-600)

⚠️ If you notice any spurious failure please add them to the blacklist! ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[Skgland]

The reason why timeouts regressions are marked as spurious is because the amount of time it takes to build a crate varies wildly between builds, depending on whether the dependencies were cached or not in the target directory. If we didn't mark those as spurious every crater run would contain spurious regressions in the regressed crates list.

This sounds like something along the lines of rust-lang/cargo#2644 might help as it would allow to make a two phase build, first deps only and then the crate itself.