Skip to content

Speed up test_ax_sweep_impl_minimal by mocking Sobol sensitivity MC samples - #5282

Closed
saitcakmak wants to merge 1 commit into
facebook:mainfrom
saitcakmak:export-D118535425
Closed

Speed up test_ax_sweep_impl_minimal by mocking Sobol sensitivity MC samples#5282
saitcakmak wants to merge 1 commit into
facebook:mainfrom
saitcakmak:export-D118535425

Conversation

@saitcakmak

Copy link
Copy Markdown
Contributor

Summary:
automl.internal.ax.ax_sweep.tests.test_ax_sweep_feature_matrix.TestAxSweepFeatureMatrix.test_ax_sweep_impl_minimal (T562950220699357) has been timing out on stress runs against TPX's 600s limit. Profiling shows the runtime is dominated by variance-based (Sobol) sensitivity analysis, not by the number of analysis cards:

  • AxSweepOrchestrator.report_results is called 4x and accounts for 110.2s of a 127.2s local run (87%): _compute_and_save_analysis_cards 67.3s, legacy get_standard_plots 38.5s.
  • Exclusive per-Analysis accounting: SensitivityAnalysisPlot 65.9s over 2 calls; every other analysis is under 0.5s.
  • ax_parameter_sens defaults to num_mc_samples = 10**4, so each call is O(num_mc_samples * num_params) posterior-mean evaluations, multiplied by the MCMC sample count on this test's fully Bayesian (SAASBO) model. mock_botorch_optimize does not help, since it cheapens fitting, not posterior evaluation.
  • It runs 4x per test: twice via get_standard_plots -> ax_parameter_sens(order="total") and twice via OverviewAnalysis -> InsightsAnalysis -> TopSurfacesAnalysis -> SensitivityAnalysisPlot, where _choose_sensitivity_order(3) selects the most expensive "second" order.

This adds an opt-in mock_sensitivity_analysis decorator / context manager to ax/utils/testing/mock.py that forces num_mc_samples=32 in compute_sobol_indices_from_model_list and compute_derivatives_from_model_list, and applies it to execute_ax_sweep_and_make_assertions. Nothing in that test asserts on sensitivity values -- only that the expected number of plots is produced -- so the reduced sample count is safe here.

Note that Ax's own TestCase.MAX_TEST_SECONDS = 120 guard does fire on this test, but the resulting TimeoutError is swallowed by Analysis.compute_result's broad except Exception (surfacing only as a "Failed to compute SensitivityAnalysisPlot" error card) and by report_results' broad handlers, so the test still reported ok. That is worth fixing separately.

Differential Revision: D118535425

…amples

Summary:
`automl.internal.ax.ax_sweep.tests.test_ax_sweep_feature_matrix.TestAxSweepFeatureMatrix.test_ax_sweep_impl_minimal` (T562950220699357) has been timing out on stress runs against TPX's 600s limit. Profiling shows the runtime is dominated by variance-based (Sobol) sensitivity analysis, not by the number of analysis cards:

- `AxSweepOrchestrator.report_results` is called 4x and accounts for 110.2s of a 127.2s local run (87%): `_compute_and_save_analysis_cards` 67.3s, legacy `get_standard_plots` 38.5s.
- Exclusive per-`Analysis` accounting: `SensitivityAnalysisPlot` 65.9s over 2 calls; every other analysis is under 0.5s.
- `ax_parameter_sens` defaults to `num_mc_samples = 10**4`, so each call is O(num_mc_samples * num_params) posterior-mean evaluations, multiplied by the MCMC sample count on this test's fully Bayesian (SAASBO) model. `mock_botorch_optimize` does not help, since it cheapens fitting, not posterior evaluation.
- It runs 4x per test: twice via `get_standard_plots` -> `ax_parameter_sens(order="total")` and twice via `OverviewAnalysis` -> `InsightsAnalysis` -> `TopSurfacesAnalysis` -> `SensitivityAnalysisPlot`, where `_choose_sensitivity_order(3)` selects the most expensive `"second"` order.

This adds an opt-in `mock_sensitivity_analysis` decorator / context manager to `ax/utils/testing/mock.py` that forces `num_mc_samples=32` in `compute_sobol_indices_from_model_list` and `compute_derivatives_from_model_list`, and applies it to `execute_ax_sweep_and_make_assertions`. Nothing in that test asserts on sensitivity values -- only that the expected number of plots is produced -- so the reduced sample count is safe here.

Note that Ax's own `TestCase.MAX_TEST_SECONDS = 120` guard does fire on this test, but the resulting `TimeoutError` is swallowed by `Analysis.compute_result`'s broad `except Exception` (surfacing only as a "Failed to compute SensitivityAnalysisPlot" error card) and by `report_results`' broad handlers, so the test still reported ok. That is worth fixing separately.

Differential Revision: D118535425
@meta-cla meta-cla Bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Sep 3, 2026
@meta-codesync

meta-codesync Bot commented Sep 3, 2026

Copy link
Copy Markdown

@saitcakmak has exported this pull request. If you are a Meta employee, you can view the originating Diff in D118535425.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 23.52941% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.57%. Comparing base (d034ba2) to head (4fe7590).

Files with missing lines Patch % Lines
ax/utils/testing/mock.py 23.52% 13 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5282      +/-   ##
==========================================
- Coverage   96.59%   96.57%   -0.03%     
==========================================
  Files         621      621              
  Lines       70894    70911      +17     
==========================================
- Hits        68481    68480       -1     
- Misses       2413     2431      +18     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@meta-codesync meta-codesync Bot closed this in 778e22f Sep 3, 2026
@meta-codesync meta-codesync Bot added the Merged label Sep 3, 2026
@meta-codesync

meta-codesync Bot commented Sep 3, 2026

Copy link
Copy Markdown

This pull request has been merged in 778e22f.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed Do not delete this pull request or issue due to inactivity. Merged meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants