Skip to content

Latest commit

 

History

History
153 lines (99 loc) · 5.82 KB

TESTING.md

File metadata and controls

153 lines (99 loc) · 5.82 KB

Testing

This is an overview of our testing infrastructure.

See also

Writing good tests

  • Test behavior that could plausibly break. Don't add tests for deprecated code paths, trivial UI shortcuts, or one case out of many that all work the same way.
  • A bug fix comes with a regression test that fails without the fix. If that is not feasible, the PR description says why.
  • Keep tests readable: one behavior per test, with setup, action, and checks visibly separated. Reviewers read tests to understand what the code is supposed to do.
  • Start each test with a one-line docstring saying which behavior it pins, unless the test name already says it all.
  • Wait for a condition, not for a fixed number of frames or a fixed sleep. In egui_kittest, harness.run() runs until the UI settles; for background work, step until the condition holds, with a timeout. A literal step count (harness.run_steps(4)) needs a comment saying what it waits for.
  • Put unit tests in a #[cfg(test)] mod tests at the bottom of the file they test, and integration tests in the crate's tests/ folder. Don't add a separate tests.rs module next to the code, or test modules in lib.rs/mod.rs.
  • Don't copy production data or constants into a test: import them, or use values that are obviously dummies.
  • A test that asserts known-wrong behavior links the issue that tracks the fix (// TODO(RR-1234): should be …).
  • Test the viewer through its UI (e.g. clicking buttons in an egui_kittest harness) rather than by writing to internal state.
  • Give non-obvious assert!s a message saying what should hold, and make failure messages print what was actually received.
  • Prefer an exact comparison (assert_eq!) over a partial one (contains).
  • Keep fixtures small (a handful of rows or messages), and keep tests fast: no waiting for many seconds.
  • For small, pure functions, consider a doc test.

Network tests in sandboxes

Use 127.0.0.1 and OS-assigned ports (port=0) for local test servers; sandboxes may reject wildcard binds. Keep wildcard binds only when testing wildcard behavior. Report blocked socket access rather than silently skipping tests.

Rust unit tests

We use the standard Rust test framework. We like to use nextest because it's faster, but cargo test should work fine.

While developing, default to using --all-features. (Our CI runs checks and tests with many feature configurations.)

Typically:

cargo nextest run -p re_XXX --all-features

where re_XXX is the crate under development.

Wasm tests

Run pixi run web-test to test all workspace packages with a wasm-bindgen-test dev-dependency. Use pixi run web-test --package re_async to select one package. CI runs this same runner.

The runner executes Node and browser passes, defaulting unmarked test binaries to the browser. Each pass skips binaries marked for the other runtime. To select Node, add this at the top of a Wasm integration-test file:

#![cfg(target_arch = "wasm32")]

wasm_bindgen_test::wasm_bindgen_test_configure!(run_in_node_experimental);

Runtime selection applies to an entire test binary, not individual functions or modules. Keep Node and browser tests in separate integration-test files when a package needs both.

⚠️ The browser fallback is set by the web-test command. Invoking wasm-bindgen-test-runner directly defaults unmarked binaries to Node instead.

Snapshot tests

We use insta for snapshot tests.

To create or update snapshots, you may use:

INSTA_FORCE_PASS=1 cargo nextest run -p re_XXX --all-features

Then, use this to review the created/updated snapshots:

cargo insta review

UI snapshot tests

We use egui_kittest extensively for UI snapshot tests.

To create or update snapshots, you may use:

UPDATE_SNAPSHOTS=1 cargo nextest run -p re_XXX --all-features

Our CI automatically provides a link in PR for kitdiff, our custom visual image diff tool.

Python unit tests

We use pytest for Python unit tests, along with syrupy for snapshot testing.

To run tests, use:

pixi run py-test

Creating or updating snapshots is done by adding --snapshot-update to the pytest command.

Redap tests

Redap stands for "Rerun data protocol." It is the interface between clients such as the Rerun viewer or SDK, and servers such as Rerun OSS or Rerun Hub.

We have several test harnesses related to redap.

re_redap_tests

This is a Rust-based compliance test suite that builds directly against the server's service handler. It is run both against the OSS server in this repository, and our Rerun's proprietary implementation Rerun Hub. This test suite does not run through an actual gRPC connection. It directly links to the servers' code.

This test suite is executed by the OSS server tests, so you can run it locally with:

cargo nextest run -p re_server --all-features

re_integration_test

This is a Rust-based test suite that runs both our Rust-based client and the OSS server. The harness spins up a local server and the tests connect to it.

Run it with:

cargo nextest run -p re_integration_test --all-features

e2e_redap_tests

This is a python-based end-to-end test suite that uses our Python SDK and connects, by default, to the OSS server. It is executed by pytest. More in formation in the e2e_redap_tests README.