Skip to content

test(ui): wait for the export's notification, not for the file it wrote - #128

Merged
jimisola merged 2 commits into
sjseth:mainfrom
jimisola:fix/flaky-export-waits
Aug 26, 2026
Merged

test(ui): wait for the export's notification, not for the file it wrote#128
jimisola merged 2 commits into
sjseth:mainfrom
jimisola:fix/flaky-export-waits

Conversation

@jimisola

Copy link
Copy Markdown
Collaborator

What & Why

Found on #126 (the setup-uv v10 bump), which had nothing to do with it: pytest (macos-latest, py3.13) went red while py3.12 and py3.14 on the same runner passed, and a plain re-run then went green. Two failures, one root cause, and it is a race that has been in the suite the whole time — it just needs the timing to land badly.

FAILED tests/unit/ui/test_e2e.py::test_demo_e_a_model_that_cannot_classify_still_lists_and_exports
        assert [] == ['Export complete']
FAILED tests/unit/ui/test_models.py::test_the_same_archive_can_be_imported_as_a_separate_copy
        assert (None is not None)      # copy.model_path

The cause

def export_to(page, window, model, dest):
    page.export_selected()
    assert drain_until(window, dest.exists), "export worker never finished"

dest.exists() is satisfied by the worker thread, mid-flight. export_model writes <dest>.tmp and then does an atomic os.replace, so the archive appears the moment the worker is done writing — while _on_exported's notify("Export complete", …) is still on its way back through the bus to the main thread. The wait returns early, with a notification pending.

That breaks two things, in different ways:

  1. Directly — the e2e test asserts window.notify.titles on the very next line, one drain too soon, and reads [].
  2. At a distance, which is the nastier oneimport_from waits for len(window.notify.calls) > before, where before was captured while the export's notification was still queued. The export's own late-arriving notify satisfies that condition, so the helper returns before the import worker has finished. import_model sets model_path during extraction, so the row reads back with model_path still None.

The second failure never mentions exports at all, which is why it looked like two unrelated flakes.

The fix

Wait for the completion that the UI actually reports, and keep the file as an assertion rather than a synchronisation point:

before = window.notify.titles.count("Export complete")
page.export_selected()
assert drain_until(window, lambda: window.notify.titles.count("Export complete") > before), (
    "export worker never finished"
)
assert dest.exists(), "the export reported success without writing the archive"

Three sites: the export_to helper in test_models.py and the two exports in test_e2e.py. import_from is left alone — its wait was the right shape all along, it was just being fed a stale before.

Deliberately not touched: test_models.py's delete test still waits on not paths.model_dir(...).exists(). That worker is run_worker(lambda: shutil.rmtree(...)) with no on_done, so the directory vanishing is the only completion signal there is. Different situation, correctly written.

The regression test

test_export_to_waits_for_the_notification_not_the_archive makes the window deterministic instead of leaving it to which runner you land on: it stubs export_model with one that writes the archive and then takes 250 ms, so the file exists long before the notification does. Against the old helper it fails with IndexError: list index out of range — there is no notification at all yet. Against the new one it passes.

The sleep is simulated slow work inside the export, not the test waiting on an async result; drain_until still does all the waiting, so this doesn't reintroduce what CLAUDE.md §5 rules out.

Test Plan

  • pytest1448 passed.
  • ruff check, ruff format --check, ty check — clean.
  • Verified both directions: reverted export_to to the old wait and confirmed the new test fails; restored it and confirmed it passes.

Not reproducible on Linux, which is the point — on this machine drain_until happens to deliver the notification in the same iteration that the file appears. macOS scheduled it differently and CI found it.

Author checklist

  • Conventional Commit title
  • DCO sign-off
  • No CLAUDE.md change — this is a test-helper fix, no architecture or boundary moved

`export_to` and the two e2e exports waited on the archive appearing. That is
satisfied by the *worker thread* — `export_model` finishes its atomic
os.replace and only afterwards posts completion — so the wait could return
with "Export complete" still travelling through the bus.

Two ways that bit, both seen on macOS py3.13 in CI while every other leg
passed:

- a test asserting `notify.titles` on the next line read `[]`, one drain early;
- `import_from` waits for "a notification that wasn't there before", and the
  export's late arrival satisfied it — so the helper returned while the import
  worker was still running and the row it had created read back with
  `model_path` still None.

Waiting on the notification fixes both, and keeps the file assertion as a
separate check rather than a synchronisation point. The regression test drives
a deliberately slow export so the window is deterministic instead of a matter
of which runner you land on; it fails with an IndexError against the old
helper.

Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>
@jimisola
jimisola requested a review from sjseth as a code owner August 19, 2026 21:54
@github-actions github-actions Bot added the tests Test suite label Aug 19, 2026
@jimisola
jimisola enabled auto-merge (squash) August 20, 2026 08:44
@jimisola
jimisola merged commit f8f474f into sjseth:main Aug 26, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests Test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants