Skip to content

WPT results: count subtests gained and lost in each direction - #1161

Merged
nicoburns merged 1 commit into
mainfrom
devin/1791562867-wpt-diff-subtest-counts
Oct 9, 2026
Merged

nicoburns merged 1 commit into
mainfrom
devin/1791562867-wpt-diff-subtest-counts

Conversation

@nicoburns

@nicoburns nicoburns commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Summary

The WPT results section in PR descriptions undercounted when subtests of one test moved in both directions. The headline summed each test's net pass-count change, and wpt diff (without --verbose) omits a test whose status and pass count are both unchanged. So a test that gains 40 subtests and loses 40 was missing from the headline and the full diff, while the by-area table (which compares subtests by name) counted it.

Example from #1046: headline "25 newly passing, 0 newly failing", by-area table +25 (+65 / -40). The difference is css/css-sizing/stretch/block-height-008.html, 80/160 before and after with 40 subtests fixed and 40 regressed.

Changes:

  • wpt.yml: the JSON diff is generated with wpt diff --verbose, which joins subtests by name and reports tests whose individual subtests changed.
  • wpt_diff_to_pr.py: gained/lost are counted per subtest from each entry's subtests list, falling back to the net change when the list is empty (tests without subtests, or a diff produced without --verbose).
  • A test with changes in both directions is shown as ! with +G/-L instead of the net:
! FAIL => FAIL  [80/160]  +40/-40  css/css-sizing/stretch/block-height-008.html
  • Entries where only non-passing subtest statuses changed (e.g. FAIL to TIMEOUT) with the same test status and pass count are left out, as before. --verbose would otherwise add them to the list.

wpt-post-results.yml runs the script from main against the artifact of the PR's run, so until a PR is rebased past this its diff has no subtest lists and renders as it does today.

Link to Devin session: https://dioxus.staging.devinenterprise.com/sessions/3a06d93089f2434ba3519a7813dea1e8
Open in Devin Desktop: https://dioxus.staging.devinenterprise.com/desktop/session/3a06d93089f2434ba3519a7813dea1e8?variant=devin-insiders
Requested by: @nicoburns

WPT results

No changes in test results compared to main.

Generated by the WPT workflow.

@staging-devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

  • Disable automatic comment, CI, and merge conflict monitoring

@nicoburns
nicoburns enabled auto-merge (squash) October 9, 2026 16:24
@nicoburns
nicoburns merged commit 1b4843a into main Oct 9, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant