Skip to content

test(it): Redesign Invoker coverage matrix - #386

Draft
keeganwitt wants to merge 3 commits into
masterfrom
codex/ref/integration-test-redesign
Draft

test(it): Redesign Invoker coverage matrix#386
keeganwitt wants to merge 3 commits into
masterfrom
codex/ref/integration-test-redesign

Conversation

@keeganwitt

@keeganwitt keeganwitt commented Jun 24, 2026

Copy link
Copy Markdown
Member

Replaces the history-shaped Maven Invoker fixture suite with 20 flat, behavior-named workflows and documents the coverage contract for future additions.

The redesign keeps Invoker coverage focused on user-visible Maven/Groovy execution boundaries: lifecycle wiring, classpath selection, reactor behavior, generated bytecode/artifacts, forked JVM behavior, shaded Groovy discovery, and Maven-plugin consumption. A follow-up provenance audit found a few old fixtures were feature/regression boundaries rather than arbitrary variants, so this version restores those behaviors under descriptive names instead of preserving old fixture names.

Conditions tested by the old suite

Old condition Representative old fixtures Provenance signal
Basic and advanced Groovy compilation basicCompile, advancedCompile, astCompile Baseline compile workflow; basicCompile later carried issue #186 annotation coverage
Compile with configuration script configScriptCompile Configuration-script workflow
Parameter metadata compilation parametersCompile GROOVY-7423 / issue #45 support
Compile test Groovy with test-scope dependency visibility testDependencyCompile Issue #358 reproduction and project-only classpath visibility
Mixed Java/Groovy compilation with stub generation mixedCompile, mixedCompile2, mixedCompileCircular Mixed compilation workflows, including Java-root and circular graph variants
Mixed Java/Groovy reactor compilation mixedCompileMultiModule, mixedCompileMultiModule2 Multi-module joint compilation graph variants
Main stub generation and removal basicGenerateStubs, advancedGenerateStubs Stub lifecycle workflow
Clean-classpath compile failure boundary cleanClasspathCompile Classpath isolation expected failure
Clean-classpath stub generation boundary cleanClasspathGenerateStubs GMAVENPLUS-17 classpath isolation expected failure
Basic and file-backed script execution basicExecute, advancedExecute Script execution workflows
Reactor script execution with classpath variants multimoduleProjectClasspath, multimodulePluginClasspath, multimodulePluginAndProjectClasspath, multimodulePluginAndProjectClasspath2 Issue #183 multi-module classloader isolation
Main GroovyDoc generation basicGroovydoc, advancedGroovydoc GroovyDoc workflow
Main GroovyDoc JAR packaging basicGroovydocJar GroovyDoc JAR goals / #124
Mixed Java/Groovy GroovyDoc generation mixedGroovydoc, mixedGroovydoc2 Mixed-source documentation graph variants
Forked compile and test compile forkedCompile, forkedCompilePluginClasspath, forkedCompileProjectAndPluginClasspath Toolchain/forked JVM coverage and later classpath cross-products
Forked stub generation forkedStubs Toolchain support for stub generation / #43
Forked GroovyDoc generation forkedGroovyDoc Toolchain support for GroovyDoc / #43
Project/plugin/project-and-plugin Groovy classpath modes pluginClasspath, pluginAndProjectClasspath, testDependencyCompile Classloader selection / #64 and all classpath modes / #140
Shaded Groovy discovery shadedGroovy GMAVENPLUS-35 shaded Groovy behavior
Groovy-authored Maven plugin consumption mavenPlugin Maven-plugin packaging and consumer execution

Conditions tested by the new suite

New condition New fixture Coverage disposition
Default Groovy compilation with runtime proof and annotation coverage compile-default Preserves baseline compile and issue #186 annotation signal
Compile and test-compile with configuration script compile-config-script Preserved directly
Parameter metadata bytecode generation compile-parameters Restored after provenance audit
Mixed Java/Groovy compilation and source-root wiring compile-mixed-java-groovy Preserved through consolidation
Mixed Java/Groovy reactor ordering compile-reactor Preserved through consolidation
Main and test stub lifecycle/source-root registration stubs-main-and-test Preserved through consolidation
Test-stub source-root removal stubs-cleanup Newly covered behavior
Expected failure for compile classpath isolation compile-classpath-isolation Preserved directly
Expected failure for stub classpath isolation stubs-classpath-isolation Restored after provenance audit
Inline and file-backed script execution execute-inline-and-file Preserved through consolidation
Reactor script execution classpath behavior execute-reactor-classpath Preserves issue #183 reactor classloader boundary
Main GroovyDoc generation and attached docs JAR groovydoc-main-and-jar Preserved/improved artifact oracle
Test GroovyDoc generation and attached test docs JAR groovydoc-test-and-test-jar Newly covered goal family
Forked JVM compile plus stub generation forked-jvm-compile-and-stubs Preserves #43 compile/stub toolchain behavior
Forked JVM GroovyDoc generation forked-jvm-groovydoc Preserves #43 GroovyDoc toolchain behavior
Project-only Groovy classpath selection classpath-project-only Preserves project-only mode and issue #358 visibility
Plugin-only Groovy classpath selection classpath-plugin-only Preserves plugin-only mode
Combined project-and-plugin Groovy classpath selection classpath-project-and-plugin Preserves combined mode
Shaded Groovy discovery from an uber JAR shaded-groovy Preserved directly
Groovy-authored Maven plugin descriptor and consumer execution maven-plugin-consumer Preserved directly

Conditions intentionally no longer tested as separate Invoker projects

Removed separate Invoker condition Current disposition
Separate basic, advanced, and AST-flavored compile projects Consolidated into compile-default, compile-config-script, and compile-parameters based on observable behavior.
Duplicate mixed Java/Groovy variants without a named regression boundary Kept one mixed single-module workflow and one mixed reactor workflow.
Java-root mixed compilation as its own project Dropped as a graph-shape variant; the source-root/stub boundary remains covered.
Circular mixed Java/Groovy dependency shape as its own project Dropped as a graph-shape variant unless a regression re-establishes it as distinct.
Second mixed reactor direction Dropped as a duplicate reactor dependency-shape variant.
Separate reactor execute fixtures for each classpath mode Consolidated into one reactor execute workflow; classpath modes are covered independently by dedicated classpath fixtures.
Mixed Java/Groovy GroovyDoc variants Main/test GroovyDoc artifact behavior is now tested directly; mixed-source doc graph variants are not kept as separate ITs.
Custom GroovyDoc styling/resources as a standalone behavior Dropped unless tied to a documented user-visible regression.
Forked compile classpath-mode cross-product Forked JVM behavior and classpath modes are tested separately rather than as a Cartesian product.
Forked stubs as a standalone project Folded into forked-jvm-compile-and-stubs.

New or restored conditions added by this redesign

Added/restored condition Fixture
groovydocTests goal produces test GroovyDoc output groovydoc-test-and-test-jar
groovydocTests-jar attaches/packages test documentation groovydoc-test-and-test-jar
removeTestStubs removes the generated test-stub source root from the Maven model stubs-cleanup
Java parameter metadata is present in compiled Groovy bytecode when requested compile-parameters
Stub generation respects project-only classpath isolation stubs-classpath-isolation
Issue #186 annotation compile signal remains in the default compile path compile-default
Forked JVM compile path also covers stub generation in the same consuming build forked-jvm-compile-and-stubs
Classpath-mode fixtures fail if the wrong Groovy visibility is selected classpath-project-only, classpath-plugin-only, classpath-project-and-plugin
GroovyDoc and documentation JAR fixtures assert generated artifacts instead of relying only on Maven exit status groovydoc-main-and-jar, groovydoc-test-and-test-jar
The suite has an explicit fixture-addition policy and coverage table src/it/README.md, AGENTS.md

Verification

  • .\mvnw.cmd '-Dinvoker.test=compile-default' invoker:run
  • .\mvnw.cmd '-Dinvoker.test=compile-parameters' invoker:run
  • .\mvnw.cmd '-Dinvoker.test=stubs-classpath-isolation' invoker:run
  • .\mvnw.cmd '-Dinvoker.test=compile-classpath-isolation' invoker:run
  • git diff --check
  • .\mvnw.cmd test
  • .\mvnw.cmd invoker:run

The full Groovy 2.5, 3, 4, and 5 matrix remains for CI.

keeganwitt and others added 3 commits June 24, 2026 00:45
Replace history-shaped fixtures with behavior-named workflows and add focused lifecycle assertions for test GroovyDoc and stub cleanup.

Co-Authored-By: Codex <noreply@openai.com>
Explain the behavior-oriented integration test matrix and direct parameter-only coverage to unit tests.

Co-Authored-By: Codex <noreply@openai.com>
Add behavior-named fixtures for parameter metadata and stub classpath isolation, and retain annotation compile coverage in the default compile fixture.

Co-Authored-By: Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant