change code coverage - #3137
Conversation
github-project-automation
Bot
added this to
S-CORE Roadmap and TLC - Technical Lead Circle
|
The created documentation from the pull request is available at: docu-html |
| Measuring code coverage exclusively on the **Linux host** (via LLVM) is considered sufficient for safety certification, provided that a **two-step verification** is performed to demonstrate equivalence on the target: | ||
|
|
||
| #. **Quantitative Verification (Host):** The complete test suite is executed on the Linux host with code coverage enabled. This demonstrates that the test cases are structurally complete (100% C0/C1 coverage) and that no dead code exists. | ||
| #. **Qualitative Verification (Target):** The identical test suite is executed on the **QNX target**, but with code coverage instrumentation turned off. This demonstrates that the software compiles, links, and behaves identically on the real target hardware (correctness of execution, no compiler/linker optimization bugs, no endianness or memory alignment issues). |
There was a problem hiding this comment.
What about when the baseline of coverage is different? For example, files that might be QNX only.
I think this argumentation is missing what happens with code that it is QNX only and code that it is Linux only.
There was a problem hiding this comment.
Good point — the argumentation was missing the case where the coverage baseline differs between host and target. I've added a new subsection "Handling Platform-Specific Code" that splits this into three cases:
- Linux-only code (mocks, emulation, host utilities): classified as QM/test-support and excluded from the ASIL-B production baseline, so it can't skew the metrics.
- QNX-only code inside mixed files (
#ifdef __QNX__blocks): marked with governed coverage-exclusion markers plus a justification that the region is verified on the target. - QNX-only whole files compiled only for QNX: excluded from the host structural-coverage baseline (to avoid false-negative 0%), and treated as a justified deviation with residual risk rather than an equivalence claim, compensated by requirements-based integration tests on the target.
The subsection also explains that the host/target coverage baselines differ by construction and how each delta is reconciled.
| This approach satisfies *ISO 26262-6* for *ASIL_B* for the following reasons: | ||
|
|
||
| * **Identical Test Results:** Passing 100% of the tests on both the host and the target demonstrates that the control flow under test is identical. | ||
| * **Mitigation of Target Instrumentation Risks:** Turning off coverage instrumentation on the target is recommended for embedded systems, as active instrumentation alters the timing behavior, memory footprint, and compiler optimizations on the target. Testing the uninstrumented code on QNX ensures that the actual production binary is verified (mitigating "Heisenbugs"). |
There was a problem hiding this comment.
That is true, instrumentation affects behavior, but it could be run, once instrumented to measure coverage and one without instrumentation just to check that the tests pass.
There was a problem hiding this comment.
Agreed. I've reworded the Mitigation of Target Instrumentation Risks bullet to make explicit that the two-step verification is exactly that pattern — measure coverage with instrumentation, then re-run without instrumentation to confirm the tests still pass — but split across the two environments: the host run is instrumented (for structural coverage) and the target run is uninstrumented (to confirm the production binary behaves identically).
The one constraint I called out in the text: qcc cannot produce LLVM source-based coverage, so an instrumented coverage run on the QNX target itself is not available; the instrumented measurement necessarily happens on the host.
|
|
||
| * **Identical Test Results:** Passing 100% of the tests on both the host and the target demonstrates that the control flow under test is identical. | ||
| * **Mitigation of Target Instrumentation Risks:** Turning off coverage instrumentation on the target is recommended for embedded systems, as active instrumentation alters the timing behavior, memory footprint, and compiler optimizations on the target. Testing the uninstrumented code on QNX ensures that the actual production binary is verified (mitigating "Heisenbugs"). | ||
| * **Equivalence Justification:** The host-based coverage measurement is justified by confirming that no platform-specific code paths (e.g., ``#ifdef QNX`` blocks) are bypassed. Any target-specific hardware abstraction layer (HAL) is verified separately via system-level integration tests. |
There was a problem hiding this comment.
The host-based coverage measurement is justified by confirming that no platform-specific code paths (e.g.,
#ifdef QNXblocks) are bypassed.
How should a developer do that? How can a developer find such paths? As mentioned above, it is not only the pragmas, but there could be files that are completly excluded from the linux build.
There was a problem hiding this comment.
You're right that this can't rely on a developer manually grepping for #ifdef guards, since whole files can be excluded from the host build. I've added an "Identifying platform-specific code" subsection describing a concrete, tool-based process:
- Build-set comparison: the set of files compiled for the host is diffed against the set compiled for QNX; files present only in the QNX build are flagged automatically as the "QNX-only files" case.
- Explicit exclusion manifest: every host-coverage exclusion (per-file or within-file region marker) is recorded in one reviewable list, so the full delta between the host baseline and the target code base is visible in a single place.
- CI gate: the comparison and the manifest are enforced in CI, so a new QNX-only file or conditional block without a documented exclusion and a compensating target verification fails the quality gate.
I also added a note that the exclusion-marker syntax must match the actual toolchain: since coverage uses llvm-cov/llvm-profdata, LCOV_EXCL_* markers are not interpreted natively and must not be assumed to work unless an lcov conversion/merge step is explicitly part of the pipeline.
There was a problem hiding this comment.
I'm only not sure, how we bring this to the infrastructure.
3 participants
This pull request updates the documentation for code coverage analysis in C++ projects, specifically clarifying the recommended approach for coverage measurement and its justification for safety certification. The main changes focus on switching from GCC/gcov-based coverage to LLVM/clang-based coverage on the host, and providing a detailed rationale for this methodology in the context of safety standards.
Code Coverage Documentation Updates:
gcov + gcovrwithllvm-cov + llvm-profdata, and removed QNX as a coverage collection target.