Skip to content

change code coverage #3137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RolandJentschETAS wants to merge 3 commits into
base: main
Choose a base branch
from
Open

change code coverage #3137

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d860229
change code coverage
RolandJentschETAS Jul 29, 2026
090cc47
fix build warning
RolandJentschETAS Jul 29, 2026
d06079b
add explanation for change
RolandJentschETAS Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 23 additions & 10 deletions docs/contribute/development/cpp/code_analysis.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,25 +102,38 @@ Adress/ Leak Sanitizer (ASAN/LSAN)

If both tools are combined at runtime memory leaks and the corresponding address can be investigated.

Coverage
========
Code Coverage
=============

As required by the verification guideline coverage needs to be calculated for the code which is used in the project. Therefore two approaches should be available:
As required by the verification guideline code coverage needs to be calculated for the code which is used in the project. Coverage is calculated on the host using LLVM's source-based coverage:

* As a quick solution it is possible to calculate the coverage on the host via gcc.
* But for a more accurate statement coverage can also be calculated with the qcc compiler with the appropriate libraries and POSIX interfaces. This method will also be used for the reporting.
* Coverage is calculated on the host via clang/llvm. This method is also used for the reporting.

Since ``qcc`` does not support LLVM's source-based coverage instrumentation, coverage is not collected on the QNX target. LLVM's source-based coverage natively supports MC/DC, which is required for the higher ASIL levels.

To enable this, following tools are used:

.. needuml::

object "Coverage" as coverage
object "gtest" as gtest
object "gcov + gcovr" as gcov
object "llvm-cov + llvm-profdata" as llvm
object "host" as host
object "QNX" as qnx

coverage --> gtest
gtest --> gcov
gcov --> host
gcov --> qnx
gtest --> llvm
llvm --> host

Argumentation: Host-based Coverage with Target Test Execution
-------------------------------------------------------------

Measuring code coverage exclusively on the **Linux host** (via LLVM) is considered sufficient for safety certification, provided that a **two-step verification** is performed to demonstrate equivalence on the target:

#. **Quantitative Verification (Host):** The complete test suite is executed on the Linux host with code coverage enabled. This demonstrates that the test cases are structurally complete (100% C0/C1 coverage) and that no dead code exists.
#. **Qualitative Verification (Target):** The identical test suite is executed on the **QNX target**, but with code coverage instrumentation turned off. This demonstrates that the software compiles, links, and behaves identically on the real target hardware (correctness of execution, no compiler/linker optimization bugs, no endianness or memory alignment issues).

@limdor limdor Jul 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about when the baseline of coverage is different? For example, files that might be QNX only.
I think this argumentation is missing what happens with code that it is QNX only and code that it is Linux only.

@RolandJentschETAS RolandJentschETAS Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point — the argumentation was missing the case where the coverage baseline differs between host and target. I've added a new subsection "Handling Platform-Specific Code" that splits this into three cases:

  1. Linux-only code (mocks, emulation, host utilities): classified as QM/test-support and excluded from the ASIL-B production baseline, so it can't skew the metrics.
  2. QNX-only code inside mixed files (#ifdef __QNX__ blocks): marked with governed coverage-exclusion markers plus a justification that the region is verified on the target.
  3. QNX-only whole files compiled only for QNX: excluded from the host structural-coverage baseline (to avoid false-negative 0%), and treated as a justified deviation with residual risk rather than an equivalence claim, compensated by requirements-based integration tests on the target.

The subsection also explains that the host/target coverage baselines differ by construction and how each delta is reconciled.


This approach satisfies *ISO 26262-6* for *ASIL_B* for the following reasons:

* **Identical Test Results:** Passing 100% of the tests on both the host and the target demonstrates that the control flow under test is identical.
* **Mitigation of Target Instrumentation Risks:** Turning off coverage instrumentation on the target is recommended for embedded systems, as active instrumentation alters the timing behavior, memory footprint, and compiler optimizations on the target. Testing the uninstrumented code on QNX ensures that the actual production binary is verified (mitigating "Heisenbugs").

@limdor limdor Jul 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is true, instrumentation affects behavior, but it could be run, once instrumented to measure coverage and one without instrumentation just to check that the tests pass.

@RolandJentschETAS RolandJentschETAS Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I've reworded the Mitigation of Target Instrumentation Risks bullet to make explicit that the two-step verification is exactly that pattern — measure coverage with instrumentation, then re-run without instrumentation to confirm the tests still pass — but split across the two environments: the host run is instrumented (for structural coverage) and the target run is uninstrumented (to confirm the production binary behaves identically).

The one constraint I called out in the text: qcc cannot produce LLVM source-based coverage, so an instrumented coverage run on the QNX target itself is not available; the instrumented measurement necessarily happens on the host.

* **Equivalence Justification:** The host-based coverage measurement is justified by confirming that no platform-specific code paths (e.g., ``#ifdef QNX`` blocks) are bypassed. Any target-specific hardware abstraction layer (HAL) is verified separately via system-level integration tests.

@limdor limdor Jul 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The host-based coverage measurement is justified by confirming that no platform-specific code paths (e.g., #ifdef QNX blocks) are bypassed.

How should a developer do that? How can a developer find such paths? As mentioned above, it is not only the pragmas, but there could be files that are completly excluded from the linux build.

@RolandJentschETAS RolandJentschETAS Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right that this can't rely on a developer manually grepping for #ifdef guards, since whole files can be excluded from the host build. I've added an "Identifying platform-specific code" subsection describing a concrete, tool-based process:

  • Build-set comparison: the set of files compiled for the host is diffed against the set compiled for QNX; files present only in the QNX build are flagged automatically as the "QNX-only files" case.
  • Explicit exclusion manifest: every host-coverage exclusion (per-file or within-file region marker) is recorded in one reviewable list, so the full delta between the host baseline and the target code base is visible in a single place.
  • CI gate: the comparison and the manifest are enforced in CI, so a new QNX-only file or conditional block without a documented exclusion and a compensating target verification fails the quality gate.

I also added a note that the exclusion-marker syntax must match the actual toolchain: since coverage uses llvm-cov/llvm-profdata, LCOV_EXCL_* markers are not interpreted natively and must not be assumed to work unless an lcov conversion/merge step is explicitly part of the pipeline.

@RolandJentschETAS RolandJentschETAS Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm only not sure, how we bring this to the infrastructure.

Loading