diff --git a/docs/contribute/development/cpp/code_analysis.rst b/docs/contribute/development/cpp/code_analysis.rst index 0a9e879a137..637bfec87f0 100644 --- a/docs/contribute/development/cpp/code_analysis.rst +++ b/docs/contribute/development/cpp/code_analysis.rst @@ -102,13 +102,19 @@ Adress/ Leak Sanitizer (ASAN/LSAN) If both tools are combined at runtime memory leaks and the corresponding address can be investigated. -Coverage -======== +Code Coverage +============= -As required by the verification guideline coverage needs to be calculated for the code which is used in the project. Therefore two approaches should be available: +As required by the verification guideline code coverage needs to be calculated for the code which is used in the project. Coverage is calculated on the host using LLVM's source-based coverage: -* As a quick solution it is possible to calculate the coverage on the host via gcc. -* But for a more accurate statement coverage can also be calculated with the qcc compiler with the appropriate libraries and POSIX interfaces. This method will also be used for the reporting. +* Coverage is calculated on the host via clang/llvm. This method is also used for the reporting. + +Since ``qcc`` does not support LLVM's source-based coverage instrumentation, coverage is not collected on the QNX target. LLVM's source-based coverage natively supports MC/DC, which is required for the higher ASIL levels. + +LLVM's source-based coverage is preferred over ``gcov``-based coverage for the following reasons: + +* **Precision with templates, generics and modern C++:** ``gcov`` is line-based, so with heavy templating, inlining and macros the mapping is coarse and multiple template instantiations collapse onto the same lines, producing imprecise or misleading results. LLVM's source-based coverage is region- and instantiation-based and therefore significantly more accurate for modern C++. +* **MC/DC support:** ``gcov`` historically provided no MC/DC support (GCC offers it only from GCC 14 via ``-fcondition-coverage``), whereas LLVM/clang supports MC/DC natively (``-fcoverage-mcdc``), which is required for the higher ASIL levels. To enable this, following tools are used: @@ -116,11 +122,68 @@ To enable this, following tools are used: object "Coverage" as coverage object "gtest" as gtest - object "gcov + gcovr" as gcov + object "llvm-cov + llvm-profdata" as llvm object "host" as host - object "QNX" as qnx coverage --> gtest - gtest --> gcov - gcov --> host - gcov --> qnx + gtest --> llvm + llvm --> host + +Argumentation: Host-based Coverage with Target Test Execution +------------------------------------------------------------- + +Measuring code coverage exclusively on the **Linux host** (via LLVM) is considered sufficient for safety certification, provided that a **two-step verification** is performed to demonstrate equivalence on the target: + +#. **Quantitative Verification (Host):** The complete test suite is executed on the Linux host with code coverage enabled. This demonstrates that the test cases are structurally complete (100% C0/C1 coverage) and that no dead code exists. +#. **Qualitative Verification (Target):** The identical test suite is executed on the **QNX target**, but with code coverage instrumentation turned off. This demonstrates that the software compiles, links, and behaves identically on the real target hardware (correctness of execution, no compiler/linker optimization bugs, no endianness or memory alignment issues). + +This approach satisfies *ISO 26262-6* for *ASIL_B* for the following reasons: + +* **Identical Test Results:** Passing 100% of the tests on both the host and the target demonstrates that the control flow under test is identical. +* **Mitigation of Target Instrumentation Risks:** The two-step verification deliberately splits the two concerns onto the environment best suited for each: the **host run is instrumented** (to measure structural coverage), and the **target run is uninstrumented** (to confirm the tests still pass on the real production binary). This is the same "measure coverage with instrumentation, then re-run without instrumentation to confirm behaviour" pattern applied across two environments. Running uninstrumented on QNX matters because active instrumentation alters timing behaviour, memory footprint and compiler optimizations on embedded targets (mitigating "Heisenbugs"). Note that ``qcc`` cannot produce LLVM source-based coverage, so an *instrumented* coverage run on the target is not an available option. +* **Equivalence Justification:** The host-based coverage measurement is justified by confirming that no platform-specific code paths (e.g., ``#ifdef QNX`` blocks) or QNX-only source files are silently bypassed by the host measurement. The reconciliation strategy for such code, and the process a developer uses to identify it, is described in `Handling Platform-Specific Code`_ below. Any target-specific hardware abstraction layer (HAL) is additionally verified via system-level integration tests. + +Handling Platform-Specific Code +------------------------------- + +Because structural coverage is measured on the **Linux host** while the QNX target is only tested for pass/fail, the coverage *baseline* (the set of source files and lines expected to be covered) differs between the two environments. Code that is strictly platform-specific therefore needs an explicit reconciliation strategy so that it neither skews the release metrics nor becomes unverified "dead" code in the safety case. + +Three cases are distinguished. + +Linux-only code (mocks, emulation, host utilities) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This code is compiled and executed only on the host and is absent on the QNX target. It must not contribute to the ASIL-B production coverage figures. + +* Linux-only helper code and mocks are kept structurally separate from production code (e.g. under dedicated ``mock/`` or ``test/`` directories) and are classified as Quality Managed (QM) / test-support code. +* Such directories are excluded when the coverage report is generated, so this code is not part of the ASIL-B production coverage baseline and cannot inflate or deflate the reported figures. + +QNX-only code inside mixed files (conditional compilation) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For files that are compiled on the host but contain small target-specific blocks (e.g. guarded by ``#ifdef __QNX__``), those blocks are not compiled into the host binary and would otherwise appear as non-covered or non-existent regions. + +* These target-only regions are marked with standardized coverage-exclusion markers understood by the coverage report generator, together with a justification stating that the region is executed and verified on the QNX target rather than on the host. +* The applicable exclusion markers and their justification format are governed centrally (rather than chosen per developer) so that exclusions remain auditable and consistent across repositories. + +.. note:: + + The exclusion-marker syntax must match the coverage tooling actually in use. Because coverage in S-CORE is produced by **LLVM source-based coverage** (``llvm-cov`` / ``llvm-profdata``), the marker mechanism supported by that toolchain (line/region exclusion) must be used; ``lcov``/``gcov`` ``LCOV_EXCL_*`` markers are *not* interpreted natively by ``llvm-cov`` and must not be assumed to work unless an ``lcov`` conversion/merge step is explicitly part of the pipeline. + +QNX-only files excluded from the host build +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some modules or files are compiled **only** for QNX and cannot be built on the host at all (e.g. sources that call QNX-specific OS APIs). These would report 0% coverage in a host-only measurement. + +* Such files are excluded from the **host structural-coverage baseline**, so they do not produce misleading false-negative 0% figures. +* Because ``qcc`` cannot produce LLVM source-based coverage, **no structural coverage can be obtained for these files**. This is a *justified deviation* from the unit-level structural-coverage objective, not a claim of equivalence: the residual risk is that unit-level statement/branch coverage evidence does not exist for this code. +* The deviation is compensated by verifying these files **on the QNX target** through requirements-based integration tests executed on the real hardware, and by arguing the safety case on the basis of a 100% pass rate of the associated functional requirements. Each such file/module and its compensating verification is recorded so the deviation is traceable and reviewable. + +Identifying platform-specific code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A developer must be able to determine which code falls into the categories above; this cannot rely on manually reading source for ``#ifdef`` guards, because whole files may be excluded from the host build. The following mechanisms are used: + +* **Build-set comparison:** The set of source files compiled for the host is compared against the set compiled for the QNX target. Files present only in the QNX build are, by construction, the "QNX-only files" case and are flagged automatically. +* **Explicit exclusion manifest:** Every host-coverage exclusion (whether a per-file exclusion or a within-file region marker) is recorded in a reviewable list, so the complete delta between the host coverage baseline and the full target code base is visible in one place rather than scattered across the sources. +* **CI gate:** The comparison and the manifest are checked in CI, so newly added QNX-only files or new conditional blocks that are not accompanied by a documented exclusion and a compensating target verification cause the quality gate to fail.