docs: add DR-009-Infra on harmonizing dependable_element with sphinx/… - #3097
docs: add DR-009-Infra on harmonizing dependable_element with sphinx/…#3097antonkri wants to merge 6 commits into
Conversation
Add a proposed decision record describing how the dependable_element concept is interlocked with the existing sphinx/sphinx-needs based S-CORE process: sphinx/sphinx-needs stays the single source of truth, requirements/AoU are exported via TRLC (with a TRLC metamodel check) into dependable_element, and architecture PlantUML files are shared between both worlds. Diagrams are stored as static SVG images under _assets/ so they render both on GitHub and in the Sphinx build.
4c9377a to
e612bbe
Compare
|
The created documentation from the pull request is available at: docu-html |
AlexanderLanin
left a comment
There was a problem hiding this comment.
Thanks a lot for the write-up. I have tried that myself and invested 40+ hours into it. With no tangible outcome, as I was not able to collect benefits of TRLC/Lobster.
| proposes to **harmonize** them so that `dependable_element` becomes the | ||
| *aggregation and traceability layer on top of* the existing sphinx-needs | ||
| toolchain, rather than a competing parallel mechanism. |
There was a problem hiding this comment.
I assume this DR is about introducing dependable_element and about how to integrate it into S-CORE docs-as-code. Not just about harmonizaton.
There was a problem hiding this comment.
In my opinion, the sphinx/sphinx-needs toolchain and dependable_element can be used together, with each having a slightly different focus. sphinx(-needs) focuses on requirements, architecture definition, and traceability more from a documentation perspective. dependable_element, on the other hand, focuses more on providing developer support to ensure, in a qualified manner, that all artifacts — including the traceability report — are valid and still reflect reality. This essentially covers what is currently handled manually through inspections and verification reports.
Some commercial solutions use a similar approach: on one hand, sphinx/sphinx-needs for the visualization of process artifacts, where validity checks are still needed to ensure that the rendered artifacts conform to the process metamodel. On the other hand, lean and easy-to-qualify solutions that produce, in a qualified manner, a traceability report — including additional checks to ensure that the artifacts and the traceability report still correspond to the reality in the source code.
| technical solutions exhibit **instability**, and the overall **degree of | ||
| automation can still be increased significantly**: | ||
|
|
||
| - support for detailed design is completely missing |
There was a problem hiding this comment.
Contradiction: all detail design requirements are fully supported in S-CORE docs-as-code. Note however that there is almost no formal detail design documentation required by S-CORE process.
There was a problem hiding this comment.
Indeed, there are currently not many requirements for detailed design in the s-core process documentation (and this is not fault of docs-as-code tooling). However, I would consider this a pitfall rather than a good design decision. dependable_element already provides a definition of a unit (which is completely missing in s-core) as well as a number of checks — for example, verifying that unit APIs correspond to the architecture diagram — that we could simply reuse.
| - consistency checks between component and detailed design diagrams versus the | ||
| real dependencies in the build system and the structure in the source code | ||
| are missing |
There was a problem hiding this comment.
Note that we typically don't expect many detailed design diagrams. See S-CORE process documentation.
There was a problem hiding this comment.
I guess, this is already covered by my comment above. I would consider this as a pitfall rather than a good design decision.
| - sphinx / sphinx-needs does not allow, at least the way it is set up right | ||
| now, to specify accurate dependencies between requirements ↔ architecture ↔ | ||
| source code ↔ tests. It treats everything as one big folder, where every |
There was a problem hiding this comment.
"accurate dependencies" are provided by S-CORE docs-as-code. That's normal traceability, isn't it?!
There was a problem hiding this comment.
docs-as-code relies on sphinx. sphinx is not aware of our build system. It has its own workspace and this comes with some disadvantages, as you have to copy or mirror the whole data into sphinx workspace:
- sphinx doesn't always know what has changed, if you want to be sure you are using newest data, you have to do a clean build. This takes time. This may be ok for automated CI workflows, but not for normal developer experience
- sphinx doesn't know how you build your software, with which compiler flags, linker flags and so on. So there is no 100% guarantee you sphinx build is taking the same config, that is e.g. used for the final target build.
- It can be difficult in the future to address variants. E.g. your modules can have different variants (= feature flags), that you can activate or deactivate. Then you will need to be very careful what are you copying or what you are not copying. "Just copying" some data from bazel output folder doesn't work. Bazel knows what belongs together, you not.
I do not say it doesn't work with sphinx(-needs), you can get this working. But I am not sure, that this is the right tool for this task.
| real dependencies in the build system and the structure in the source code | ||
| are missing | ||
| - any kind of dependencies between tests / test executions and the generated | ||
| test reports in sphinx-needs are missing |
There was a problem hiding this comment.
full traceability is provided by S-CORE docs-as-code. See HowTo: Reference Docs in Tests
There was a problem hiding this comment.
I guess, I've addressed this already in my previous comments
| - the framework that most of the current process automation is built on — | ||
| **sphinx and sphinx-needs** — **cannot be properly qualified** for use in a | ||
| safety-critical context. It is a large, dynamically extensible documentation |
There was a problem hiding this comment.
To my knowledge we had this discussion already, and clarified that it is and it can be used in safety-critical context. This paragraph needs a little more clarity with correct wording by safety experts.
There was a problem hiding this comment.
I would agree an this statement because of https://safety.useblocks.com/index.html. But we can always discuss potential concern in the process community meetings.
There was a problem hiding this comment.
To my understanding, as written here https://safety.useblocks.com/usage/qualification.html#qualification-method, a second lean implementation is used to verify the output of sphinx/sphinx-needs. I would say, the proposed solution in this DR goes exactly in the same direction with the only difference, that we do not use needs.json but lobster.json for traceability documentation.
I guess, the problem is not to qualify all the tooling around sphinx-needs and metamodel.yaml we've built in score. That should work well. The problem is that needs.json, that we check, is produced by sphinx - a huge large framework with a lot of additional packages and config options you can use. To qualify sphinx can be very challenging.
If you guys know someone, who has done qualification of sphinx without building additional tooling to parse sphinx-needs in rst files separately from sphinx, be my guest.
| code → tests). This makes the relationships first-class, machine-checkable | ||
| edges instead of implicit links inside a documentation folder, and enables | ||
| incremental, cached re-evaluation of only the affected artifacts. |
There was a problem hiding this comment.
Current relationships are also "first-class, machine-checkable edges". Please clarify the difference and why it is an improvement.
There was a problem hiding this comment.
explained here: #3097 (comment)
You do copy/link things from bazel build folder to sphinx workspace "manually", at least for verification report. I would not call it "first-class, machine-checkable edges".
| - **Deterministic, hermetic checks.** Consistency and traceability checks run | ||
| as ordinary build actions and are therefore reproducible and qualifiable, | ||
| rather than living inside dynamic documentation plugins. |
There was a problem hiding this comment.
Current checks also run "deterministic and hermetic". Please clarify the difference and why it is an improvement.
There was a problem hiding this comment.
To my understanding, sphinx has its own logic for resolution of links and dependencies. So we have like one "build-system" running in context of another. I am not sure it is deterministic and hermetic.
| - **LOBSTER traceability aggregation.** The typed requirement and test | ||
| artifacts are converted to LOBSTER and combined into a per-element | ||
| traceability report, with the tracing tiers (component → feature → | ||
| stakeholder / assumed-system) enforced by the tooling. |
There was a problem hiding this comment.
Same as S-CORE docs-as-code, so this doesn't sound like a difference. Please clarify how this differs.
There was a problem hiding this comment.
In case of docs-as-code, we have one central needs.json, as a result, the checks can be done only on this one needs.json. It means if you changes smth. in detailed design diagram and wants to know, whether this is still compliant with the implementation, then you will need to generate the whole needs.json and for this you will to build all the targets, e.g. verification report and for this to execute test and so on and so on.
In case of lobster, every bazel macro produces valid lobster.json reports, that can be used for validation. It means, to check that your detailed design is still valid, you will need to build only the lobster.json for detailed design. Only later, in the dependable_element, all lobster files are merged together in one for the final traceability check. That's the difference.
| - **Aggregated, self-contained deliverable.** `dependable_element` collects all | ||
| artifacts of a *Safety Element out of Context (SEooC)* — requirements, | ||
| architecture, assumptions of use, dependability analysis, components, tests, | ||
| checklists and glossary — into a single, versioned deliverable with its own | ||
| HTML documentation and traceability report. |
There was a problem hiding this comment.
Same as S-CORE docs-as-code, so this doesn't sound like a difference. Please clarify how this differs.
There was a problem hiding this comment.
yes, here at the end we get a complete lobster.json file, that is comparable with need.json. For the difference, please see the comment before.
|
In my opinion this makes no sense to even start defining. Older meeting minutes here: eclipse-score/communication#545 (comment) This meeting has not yet happened even after numerus attempts. I can not work on something that I do not know the requirements of, simple as that. |
PandaeDo
left a comment
There was a problem hiding this comment.
Thank you for your perspective. But I would not agree from my experiences to your concerns against the actual solution, especially regarding qualification.
| the actual C++ sources (via a libclang toolchain) and validate the declared | ||
| architecture diagrams against the real dependency and code structure, | ||
| closing a gap that pure sphinx-needs cannot cover. | ||
| - **LOBSTER traceability aggregation.** The typed requirement and test |
There was a problem hiding this comment.
I'm not a lawyer. But I guess we need to check if there is no licence conflict.
| The **merging / interlocking of these two approaches** — sphinx-needs as the | ||
| low-friction authoring surface and single source of truth, `dependable_element` | ||
| as the build-anchored automation and traceability layer on top of it — is | ||
| expected to **drastically accelerate the roll-out of the S-CORE process into |
There was a problem hiding this comment.
Would not confirm this statement from the experiences we gained.
There was a problem hiding this comment.
can you please be more concrete?
| component requirements, architecture, assumptions of use, safety analyses, | ||
| checklists — as **Sphinx-Needs** directives (`.. feat_req::`, `.. comp_req::`, | ||
| …) inside `.rst`/`.md` sources. The build emits a project-wide `needs.json`, | ||
| which serves two purposes: it is used both to **check that the need elements |
There was a problem hiding this comment.
Needs.json is only an output that can be dependency as an input to other docuementation builds.
The check that it is conform is done inside the sphinx-build and the needs.json might be part of that if it comes from an external source, but the internal local one does not play a role.
There was a problem hiding this comment.
I agree and I don't see this as controversial to what I've written.
| assumptions of use) from them, and convert those to TRLC on the fly. As a | ||
| side benefit, the generated files are then also checked against the TRLC | ||
| metamodel, adding a second, independent validation layer on top of the | ||
| sphinx-needs metamodel check. |
There was a problem hiding this comment.
So we then have to upkeep and keep in sync and verify two different metamodels?
There was a problem hiding this comment.
yes, true, that would be the consequence
There was a problem hiding this comment.
Sounds like overengineering. I do not understand the benefit for introduction such a complex solution.
| - **Explicit, fine-grained dependencies.** Artifacts are wired together through | ||
| Bazel dependencies (requirements → architecture → components/units → source | ||
| code → tests). This makes the relationships first-class, machine-checkable | ||
| edges instead of implicit links inside a documentation folder, and enables | ||
| incremental, cached re-evaluation of only the affected artifacts. |
There was a problem hiding this comment.
What exactly is the difference here, compared to current implementation?
Why does the current implementation not work / provide what is needed?
There was a problem hiding this comment.
I guess, I've explained this in my previous comments:
| - **Architecture-vs-reality validation.** The component and unit rules parse | ||
| the actual C++ sources (via a libclang toolchain) and validate the declared | ||
| architecture diagrams against the real dependency and code structure, | ||
| closing a gap that pure sphinx-needs cannot cover. |
There was a problem hiding this comment.
What about python and rust code, there this is not doable and therefore an issue?
Why is this a gap in the current implementation, was this ever a requirement to counter check the actual implementation against the needs you have defined?
IMO that should be part of the review process.
There was a problem hiding this comment.
- Yes, we will need to extend it for Rust.
- Review process is a manual step, it is better, if you have this automated.
| source code ↔ tests. It treats everything as one big folder, where every | ||
| change forces everything to be regenerated |
There was a problem hiding this comment.
| source code ↔ tests. It treats everything as one big folder, where every | |
| change forces everything to be regenerated | |
| source code ↔ tests. | |
| - It treats everything as one big folder, where every | |
| change forces everything to be regenerated |
This seems to be two arguments mashed into one bullet point.
There was a problem hiding this comment.
yes, will need to fix it, if the decision will be taken to merge this PR
| behaviour depends on a broad set of third-party Python extensions and | ||
| configuration that can change the output in ways that are difficult to | ||
| constrain and reproduce, and the validation logic (both the metamodel and | ||
| the traceability checks) lives inside these dynamic extensions rather than in |
There was a problem hiding this comment.
We don't need to qualify sphinx or sphinx-needs. We need to qualify score_docs_as_code (which internally uses sphinx and sphinx-needs).
|
|
||
| ### Non-Goals | ||
|
|
||
| ## Options Considered |
There was a problem hiding this comment.
I see two options to consider:
- integrate TLRC and LOBSTER
- extend score_docs_as_code
There was a problem hiding this comment.
I've kept this empty intentionally, I just wanted to have the discussion first.
There was a problem hiding this comment.
We cant really have a DR without options?
There was a problem hiding this comment.
The two options seem pretty obvious to me:
- use TLRC
- do not use TLRC and implement whatever is missing with Sphinx-Needs
Most of the description above should be moved into the TLRC option here.
There was a problem hiding this comment.
I introduce two options, but mainly I've provided the description of the goals for this DR in this commit: c1b488f
I hope, it makes it clearer.
|  | ||
|
|
||
| ### Goals and Requirements | ||
|
|
There was a problem hiding this comment.
My understanding after the Infra meeting what this proposal aims for:
| - easy tool qualification | |
| - enable unit design and traceability around it | |
| - provide traceability report | |
| ## Options Considered | ||
|
|
||
| ## Evaluation | ||
|
|
There was a problem hiding this comment.
As a note: build speed should be an evaluation criterium
Add a mermaid classDiagram to DR-009-infra breaking dependable_element down into the individual rules_score rules, showing per-rule checks, the artifacts each rule contributes to the aggregated lobster.json, and the macro argument on each edge.
Agree, that's really a pity, that the meeting has not happened. This DR is an attempt to refresh the discussion. |
0808f93 to
8bf0361
Compare
| report (empty when the rule adds nothing). Each arrow is labelled with the | ||
| macro argument that wires the child rule into its parent. | ||
|
|
||
| ```mermaid |
There was a problem hiding this comment.
| ```mermaid | |
| ```{mermaid} |
This should fix the Sphinx build.
| labelled with the kind of check (per element, or graph over the whole model) or | ||
| the relationship to the metamodel. | ||
|
|
||
| ```mermaid |
There was a problem hiding this comment.
| ```mermaid | |
| ```{mermaid} |
This should fix the Sphinx build.
| S-CORE currently documents its process artifacts — stakeholder, feature and | ||
| component requirements, architecture, assumptions of use, safety analyses, | ||
| checklists — as **Sphinx-Needs** directives (`.. feat_req::`, `.. comp_req::`, | ||
| …) inside `.rst`/`.md` sources. The build emits a project-wide `needs.json`, |
There was a problem hiding this comment.
"The build emits" maybe better "The documentation build emits"
| sphinx / sphinx-needs on the basis of `needs.json`. | ||
|
|
||
| In parallel, a newer set of Bazel rules (`rules_score`) introduces the | ||
| **`dependable_element`** concept: a macro that aggregates all safety-relevant |
There was a problem hiding this comment.
Is this really a "macro" ??? Better a "bazel command" ?
| artifacts of a *Safety Element out of Context (SEooC)* into one deliverable — | ||
| requirements, architectural design, assumptions of use, dependability | ||
| analysis, components, tests, checklists and glossary — and produces both a | ||
| consolidated LOBSTER traceability report and a self-contained Sphinx HTML |
There was a problem hiding this comment.
It is not defined and linked here, what LOBSTER means and why this has any advantage against the current sphinx needs process.
There was a problem hiding this comment.
Lobster is a tool for traceability report: https://github.com/bmw-software-engineering/lobster
It is also explained here: https://github.com/eclipse-score/tooling/tree/main/bazel/rules/rules_score
|
|
||
| Today these two worlds overlap but are not formally reconciled: the process | ||
| description assumes a "flat" sphinx-needs documentation, while | ||
| `dependable_element` imposes a typed, per-element aggregation model. This DR |
There was a problem hiding this comment.
Isn't the dependable element already specified in the process ?
There was a problem hiding this comment.
To my knowledge it is not defined in the process but the tooling was already introduced: https://github.com/eclipse-score/tooling/tree/main/bazel/rules/rules_score
| automation can still be increased significantly**: | ||
|
|
||
| - support for detailed design is completely missing | ||
| - consistency checks between component and detailed design diagrams versus the |
There was a problem hiding this comment.
This is done by checklists and is not required by the ISO for ASIL B. So we had removed sphinx needs for unit to avoid slow down of development by process overhead and the code first approach.
There was a problem hiding this comment.
Yes, but I guess we all agree, that an automation is better as checklists, especially in open source project, where liability for manually created checklists is 0.0
| source code ↔ tests. It treats everything as one big folder, where every | ||
| change forces everything to be regenerated | ||
| - the framework that most of the current process automation is built on — | ||
| **sphinx and sphinx-needs** — **cannot be properly qualified** for use in a |
There was a problem hiding this comment.
It is already qualified and used in safety critical context. What kind of qualification is missing ? Therefore we have a tool classification.
There was a problem hiding this comment.
Currently I can speak for only one commercial solution, that I am aware about, and this is the one from use-blocks. For me, use-blocks is the company, that has exhaustive experience and understanding of how sphinx/sphinx-needs works. Based on their documentation, that is available on web, they use different lightweight tooling to parse rst files and to compare final result with what sphinx/sphinx-needs produces. So you do not qualify sphinx, you qualify another tool and then use this tool to ensure, that the output of sphinx is correct. This approach is comparable to what we purpose with this DR.
| **sphinx and sphinx-needs** — **cannot be properly qualified** for use in a | ||
| safety-critical context. It is a large, dynamically extensible documentation | ||
| toolchain that was not designed with tool qualification in mind: its | ||
| behaviour depends on a broad set of third-party Python extensions and |
There was a problem hiding this comment.
What makes it better with introduction of "lobster", which is only a more complex third party extension ? Sounds for me like "to cast out devils by Beelzebub".
There was a problem hiding this comment.
...aaa @RolandJentschETAS , I would disagree. And I think that's basically the main source of misunderstanding here. It is not about tooling. I would agree that everything you can do with dependable_element can be done with Sphinx and vice versa. Therefore, also all those comparison tables do not make much sense. It is about how you develop your software. The Sphinx approach we currently have is going more in the classic direction of software development in automotive, how we have it right now, e.g., with CodeBeamer. There is a big monolithic tooling that is not connected to the real build system or code at all. You fill in requirements and architecture and then we have tons of checklists that we need to fill manually. And if you want requirements and architecture being connected with source code, then buy commercial extensions and some IDE. Welcome back to the 20th century.
The creators of Lobster and dependable_element believe that requirements/architecture definition, code implementation, and testing should go hand in hand. The assumption is that as few manual checks as possible should be done — ideally zero. All should be checked automatically. Scripts for automated checks can be reviewed/evaluated later by the commercial distributor and reused. Manual reviews are for the bin.
| The recently introduced approach named **`dependable_element`** provides | ||
| exactly what sphinx/sphinx-needs is missing: a **high degree of automation | ||
| anchored directly in the build system** (Bazel), where work products are | ||
| declared as typed rules, their relationships are checked deterministically, and |
There was a problem hiding this comment.
What means "deterministically" here ?
| At a high level, `dependable_element` works as follows: | ||
|
|
||
| - **Typed work-product rules.** Each process artifact has its own Bazel rule | ||
| (`feature_requirements`, `component_requirements`, |
There was a problem hiding this comment.
Sounds for me like an artificial meta-meta-model.
There was a problem hiding this comment.
Not a meta-meta-model, but implementation/support of the score metamodel in bazel, ensuring every module defined its components , requirements and architecture in the same manner using the same tooling.
| the same two reasons: | ||
|
|
||
| - **Fast turn-around times.** As shown by | ||
| [useblocks ubTrace](https://useblocks.com/products/ubtrace), the traceability |
There was a problem hiding this comment.
Sounds like promotion and support for useblocks. So we add here an dependency to an external commercial company ?
There was a problem hiding this comment.
Oh no, not at all. That's only the attempt to show, that also commercial solutions are using comparable approach as proposed by this DR.
| deterministic component is far easier to qualify for use in a safety context. | ||
|
|
||
| In other words, these commercial solutions rely on an architecture comparable to | ||
| `dependable_element` / LOBSTER: a separate, lightweight, easy-to-qualify |
There was a problem hiding this comment.
Thats marketing ...
There was a problem hiding this comment.
same as above, not marketing, use-blocks is used here as reference.
c1b488f to
8ccf48d
Compare


Add a proposed decision record describing how the dependable_element concept is interlocked with the existing sphinx/sphinx-needs based S-CORE process: sphinx/sphinx-needs stays the single source of truth, requirements/AoU are exported via TRLC (with a TRLC metamodel check) into dependable_element, and architecture PlantUML files are shared between both worlds. Includes overview and merge diagrams.