docs: Add AI SDLC / SpecKit tooling evaluation comparison matrix (Iss…

[aryansingh0012]

Add AI SDLC tooling comparison matrix for Issue #3115

[github-actions]

The created documentation from the pull request is available at: docu-html

[masc2023]

Check existing DRs, and template, this DR does not follow the required rules, please update
https://eclipse-score.github.io/score/main/design_decisions/index.html

https://eclipse-score.github.io/process_description//main/folder_templates/platform/docs/change/decision_record.html

[FScholPer]

Naming Convention: https://github.com/eclipse-score/score/blob/main/docs/design_decisions/index.rst

[aryansingh0012]

Added two new decision records under design_decisions and verified the docs build in WSL with bazel run //:docs.

[dirkmacke]

Do I understand DR-10 correctly, that the decision (or recommendation) is based solely on an agent's summary of the tools' repos and no real-world experience with these tools on the S-CORE codebase, or on how they scale with a rather large community and high-frequency changes like S-CORE? I might be misinterpreting the "evidence" section, though, hence the question.

[praveen-ltts]

Do I understand DR-10 correctly, that the decision (or recommendation) is based solely on an agent's summary of the tools' repos and no real-world experience with these tools on the S-CORE codebase, or on how they scale with a rather large community and high-frequency changes like S-CORE? I might be misinterpreting the "evidence" section, though, hence the question.

@dirkmacke , a good study has been done on these tools to be part of SCORE ecosystem and evaluated which of them is more suitable. Also, we have created an EPIC to add these tools in SCORE ecosystem.

@FScholPer , please add more information if I missed something.

[dirkmacke]

@dirkmacke , a good study has been done on these tools to be part of SCORE ecosystem and evaluated which of them is more suitable. Also, we have created an EPIC to add these tools in SCORE ecosystem.

No doubts about that, but did you use/test-drive them, too?

I did spent quite some time with test-driving Openspec (BTW, I was surprised not to see it on the list) and did some evaluation on BMAD as well. Goal was a PoC for series SW development - no safety requirements/pure PoC, but we wanted to see, how SDD scales for concurrent development (multiple devs/agents working in parallel on a design base). Our findings in a nutshell: It does not plus getting Openspec to adhere even to its own very simple SDLC process requires constant supervision and/or supporting tooling. Both in plain scripting and agent/skill plumbing.

Key reasons for this: All the SDD tools do not forsee loopbacks within an ongoing "feature" (or "change", as Openspec calls it), so after you have your tasks, the agent implements. And iterates, And iterates, And iterates. If you still have implemented what was originally spec'd, you'll need to check, since the process/tool will not loop back to check if all the changes done/encountered during implementation and testing are still in line with what was initially spec'd. You'll also not be looped back to spec should missing requirements, e.g. for non-functional testing, being discovered in a later phase of the process. You'll have to notice that and force them. At one point I had an agent so unhappy with doing that, that his recommendation was to re-write Openspec in order not to get the spec fixed and - of course - to not start the whole process again (check that plan & tasks are in line, check that implementation is still in line with the updated tasks etc.) which his own skills should have forced him to do - but they didn't get used, because at least in Openspec there's no plumbing in the agents for that.

I have still very limited understanding of the sphinx/sphinx-needs toolchain, but just using Openspec or Speckit most likely won't work as intended, since you'll basically need to fork the framework to replace the default process and artifacts with the S-CORE ones. Otherwise you'll end up with a lot of additional i.e. Speckit artifacts just to create the S-CORE artifacts, like DRs or test specs etc. to ultimately at some point create some code (and you'll have some unaligned artifacts for this, too).

Maybe this is what is intended (=writing a "S-CORE Speckit") and I simply didn't read it out of the DRs and the associated PR.

Thanks for reading until here anyways 😉

[qor-lb]

In this case I would wait with the DR until this PoC is setup. We should only merge decisions where we know that the implementation is technically feasible.

[aryansingh0012]

Hi @qor-lb ,
APM is pinned and supported tool for s-core DevContainer ecosystem.
Created separate pr #3188

[qor-lb]

I am missing the context/motivation this DR is based on. What problem do we try to solve here?

[aryansingh0012]

Thanks for the feedback. The motivation behind this DR is:
Evaluate existing AI SDLC frameworks for their suitability within the S-CORE project.
Identify a workflow that supports end-to-end traceability (Requirements → Specifications → ADRs → Code → Tests → Documentation).
Ensure the workflow is compatible with S-CORE engineering goals, including ASPICE alignment, governance, reproducibility, and long-term maintainability.
Compare the strengths and limitations of candidate frameworks (SpecKit, Syspilot, BMAD, and Pharaoh).