Log workflow output by erlenlh · Pull Request #14301 · equinor/ert

erlenlh · 2026-08-27T07:18:05Z

Issue
Part of #13320
Closes #14308

Approach
First of two PRs splitting the original "Store workflow output" work. This one
covers capturing workflow job output and writing it to the ERT log; the
follow-up (#14062) adds the
WorkflowEvent and persists events to experiment storage.

Workflow job output previously only reached the terminal ERT was started from,
so it was lost once that terminal was gone, and a job run outside a terminal
left no record at all.

Replaces sys.stdout/sys.stderr with a thread-aware _CaptureProxy while a
job runs, recording what the calling thread writes while still forwarding it
to the real stream. Concurrently running jobs do not pick up each other's
output.
Adds WorkflowJobResult, recording name, index, arguments, output, status and
timestamp per invocation. workflowReport() is keyed by job name, so a
workflow running the same job twice previously kept only the last invocation's
output.
Names workflows after their LOAD_WORKFLOW alias, so reporting can use the
name users recognise rather than the source file path.
Writes one log entry per job invocation, naming the hook, workflow and job.
Covers hooked workflows, ert workflow and the GUI Run workflow tool, since
all three go through WorkflowRunner.

Also fixes two latent bugs found along the way: the running flag was not cleared
when a job was rejected for bad arguments, and a job that stopped the workflow
had its output dropped before it could be recorded.

PR title captures the intent of the changes, and is fitting for release notes.
Added appropriate release note label
Commit history is consistent and clean, in line with the contribution guidelines.
Make sure unit tests pass locally after every commit (git rebase -i main --exec 'just rapid-tests')

When applicable

When screenshots are changed: Review screenshot-PR in ert-testdata,
merge screenshot-PR in ert-testdata before merging this PR.
When there are user facing changes: Updated documentation
New behavior or changes to existing untested code: Ensured that unit tests are added (See Ground Rules).
Large PR: Prepare changes in small commits for more convenient review
Bug fix: Add regression test for the bug
Bug fix: Add backport label to latest release (format: 'backport release-branch-name')

codecov-commenter · 2026-08-27T07:24:54Z

Codecov Report

❌ Patch coverage is 98.36066% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.20%. Comparing base (074eb57) to head (da83710).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/ert/config/_capture_output.py	96.05%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #14301      +/-   ##
==========================================
+ Coverage   91.17%   91.20%   +0.03%     
==========================================
  Files         495      496       +1     
  Lines       36095    36244     +149     
==========================================
+ Hits        32908    33055     +147     
- Misses       3187     3189       +2

Flag	Coverage Δ
cli-tests	`35.92% <89.61%> (+0.22%)`	⬆️
fuzz	`43.74% <37.15%> (+<0.01%)`	⬆️
gui-tests	`58.62% <83.60%> (+0.11%)`	⬆️
performance-and-unit-tests	`81.13% <98.36%> (+0.07%)`	⬆️
test	`45.60% <81.42%> (+0.16%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/ert/config/ert_config.py	`95.95% <ø> (ø)`
src/ert/config/ert_script.py	`88.65% <100.00%> (+1.35%)`	⬆️
src/ert/config/external_ert_script.py	`100.00% <100.00%> (ø)`
src/ert/config/workflow.py	`98.52% <100.00%> (+0.19%)`	⬆️
src/ert/run_models/run_model.py	`92.48% <ø> (ø)`
src/ert/workflow_runner.py	`94.91% <100.00%> (+2.85%)`	⬆️
src/ert/config/_capture_output.py	`96.05% <96.05%> (ø)`

xjules · 2026-08-27T08:08:30Z

+    @contextlib.contextmanager
+    def capture(self) -> Iterator[io.StringIO]:
+        """Record what the calling thread writes for the duration of the block."""
+        buffer = io.StringIO()


shouldn't all of this be in the try block rather than only the yield?

not sure i understand. The append should happen before, such that the remove in finally has something to remove. Remove should be in finally so that we know it happens. Follows pattern from contextmanager docs

I meant more like if this fails buffers = self._buffers, but I guess then we don't want to remove it from the list.
So it might be good 👍

codspeed-hq · 2026-08-27T10:21:06Z

Merging this PR will not alter performance

✅ 36 untouched benchmarks

_{Comparing log-workflow-output (da83710) with main (074eb57)}

xjules · 2026-09-01T07:37:19Z

+    CANCELLED = "cancelled"
+
+
+def _job_description(


This can't be a WorkflowJobResult member / function?

i think putting it on result is wrong, since it is used before we actually run the runner.

Maybe we can put it on the runner? agree that its weird to be free

xjules

It looks already very good!
Had only a smaller comments.
Also, there are way to many articles in the test names: remove the, a and an

xjules

Nice work @erlenlh ! 🚀

Workflow carried no name of its own, so anything reporting on a workflow had to fall back to the source file path. LOAD_WORKFLOW lets a workflow be given a name, and that is the name users recognise. Store the name on Workflow, defaulting to the file name when none is given, so it can be reported alongside the jobs that were run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

WorkflowJobRunner.run() set the running flag before validating its arguments but only cleared it on the success path, so a job rejected for too few or too many arguments kept reporting that it was still running. workflowReport() is also keyed by job name, so a workflow running the same job more than once kept only the output of the last invocation. Add WorkflowJobResult, recording name, index, arguments, output, status and timestamp per invocation, and expose the ordered list through workflow_job_results(). Status is modelled as a WorkflowJobStatus enum rather than a failed/cancelled boolean pair, since the booleans could express states that do not exist; a job interrupted mid-run is now reported as cancelled instead of failed. Jobs skipped because the workflow was cancelled are recorded too, so a cancelled workflow can be told apart from one that never ran. A job that stops the workflow is recorded before the error is raised, so its output is not lost. workflowReport() is left as it is, since the run workflow tool and the CLI still use it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Internal python workflow jobs wrote straight to sys.stdout/sys.stderr, so their output was visible in the terminal but not available to ERT itself, and was lost once that terminal was gone. External jobs captured their subprocess output but did not pass stdout on to the terminal. Replace sys.stdout/sys.stderr with a proxy while a job runs, recording what the calling thread writes while still forwarding it to the real stream. Capture is per-thread, so concurrently running jobs do not pick up each other's output. The proxy and its context manager live in their own module, since they are self-contained and have nothing to do with ErtScript beyond the single call that uses them. Write one log entry per job invocation to the ERT log, naming the hook, the workflow and the job, and quoting the arguments, stdout and stderr, built on WorkflowRunner so the starting and result lines share one format. This covers hooked workflows, 'ert workflow' and the GUI Run workflow tool, since all three go through WorkflowRunner. Also stop reporting a failing external job with an ERT stack trace: the new ExternalScriptError carries the exit code, since the traceback would only show ERT internals. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Test names introduced in this PR used "a", "an" and "the", which does not follow this repo's naming convention and was flagged repeatedly on the sibling PR #14301. Rename them to read as plain behaviour specifications. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

erlenlh mentioned this pull request Aug 27, 2026

Store workflow output #14062

Open

10 tasks

xjules reviewed Aug 27, 2026

View reviewed changes

Comment thread src/ert/config/ert_script.py Outdated

xjules reviewed Aug 27, 2026

View reviewed changes

Comment thread src/ert/config/ert_script.py Outdated

xjules reviewed Aug 27, 2026

View reviewed changes

Comment thread src/ert/config/ert_script.py Outdated

xjules reviewed Aug 27, 2026

View reviewed changes

Comment thread src/ert/config/ert_script.py Outdated

xjules reviewed Aug 27, 2026

View reviewed changes

Comment thread src/ert/config/ert_script.py Outdated

xjules reviewed Aug 27, 2026

View reviewed changes

Comment thread src/ert/config/workflow.py

xjules reviewed Aug 27, 2026

View reviewed changes

Comment thread src/ert/workflow_runner.py Outdated

berland reviewed Aug 27, 2026

View reviewed changes

Comment thread docs/ert/reference/workflows/complete_workflows.rst Outdated

berland reviewed Aug 27, 2026

View reviewed changes

Comment thread src/ert/workflow_runner.py Outdated

berland reviewed Aug 28, 2026

View reviewed changes

Comment thread src/ert/workflow_runner.py Outdated

berland reviewed Aug 28, 2026

View reviewed changes

Comment thread src/ert/workflow_runner.py Outdated

berland reviewed Aug 28, 2026

View reviewed changes

Comment thread src/ert/workflow_runner.py Outdated

berland reviewed Aug 28, 2026

View reviewed changes

Comment thread src/ert/config/ert_script.py Outdated

berland reviewed Aug 28, 2026

View reviewed changes

Comment thread src/ert/config/ert_script.py Outdated

berland reviewed Aug 28, 2026

View reviewed changes

Comment thread src/ert/config/ert_script.py Outdated

berland reviewed Aug 28, 2026

View reviewed changes

Comment thread src/ert/config/ert_script.py Outdated

xjules reviewed Aug 28, 2026

View reviewed changes

Comment thread src/ert/config/ert_script.py Outdated

xjules reviewed Aug 28, 2026

View reviewed changes

Comment thread tests/ert/unit_tests/cli/test_cli_workflow.py Outdated

xjules reviewed Aug 28, 2026

View reviewed changes

Comment thread tests/ert/unit_tests/config/test_ert_config.py Outdated

xjules reviewed Sep 1, 2026

View reviewed changes

Comment thread src/ert/config/_capture_output.py Outdated

xjules reviewed Sep 1, 2026

View reviewed changes

Comment thread tests/ert/unit_tests/config/test_ert_config.py Outdated

xjules reviewed Sep 1, 2026

View reviewed changes

Comment thread tests/ert/unit_tests/workflow_runner/test_ert_script.py Outdated

xjules reviewed Sep 1, 2026

View reviewed changes

Comment thread tests/ert/unit_tests/workflow_runner/test_ert_script.py Outdated

xjules reviewed Sep 1, 2026

View reviewed changes

Comment thread tests/ert/unit_tests/workflow_runner/test_ert_script.py Outdated

xjules requested changes Sep 1, 2026

View reviewed changes

erlenlh force-pushed the log-workflow-output branch 4 times, most recently from 33034fb to 6b7bf43 Compare September 7, 2026 12:20

xjules approved these changes Sep 7, 2026

View reviewed changes

erlenlh and others added 3 commits September 7, 2026 22:15

erlenlh force-pushed the log-workflow-output branch from 6b7bf43 to da83710 Compare September 7, 2026 20:23

erlenlh merged commit d8d93af into main Sep 8, 2026
42 checks passed

erlenlh deleted the log-workflow-output branch September 8, 2026 07:53

Uh oh!

Conversation

erlenlh commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

When applicable

Uh oh!

codecov-commenter commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

xjules Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

erlenlh Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

xjules Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codspeed-hq Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xjules Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

erlenlh Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xjules left a comment

Choose a reason for hiding this comment

Uh oh!

xjules left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

erlenlh commented Aug 27, 2026 •

edited

Loading

codecov-commenter commented Aug 27, 2026 •

edited

Loading

codspeed-hq Bot commented Aug 27, 2026 •

edited

Loading