Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
7a832d8
Fix typos in changelogs
ArBridgeman Jul 9, 2026
c37d96c
Make permissions block present in workflows that call the custom work…
ArBridgeman Jul 9, 2026
5d7f0ed
Add documentation for new feature
ArBridgeman Jul 9, 2026
620fde3
Add jinja test
ArBridgeman Jul 9, 2026
24cfb7b
Prepare config test
ArBridgeman Jul 9, 2026
bea075f
Add extract_permissions
ArBridgeman Jul 9, 2026
92a1ae4
Remove not-maintained workflows from get_workflow_templates - need to…
ArBridgeman Jul 10, 2026
c3d2872
Remove now unneeded custom exception as not maintained workflows not …
ArBridgeman Jul 10, 2026
0a36b86
Remove redundant check
ArBridgeman Jul 10, 2026
1ac5e10
Ensure NOT_MAINTAINED_WORKFLOWS are seeded first
ArBridgeman Jul 10, 2026
69e8d14
Tighten wording in documentation
ArBridgeman Jul 10, 2026
30a3ac2
Merge branch 'main' into feature/922_make_permissions_configurable_fo…
ArBridgeman Jul 10, 2026
08f30e6
Apply text changes from reviewers
ArBridgeman Jul 14, 2026
70a9f21
Fix section headings
ArBridgeman Jul 14, 2026
c1028c5
Introduce macro for workflow_permissions
ArBridgeman Jul 14, 2026
cfc71fe
Introduce macro for workflow_passed_secrets
ArBridgeman Jul 14, 2026
29d22ac
Reduce enum usage
ArBridgeman Jul 14, 2026
98dbec0
Resolve sonar finding
ArBridgeman Jul 14, 2026
3663d2b
Fix type hints
ArBridgeman Jul 14, 2026
5087874
Resolve sonar finding - attempt 2
ArBridgeman Jul 14, 2026
d9b094c
Set autoescape=False as unneeded
ArBridgeman Jul 14, 2026
85d2873
Simplify tests as already covered
ArBridgeman Jul 14, 2026
f9680a8
Resolve sonar finding - attempt 3
ArBridgeman Jul 14, 2026
923c88e
Resolve sonar finding - attempt 4
ArBridgeman Jul 14, 2026
4e01cdf
Resolve new sonar finding
ArBridgeman Jul 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 7 additions & 12 deletions doc/changes/changes_10.0.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,21 +42,20 @@ This release fixes vulnerabilities by updating dependencies:
| cryptography | GHSA-537c-gmf6-5ccf | 48.0.0 | 48.0.1 |
| msgpack | GHSA-6v7p-g79w-8964 | 1.1.2 | 1.2.1 |

## Feature
* #867: Fixed zizmor linting results

## Features

* #874: Added the `security` label to dependency update PR creation
* #699: Added `all-extras` support to the Python environment GitHub action
* #875: Added `name` attribute to generated workflow jobs using `-extension.yml` workflows
* #878: Added Nox session `workflow:audit` which uses `zizmor` and added it in `checks.yml`
* #872: Added `custom_workflows` to `github_template_dict` for automatic custom workflow secret extraction

## Bug
## Bugfixes

* #744: Updated nox DB-version handling to use `BaseConfig.minimum_exasol_version` instead hardcoded `7.1.9`

## Feature

* #878: Added Nox session `workflow:audit` which uses `zizmor` and added it in `checks.yml`
* #872: Added `custom_workflows` to `github_template_dict` for automatic custom workflow secret extraction

## Refactoring

* #744: Extracted shared minimum-version selection logic into `minimum_declared_version()`
Expand All @@ -67,10 +66,6 @@ This release fixes vulnerabilities by updating dependencies:
* #828: Removed the legacy migration page and merged the useful guidance into getting started
* #789: Consolidated the metrics and Sonar documentation to reflect the current PTB reporting flow

## Security

* #867: Fixed zizmor linting results

## Dependency Updates

### `main`
Expand All @@ -80,4 +75,4 @@ This release fixes vulnerabilities by updating dependencies:
* Updated dependency `pip-audit:2.10.0` to `2.10.1`
* Updated dependency `pylint:4.0.5` to `4.0.6`
* Updated dependency `pytest:9.0.3` to `9.1.1`
* Updated dependency `zizmor:1.25.2` to `1.26.1`
* Updated dependency `zizmor:1.25.2` to `1.26.1`
2 changes: 1 addition & 1 deletion doc/changes/changes_10.2.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This minor release adds automated vulnerability updates through Nox session
`vulnerabilities:update` and improves the `dependency-update.yml`. It also includes a
few workflow-related bug fixes and documentation updates.

## Bug
## Bugfixes

* #909: Updated `cd.yml` workflow so that `cd-extension.yml` workflow depends on `build-and-publish`. This ensures that the custom release workflow only runs when the PyPi release was successful.
* #910: Added `gh-pages.yml` to be ignored when `has_documentation=False` in the `PROJECT_CONFIG`
Expand Down
4 changes: 1 addition & 3 deletions doc/changes/changes_10.2.1.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
# 10.2.1 - 2026-07-08

## Summary

## Bug Fix
## Bugfixes

* #920: Ensured extracted secrets are unique and alphabetically sorted from the custom workflows
4 changes: 4 additions & 0 deletions doc/changes/unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

## Summary

## Features

* #922: Extended `custom_workflows` of `github_template_dict` for automatic custom workflow permissions extraction

## Refactoring

* #924: Removed `lint:dependencies` usage from `report.yml` and added deprecation notice
45 changes: 41 additions & 4 deletions doc/user_guide/features/github_workflows/workflow_variables.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,22 @@ standardized baseline that can be overridden in individual projects.
:start-at: github_template_dict
:end-before: @computed_field

.. _custom_workflow_metadata:

Custom Workflow Metadata
------------------------

The PTB extracts metadata from reusable custom workflow files and exposes it
Comment thread
ArBridgeman marked this conversation as resolved.
through :py:attr:`exasol.toolbox.config.BaseConfig.github_template_dict` under the
``custom_workflows`` entry. PTB-controlled workflow templates use that metadata
when they call reusable workflows.

.. _custom_workflow_secrets:

Custom Workflow Secrets
^^^^^^^^^^^^^^^^^^^^^^^
Secrets
^^^^^^^

The PTB extracts secret names from reusable custom workflow files and exposes them
The PTB extracts the names of secrets from custom workflow files and exposes them
through :py:attr:`exasol.toolbox.config.BaseConfig.github_template_dict` under the
``custom_workflows`` entry. PTB-controlled workflow templates use those extracted
names when they call reusable workflows and forward secrets via ``secrets:``.
Expand All @@ -50,6 +60,33 @@ For example, ``slow-checks.yml`` can define its reusable workflow header like th
Those extracted secret names are then made available to the PTB templates that
reference the custom workflow.

.. _custom_workflow_permissions:

Permissions
^^^^^^^^^^^

The PTB extracts the permissions required by custom workflow files. It reads every job's
``permissions`` block and combines the results into a single ordered mapping, where
the most permissive level wins.

Please only configure the minimum required permissions by granting the least required
access. In practice, ``contents: read`` is the most common baseline for workflows, and
other permissions should only be added when it is truly required.

For example, a custom workflow can declare permissions like this:

.. code-block:: yaml

name: Slow-Checks

on:
workflow_call:

jobs:
run-integration-tests:
permissions:
contents: read

.. _workflow_matrix:

Matrix Combinations
Expand All @@ -64,7 +101,7 @@ Extending the Matrix
^^^^^^^^^^^^^^^^^^^^

If you need to expose additional values via the ``matrix.yml``, you can extend
:class:`exasol.toolbox.config.BaseConfig`.
:class:`exasol.toolbox.config.BaseConfig`.

The example adds two additional matrix dimensions: A declared one
`extra_matrix_value` and a computed one `computed_matrix_value`. Each of them
Expand Down
31 changes: 16 additions & 15 deletions exasol/toolbox/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,12 @@
PLUGIN_ATTR_NAME,
)
from exasol.toolbox.util.version import Version
from exasol.toolbox.util.workflows.custom_workflow_extractor import (
CustomWorkflowExtractor,
)
from exasol.toolbox.util.workflows.render_yaml import GithubTemplateContext

WORKFLOW_HEADER_PREFIX = (
"# Generated and maintained by the exasol-toolbox.\n"
"# Last generated with exasol-toolbox version "
)

WORKFLOW_HEADER_PATTERN = rf"\A{re.escape(WORKFLOW_HEADER_PREFIX)}[^\n]+\.\n"


Expand Down Expand Up @@ -323,26 +320,30 @@ def github_workflow_directory(self) -> Path:

@computed_field # type: ignore[misc]
@property
def github_template_dict(self) -> dict[str, Any]:
def github_template_dict(self) -> GithubTemplateContext:
"""
Dictionary of variables to dynamically render Jinja2 templates into valid YAML
configurations.
"""
from exasol.toolbox.util.workflows.custom_workflow_extractor import (
CustomWorkflowExtractor,
)

custom_workflow_extractor = CustomWorkflowExtractor(
github_workflow_directory=self.github_workflow_directory,
sonar_token_name=self.sonar_token_name,
)

return {
"custom_workflows": custom_workflow_extractor.build_custom_workflow_dict(),
"dependency_manager_version": self.dependency_manager.version,
"has_documentation": self.has_documentation,
"minimum_python_version": self.minimum_python_version,
"os_version": self.os_version,
"python_versions": self.python_versions,
"sonar_token_name": self.sonar_token_name,
"workflow_header": f"{WORKFLOW_HEADER_PREFIX}{__version__}.",
}
return GithubTemplateContext(
custom_workflows=custom_workflow_extractor.build_custom_workflow_dict(),
dependency_manager_version=self.dependency_manager.version,
has_documentation=self.has_documentation,
minimum_python_version=self.minimum_python_version,
os_version=self.os_version,
python_versions=self.python_versions,
sonar_token_name=self.sonar_token_name,
workflow_header=f"{WORKFLOW_HEADER_PREFIX}{__version__}.",
)

@computed_field # type: ignore[misc]
@property
Expand Down
25 changes: 25 additions & 0 deletions exasol/toolbox/templates/github/workflows/_workflow_macros.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
(% macro workflow_permissions(workflow_name) -%)
(% set permissions = custom_workflows[workflow_name].permissions -%)
(% if permissions -%)
(% set rendered_permissions -%)
permissions:
(% for permission_name, permission_level in permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor -%)
(% endset -%)
(( rendered_permissions | trim | indent(width=4, first=true) ))
(% endif -%)
(% endmacro %)

(% macro workflow_passed_secrets(workflow_name) -%)
(% set secrets = custom_workflows[workflow_name].secrets -%)
(% if secrets -%)
(% set rendered_secrets -%)
secrets:
(% for secret_name in secrets %)
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor -%)
(% endset -%)
(( rendered_secrets | trim | indent(width=4, first=true) ))
(% endif %)
(% endmacro %)
11 changes: 3 additions & 8 deletions exasol/toolbox/templates/github/workflows/cd.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
(( workflow_header ))
(% from "_workflow_macros.j2" import workflow_passed_secrets, workflow_permissions with context %)
name: CD

on:
Expand Down Expand Up @@ -32,14 +33,8 @@ jobs:
needs:
- check-release-tag
- build-and-publish
(% if custom_workflows["cd-extension"].secrets %)
secrets:
(% for secret_name in custom_workflows["cd-extension"].secrets %)
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
permissions:
contents: write
(( workflow_passed_secrets("cd-extension") -))
(( workflow_permissions("cd-extension") ))
(% endif %)

(% if has_documentation %)
Expand Down
12 changes: 3 additions & 9 deletions exasol/toolbox/templates/github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
(( workflow_header ))
(% from "_workflow_macros.j2" import workflow_passed_secrets, workflow_permissions with context %)
name: CI

on:
Expand All @@ -9,15 +10,8 @@ jobs:
merge-gate:
name: Merge Gate
uses: ./.github/workflows/merge-gate.yml
(% if custom_workflows["merge-gate"].secrets %)
secrets:
(% for secret_name in custom_workflows["merge-gate"].secrets %)
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
permissions:
contents: read

(( workflow_passed_secrets("merge-gate") -))
(( workflow_permissions("merge-gate") ))
report:
name: Report
needs:
Expand Down
11 changes: 3 additions & 8 deletions exasol/toolbox/templates/github/workflows/fast-tests.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
(( workflow_header ))
(% from "_workflow_macros.j2" import workflow_passed_secrets, workflow_permissions with context %)
name: Fast-Tests

on:
Expand Down Expand Up @@ -46,12 +47,6 @@ jobs:
fast-tests-extension:
name: Extension
uses: ./.github/workflows/fast-tests-extension.yml
(% if custom_workflows["fast-tests-extension"].secrets %)
secrets:
(% for secret_name in custom_workflows["fast-tests-extension"].secrets %)
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
permissions:
contents: read
(( workflow_passed_secrets("fast-tests-extension") -))
(( workflow_permissions("fast-tests-extension") ))
(% endif %)
23 changes: 5 additions & 18 deletions exasol/toolbox/templates/github/workflows/merge-gate.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
(( workflow_header ))
(% from "_workflow_macros.j2" import workflow_passed_secrets, workflow_permissions with context %)
name: Merge-Gate

on:
Expand Down Expand Up @@ -53,29 +54,15 @@ jobs:
needs:
- approve-run-slow-tests
uses: ./.github/workflows/slow-checks.yml
(% if custom_workflows["slow-checks"].secrets %)
secrets:
(% for secret_name in custom_workflows["slow-checks"].secrets %)
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
permissions:
contents: read

(( workflow_passed_secrets("slow-checks") -))
(( workflow_permissions("slow-checks") ))
(% if custom_workflows["merge-gate-extension"].exists %)
merge-gate-extension:
name: Extension
uses: ./.github/workflows/merge-gate-extension.yml
(% if custom_workflows["merge-gate-extension"].secrets %)
secrets:
(% for secret_name in custom_workflows["merge-gate-extension"].secrets %)
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
permissions:
contents: read
(( workflow_passed_secrets("merge-gate-extension") -))
(( workflow_permissions("merge-gate-extension") ))
(% endif %)

# This job ensures inputs have been executed successfully.
allow-merge:
name: Allow Merge
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
(( workflow_header ))
(% from "_workflow_macros.j2" import workflow_passed_secrets, workflow_permissions with context %)
name: Periodic-Validation

on:
Expand Down Expand Up @@ -45,15 +46,8 @@ jobs:
uses: ./.github/workflows/slow-checks.yml
needs:
- restrict-to-default-branch
(% if custom_workflows["slow-checks"].secrets %)
secrets:
(% for secret_name in custom_workflows["slow-checks"].secrets %)
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
permissions:
contents: read

(( workflow_passed_secrets("slow-checks") -))
(( workflow_permissions("slow-checks") ))
report:
name: Report
needs:
Expand Down
Loading