Skip to content
Open
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
7a832d8
Fix typos in changelogs
ArBridgeman Jul 9, 2026
c37d96c
Make permissions block present in workflows that call the custom work…
ArBridgeman Jul 9, 2026
5d7f0ed
Add documentation for new feature
ArBridgeman Jul 9, 2026
620fde3
Add jinja test
ArBridgeman Jul 9, 2026
24cfb7b
Prepare config test
ArBridgeman Jul 9, 2026
bea075f
Add extract_permissions
ArBridgeman Jul 9, 2026
92a1ae4
Remove not-maintained workflows from get_workflow_templates - need to…
ArBridgeman Jul 10, 2026
c3d2872
Remove now unneeded custom exception as not maintained workflows not …
ArBridgeman Jul 10, 2026
0a36b86
Remove redundant check
ArBridgeman Jul 10, 2026
1ac5e10
Ensure NOT_MAINTAINED_WORKFLOWS are seeded first
ArBridgeman Jul 10, 2026
69e8d14
Tighten wording in documentation
ArBridgeman Jul 10, 2026
30a3ac2
Merge branch 'main' into feature/922_make_permissions_configurable_fo…
ArBridgeman Jul 10, 2026
08f30e6
Apply text changes from reviewers
ArBridgeman Jul 14, 2026
70a9f21
Fix section headings
ArBridgeman Jul 14, 2026
c1028c5
Introduce macro for workflow_permissions
ArBridgeman Jul 14, 2026
cfc71fe
Introduce macro for workflow_passed_secrets
ArBridgeman Jul 14, 2026
29d22ac
Reduce enum usage
ArBridgeman Jul 14, 2026
98dbec0
Resolve sonar finding
ArBridgeman Jul 14, 2026
3663d2b
Fix type hints
ArBridgeman Jul 14, 2026
5087874
Resolve sonar finding - attempt 2
ArBridgeman Jul 14, 2026
d9b094c
Set autoescape=False as unneeded
ArBridgeman Jul 14, 2026
85d2873
Simplify tests as already covered
ArBridgeman Jul 14, 2026
f9680a8
Resolve sonar finding - attempt 3
ArBridgeman Jul 14, 2026
923c88e
Resolve sonar finding - attempt 4
ArBridgeman Jul 14, 2026
4e01cdf
Resolve new sonar finding
ArBridgeman Jul 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/changes/changes_10.0.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ This release fixes vulnerabilities by updating dependencies:
* #699: Added `all-extras` support to the Python environment GitHub action
* #875: Added `name` attribute to generated workflow jobs using `-extension.yml` workflows

## Bug
## Bug Fix
Comment thread
ArBridgeman marked this conversation as resolved.
Outdated

* #744: Updated nox DB-version handling to use `BaseConfig.minimum_exasol_version` instead hardcoded `7.1.9`

Expand Down Expand Up @@ -80,4 +80,4 @@ This release fixes vulnerabilities by updating dependencies:
* Updated dependency `pip-audit:2.10.0` to `2.10.1`
* Updated dependency `pylint:4.0.5` to `4.0.6`
* Updated dependency `pytest:9.0.3` to `9.1.1`
* Updated dependency `zizmor:1.25.2` to `1.26.1`
* Updated dependency `zizmor:1.25.2` to `1.26.1`
2 changes: 1 addition & 1 deletion doc/changes/changes_10.2.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This minor release adds automated vulnerability updates through Nox session
`vulnerabilities:update` and improves the `dependency-update.yml`. It also includes a
few workflow-related bug fixes and documentation updates.

## Bug
## Bug Fix

* #909: Updated `cd.yml` workflow so that `cd-extension.yml` workflow depends on `build-and-publish`. This ensures that the custom release workflow only runs when the PyPi release was successful.
* #910: Added `gh-pages.yml` to be ignored when `has_documentation=False` in the `PROJECT_CONFIG`
Expand Down
2 changes: 0 additions & 2 deletions doc/changes/changes_10.2.1.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
# 10.2.1 - 2026-07-08

## Summary

## Bug Fix

* #920: Ensured extracted secrets are unique and alphabetically sorted from the custom workflows
4 changes: 4 additions & 0 deletions doc/changes/unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

## Summary

## Feature
Comment thread
ArBridgeman marked this conversation as resolved.
Outdated

* #922: Extended `custom_workflows` of `github_template_dict` for automatic custom workflow permissions extraction
Comment thread
ArBridgeman marked this conversation as resolved.
Outdated

## Refactoring

* #924: Removed `lint:dependencies` usage from `report.yml` and added deprecation notice
45 changes: 41 additions & 4 deletions doc/user_guide/features/github_workflows/workflow_variables.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,22 @@ standardized baseline that can be overridden in individual projects.
:start-at: github_template_dict
:end-before: @computed_field

.. _custom_workflow_metadata:

Custom Workflow Metadata
^^^^^^^^^^^^^^^^^^^^^^^^

The PTB extracts metadata from reusable custom workflow files and exposes it

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the general and technical description.

Could we, for more straight-forward and impatient users, add a small section with concrete instructions?

  • What do I need to do?
  • When? (In which Scenario / Use Case / Problem)
  • Example, maybe a pointer to one of the PTB's workflow templates illustrating the usage?

@ArBridgeman ArBridgeman Jul 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for impatient users they can use codex to summarize what's already there.
There really isn't in my opinion too much text.
Each section already has an example showing what to do.

@ArBridgeman ArBridgeman Jul 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps, there was some confusion as the headers weren't properly defined. I've fixed that in a commit so it looks like.

image

through :py:attr:`exasol.toolbox.config.BaseConfig.github_template_dict` under the
``custom_workflows`` entry. PTB-controlled workflow templates use that metadata
when they call reusable workflows.

.. _custom_workflow_secrets:

Custom Workflow Secrets
^^^^^^^^^^^^^^^^^^^^^^^
Secrets
-------

The PTB extracts secret names from reusable custom workflow files and exposes them
The PTB extracts secret names from custom workflow files and exposes them
Comment thread
ArBridgeman marked this conversation as resolved.
Outdated
through :py:attr:`exasol.toolbox.config.BaseConfig.github_template_dict` under the
``custom_workflows`` entry. PTB-controlled workflow templates use those extracted
names when they call reusable workflows and forward secrets via ``secrets:``.
Expand All @@ -50,6 +60,33 @@ For example, ``slow-checks.yml`` can define its reusable workflow header like th
Those extracted secret names are then made available to the PTB templates that
reference the custom workflow.

.. _custom_workflow_permissions:

Permissions
-----------

The PTB extracts the permissions required by custom workflow files. It reads every job's
``permissions`` block and combines the results into a single ordered mapping, where
the most permissive level wins.

Please only configure the minimum required permissions by granting the least required
access. In practice, ``contents: read`` is the most common baseline for workflows, and
other permissions should only be added when it is truly required.

For example, a custom workflow can declare permissions like this:

.. code-block:: yaml

name: Slow-Checks

on:
workflow_call:

jobs:
run-integration-tests:
permissions:
contents: read

.. _workflow_matrix:

Matrix Combinations
Expand All @@ -64,7 +101,7 @@ Extending the Matrix
^^^^^^^^^^^^^^^^^^^^

If you need to expose additional values via the ``matrix.yml``, you can extend
:class:`exasol.toolbox.config.BaseConfig`.
:class:`exasol.toolbox.config.BaseConfig`.

The example adds two additional matrix dimensions: A declared one
`extra_matrix_value` and a computed one `computed_matrix_value`. Each of them
Expand Down
6 changes: 5 additions & 1 deletion exasol/toolbox/templates/github/workflows/cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["cd-extension"].permissions %)
permissions:
contents: write
(% for permission_name, permission_level in custom_workflows["cd-extension"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)
(% endif %)

(% if has_documentation %)
Expand Down
6 changes: 5 additions & 1 deletion exasol/toolbox/templates/github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["merge-gate"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["merge-gate"].permissions.items() %)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this pattern repeats 3-4 times - would it make sense to move the functionality into an API function
and in the jinja template only have a call like this (not sure if this is possible)?:

(% workflow_permissions("merge-gate") %)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented this separately for secrets being passed and permissions.

(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)

report:
name: Report
Expand Down
6 changes: 5 additions & 1 deletion exasol/toolbox/templates/github/workflows/fast-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,10 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["fast-tests-extension"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["fast-tests-extension"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)
(% endif %)
12 changes: 10 additions & 2 deletions exasol/toolbox/templates/github/workflows/merge-gate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["slow-checks"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["slow-checks"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)

(% if custom_workflows["merge-gate-extension"].exists %)
merge-gate-extension:
Expand All @@ -72,8 +76,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["merge-gate-extension"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["merge-gate-extension"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)
(% endif %)

# This job ensures inputs have been executed successfully.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["slow-checks"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["slow-checks"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)

report:
name: Report
Expand Down
59 changes: 58 additions & 1 deletion exasol/toolbox/util/workflows/custom_workflow.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

from enum import IntEnum
from pathlib import Path

from pydantic import (
Expand All @@ -13,12 +14,43 @@
from exasol.toolbox.util.workflows.render_yaml import parse_yaml_text


class PermissionRank(IntEnum):
"""GitHub permission levels ranked from least to most permissive."""

NONE = 0
READ = 1
WRITE = 2

@classmethod
def _missing_(cls, value: object) -> PermissionRank:
Comment thread
ArBridgeman marked this conversation as resolved.
Outdated
"""Convert a GitHub permission string to its rank."""
if isinstance(value, str):
for rank in cls:
if rank.name.lower() == value:
return rank
raise ValueError(f"Unknown GitHub permission level: {value!r}")


def merge_permissions(permission_maps: list[dict[str, str]]) -> dict[str, str]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I can get the goal of the implementation, but my stomach says there must be a simpler, shorter way.

How about this approach?

Permissions: TypeAlias = dict[str, str]
"""
Each permission is defined by a name as it key and a value, which is
either None, or "read" or "write".
"""

def merge_permissions(permission_maps: list[Permissions]) -> Permissions:
    """Merge permission maps to keep the greater permission."""
    rank = {None: 0, "read": 1, "write": 2}
    result = {}
    def max_permission(perm: str, value: str) -> str:
        current = result.get(perm)
        return value if rank[value] > rank[current] else current

    for other in permission_maps:
        result |= {p: max_permission(p, v) for p, v in other.items()}
    return result

No enum required, < 70% characters, tests still green.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I removed the enum and simplified it.

"""Merge permission maps to keep the greater permission."""
merged_permissions: dict[str, str] = {}
for permission_map in permission_maps:
for permission_name, requested_level in permission_map.items():
Comment thread
ArBridgeman marked this conversation as resolved.
current_level = merged_permissions.get(permission_name, None)
if current_level is None:
merged_permissions[permission_name] = requested_level
continue
if PermissionRank(requested_level) > PermissionRank(current_level): # type: ignore[arg-type]
merged_permissions[permission_name] = requested_level
return merged_permissions


class CustomWorkflow(BaseModel):
"""A project-owned workflow used for seeded workflows and extensions.

These workflows are seeded by the PTB or extend PTB-provided workflows, but
they are maintained by the project itself rather than the PTB. See
`Custom Workflows <file:///home/chku/git/ptb/.html-documentation/user_guide/features/github_workflows/index.html#custom-workflows>`__.
`Custom Workflows <https://exasol.github.io/python-toolbox/main/user_guide/features/github_workflows/index.html#custom-workflows>`__.
Comment thread
ArBridgeman marked this conversation as resolved.
"""

model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True)
Expand Down Expand Up @@ -53,3 +85,28 @@ def extract_secrets(self) -> tuple[str, ...]:
if secrets := workflow_call.get("secrets", {}):
return tuple(secrets.keys())
return ()

def extract_permissions(self) -> dict[str, str]:
"""Return the effective job permissions required by the workflow.

The extractor scans all jobs and merges their ``permissions`` blocks into a
single mapping. When the same permission appears multiple times, the more
permissive level wins while preserving the first-seen order of the keys.

For example, a custom workflow can declare permissions like this:

.. code-block:: yaml

name: Slow-Checks

on:
workflow_call:

jobs:
run-integration-tests:
permissions:
contents: read
"""
jobs = self.yaml_content.get("jobs", {})
permission_maps = [job.get("permissions", {}) for job in jobs.values()]
return merge_permissions(permission_maps)
Comment thread
ArBridgeman marked this conversation as resolved.
28 changes: 26 additions & 2 deletions exasol/toolbox/util/workflows/custom_workflow_extractor.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,21 +8,35 @@
field_validator,
)

from exasol.toolbox.util.workflows.custom_workflow import CustomWorkflow
from exasol.toolbox.util.workflows.custom_workflow import (
CustomWorkflow,
merge_permissions,
)

MINIMUM_GITHUB_PERMISSIONS: dict[str, str] = {"contents": "read"}


class CustomWorkflowEntry(BaseModel):
model_config = ConfigDict(frozen=True)

exists: bool
secrets: tuple[str, ...]
permissions: dict[str, str]

@field_validator("secrets", mode="before")
@classmethod
def _normalize_secrets(cls, secrets: tuple[str, ...]) -> list[str]:
"""Return unique secret names in alphabetical order."""
return sorted(set(secrets))

@field_validator("permissions", mode="before")
@classmethod
def _normalize_permissions(
cls, permissions: list[dict[str, str]]
) -> dict[str, str]:
"""Merge permission maps while preserving first-seen order."""
return merge_permissions(permissions)


class CustomWorkflowExtractor(BaseModel):
model_config = ConfigDict(frozen=True)
Expand All @@ -43,13 +57,18 @@ def _build_custom_workflow_entry(
file_path = self.github_workflow_directory / f"{workflow}.yml"

secrets: tuple[str, ...] = ()
permissions: list[dict[str, str]] = []
if file_path.is_file():
custom_workflow = CustomWorkflow.load_from_file(file_path=file_path)
secrets = custom_workflow.extract_secrets()

permissions = [
MINIMUM_GITHUB_PERMISSIONS,
custom_workflow.extract_permissions(),
]
return CustomWorkflowEntry(
exists=file_path.exists(),
secrets=secrets,
permissions=permissions,
)

def _build_merge_gate_entry(
Expand All @@ -61,6 +80,11 @@ def _build_merge_gate_entry(
+ custom_workflows_dict["slow-checks"].secrets
# from the `report.yml`
+ (self.sonar_token_name,),
permissions=[
MINIMUM_GITHUB_PERMISSIONS,
custom_workflows_dict["merge-gate-extension"].permissions,
custom_workflows_dict["slow-checks"].permissions,
],
)

def build_custom_workflow_dict(
Expand Down
12 changes: 0 additions & 12 deletions exasol/toolbox/util/workflows/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,18 +99,6 @@ def __init__(self, workflow_name: str, valid_workflows):
)


class NotMaintainedWorkflowError(ValueError):
"""
Raised when a PTB-seeded workflow is requested in an existing project.
"""

def __init__(self, workflow_name: str):
super().__init__(
f"Workflow '{workflow_name}' is a PTB-seeded workflow that is "
"originally provided by the PTB and can only be seeded for a new project."
)


class YamlKeyError(Exception):
"""
Base exception for when a specified value cannot be found in a YAML.
Expand Down
Loading