Skip to content

docs: Document GitHub workflows and jobs missed in previous PRs. #2498

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kirkrodrigues wants to merge 2 commits into
base: main
Choose a base branch
from
Open

docs: Document GitHub workflows and jobs missed in previous PRs. #2498

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1d7fab6
docs: Document GitHub workflows and jobs missed in previous PRs.
kirkrodrigues Aug 24, 2026
bd7e70e
Address CodeRabbit's comments.
kirkrodrigues Aug 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 89 additions & 16 deletions docs/src/dev-docs/tooling-gh-workflows.md
100644 → 100755
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,17 @@ builds, testing, and linting. We briefly describe each workflow below.

## clp-artifact-build

This workflow is responsible for:
This workflow is responsible for the following:

1. building (Linux) container images containing CLP-core's dependencies,
2. building CLP-core and running its unit & integration tests, and
3. building a container image containing CLP's package components.
1. Building container images containing the dependencies necessary to build all CLP artifacts.
2. Building CLP-core, the CLP-core Python wheels, and the CLP package.
3. Building container images containing:

* CLP-core;
* the CLP package; and
* the binaries necessary for the Spider worker.

4. Running C++ linting checks, unit tests, and integration tests on the built artifacts.

To minimize build times, the jobs in the workflow are organized in the directed acyclic graph (DAG)
shown below.
Expand All @@ -29,24 +35,48 @@ shown below.
}
}%%
flowchart LR
%% CLP-artifact-dependency container build jobs
filter-relevant-changes --> centos-stream-9-deps-image
filter-relevant-changes --> manylinux_2_28-deps-image
filter-relevant-changes --> musllinux_1_2-deps-image
filter-relevant-changes --> ubuntu-jammy-deps-image
filter-relevant-changes --> ubuntu-jammy-aarch64-deps-image
filter-relevant-changes --> ubuntu-jammy-x86_64-deps-image
manylinux_2_28-deps-image --> manylinux_2_28-deps-image-merge
musllinux_1_2-deps-image --> musllinux_1_2-deps-image-merge

%% CLP-core build jobs
filter-relevant-changes --> centos-stream-9-binaries
filter-relevant-changes --> manylinux_2_28-x86_64-binaries
filter-relevant-changes --> musllinux_1_2-x86_64-binaries
filter-relevant-changes --> ubuntu-jammy-binaries
centos-stream-9-deps-image --> centos-stream-9-binaries
manylinux_2_28-deps-image --> manylinux_2_28-deps-image-merge
filter-relevant-changes --> manylinux_2_28-x86_64-binaries
manylinux_2_28-deps-image --> manylinux_2_28-x86_64-binaries
manylinux_2_28-deps-image-merge --> manylinux_2_28-x86_64-binaries
musllinux_1_2-deps-image --> musllinux_1_2-deps-image-merge
filter-relevant-changes --> musllinux_1_2-x86_64-binaries
musllinux_1_2-deps-image --> musllinux_1_2-x86_64-binaries
musllinux_1_2-deps-image-merge --> musllinux_1_2-x86_64-binaries
ubuntu-jammy-deps-image --> ubuntu-jammy-binaries
ubuntu-jammy-deps-image --> package-image
filter-relevant-changes --> ubuntu-jammy-binaries
ubuntu-jammy-x86_64-deps-image --> ubuntu-jammy-binaries

%% CLP-core binaries container build jobs
ubuntu-jammy-binaries --> ubuntu-jammy-binaries-image

%% CLP-core Python-wheel build jobs
filter-relevant-changes --> manylinux_2_28-x86_64-python-wheels
manylinux_2_28-deps-image --> manylinux_2_28-x86_64-python-wheels
manylinux_2_28-deps-image-merge --> manylinux_2_28-x86_64-python-wheels

%% CLP-package container build jobs
filter-relevant-changes --> package-image
ubuntu-jammy-aarch64-deps-image --> package-image
ubuntu-jammy-x86_64-deps-image --> package-image
package-image --> package-image-multiarch-manifest

%% Spider-worker container build jobs
filter-relevant-changes --> spider-worker-image
ubuntu-jammy-x86_64-deps-image --> spider-worker-image

%% Lint & test jobs
filter-relevant-changes --> ubuntu-jammy-lint
ubuntu-jammy-x86_64-deps-image --> ubuntu-jammy-lint
ubuntu-jammy-binaries --> ubuntu-jammy-integration-tests-core
:::

Expand All @@ -68,21 +98,28 @@ Arrows between jobs indicate a dependency. The jobs are as follows:
arch-suffixed tag (e.g. `:main-amd64`).
* `musllinux_1_2-deps-image-merge`: On push to `main`, merges the per-arch tags produced by
`musllinux_1_2-deps-image` into a single multi-arch `:main` manifest.
* `ubuntu-jammy-deps-image`: Builds a container image containing the dependencies necessary to build
CLP-core in an Ubuntu Jammy x86 environment.
* `ubuntu-jammy-x86_64-deps-image` / `ubuntu-jammy-aarch64-deps-image`: Builds a container image
containing the dependencies necessary to build CLP-core in an Ubuntu Jammy x86/aarch64
environment.
Comment on lines +101 to +103

@quinntaylormitchell quinntaylormitchell Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be worth documenting that the aarch64 job only runs on main (rather than on PRs as well)? Completely up to you, I don't feel strongly.

* `centos-stream-9-binaries`: Builds the CLP-core binaries in the built CentOS Stream 9 container
and runs core's unit tests.
* `manylinux_2_28-x86_64-binaries`: Builds the CLP-core binaries in the built manylinux_2_28
container and runs core's unit tests.
* `musllinux_1_2-x86_64-binaries`: Builds the CLP-core binaries in the built musllinux_1_2 container
and runs core's unit tests.
* `package-image`: Builds a container image containing CLP's package components.
* `manylinux_2_28-x86_64-python-wheels`: Builds the `yscope-clp-core` wheels in the built
`manylinux_2_28` container.
* `ubuntu-jammy-binaries`: Builds the CLP-core binaries in the built Ubuntu Jammy container and runs
core's unit tests.
* `ubuntu-jammy-binaries-image`: Builds an Ubuntu Jammy container image containing CLP-core's
binaries built in the `ubuntu-jammy-binaries` job.
* `ubuntu-jammy-lint`: Runs C++ linting checks in the built ubuntu-jammy container.
* `ubuntu-jammy-integration-tests-core`: Runs CLP-core's integration tests using the binaries built
in the `ubuntu-jammy-binaries` job, and then uploads the logs from the tests.
* `package-image`: Builds the CLP package container image.
* `package-image-multiarch-manifest`: When run on `main`, merges the per-arch tags produced by
`package-image` into a single multi-arch manifest.
* `spider-worker-image`: Builds a container image containing CLP-core and `clp-tdl-package`.

@quinntaylormitchell quinntaylormitchell Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though this workflow does build core and clp-tdl-package, I think the clp-spider-worker Dockerfile only puts clp-s, indexer, log-converter into the container image. Might be pedantic, up to you if you want to change it


When the PR or commit doesn't change any of the files that affect CLP's dependencies (or the
dependency container images), then the dependency container images won't be rebuilt; instead the
Expand Down Expand Up @@ -116,9 +153,45 @@ details, see [GitHub-hosted runners][gh-hosted-runners].

This workflow builds CLP-core on macOS and runs its unit tests.

## clp-docs

This workflow validates that the docs site can be built.

## clp-docs-generated-code-checks

This workflow generates the OpenAPI docs and validates that they don't differ from the committed
OpenAPI docs.

## clp-lint

This workflow runs linting checks on the codebase.
This workflow runs all JavaScript, Python, and YAML linting checks on the codebase.

:::{note}
C++, Rust, and Helm linting checks are run in the `clp-artifact-build`, `clp-rust-checks`, and
`clp-package-helm` workflows, respectively.
Comment on lines +170 to +171

@quinntaylormitchell quinntaylormitchell Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is tricky because C++ linting also happens in clp-core-build-macos, not just clp-artifact-build; it's more verbose, but how about the following:

Suggested change
C++, Rust, and Helm linting checks are run in the `clp-artifact-build`, `clp-rust-checks`, and
`clp-package-helm` workflows, respectively.
Further linting checks on the codebase are run as appropriate in other workflows: C++ linting in
`clp-artifact-build` and `clp-core-build-macos`, Rust linting in `clp-rust-checks`, and Helm linting
in `clp-package-helm`.

:::

## clp-package-helm

This workflow contains two jobs for linting, building, and publishing the Helm chart:

* `lint` runs Helm linting checks on the chart.
* `publish` builds the chart; then on pushes to `main` and semantic-version branches, the job
publishes the built chart to the `gh-pages` branch.
Comment on lines +179 to +180

@quinntaylormitchell quinntaylormitchell Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that both the building and publishing are gated under if: github.event_name != 'pull_request', but this wording makes it seem like only the publishing part is gated. How about

Suggested change
* `publish` builds the chart; then on pushes to `main` and semantic-version branches, the job
publishes the built chart to the `gh-pages` branch.
* `publish` builds the chart and publishes it to the `gh-pages` branch; `publish` runs only on
pushes to `main` and semantic-version branches.


## clp-pr-title-checks

This workflow validates pull request titles against the Conventional Commits specification.

## clp-rust-checks

This workflow validates Rust's lock files, runs all Rust linting checks, and runs all Rust unit
tests.
Comment on lines +188 to +189

@quinntaylormitchell quinntaylormitchell Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that clp-rust-checks runs task tests:rust-all, so maybe this would be more appropriate:

Suggested change
This workflow validates Rust's lock files, runs all Rust linting checks, and runs all Rust unit
tests.
This workflow validates Rust's lock files, runs all Rust linting checks, and runs all Rust tests.


## clp-s-generated-code-checks

This workflow generates the KQL and SQL ANTLR parsers and validates that they don't differ from the
committed parsers.

## clp-uv-checks

Expand Down
Loading