Skip to content

Add dedicated cas suite to regression and split lightweight_delete CAS jobs - #2327

Merged
strtgbb merged 8 commits into
antalya-26.6from
feature/antalya-26.6/split-cas-lightweight-delete
Sep 9, 2026
Merged

Add dedicated cas suite to regression and split lightweight_delete CAS jobs#2327
strtgbb merged 8 commits into
antalya-26.6from
feature/antalya-26.6/split-cas-lightweight-delete

Conversation

@alsugiliazova

@alsugiliazova alsugiliazova commented Sep 8, 2026

Copy link
Copy Markdown
Member

Add the dedicated cas / cas_s3_cache suite from clickhouse-regression to the Antalya 26.6 pipeline, gated on the same ci_regression_cas checkbox as the other CAS jobs.

Match the clickhouse-regression release CAS job splits:

  • lightweight_delete: four shards (disk space, hard restart, leftover, and the slow features), each with a 5-hour timeout.
  • attach partition: three parts (part 1, part 2, part 3).

Related: #2234

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

...

Documentation entry for user-facing changes

  • I will add a documentation PR to ClickHouse/clickhouse-docs right after this PR is merged.
  • I need help to write the documentation.
  • No documentation needed.

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

alsugiliazova and others added 2 commits September 8, 2026 12:03
…use-regression.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@alsugiliazova alsugiliazova added antalya-26.6 cicd Improvements and fixes to the CICD process labels Sep 8, 2026
Co-authored-by: Cursor <cursoragent@cursor.com>
@alsugiliazova alsugiliazova changed the title Split CAS lightweight_delete regression jobs into three shards Add dedicated cas suite to regression and split lightweight_delete CAS jobs Sep 8, 2026
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

Workflow [PR], commit [f14cedc]

CarlosFelipeOR and others added 5 commits September 8, 2026 21:55
Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com>
…_delete` and attach partition.

Use four `lightweight_delete` shards with a 3-hour timeout, and add attach partition part 3 so the leftover Repeat(100) work is no longer folded into part 1.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…as `release`.

Keep attach `part 3` in the regular `Alter` job so those tests still run, and take `lightweight_delete` out of `Common` into the same four shards with a 5-hour timeout.

Co-authored-by: Cursor <cursoragent@cursor.com>
@CarlosFelipeOR

CarlosFelipeOR commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

I addressed some fixes/improvements:

1 — Selects suite

cas_selects was failing on /selects/final/force/concurrent, not because of this PR: the CAS disk exhausts the container's ephemeral ports against the store and the mount lease gets fenced — #2243, 15 of 47 job executions since 2026-08-26. Fixed in clickhouse-regression by setting http_keep_alive_timeout 60s and http_keep_alive_max_requests 1000 on the CAS disk (69be3aa72), in the shared helper so it covers every --cas suite. Rerun is green, 3020/3020, after three failed attempts on the same build.

Failing job: https://github.com/Altinity/ClickHouse/actions/runs/34214700789/job/102257899198

2 — Engines suite

cas_engines was failing on /engines/replicated_replacing_merge_tree/without is deleted, not because of this PR: the scenario reads the second replica right after inserting on the first, with no SYSTEM SYNC REPLICA. Under CAS the replica fetch takes ~267 ms against ~11 ms on local disk, so the SELECT lands before the part is committed and returns 0 rows — 9 failures in 109 CAS runs since 2026-08-18, 0 in 590 non-CAS runs. Fixed in clickhouse-regression by syncing the replica before reading it (3830fc45e), applied to all five scenarios of the feature rather than just the failing one — Altinity/clickhouse-regression#165. 15 of 15 passes on the build that failed 13 of 16 before. Rerun is green, 37/37, after two failed attempts on the same build.

Failing job: https://github.com/Altinity/ClickHouse/actions/runs/34214700789/job/102128879154

3 — CAS jobs were overwriting each other's S3 reports

engines, cas_engines and cas_s3_cache_engines all uploaded to the same prefix, and so did every other CAS/non-CAS pair: 38 of the 98 concrete jobs shared 11 prefixes. setup.sh already builds a /cas or /cas-s3-cache suffix, but it reads $regression_args, which this workflow never exposed to the job environment — the input only becomes REGRESSION_ARGS in a step that runs after setup.sh, so the suffix was always empty. A job retry overwrote the failing attempt too, which is why the server logs of a CAS failure were routinely lost and report_url in gh-data.clickhouse_regression_results pointed at the wrong job. Fixed here in 0fa0a34 by exposing regression_args to setup.sh, plus giving CASAlter/CASS3CacheAlter the storage_path the non-CAS Alter job already has (their replace and move variants share suite=alter with an empty part, so the suffix alone would not separate them). Companion fix in clickhouse-regression (a0e488a1): the guard that skips the suffix compared against STORAGE=/cas while tiered_storage_cas actually sets /with_cas. 99 jobs now resolve to 99 distinct prefixes.

@CarlosFelipeOR CarlosFelipeOR left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CarlosFelipeOR CarlosFelipeOR added the verified Approved for release label Sep 9, 2026
@strtgbb
strtgbb merged commit 5181377 into antalya-26.6 Sep 9, 2026
295 of 301 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

antalya antalya-26.6 antalya-26.6.4.20001 cicd Improvements and fixes to the CICD process verified Approved for release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants