pulp-cleanup: add nightly Jenkins job for stale Pulp resource cleanup by vamahaja · Pull Request #2644 · ceph/ceph-build

vamahaja · 2026-07-08T07:30:58Z

Summary

Add a nightly pulp-cleanup-nightly Jenkins job that purges stale ceph project Pulp repositories and distributions using scripts/purge-policy.yaml.
Cleanup excludes named policy refs via label-select, deletes remaining resources past the default retention days (distributions before repositories), then runs orphan cleanup.
Update pulp_upload.sh so repository version retention uses per BRANCH & keep_minimum from the purge policy, with fallback to the default.

Job config

Job: pulp-cleanup-nightly, triggered @midnight.
Parameters: CEPH_BUILD_BRANCH (default main), CEPH bool (default true) to enable ceph project cleanup.
Pipeline sets up the Pulp client, then runs cleanup.sh when CEPH is true.

Purge policy (`scripts/purge-policy.yaml`)

Project root: ceph
Default retention: days: 14, keep_minimum: 3
Per-ref overrides for release and special branches
Cleanup uses named refs as exclusions; uploads use keep_minimum for repository version retention

djgalloway · 2026-07-08T17:55:00Z

I think we need something more like chacra's retention map.

Or at least leave this PR as-is but add KEEP_MINIMUM or RETAIN_MINIMUM on top of RETAIN_DAYS.

vamahaja · 2026-07-10T11:20:37Z

I think we need something more like chacra's retention map.

Or at least leave this PR as-is but add KEEP_MINIMUM or RETAIN_MINIMUM on top of RETAIN_DAYS.

Updated logic PTAL.

Add a nightly Jenkins job and cleanup script that prune stale `Ceph` project Pulp repositories and distributions using `scripts/purge-policy.yaml`. Also update `pulp_upload.sh` so repository version retention uses `keep_minimum` from the purge policy for the current `BRANCH`, falling back to the default when the ref is not listed. Signed-off-by: Vaibhav Mahajan <vamahaja@redhat.com>

djgalloway · 2026-08-06T19:57:19Z

I think it makes more sense to have this in pulp-cleanup/config

djgalloway · 2026-08-06T20:48:08Z

+  default:
+    days: 14
+    keep_minimum: 3
+  ref:


Add main please. Give it the same retention as a release branch

djgalloway · 2026-08-06T20:50:57Z

keep_minimum is never read from anywhere in here

djgalloway · 2026-08-06T20:56:55Z

+readonly PULP_RESOURCES=(distribution repository)
+
+# Default protection time in hours
+readonly PROTECTION_TIME=24


Minutes. Do 1440.

Options: --content-hrefs TEXT List of specific Contents to delete if they are orphans --protection-time INTEGER How long in minutes Pulp should hold orphan Content and Artifacts before becoming candidates for cleanup task

djgalloway · 2026-08-06T20:57:46Z

+
+    pulp "${type}" "${resource}" list \
+        --limit "${PULP_LIST_LIMIT}" \
+        --ordering '-pulp_created' \


-pulp_created fetches the newest N. We want to delete the oldest N.

djgalloway · 2026-08-06T20:58:31Z

+        --limit "${PULP_LIST_LIMIT}" \
+        --ordering '-pulp_created' \
+        --label-select project="${project}","${label_select}" \
+        2>&1 | sed -n '/^\[/,$p'


The job/script will fail silently with this. Make a failed pulp call fatal.

djgalloway · 2026-08-06T21:04:16Z

+    echo "${pulp_json}" | jq -r --arg days "${days}" '
+        .[]
+        | select(
+            (.pulp_created | split(".")[0] + "Z" | fromdateiso8601)
+            < (now - ($days | tonumber) * 86400)
+          )
+        | .name
+    '


We should filter on pulp_last_updated instead.

Also, we'll have a ceph-tentacle-rocky-10-x86_64 repository forever. We want to clean up all of the distrubutions inside that are older than 14 days but keep a minimum of 10.

So if the repository matches any of the named branches in purge-policy.yaml, we should proceed with cleaning up the distributions inside by following the policy.

If the repository doesn't match a named branch in purge_policy.yaml and its pulp_last_updated is over 14 days ago, delete the repository. If newer than 14 days, we should check inside for distributions and clean up any older than 14 days.

djgalloway · 2026-08-06T21:09:12Z

+                script {
+                    echo "Setting up yq..."
+                    sh """#!/bin/bash -ex
+                        source ./scripts/setup_yq.sh


Sourcing the script just defines the function. Just run it directly.

djgalloway · 2026-08-06T21:10:13Z

+    display-name: 'Pulp: Cleanup Nightly'
+    project-type: pipeline
+    description: >
+      Nightly parameterized cleanup of stale Teuthology Pulp resources.


Suggest Nightly parameterized cleanup of stale ceph builds in Pulp

djgalloway · 2026-08-06T21:10:28Z

@@ -0,0 +1,37 @@
+- job:
+    name: pulp-cleanup-nightly
+    disabled: true


Is this intentional?

djgalloway · 2026-08-06T21:11:09Z

The other job definitions are .yml

djgalloway · 2026-08-06T21:17:54Z

+    mapfile -t policy_refs < <(get_policy_refs "${project}" "${label}")
+    _label_select=$(
+        printf "${label}!=%s\n" "${policy_refs[@]}" | paste -sd, -
+    )


If anything goes wrong here, this falls back to "delete everything"

Gate on a yq failure, missing file, etc. Any error.

djgalloway · 2026-08-06T21:21:19Z

Maybe implement a DRY_RUN. "Would delete repository ceph-wip-nbala-test-1-centos-9-default-x86_64"

"Leaving distributions X Y Z in repository ceph-tentacle-rocky-10-x86_64 to satisfy keep_minimum: 10"

Something like those. I'd like to see some dry-run output before we let this loose even though the artifacts in pulp aren't terribly important yet.

Finished with my review for now.

vamahaja marked this pull request as draft July 8, 2026 07:31

vamahaja force-pushed the pulp-cleanup branch 5 times, most recently from f47e5f6 to 72d89d9 Compare July 8, 2026 09:39

vamahaja marked this pull request as ready for review July 8, 2026 13:03

vamahaja requested review from djgalloway and zmc July 8, 2026 13:03

vamahaja marked this pull request as draft July 8, 2026 13:06

vamahaja force-pushed the pulp-cleanup branch from 72d89d9 to 22faa03 Compare July 10, 2026 11:16

vamahaja force-pushed the pulp-cleanup branch 6 times, most recently from c62087e to 9d67a99 Compare July 14, 2026 11:24

vamahaja force-pushed the pulp-cleanup branch 3 times, most recently from a0b55c6 to a83e10c Compare July 20, 2026 10:49

vamahaja force-pushed the pulp-cleanup branch from a83e10c to c32aab1 Compare July 20, 2026 11:18

vamahaja marked this pull request as ready for review July 20, 2026 11:22

djgalloway reviewed Aug 6, 2026

View reviewed changes

Comment thread scripts/purge-policy.yaml

djgalloway Aug 6, 2026

Copy link
Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes more sense to have this in pulp-cleanup/config

djgalloway reviewed Aug 6, 2026

View reviewed changes

Comment thread pulp-cleanup/build/cleanup.sh

djgalloway Aug 6, 2026

Copy link
Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keep_minimum is never read from anywhere in here

djgalloway reviewed Aug 6, 2026

View reviewed changes

Comment thread pulp-cleanup/config/definitions/pulp-cleanup.yaml

djgalloway Aug 6, 2026

Copy link
Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other job definitions are .yml

djgalloway reviewed Aug 6, 2026

View reviewed changes

Uh oh!

Conversation

vamahaja commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Job config

Purge policy (scripts/purge-policy.yaml)

Uh oh!

djgalloway commented Jul 8, 2026

Uh oh!

vamahaja commented Jul 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

djgalloway commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vamahaja commented Jul 8, 2026 •

edited

Loading

Purge policy (`scripts/purge-policy.yaml`)