ceph-windows: cross-compile in a podman container instead of a VM

ceph-pr-pipeline/README.md

@@ -12,7 +12,7 @@ rules don't change.
 | `Unmodified Submodules` | [ceph-pr-submodules](../ceph-pr-submodules/) | GitHub API only, no clone, seconds |
 | `make check` | [ceph-pull-requests](../ceph-pull-requests/) | shared build → ctest on its own builder |
 | `ceph API tests` | [ceph-pr-api](../ceph-pr-api/) | shared build → API tests on its own builder |
-| `ceph windows tests` | [ceph-windows-pull-requests](../ceph-windows-pull-requests/) | parallel leg, own (win32) build |
+| `ceph windows tests` | [ceph-windows-pull-requests](../ceph-windows-pull-requests/) | parallel leg, containerized builds + one Windows VM (see below) |
 | `make check (arm64)` | [ceph-pull-requests-arm64](../ceph-pull-requests-arm64/) | parallel leg, build+test on one arm64 node |
 
 ## Flow
@@ -24,7 +24,7 @@ webhook (pull_request / issue_comment)
             prepare                   resolve PR, classify files, post pending statuses
             ├─ Signed-off-by          GitHub API
             ├─ Unmodified Submodules  GitHub API
-            ├─ ceph windows tests     (libvirt) own clone + build + tests
+            ├─ ceph windows tests     (noble+libvirt) container builds + Windows VM
             ├─ build and test (arm64) (arm64)  restore-or-build + tests
             └─ build and test
                  build ceph           (huge)  ONE clone, bwc -e buildtests, tree → S3
@@ -42,6 +42,85 @@ webhook (pull_request / issue_comment)
 - **Checkout time:** set `CEPH_REFERENCE_REPO` to a local ceph.git mirror on
   the builders to cut the remaining clones to seconds.
 
+## The windows leg
+
+Historically ([ceph-windows-pull-requests](../ceph-windows-pull-requests/))
+this check provisioned three VMs per run: an Ubuntu VM to cross-compile
+Ceph for Windows in, a second Ubuntu VM to build and host a vstart cluster,
+and a Windows VM to run the tests — 114–177 minutes, most of it VM
+provisioning and two from-scratch builds.  Now only the Windows client is a
+VM (it loads the wnbd/Dokan kernel drivers; nothing but a real Windows
+kernel can); both builds run in containers on the builder host with
+persistent caches.  Measured: 79 min on a node with cold caches, ~60 warm.
+
+Steps, in order (all scripts in [scripts/ceph-windows](../scripts/ceph-windows/)):
+
+1. **Checkout**: ceph-build plus the PR merge ref, then submodules.
+2. **Cross-compile** (`win32_build_container`): runs the PR's own
+   `win32_build.sh` in a podman container built from
+   `win32-build.containerfile` (just the mingw toolchain packages; podman's
+   layer cache makes the image build a no-op after each builder's first
+   run).  Produces
+   `$WORKSPACE/ceph.zip`; the build tree is then removed so `do_cmake.sh`
+   can run later.  Two per-builder caches under `~/.cache/ceph-win32-build/`:
+   - `deps/<sha>`: the compiled mingw deps (mingw-llvm, boost, openssl,
+     ...), keyed by the sha of the branch's `win32_deps_build.sh` +
+     `mingw_conf.sh` + the containerfile, so a deps change in any branch
+     just builds a new entry (flock-guarded); `win32_build.sh` skips the
+     whole deps build when the cache's `completed` marker exists.
+   - `ccache`: compiler cache for the tree itself.  Mount paths are fixed
+     (`/workspace`, `/depscache`) so object paths stay stable across builds
+     and nodes.  `MINGW_LLVM_DIR` must be exported into the container:
+     `mingw_conf.sh` defaults it relative to the ceph tree, not `DEPS_DIR`.
+3. **`setup_libvirt`**: ensures the libvirt default NAT network (virbr0,
+   192.168.122.1) and the ssh helpers.
+4. **`setup_libvirt_windows_vm`**: boots the pristine Windows Server 2019
+   image as a QEMU/KVM guest on that network.
+5. **`setup_ceph_vstart_container`**: compiles the PR's own source into
+   the Ceph cluster the Windows client is tested against, and starts it.
+   Nothing prebuilt is downloaded and no release is involved: bwc starts
+   a container that has Ceph's build dependencies installed, mounts the
+   same `$WORKSPACE/ceph` checkout (the PR merged onto its target
+   branch) inside it, and runs `do_cmake.sh && ninja vstart` on it —
+   `CMAKE_BUILD_TYPE=Release` just means an optimized compile of the
+   PR's code.  Because both the client zip and the cluster come from
+   this one checkout, a PR that changes mon/osd/mds behavior is tested
+   on both sides of the wire, exactly like the VM flow was.  ccache is
+   what makes the compile fast: a per-builder stash of object files from
+   previous runs, so source files the PR didn't change are cache hits
+   and only the PR's actual changes recompile (~5 min typical; ~10 min
+   on a builder compiling for the first time).  vstart.sh then launches
+   the daemons (mon/mgr/osd/mds) inside a second, long-lived container
+   named `ceph_build`.  That container shares the host's network
+   (`--net=host`) and the daemons bind 192.168.122.1 — the builder's own
+   address on the libvirt NAT bridge — which is an address the Windows
+   VM can reach, so the client talks to the cluster like it would any
+   remote one.  The leg waits (bounded) for a READY marker the vstart
+   script drops once pools are set up, and fails immediately if the
+   cluster container dies instead.  Two vstart behaviors worth knowing:
+   - vstart writes `mds root ino uid/gid = $(id -u/-g)` into the MDS conf,
+     and the Windows client mounts with `client_mount_uid/gid = 1000` and
+     `client_permissions = true` — the cluster must be started by uid 1000
+     (as the VM's ubuntu user silently always did) or every cephfs write
+     from the client gets EACCES.
+   - vstart writes outside its data dir: `logrotate.conf`/`.state` in the
+     build dir and `vstart_environment.sh` next to itself in src.  The
+     vstart user owns the build dir wholesale plus that one src file; the
+     host-side conf/keyring/log extraction goes through `podman cp`/`podman
+     exec` because those files are subuid-owned on the host.
+6. **`run_tests`**: uploads `ceph.zip` + conf/keyring to the Windows VM,
+   runs `run_tests.ps1` and the qa/workunits/windows suite.  With
+   `VSTART_CONTAINER` set, its cluster-side diagnostics use `podman exec`
+   and local log copies; unset, the old ssh-into-the-VM behavior is
+   unchanged (the old freestyle job shares this script).
+7. **Cleanup** (post): remove both containers, destroy the VM, wipe the
+   workspace.
+
+The leg needs `installed-os-noble` builders: the container builds need a
+modern podman, and on jammy `setup_container_runtime.sh` can only provide
+docker.  The `ceph-windows-container-build` job runs step 2 standalone for
+validation or one-off manual windows builds.
+
 ## Gating
 
 - PR author with write/admin on ceph/ceph (the same check as ceph.git's

ceph-pr-pipeline/build/Jenkinsfile

@@ -499,18 +499,38 @@ def doWindowsTests() {
   checkoutCephBuild()
   checkoutCephPr()
   withStatus("windows", "ceph windows tests") {
-    // Same script chain as the old ceph-windows-pull-requests job (which
-    // concatenated these into one shell); check_docs_pr_only is handled by
-    // the prepare stage instead.
+    // The cross-compile runs in a podman container on this host (with the
+    // mingw deps and ccache persisted between builds), replacing the
+    // throwaway build VM the old ceph-windows-pull-requests job provisioned
+    // on every run.  Only the vstart cluster host and the Windows client
+    // remain libvirt VMs.
+    postStatus("windows", "pending", "cross-compiling ceph for windows")
+    sh "./ceph-build/scripts/setup_container_runtime.sh"
+    sh(
+      label: "cross-compile ceph for windows (container)",
+      script: """#!/bin/bash -ex
+        export DOCKER_HUB_USERNAME="\$DOCKER_HUB_CREDS_USR"
+        export DOCKER_HUB_PASSWORD="\$DOCKER_HUB_CREDS_PSW"
+        bash ./ceph-build/scripts/ceph-windows/win32_build_container
+        # The cross-build outputs are all in ceph.zip; drop the build tree
+        # so setup_ceph_vstart's rsync doesn't ship it to the cluster VM
+        # (do_cmake.sh refuses to run in a tree with a build dir).
+        rm -rf ceph/build
+      """
+    )
+    // The vstart cluster builds and runs in a bwc container too (Release,
+    // same configuration the VM used), so the only VM left is the Windows
+    // client itself.
+    postStatus("windows", "pending", "running windows tests")
     sh """#!/bin/bash -ex
+      export DOCKER_HUB_USERNAME="\$DOCKER_HUB_CREDS_USR"
+      export DOCKER_HUB_PASSWORD="\$DOCKER_HUB_CREDS_PSW"
+      export GIT_BRANCH="origin/pr/${params.ghprbPullId}/merge"
       source ./ceph-build/scripts/build_utils.sh
+      source ./ceph-build/scripts/bwc.sh
       source ./ceph-build/scripts/ceph-windows/setup_libvirt
-      source ./ceph-build/scripts/ceph-windows/setup_libvirt_ubuntu_vm
-      source ./ceph-build/scripts/ceph-windows/win32_build
-      source ./ceph-build/scripts/ceph-windows/cleanup_libvirt_ubuntu_vm
-      source ./ceph-build/scripts/ceph-windows/setup_libvirt_ubuntu_vm
       source ./ceph-build/scripts/ceph-windows/setup_libvirt_windows_vm
-      source ./ceph-build/scripts/ceph-windows/setup_ceph_vstart
+      source ./ceph-build/scripts/ceph-windows/setup_ceph_vstart_container
       source ./ceph-build/scripts/ceph-windows/run_tests
     """
   }
@@ -585,7 +605,10 @@ pipeline {
         }
         stage("ceph windows tests") {
           when { beforeAgent true; expression { run_windows } }
-          agent { label "x86_64 && (installed-os-noble || installed-os-jammy) && libvirt" }
+          // noble only: the containerized cross-build needs a modern podman,
+          // which setup_container_runtime.sh can't provide on jammy (it falls
+          // back to docker there).
+          agent { label "x86_64 && installed-os-noble && libvirt" }
           environment {
             CEPH_WIN_CI_KEY = credentials("ceph_win_ci_private_key")
           }
@@ -594,6 +617,11 @@ pipeline {
           }
           post {
             always {
+              sh """#!/bin/bash
+                podman rm -f "ceph-win32-build-\$BUILD_ID" ceph_build 2>/dev/null || true
+                mkdir -p artifacts
+                cp ceph/build/*.log vstart-container.log artifacts/ 2>/dev/null || true
+              """
               archiveArtifacts(artifacts: "artifacts/**", allowEmptyArchive: true)
               sh """#!/bin/bash
                 source ./ceph-build/scripts/build_utils.sh

ceph-windows-container-build/README.md

@@ -0,0 +1,26 @@
+# ceph-windows-container-build
+
+Builds Ceph for Windows (ceph.git's `win32_build.sh`) inside a podman
+container on the builder host, via
+`scripts/ceph-windows/win32_build_container`, instead of the throwaway
+libvirt Ubuntu VM the windows CI used historically.  Produces the same
+`ceph.zip` that `setup_libvirt_windows_vm`/`run_tests` consume.
+
+Two host-side caches make repeat builds fast:
+
+- **deps** (`~/.cache/ceph-win32-build/deps/<hash>`): the compiled mingw
+  dependencies (mingw-llvm, boost, openssl, ...), keyed by the sha of the
+  branch's `win32_deps_build.sh` + `mingw_conf.sh` + the containerfile.
+  `win32_build.sh` skips the whole deps build when the cache's `completed`
+  marker is present.
+- **ccache** (`~/.cache/ceph-win32-build/ccache`): compiler cache for the
+  ceph tree itself.  Container mount paths are fixed so object paths are
+  stable across builds.
+
+This job runs the cross-compile step of ceph-pr-pipeline's windows leg
+standalone — for validating changes to the container build without a full
+pipeline run, and for one-off manual windows builds of a branch or PR (set
+`CEPH_BRANCH` or `ghprbPullId`).  How the full leg fits together
+(cross-compile container, vstart cluster container, Windows VM, and the
+network between them) is documented in
+[ceph-pr-pipeline's README](../ceph-pr-pipeline/README.md#the-windows-leg).

ceph-windows-container-build/build/Jenkinsfile

@@ -0,0 +1,79 @@
+// ceph-windows-container-build: validation job for the containerized Ceph
+// Windows cross-build.  Runs ceph.git's win32_build.sh inside a podman
+// container on the builder host (scripts/ceph-windows/win32_build_container)
+// instead of the throwaway libvirt Ubuntu VM the windows CI has used so far,
+// and produces the same ceph.zip the windows test flow consumes.
+//
+// Testbed for moving ceph-pr-pipeline's windows leg off VMs; also usable for
+// one-off manual windows builds of a branch or PR.
+
+def checkoutCeph() {
+  if (params.ghprbPullId) {
+    sh "rm -rf ceph"
+    checkout scmGit(
+      branches: [[name: "origin/pr/${params.ghprbPullId}/merge"]],
+      browser: [$class: 'GithubWeb', repoUrl: 'https://github.com/ceph/ceph'],
+      userRemoteConfigs: [[
+        url: "https://github.com/ceph/ceph.git",
+        refspec: "+refs/pull/${params.ghprbPullId}/*:refs/remotes/origin/pr/${params.ghprbPullId}/*",
+      ]],
+      extensions: [
+        [$class: 'RelativeTargetDirectory', relativeTargetDir: 'ceph'],
+        [$class: 'CloneOption', shallow: true, noTags: true, honorRefspec: true,
+         timeout: 20],
+      ],
+    )
+  } else {
+    sh """#!/bin/bash -ex
+      rm -rf ceph
+      git clone --depth 1 --no-tags -b "${params.CEPH_BRANCH}" https://github.com/ceph/ceph.git ceph
+    """
+  }
+}
+
+pipeline {
+  agent { label params.NODE_LABEL }
+  options {
+    skipDefaultCheckout(true)
+    timeout(time: 5, unit: "HOURS")
+    timestamps()
+  }
+  environment {
+    TERM = "xterm"
+    DOCKER_HUB_CREDS = credentials("dgalloway-docker-hub")
+  }
+  stages {
+    stage("checkout") {
+      steps {
+        sh """#!/bin/bash -ex
+          rm -rf ceph-build
+          git clone --depth 1 --no-tags -b "${params.CEPH_BUILD_BRANCH}" https://github.com/ceph/ceph-build ceph-build
+        """
+        script { checkoutCeph() }
+      }
+    }
+    stage("build ceph windows") {
+      steps {
+        sh "./ceph-build/scripts/setup_container_runtime.sh"
+        sh """#!/bin/bash -ex
+          export DOCKER_HUB_USERNAME="\$DOCKER_HUB_CREDS_USR"
+          export DOCKER_HUB_PASSWORD="\$DOCKER_HUB_CREDS_PSW"
+          export NUM_WORKERS="${params.NUM_WORKERS}"
+          bash ./ceph-build/scripts/ceph-windows/win32_build_container
+        """
+      }
+    }
+  }
+  post {
+    always {
+      sh """#!/bin/bash
+        podman rm -f "ceph-win32-build-\$BUILD_ID" 2>/dev/null || true
+        rm -rf artifacts
+        mkdir -p artifacts
+        cp ceph/build/*.log artifacts/ 2>/dev/null || true
+        ls -lh ceph.zip 2>/dev/null || true
+      """
+      archiveArtifacts(artifacts: "artifacts/**", allowEmptyArchive: true)
+    }
+  }
+}

ceph-windows-container-build/config/definitions/ceph-windows-container-build.yml

@@ -0,0 +1,62 @@
+- job:
+    name: ceph-windows-container-build
+    description: |
+      Validation job for the containerized Ceph Windows cross-build: runs
+      ceph.git's win32_build.sh inside a podman container on the builder host
+      (instead of the libvirt Ubuntu VM the windows CI used historically) and
+      produces the ceph.zip consumed by the windows tests.  Compiled mingw
+      dependencies and ccache persist on the builder between runs.
+      Testbed for ceph-pr-pipeline's windows leg; also usable for one-off
+      manual windows builds of a branch or PR.
+    project-type: pipeline
+    quiet-period: 1
+    concurrent: true
+
+    properties:
+      - build-discarder:
+          days-to-keep: 15
+          num-to-keep: 50
+
+    pipeline-scm:
+      scm:
+        - git:
+            url: https://github.com/ceph/ceph-build
+            branches:
+              - ${{CEPH_BUILD_BRANCH}}
+            shallow-clone: true
+            submodule:
+              disable: true
+            wipe-workspace: true
+      script-path: ceph-windows-container-build/build/Jenkinsfile
+      lightweight-checkout: true
+      do-not-fetch-tags: true
+
+    parameters:
+      - string:
+          name: CEPH_BRANCH
+          default: main
+          description: "ceph.git branch to build (ignored if ghprbPullId is set)"
+
+      - string:
+          name: ghprbPullId
+          description: "optional: build this ceph PR's merge ref instead of CEPH_BRANCH"
+
+      - string:
+          name: NODE_LABEL
+          default: "x86_64 && installed-os-noble && libvirt"
+          description: "builder label expression (noble only: the container build needs a modern podman, which setup_container_runtime.sh can't provide on jammy)"
+
+      - string:
+          name: NUM_WORKERS
+          default: ""
+          description: "parallel build jobs (default: nproc, capped at 32)"
+
+      - string:
+          name: CEPH_BUILD_BRANCH
+          default: main
+          description: "Use the Jenkinsfile and scripts from this ceph-build branch"
+
+    wrappers:
+      - inject-passwords:
+          global: true
+          mask-password-params: true