ceph-windows: cross-compile in a podman container instead of a VM - #2694
Open
djgalloway wants to merge 8 commits into
Open
ceph-windows: cross-compile in a podman container instead of a VM#2694djgalloway wants to merge 8 commits into
djgalloway wants to merge 8 commits into
Conversation
The windows CI cross-compiles Ceph inside a freshly provisioned libvirt Ubuntu VM on every run: download the cloud image, boot, cloud-init, rsync the tree in, build everything (including all the mingw deps) from scratch, rsync ceph.zip back out, destroy the VM. That was ~50 minutes of a recent successful run before the cluster-side work even started. win32_build.sh is a plain Linux mingw cross-compile, so run it in a podman container on the builder host instead: a static toolchain image (the apt packages from win32_deps_build.sh's ubuntu branch), the workspace bind- mounted, and two host-side caches that survive between builds: - the compiled mingw deps (mingw-llvm, boost, openssl, ...), keyed by the sha of the branch's win32_deps_build.sh + mingw_conf.sh + the containerfile, so any deps change simply builds a new cache entry; win32_build.sh already skips the deps build when it finds the cache's completed marker - ccache for the ceph tree itself (CMake picks the launcher up from CMAKE_C/CXX_COMPILER_LAUNCHER), with fixed container mount paths so object paths stay stable across builds and nodes The new ceph-windows-container-build job validates this on the same builder pool the windows test jobs use, without touching ceph-pr-pipeline or the existing windows jobs; it doubles as a one-off manual windows build job for a branch or PR. Once proven, ceph-pr-pipeline's windows leg will switch to it, leaving libvirt only for the Windows client VM. Assisted-by: Claude Fable 5 (claude-fable-5) Signed-off-by: David Galloway <david.galloway@ibm.com>
mingw_conf.sh defaults MINGW_LLVM_DIR to $CEPH_DIR/build.deps/mingw-llvm, which only coincided with the deps location while DEPS_DIR used its default. With the deps cache mounted at /depscache the llvm-mingw toolchain was downloaded there but never found, and the deps build died at the first x86_64-w64-mingw32-clang invocation (lz4). Assisted-by: Claude Fable 5 (claude-fable-5) Signed-off-by: David Galloway <david.galloway@ibm.com>
Replace the windows leg's first libvirt Ubuntu VM (provisioned, built in, and destroyed on every run) with win32_build_container: the mingw cross-compile now runs in a podman container on the builder host with the compiled deps and ccache persisted between builds. Validated standalone by the ceph-windows-container-build job: 17.7 min cold / 5.5 min warm for what took ~50 min in the VM. Only the vstart cluster host and the Windows client remain libvirt VMs. The cross-build tree is dropped after zipping so setup_ceph_vstart's rsync doesn't ship it to the cluster VM (do_cmake.sh refuses to run in a tree with a build dir); on failure the build logs are archived before cleanup wipes the workspace. Assisted-by: Claude Fable 5 (claude-fable-5) Signed-off-by: David Galloway <david.galloway@ibm.com>
Run 88 landed on braggi12 (jammy), where setup_container_runtime.sh falls back to docker because jammy's podman is too old, and the container build died on "podman: command not found". All but four of the libvirt builders are noble, where podman works -- require noble, like the bwc-based legs already do. Assisted-by: Claude Fable 5 (claude-fable-5) Signed-off-by: David Galloway <david.galloway@ibm.com>
Contributor
Author
|
Now integrated into ceph-pr-pipeline's windows leg and validated end to end: pipeline run 89 ran the full leg for ceph PR 71382 (windows check only,
Old ceph-windows-pull-requests successes ran 114–177 min. Also in this branch: the windows leg now requires |
The windows leg's remaining Ubuntu VM existed only to build (32 minutes, from scratch, every run) and host the vstart cluster. Build and run it in a bwc container on the builder host instead: same Release build and vstart configuration as the VM flow, but the bwc image has the build deps baked in and a persistent host-side ccache makes repeat builds fast. The cluster container runs with --net=host and vstart binds the libvirt bridge address, which the Windows client VM reaches directly over the default NAT network. The only VM left is the Windows client itself. The container path is self-contained per run (no coupling to the shared build leg or the S3 cache), so it behaves identically in full runs and windows-only re-runs. run_tests grows a VSTART_CONTAINER mode for its cluster-side diagnostics (podman exec / local log copies instead of ssh into the VM); with the variable unset its behavior is unchanged for the old ceph-windows-pull-requests job, which shares the script. Assisted-by: Claude Fable 5 (claude-fable-5) Signed-off-by: David Galloway <david.galloway@ibm.com>
Run 92's dokan tests all failed with EACCES: vstart writes "mds root ino uid/gid = $(id -u/-g)" into the MDS config, so a cluster started by container root creates a cephfs whose root inode is owned by 0:0 -- and the windows client mounts with client_mount_uid/gid = 1000 and client_permissions enforced, so every write is denied. The VM flow never hit this because vstart ran as the ubuntu user (uid 1000). Start vstart as a uid-1000 user in the container, exactly like the VM flow. The cluster's files on the host are then owned by a subuid, so the conf/keyring copies go through podman cp and the log collection through podman exec + tar. Assisted-by: Claude Fable 5 (claude-fable-5) Signed-off-by: David Galloway <david.galloway@ibm.com>
Contributor
Author
|
vstart-in-container is now validated end to end as well: pipeline run 99 ran the full windows leg on soko12 with completely cold caches — 79 min, SUCCESS, all tests passed (ceph_test_dokan ~15 min, matching the VM-cluster baseline). Cold breakdown: ~12.5 min cross-compile (image + deps + build, fresh node), ~13 min windows VM + vstart build + cluster up, ~48 min tests, ~4 min artifacts. Warm nodes should land around 60-65 min. Two vstart behaviors surfaced during validation (runs 92/97/98), for the record:
The READY wait also fails fast now when the container exits, instead of sitting out its timeout. Whole leg, measured: 114–177 min (old VM flow) → 79 min cold / ~60 min warm, with only the Windows client VM remaining as a VM. |
djgalloway
force-pushed
the
wip-ceph-windows-container-build
branch
from
August 28, 2026 20:04
478708a to
a4d723c
Compare
Runs 97 and 98 failed one file at a time: vstart writes an environment helper (vstart_environment.sh) next to itself in the source tree, and logrotate.conf (plus all of VSTART_DEST) in the build dir -- both root-owned in the container, so the uid-1000 cluster died on Permission denied after coming fully up. Auditing vstart.sh's write surface: everything lands under the build dir, src/vstart_environment.sh, or /tmp -- so chown the build dir to the vstart user wholesale and hand it that one src file. The host-side cleanup already uses sudo rm, so subuid-owned files are fine. Also fail the READY wait as soon as bwc reports the container command failed, instead of sitting out the full 900s timeout on a cluster that already died. Assisted-by: Claude Fable 5 (claude-fable-5) Signed-off-by: David Galloway <david.galloway@ibm.com>
Walk through the leg's steps and how the pieces fit together: the cross-compile container and its two per-builder caches, the libvirt network, the vstart cluster container (--net=host on the virbr0 address, started as uid 1000, and why), the Windows VM as the one remaining VM, and the run_tests VSTART_CONTAINER mode. The vstart step is spelled out in plain terms: nothing prebuilt is downloaded and "Release" is a cmake build type, not a Ceph release -- the PR's own merged tree is compiled inside a container and started as the cluster, so daemon-side changes are tested on both sides of the wire exactly like the VM flow; ccache is a per-builder stash of object files, so only the PR's actual changes recompile. Point the standalone job's README at this section and refresh the stale windows rows in the flow overview. Assisted-by: Claude Fable 5 (claude-fable-5) Signed-off-by: David Galloway <david.galloway@ibm.com>
djgalloway
force-pushed
the
wip-ceph-windows-container-build
branch
from
August 28, 2026 20:05
a4d723c to
149b429
Compare
Contributor
Author
|
The rework at a glance — old VM flow vs this branch vs the goal, on one time scale, plus how the containers, caches, and the Windows VM fit together:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The windows CI cross-compiles Ceph inside a freshly provisioned libvirt Ubuntu VM on every run: download the cloud image, boot, cloud-init, rsync the tree in, build everything (including all the mingw deps) from scratch, rsync ceph.zip back out, destroy the VM. That was ~50 minutes of a recent successful run (ceph-windows-pull-requests #86724) before the cluster-side work even started.
win32_build.shis a plain Linux mingw cross-compile, so this runs it in a podman container on the builder host instead:scripts/ceph-windows/win32-build.containerfile) with the apt packages fromwin32_deps_build.sh's ubuntu branchscripts/ceph-windows/win32_build_containerbind-mounts the workspace and produces the same$WORKSPACE/ceph.zipthatsetup_libvirt_windows_vm/run_testsconsumewin32_deps_build.sh+mingw_conf.sh+ the containerfile, so any deps change simply builds a new cache entry;win32_build.shalready skips the deps build when it finds the cache's completed marker (a flock guards the first build per key)The new
ceph-windows-container-buildjob validates this on the same builder pool the windows test jobs use, without touching ceph-pr-pipeline or the existing windows jobs; it doubles as a one-off manual windows build job for a branch or PR.Validation on braggi07 (ceph main):
vs ~50 minutes for the VM-based build phase it replaces. Not yet validated: running the produced ceph.zip through the actual windows tests — that's the next step before switching ceph-pr-pipeline's windows leg over.
Assisted-by: Claude Fable 5 (claude-fable-5)