From 7eb5f007b20151b525b2915d9e67a735ca6a8769 Mon Sep 17 00:00:00 2001 From: topkoa Date: Sun, 12 Jul 2026 12:36:04 -0400 Subject: [PATCH 1/2] fix(docker): the published image crash-loops on a fresh volume; smoke-test that it runs The image we published an hour ago cannot start. It builds perfectly, then: PermissionError: [Errno 13] Permission denied: '/app/cache/_roformer-models' [entrypoint] Server exited and auto-update disabled. Exiting. ...and `restart: unless-stopped` turns that into a crash-loop. Cause. /app/cache does not exist in the image (no cache/ in the repo, and it is .dockerignore'd), so `chown -R appuser /app` never touches it. Docker initializes a fresh named volume from whatever sits at the mountpoint in the image - ownership included - and with nothing there it creates the directory as root:root. The container runs as appuser (uid 10001) and dies on its first write. Verified, not inferred: the volume's contents are owned by 0:0, and `id` in the image reports uid=10001. This breaks the DOCUMENTED path exactly as hard: `docker compose up -d` from the README does the same thing. Credit to @mbanks850, whose #5 ("Fix build and permission errors in Dockerfile") hit this - I misread it as build-only and owe them a correction. Fix: create /app/cache and chown it BEFORE the VOLUME instruction, so a fresh volume inherits uid 10001. A green build certified a broken image, so also: a SMOKE TEST in CI. It starts the container the way a user does and requires it to serve /health. Two things about it are deliberate: * It mounts a FRESH NAMED VOLUME. That is the load-bearing part. Without it the container writes to the image's own filesystem, the permission bug never fires, and the test passes on the very bug it exists to catch. * It does not stop at /health. A 200 there is necessary but not sufficient - the server only touches the cache once a job arrives, so it can answer /health and still be broken. The test also writes a probe file into /app/cache AS THE RUNTIME USER, which is the precise thing that failed. It fails fast if the container leaves `running`, rather than burning two minutes polling something a restart policy is bouncing in a loop. README: pulling the fixed image is NOT enough on its own. Docker sets a volume's ownership only when it first creates it, so a volume made by the broken image stays root-owned forever and will keep crash-looping on a perfectly good image. Users must `docker volume rm feedback-demucs-cache`. Without that note, a fixed image still generates "still broken" reports. Signed-off-by: topkoa --- .github/workflows/docker-build.yml | 69 ++++++++++++++++++++++++++++++ Dockerfile | 13 ++++++ README.md | 26 ++++++++++- 3 files changed, 107 insertions(+), 1 deletion(-) diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index bfa59ab..af7565c 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -70,6 +70,75 @@ jobs: cache-from: type=gha cache-to: type=gha,mode=max + # ── Does the image actually RUN? ──────────────────────────────────────── + # + # A green build says the layers assembled. It says nothing about whether the thing + # starts. We published an image that built perfectly and then crash-looped on its + # first run, because a fresh named volume is created root-owned while the container + # runs as uid 10001: + # + # PermissionError: [Errno 13] Permission denied: '/app/cache/_roformer-models' + # + # That is exactly what `docker compose up -d` does, and no amount of building would + # have caught it. So: start the container the way a user does, and require /health + # to answer. + # + # The FRESH NAMED VOLUME is the load-bearing part of this test. Without it the + # container writes to the image's own filesystem, the permission bug never fires, + # and this test would happily pass on the very bug it exists to catch. + # + # SKIP_WARMUP=true: /health binds immediately and answers while the models load; + # warming up would pull ~1.5 GB of weights we don't need in order to answer + # "did the process survive its first write?". + - name: Smoke test — the container must start and serve /health + run: | + set -euo pipefail + docker volume rm -f smoke-cache >/dev/null 2>&1 || true + docker run -d --name smoke \ + -e SKIP_WARMUP=true \ + -v smoke-cache:/app/cache \ + -p 7865:7865 \ + "${{ steps.tag.outputs.scan }}" + + ok="" + for i in $(seq 1 60); do + if curl -fsS --max-time 3 http://127.0.0.1:7865/health >/dev/null 2>&1; then + ok=1; break + fi + # Fail fast rather than burning two minutes on a container that is already dead + # (or, worse, on one that `restart: unless-stopped` is bouncing in a loop). + state="$(docker inspect -f '{{.State.Status}}' smoke 2>/dev/null || echo gone)" + if [ "$state" != "running" ]; then + echo "::error::container is '$state' - it did not stay up" + break + fi + sleep 2 + done + + echo "--- container logs ---" + docker logs --tail 60 smoke 2>&1 || true + echo "----------------------" + + if [ -z "$ok" ]; then + echo "::error::/health never answered. The image builds but does not run." + docker rm -f smoke >/dev/null 2>&1 || true + docker volume rm -f smoke-cache >/dev/null 2>&1 || true + exit 1 + fi + + echo "/health responded:" + curl -fsS --max-time 5 http://127.0.0.1:7865/health + + # Prove the cache is actually WRITABLE as the unprivileged runtime user - the + # precise thing that was broken. /health answering is necessary but not + # sufficient: the server only touches the cache once a job arrives. + docker exec smoke sh -c 'touch /app/cache/.write-probe && rm /app/cache/.write-probe' \ + || { echo "::error::/app/cache is not writable by the runtime user"; exit 1; } + echo "cache dir is writable by the runtime user" + + docker rm -f smoke >/dev/null 2>&1 || true + docker volume rm -f smoke-cache >/dev/null 2>&1 || true + - name: Create output directory run: mkdir -p sbom/ diff --git a/Dockerfile b/Dockerfile index adaa838..5b6a5e3 100644 --- a/Dockerfile +++ b/Dockerfile @@ -99,7 +99,20 @@ RUN out="$(pip install --no-cache-dir --no-deps --only-binary=:all: 'diffq-fixed COPY . . # ---- Least privilege runtime user ---- +# +# /app/cache MUST exist in the image, owned by appuser, BEFORE the VOLUME below. +# +# Docker initializes a fresh named volume from whatever sits at that mountpoint in the +# image — ownership included. With nothing there, it creates the directory as root:root. +# The container runs as appuser (uid 10001), so the server dies on its first write: +# +# PermissionError: [Errno 13] Permission denied: '/app/cache/_roformer-models' +# +# ...and `restart: unless-stopped` turns that into a crash-loop. Not theoretical: it is +# what the freshly published image did on its very first run, and it breaks the documented +# `docker compose up -d` path exactly as hard as anything else. RUN useradd --create-home --uid 10001 appuser \ + && mkdir -p /app/cache \ && chown -R appuser:appuser /app diff --git a/README.md b/README.md index bd05c9e..ffee97f 100644 --- a/README.md +++ b/README.md @@ -190,10 +190,34 @@ docker run --gpus all -p 7865:7865 slopsmith-demucs-server # Pull from GHCR and run (CPU) docker compose up -d -# GPU mode: uncomment runtime: nvidia + NVIDIA_* env vars in compose file +# GPU mode: uncomment `gpus: all` in the compose file (needs nvidia-container-toolkit; +# Linux or Windows/WSL2 only — macOS cannot pass a GPU through at all) docker compose up -d ``` +> #### ⚠️ Ran an image from before 2026-07-12? Delete the cache volume. +> +> Early images created their model-cache volume **owned by root**, while the server runs as +> an unprivileged user (uid 10001). The container would start and then immediately die with +> +> ``` +> PermissionError: [Errno 13] Permission denied: '/app/cache/_roformer-models' +> ``` +> +> ...which `restart: unless-stopped` turns into a crash-loop. +> +> The image is fixed — but **pulling the new image is not enough on its own**. Docker sets a +> volume's ownership only when it *first creates* it, so a volume made by an older image stays +> root-owned forever and will keep crash-looping on a perfectly good image. Remove it: +> +> ```bash +> docker compose down +> docker volume rm feedback-demucs-cache # or: slopsmith-demucs-server_demucs-cache +> docker compose up -d +> ``` +> +> The volume only holds cached model weights — deleting it costs you a re-download, nothing else. + ### Persistent model cache Model weights are stored in `/app/cache` inside the container. The compose file maps this to a persistent volume so weights survive restarts: From 6a984bdb0674f9adb5aa9be68f1e8d56ffe34b56 Mon Sep 17 00:00:00 2001 From: topkoa Date: Sun, 12 Jul 2026 12:44:07 -0400 Subject: [PATCH 2/2] docs: tag the fenced block (MD040) The PermissionError sample in the volume-permissions warning had no language on its fence. Signed-off-by: topkoa --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ffee97f..1fb8cf2 100644 --- a/README.md +++ b/README.md @@ -200,7 +200,7 @@ docker compose up -d > Early images created their model-cache volume **owned by root**, while the server runs as > an unprivileged user (uid 10001). The container would start and then immediately die with > -> ``` +> ```text > PermissionError: [Errno 13] Permission denied: '/app/cache/_roformer-models' > ``` >