Summary
When skills-init fails partway through its gitRefs list (e.g. a transient network error, or an auth failure on a later private repo), every subsequent restart of the init container fails at the first already-cloned ref with:
fatal: destination path '/skills/demo' already exists and is not an empty directory.
skills-init: clone https://github.com/giantswarm/agent-skills: exit status 128
The pod then sits in Init:CrashLoopBackOff forever — the retry loop can never succeed, even if the original cause (network blip, rate limit) has long cleared. The only recovery is deleting the pod by hand to get a fresh emptyDir.
Root cause
The skills volume is an emptyDir, which persists across container restarts within the same pod. The comment on Run() in go/core/internal/skillsinit/runner.go assumes otherwise:
successful operations before the failure are left in place on disk (the container restarts and re-runs from scratch)
The re-run is not from scratch: earlier successful clones are still on disk, and CloneGit does not tolerate an existing non-empty destination. The same applies to the OCI export and S3 fetch stages.
Reproduction
- Agent with ≥2
gitRefs, where ref 2 fails (easiest: a private repo with an invalid token in gitAuthSecretRef).
- Attempt 1: ref 1 clones fine, ref 2 fails → init container exits.
- Attempt 2+: ref 1 fails with
destination path already exists, masking the real error. Pod is stuck permanently.
Observed on skills-init:0.9.9.
Suggested fix
Make each fetch idempotent, e.g. os.RemoveAll(ref.Dest) before cloning (or clone into a temp dir and rename into place). Alternatively wipe the skills root at the start of Run(). This also keeps the real error visible in the logs on every retry instead of the misleading "already exists" failure.
Summary
When
skills-initfails partway through itsgitRefslist (e.g. a transient network error, or an auth failure on a later private repo), every subsequent restart of the init container fails at the first already-cloned ref with:The pod then sits in
Init:CrashLoopBackOffforever — the retry loop can never succeed, even if the original cause (network blip, rate limit) has long cleared. The only recovery is deleting the pod by hand to get a freshemptyDir.Root cause
The skills volume is an
emptyDir, which persists across container restarts within the same pod. The comment onRun()ingo/core/internal/skillsinit/runner.goassumes otherwise:The re-run is not from scratch: earlier successful clones are still on disk, and
CloneGitdoes not tolerate an existing non-empty destination. The same applies to the OCI export and S3 fetch stages.Reproduction
gitRefs, where ref 2 fails (easiest: a private repo with an invalid token ingitAuthSecretRef).destination path already exists, masking the real error. Pod is stuck permanently.Observed on
skills-init:0.9.9.Suggested fix
Make each fetch idempotent, e.g.
os.RemoveAll(ref.Dest)before cloning (or clone into a temp dir and rename into place). Alternatively wipe the skills root at the start ofRun(). This also keeps the real error visible in the logs on every retry instead of the misleading "already exists" failure.