Basic authentication probe fails for OCI image indexes when the registry requires an explicit Accept header

[crliu3227]

Description

Enroot 4.2.1 fails to import an OCI image index from a private registry using Basic authentication.
The credentials are valid, the repository is accessible, and the requested tag exists. However, the registry returns 404 Not Found when the manifest is requested without an explicit OCI Accept header. The same request succeeds with 200 OK when the appropriate Accept header is provided.
Enroot performs an authenticated manifest request without an Accept header during docker::_authenticate(). This request fails before Enroot reaches the later manifest-download code, which does provide the correct Docker and OCI media types.
The underlying HTTP error is also hidden by the pipeline used in the Basic authentication branch, making the failure appear to be a generic credential problem.

Environment

• Enroot: 4.2.1
• Enroot library: /usr/lib/enroot/docker.sh
• Authentication method: HTTP Basic
• Registry: private Docker Registry V2-compatible registry
• Image media type: application/vnd.oci.image.index.v1+json
• Enroot invoked through Pyxis/Slurm
• Credentials supplied through $ENROOT_CONFIG_PATH/.credentials

Credentials format:
machine registry.example.com login admin password <redacted>

Steps to reproduce

Run a containerized Slurm job using an OCI image index hosted in a private registry:

srun \
  --container-image=registry.example.com/project/alpine:test-tag \
  cat /etc/os-release

Alternatively, the same authentication path can be exercised directly through Enroot:

enroot digest \
  "docker://registry.example.com#project/alpine:test-tag"

Actual behavior

Pyxis reports:

pyxis: importing docker image: registry.example.com/project/alpine:test-tag
error: pyxis: child failed with error code: 1
error: pyxis: failed to import docker image
error: pyxis: printing enroot log file:
error: pyxis:     [INFO] Querying registry for permission grant
error: pyxis:     [INFO] Authenticating with user: admin
error: pyxis:     [INFO] Using credentials from file: /path/to/.credentials
error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
error: Failed to invoke spank plugin stack

There is no Authentication succeeded message and no HTTP error in the Enroot log.

Registry checks

The registry advertises Basic authentication:

$ curl -sS -I \
    https://registry.example.com/v2/project/alpine/manifests/test-tag

HTTP/2 401
www-authenticate: Basic realm="Registry Realm"

The credentials are valid:

$ curl -sS \
    --netrc-file .credentials \
    -o /dev/null \
    -w "%{http_code}\n" \
    https://registry.example.com/v2/

200

The repository is accessible:

$ curl -sS \
    --netrc-file .credentials \
    -o /dev/null \
    -w "%{http_code}\n" \
    https://registry.example.com/v2/project/alpine/tags/list

200

The requested tag is present in the tags response.
An authenticated manifest request without an explicit Accept header returns 404:

$ curl -sS \
    --netrc-file .credentials \
    -o /dev/null \
    -w "%{http_code}\n" \
    https://registry.example.com/v2/project/alpine/manifests/test-tag

404

The same request succeeds when OCI and Docker manifest media types are advertised:

$ curl -sS \
    --netrc-file .credentials \
    -H "Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json" \
    -o /dev/null \
    -w "%{http_code} %{content_type}\n" \
    https://registry.example.com/v2/project/alpine/manifests/test-tag

200 application/vnd.oci.image.index.v1+json

Suspected root cause

In Enroot 4.2.1, the Basic authentication branch performs an authenticated request to the manifest URL without an Accept header:

Basic)
    common::curl "${curl_opts[@]}" -G -v \
      ${req_params[@]+"${req_params[@]}"} \
      -- "${url}" 2>&1 > /dev/null \
      | awk '/Authorization: Basic/ { sub(/\r/, "", $4); print $4 }' \
      | common::read -r token
    ;;

Source:
https://github.com/NVIDIA/enroot/blob/v4.2.1/src/docker.sh#L66-L71
The correct manifest media types are defined later in docker::_download():

local accept_manifest_list=(
    "-H"
    "Accept: application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.index.v1+json"
)

local accept_manifest=(
    "-H"
    "Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.oci.image.manifest.v1+json"
)

However, execution never reaches that code because the Basic authentication probe already failed with HTTP 404.
Additionally, the Basic authentication pipeline redirects stderr into awk, which only preserves the Authorization: Basic line. The actual 404 error generated by common::curl is therefore discarded. With pipefail enabled, Enroot exits with status 1 without printing the underlying HTTP error.

Expected behavior

Enroot should successfully authenticate and import the OCI image index.
The Basic authentication probe should either:
1. Send the same supported manifest media types as the subsequent manifest requests; or
2. Validate Basic credentials against the registry /v2/ endpoint instead of the manifest URL.
Enroot should also preserve and report HTTP errors from the Basic authentication request.

Possible fix

One possible minimal fix is to add the supported manifest media types to the Basic authentication request:

 Basic)
     # Check that we have valid credentials and save them if successful.
-    common::curl "${curl_opts[@]}" -G -v ${req_params[@]+"${req_params[@]}"} -- "${url}" 2>&1 > /dev/null \
+    common::curl "${curl_opts[@]}" -G -v \
+      -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.oci.image.manifest.v1+json" \
+      ${req_params[@]+"${req_params[@]}"} -- "${url}" 2>&1 > /dev/null \
       | awk '/Authorization: Basic/ { sub(/\r/, "", $4); print $4 }' \
       | common::read -r token
     ;;

This change was tested temporarily against the affected registry and allowed the authenticated manifest request to return the OCI image index successfully.