Skip to content

ci(render-quarto): pass GITHUB_TOKEN as BuildKit secret to avoid Docker API rate limit#4028

Open
anshul23102 wants to merge 1 commit into
PecanProject:developfrom
anshul23102:ci/render-quarto-github-token
Open

ci(render-quarto): pass GITHUB_TOKEN as BuildKit secret to avoid Docker API rate limit#4028
anshul23102 wants to merge 1 commit into
PecanProject:developfrom
anshul23102:ci/render-quarto-github-token

Conversation

@anshul23102

Copy link
Copy Markdown
Contributor

Summary

The render-notebook CI job was failing intermittently with:

Error: HTTP error 403.
  API rate limit exceeded for <runner IP>.

This happened inside the Docker build step at modules/data.atmosphere, where confirm_deps.R calls remotes::install_deps() when a package dependency is not already cached in pecan/depends:latest. Those calls hit the unauthenticated GitHub API (60 req/hr limit), which is exhausted during concurrent CI runs.

Fix: Enable Docker BuildKit and pass GITHUB_TOKEN from the GitHub Actions environment as a BuildKit secret. The secret is mounted read-only inside the single RUN layer via --mount=type=secret, exported as GITHUB_PAT (the variable remotes consults automatically), and is never written into any image layer -- it does not appear in docker history or image metadata.

Authenticated requests receive 5,000 req/hr, well above what the make dependency checks need.

Changes

  • .github/workflows/render-quarto.yml: enable DOCKER_BUILDKIT=1, pass GITHUB_TOKEN as a BuildKit secret to docker build.
  • docker/base/Dockerfile: use --mount=type=secret,id=GITHUB_PAT in the RUN layer so the token is available as $GITHUB_PAT during make.

Test plan

  • render-notebook CI passes on this PR
  • Verify Docker image still builds correctly (secret is consumed, not stored)

…rate limit

The Docker build step in render-quarto.yml runs make inside the container,
which calls confirm_deps.R -> remotes::install_deps() when any package
dependency is not already present in pecan/depends:latest. Those calls hit
the unauthenticated GitHub API and fail with HTTP 403 when concurrent CI
jobs exhaust the 60 req/hr limit.

Fix: enable Docker BuildKit and pass GITHUB_TOKEN from the GitHub Actions
environment as a BuildKit secret (id=GITHUB_PAT). The secret is mounted
read-only inside the single RUN layer via --mount=type=secret, exported as
GITHUB_PAT (the variable remotes respects), and never written into any image
layer, so it does not appear in docker history or image metadata.
@infotroph

Copy link
Copy Markdown
Member

@anshul23102 Thanks for working on this and for your patience with the wait for review.

I'm curious why you're passing the token only to the base build and not to other builds, especiallydepends since that's where I believe the majority of the API calls in a Docker build session originate.

For context, when we've used this approach in the past (see #3083) we passed the secret to all three of the api, depends, and base builds. They were removed in our last major overhaul of the Docker build (#3354) because it didn't seem needed at the time; now that it seems needed again I'm on board with bringing it back.

Ping @robkooper: Any other factors we should consider before re-adding secrets?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants