Skip to content

Measure Go cache strategies - #2702

Closed
yuandrew wants to merge 7 commits into
mainfrom
codex/ci-cache-experiment
Closed

yuandrew wants to merge 7 commits into
mainfrom
codex/ci-cache-experiment

Conversation

@yuandrew

Copy link
Copy Markdown
Contributor

What was changed

Add a temporary CI workflow comparing current setup-go, CloudX, and split
rolling build caches on Linux, macOS, and Windows. Measure compile-only work,
mandatory tests, transfers, cache sizes, and an Ubuntu prebuild lane.

Why?

Determine whether fresher Go build caches reduce end-to-end CI time without
causing storage churn. This draft PR is experimental and must not merge.

Checklist

  1. Closes: N/A

  2. How was this tested:

  • go run . check
  • go run . unit-test -run '^$'
  • go test -count 1 -race -run '^$' ./... from test
  • JavaScript syntax and trimming fixture checks
  1. Any docs updates needed? No.

Compare current, rolling split, and CloudX cache strategies across representative CI runners. Capture compilation, mandatory test, transfer, storage, and prebuild costs.
Restore source and dependency files after collecting cold, warm, source, and dependency CI measurements.
@yuandrew

yuandrew commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor Author

sdk-go Go cache proposal

Decision

Do not adopt cloudx-io/setup-go or enable rolling caches across the full
matrix. Pilot a weekly-refreshed, workload-isolated cache only for the Linux
x64 / stable unit lane. Keep every PR restore-only; let one trusted main
job write. Keep GOMODCACHE immutable and separate from GOCACHE.

Expected pilot effect: little change when the current cache is fresh, but a
material reduction in stale-cache tail latency. In six Linux unit trials,
current and split were equal on a freshly seeded dependency repeat (217 vs
218 s), while split cut stale/source trials from 516/524 s to 211/240 s.
The broad rollout is rejected because macOS and Windows rolling entries were
evicted before the immediate dependency repeat.

Evidence set

  • Historical baseline: 20 successful main CI runs, run IDs
    34991624608, 34923306274, 34900073798, 34829597969,
    34809343629, 34681938011, 34295799351, 34255776899,
    34244427267, 34072004965, 33645466289, 33643692215,
    33539380752, 33186142418, 33184352867, 33121087245,
    33082674159, 33011764157, 32998333434, 32884527617.
  • Experiment: draft PR #2702,
    branch codex/ci-cache-experiment, final cleanup SHA c1b276a8, Go 1.27.1.
  • Runs: cold,
    warm 1,
    warm 2,
    source,
    dependency,
    dependency repeat.
  • Merge SHAs by phase: 703a88c0, 1ef5610f, 0f72f399, 2aa8b1fb,
    16541ba0, c0c03242.
  • Standard public GitHub-hosted runners: Linux x64, Windows x64, macOS Intel
    x64. Historical data also covers macOS arm64 and Go oldstable.

Percentiles use linear interpolation. Times are seconds and shown as
p50/p75/p90.

Historical baseline

Twenty successful samples per row. work is the mandatory check/test step;
post is Post Setup Go.

Job Runner Go Setup Work Post Total
check Linux x64 stable 20/27/31 103/122/244 0/1/10 127/147/269
check macOS arm64 stable 52/61/68 121/146/202 1/1/32 184/225/263
check Windows x64 stable 106/114/133 181/269/394 0/1/25 305/453/462
unit Linux x64 stable 22/31/34 309/446/460 0/10/11 344/473/484
unit Linux x64 oldstable 9/14/24 362/440/452 9/10/13 384/456/474
unit macOS Intel x64 stable 110/151/169 322/631/841 1/1/68 498/704/923
unit macOS Intel x64 oldstable 118/142/179 387/657/739 1/11/46 586/734/827
unit macOS arm64 stable 45/52/54 216/438/457 0/1/38 281/495/526
unit macOS arm64 oldstable 42/55/66 230/318/378 0/1/22 301/363/411
unit Windows x64 stable 121/135/165 294/513/658 1/1/29 445/627/753
unit Windows x64 oldstable 106/117/131 285/563/607 1/1/31 435/625/684
integration, default Linux x64 stable 26/30/34 631/642/669 0/1/2 666/674/687
integration, default Linux x64 oldstable 8/12/14 599/672/681 0/2/3 617/684/694
integration, default macOS Intel x64 stable 70/73/87 672/729/782 0/1/8 764/815/865
integration, default macOS Intel x64 oldstable 55/67/76 683/750/824 1/2/21 761/860/893
integration, default macOS arm64 stable 48/53/57 649/691/730 0/0/7 700/757/795
integration, default macOS arm64 oldstable 25/27/30 625/657/693 0/1/9 661/691/735
integration, default Windows x64 stable 117/129/145 671/678/717 1/1/5 801/818/837
integration, default Windows x64 oldstable 45/55/64 621/631/715 1/1/9 692/717/753
integration, cache off Linux x64 stable 22/27/29 647/654/679 0/1/2 673/687/699
integration, cache off Linux x64 oldstable 7/11/12 607/664/673 1/2/3 625/677/688
integration, cache off macOS Intel x64 stable 66/80/90 703/722/814 1/1/12 800/841/904
integration, cache off macOS Intel x64 oldstable 45/64/75 669/724/815 0/3/13 746/822/878
integration, cache off macOS arm64 stable 51/54/61 653/671/688 0/1/6 711/732/755
integration, cache off macOS arm64 oldstable 23/29/31 615/632/664 0/0/7 651/674/693
integration, cache off Windows x64 stable 112/129/135 671/683/714 0/1/7 800/818/845
integration, cache off Windows x64 oldstable 42/57/93 632/655/724 0/1/9 704/736/768

Stable resolved to Go 1.27.1 during the experiment; oldstable cache keys show
Go 1.26.8. Current unit archives were about 833–882 MB compressed. Check and
root-only integration archives were about 240–250 MB. Six current multi-module
unit archives occupied about 5.1 GiB.

Controlled experiment

Every unit lane ran canonical compile-only and full commands:
go run . unit-test -run '^$', then go run . unit-test. Integration ran a
compile-only pass, then go run . integration-test -dev-server. The build tool
kept -count 1, -race, coverage wiring, all modules, and all tests.

Phase Cache state Linux current/split macOS current/split Windows current/split Integration current/split
cold split miss 273/376 fail/1189 608/1023 685/676
warm 1 prior build restored 516/211 484/685 541/464 707/621
warm 2 repeated source 217/240 532/445 874/521 682/619
source only one private constant, same value 524/240 478/560 859/464 672/636
dependency all go.sum keys cold 513/521 1163/1208 867/1024 660/714
dependency repeat immediate repeat 217/218 379/fail (1529) 431/995 677/629

Across all successful phases:

Workload/runner Strategy n/fail Restore Compile Mandatory test Save Total
unit Linux x64 current 6/0 16/24/29 219/348/354 144/144/145 4/8/9 393/515/520
unit Linux x64 split 6/0 21/30/32 64/195/292 130/141/146 7/11/12 240/342/449
unit Linux x64 CloudX 4/0 23/28/32 63/147/271 145/145/145 13/13/13 250/333/451
unit macOS Intel current 5/1 118/140/141 88/122/547 237/252/265 1/2/19 484/532/911
unit macOS Intel split 5/1 104/128/169 121/799/818 249/260/295 53/63/67 685/1189/1200
unit macOS Intel CloudX 1/3 48/48/48 641/641/641 186/186/186 50/50/50 942/942/942
unit Windows x64 current 6/0 66/116/135 392/580/582 205/207/208 15/29/31 734/865/871
unit Windows x64 split 6/0 94/121/122 394/697/736 205/214/217 20/25/29 758/1016/1024
unit Windows x64 CloudX 3/1 115/121/125 97/328/466 205/205/205 36/114/161 648/742/798
integration Linux x64 current 6/0 15/20/21 61/62/62 599/607/619 1/3/3 680/684/696
integration Linux x64 split 6/0 16/22/23 6/36/54 592/603/617 5/5/6 633/666/695
integration Linux x64 CloudX 4/0 15/16/16 5/21/46 597/606/621 6/6/7 630/654/689

The warm Linux split samples separated restore, compile, test, and save. For
example warm 1 was 21 s restore, 41 s compile, 122 s mandatory tests, and 5 s
save; source-only was 18/70/130/6 s. Current warm 2 was 33/44/131/0 s. Thus a
fresh current cache is as fast; rolling helps when the immutable cache has
aged across source changes.

-count=1 disables Go test-result reuse. Mandatory test time remained roughly
118–147 s on Linux. The gain is compiled package reuse only.

Cache bytes and retention

Runner/workload Modules Build Combined CloudX/current Warm transfer
Linux unit 400 MB 478–539 MB 892–939 MB about 878–939 MB
macOS Intel unit 395 MB 506–556 MB about 901 MB about 901 MB
Windows unit 391 MB 488–535 MB 879 MB about 879 MB
Linux integration shared 400 MB 195–205 MB 254–265 MB current root cache about 595 MB split

Separation does not materially reduce restore bytes for a unit job. It reduces
each rolling upload from roughly 0.9 GB to roughly 0.5 GB and lets unit and
integration share the immutable module archive.

Repository cache use began at 10,794,448,681 bytes/16 entries, reached
16,175,664,368 bytes/25 entries after two rolling generations, and fell to
11,380,779,408 bytes/18 entries after deleting 5.4 GB of superseded experiment
entries. GitHub evicted other entries during the trials. In the dependency
repeat, Linux build entries remained, but the macOS build+module and Windows
build entries had already disappeared. Their “repeat” was cold.

Cleanup deleted 15.8 GB of superseded/final PR-scoped cache entries. The final
repository snapshot was 9,925,584,826 bytes/18 entries, with zero PR #2702
cache entries. Main-scoped caches were not deleted.

CloudX

CloudX matched split on warm Linux (211 vs 211 s) but stores modules and build
outputs together, writes a new combined archive every successful run, and keys
the requested stable string rather than the resolved Go version. Windows
warm 2 spent 115 s restoring and 192 s saving; total was 648 s versus 521 s
split. macOS had two test/network failures and required a cold 942 s seed.
The useful idea is rolling, workload-specific build state—not this action as-is.

Prebuild

The prebuild producer and consumer are serial. Their critical-path and total
runner seconds were 420 s cold, then 571, 561, and 545 s. Direct split unit was
376, 211, 240, and 240 s. There is only one unit consumer per OS/Go lane, and
integration compile cost is 2–62 s, so prebuild adds wall and runner time. This
public repository pays $0 for standard runners, but the extra runner occupancy
and latency remain.

Failures

  • Cold macOS current: contrib/googleadk timed out downloading/extracting the
    Temporal CLI; unrelated to cache correctness.
  • Cold and warm-1 macOS CloudX: proxy.golang.org/download failures.
  • Source macOS and Windows CloudX: mandatory test failures after successful
    setup/compile; excluded from successful quantiles.
  • Dependency-repeat macOS split: the same external Temporal CLI download
    timeout after a 995 s cold compile; excluded from successful quantiles.
  • All current and split dependency jobs passed. Local canonical check and
    compile-only unit runs passed before each source/dependency trial.

Recommendation

  1. Pilot Linux x64 / stable unit only for two weeks.
  2. Use official actions/setup-go with cache: false, then pinned
    actions/cache/restore and actions/cache/save.
  3. Key immutable modules by OS, architecture, resolved Go version, and all
    go.sum files.
  4. Key GOCACHE by the same fields plus unit and an ISO week bucket. Restore
    the newest matching dependency/workload prefix.
  5. Save only from a successful push to main; PRs only restore. One job is
    the sole writer. Trim build entries unused since two hours before job start.
  6. Emit cache hit, restored/saved bytes, raw bytes, and step durations. Compare
    weekly p50/p75/p90 against the baseline above.
  7. Continue only if p90 improves at least 20% without pushing repository cache
    use above its existing limit or raising failure rate. Otherwise revert the
    custom steps to the current actions/setup-go block.
  8. If the pilot holds, test the shared Linux/stable integration build cache.
    Do not expand to macOS/Windows without demonstrated retention and net gain.

Weekly rotation bounds writes while refreshing source state. A source-only
change showed safe partial invalidation; a dependency change intentionally
starts a new module/build generation. PR restore-only follows GitHub cache
scope: PRs can read base/default-branch caches but PR-created caches are scoped
to the merge ref and are not useful to main.

Exact pilot change

In the Linux/stable unit matrix lane, replace built-in setup caching with:

- name: Setup Go
  if: matrix.os != 'ubuntu-latest' || matrix.go-version != 'stable'
  uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
  with:
    go-version: ${{ matrix.go-version }}
    cache-dependency-path: '**/go.sum'

- name: Setup Go
  if: matrix.os == 'ubuntu-latest' && matrix.go-version == 'stable'
  id: go
  uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
  with:
    go-version: ${{ matrix.go-version }}
    cache: false

- name: Resolve Go cache
  if: matrix.os == 'ubuntu-latest' && matrix.go-version == 'stable'
  id: go-cache
  shell: bash
  run: |
    echo "build=$(go env GOCACHE)" >> "$GITHUB_OUTPUT"
    echo "modules=$(go env GOMODCACHE)" >> "$GITHUB_OUTPUT"
    echo "week=$(date -u +%G-%V)" >> "$GITHUB_OUTPUT"
    echo "CACHE_JOB_START_MS=$(node -e 'process.stdout.write(String(Date.now()))')" >> "$GITHUB_ENV"

- name: Restore Go modules
  if: matrix.os == 'ubuntu-latest' && matrix.go-version == 'stable'
  id: modules
  uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5.0.3
  with:
    path: ${{ steps.go-cache.outputs.modules }}
    key: go-mod-${{ runner.os }}-${{ runner.arch }}-${{ steps.go.outputs.go-version }}-${{ hashFiles('**/go.sum') }}

- name: Restore Go build cache
  if: matrix.os == 'ubuntu-latest' && matrix.go-version == 'stable'
  id: build
  uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5.0.3
  with:
    path: ${{ steps.go-cache.outputs.build }}
    key: go-build-${{ runner.os }}-${{ runner.arch }}-unit-${{ steps.go.outputs.go-version }}-${{ hashFiles('**/go.sum') }}-${{ steps.go-cache.outputs.week }}
    restore-keys: |
      go-build-${{ runner.os }}-${{ runner.arch }}-unit-${{ steps.go.outputs.go-version }}-${{ hashFiles('**/go.sum') }}-

After the unchanged canonical unit step, trim and save only on trusted main:

- name: Trim Go build cache
  if: success() && matrix.os == 'ubuntu-latest' && matrix.go-version == 'stable' && github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.build.outputs.cache-hit != 'true'
  run: node .github/scripts/trim-go-cache.js "${{ steps.go-cache.outputs.build }}"

- name: Save Go modules
  if: success() && matrix.os == 'ubuntu-latest' && matrix.go-version == 'stable' && github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.modules.outputs.cache-hit != 'true'
  uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5.0.3
  with:
    path: ${{ steps.go-cache.outputs.modules }}
    key: ${{ steps.modules.outputs.cache-primary-key }}

- name: Save Go build cache
  if: success() && matrix.os == 'ubuntu-latest' && matrix.go-version == 'stable' && github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.build.outputs.cache-hit != 'true'
  uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5.0.3
  with:
    path: ${{ steps.go-cache.outputs.build }}
    key: ${{ steps.build.outputs.cache-primary-key }}

Keep current actions/setup-go caching for every non-pilot lane. The production
trim script should accept the recorded start time, retain the two-hour Go mtime
granularity margin, and log removed/stored bytes.

Sources and limitations

Limitations: six phase samples are enough to expose direction, cold cost, and
eviction, but not seasonal runner/network variance. macOS CloudX has too few
successful warm samples. The weekly cadence is inferred from per-run rolling
measurements and must be validated by the pilot. Experiment cache writes
temporarily perturbed current-cache retention; that effect is itself evidence
against a full-matrix rollout.

Created resources

  • Remote branch: codex/ci-cache-experiment; final SHA c1b276a8.
  • Draft PR: #2702, open and
    unmerged.
  • Proposal comment: issuecomment-5713293201.
  • Local worktree: /private/tmp/sdk-go-cache-experiment.
  • Local report: /private/tmp/sdk-go-cache-proposal.md.
  • Local ignored test logs: /private/tmp/sdk-go-cache-experiment/.build/test-logs.
  • Local temporary tidy cache: /private/tmp/sdk-go-cache-tidy-build.
  • PR-scoped remote caches: all 20 final entries (10,401,171,086 bytes) and
    15.8 GB total across cleanup waves were deleted; they are not recoverable.

All workflow runs created by the branch are below. Other PR and PR target
are repository workflows whose API names are empty.

Phase/SHA Experiment CI Changelog Govulncheck Other PR PR target
cold 703a88c0 35203165686 fail 35203166329 pass 35203166316 fail 35203165561 pass 35203165509 pass 35203165530 skip
warm 1 1ef5610f 35205207596 fail 35205208148 fail 35205208248 fail 35205207525 pass 35205207514 pass 35205203297 skip
warm 2 0f72f399 35206309553 pass 35206310339 pass 35206310278 fail 35206309774 pass 35206309629 pass 35206307883 skip
source 2aa8b1fb 35207328587 fail 35207330880 fail 35207331565 fail 35207328611 pass 35207328458 pass 35207326162 skip
dependency 16541ba0 35208494210 pass 35208494656 fail 35208494710 fail 35208494176 pass 35208494212 pass 35208492644 skip
dependency repeat c0c03242 35211027388 fail 35211027930 fail 35211028463 fail 35211027547 pass 35211027532 pass 35211024222 skip
cleanup c1b276a8 — — — — — 35213627940 skip

The original worktree /Users/andrewyuan/all/sdk-go-cache remains clean on
investigate-go-cache at 34a4ec27. The isolated worktree is also clean.

@yuandrew yuandrew closed this Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant