fix(git): bound protobuf git-sync memory (shallow clone) + allow disabling refresh (#2521) [backport release-2.8]#2522
Merged
Conversation
…bling refresh (#2521) * fix(git): shallow-clone protobuf git-sync to bound in-memory usage Console clones the configured Git repository into go-git's in-memory object store to read protobuf schema files. The clone had no depth limit, so the entire repository history was materialized (decompressed and un-deltified) in RAM, and the periodic refresh used an incremental tree.Pull() that kept appending every newly fetched object to the same never-pruned store. For a large, active monorepo this is catastrophic: heap profiling of a production Console pod showed ~6.2 GB (92.75% of the heap) held by go-git's MemoryObject.Write under CloneRepository. The target repo packs to ~600 MB on disk but inflates to ~8.5 GB across full history, growing ~0.5 GB/month, eventually OOM-killing the node. Console only ever reads the current revision of the tracked files, so: - Clone shallowly: SingleBranch + Depth 1 + NoTags. Fetches just the tip snapshot instead of the full history. - Refresh by re-cloning instead of pulling. A fresh shallow clone keeps the in-memory store bounded to a single revision (the previous store is released for GC) and avoids go-git's unreliable shallow-pull behavior. CloneRepository already refreshes the file cache and fires OnFilesUpdatedHook; on failure the last good cache is retained. This cuts the footprint from multiple GB to roughly one checkout and makes it insensitive to repository history growth. * feat(git): allow refreshInterval 0 to disable periodic refresh Console proto schemas change rarely, so a frequent git refresh is often unnecessary and — now that each refresh is a shallow re-clone — wasteful on bandwidth. Operators may legitimately want "clone once at startup, refresh only on restart", but Validate() rejected refreshInterval: 0 even though SyncRepo already treated 0 as disabled (and the sample config documented 0 as the way to disable it). Allow it: - config: drop the Validate() rejection of RefreshInterval == 0; document on the field that 0 disables periodic refresh (clone once at startup). - service: guard SyncRepo with <= 0 instead of == 0 so a negative value also disables sync and can't panic time.NewTicker; make the log generic. Default remains 1m, so behavior is unchanged unless 0 is set explicitly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #2521 onto
release-2.8.Cherry-picks
935e3ac0e(fix(git): bound protobuf git-sync memory (shallow clone) + allow disabling refresh). Adapted to the 2.8 codebase, which still useszap(notslog) for logging.What it does
CloneRepositorynow usesSingleBranch + Depth 1 + NoTags, so only the tip snapshot is materialized in go-git's in-memory object store instead of full repo history. Production heap profiling showed ~6.2 GB held byMemoryObject.WriteunderCloneRepositoryfor a large monorepo; this bounds it to roughly one checkout.SyncRepore-clones the shallow snapshot on each tick instead oftree.Pull(), releasing the previous in-memory store for GC and avoiding go-git's unreliable shallow-pull.CloneRepositoryalready updates the file cache and firesOnFilesUpdatedHook; on failure the last good cache is retained.refreshInterval: 0(or negative) is now permitted and disables periodic refresh (clone once at startup). Removed theValidate()rejection of0;SyncRepoguards with<= 0. Default remains 1m, so behavior is unchanged unless0is set explicitly.Backport notes
zap.Error(err)/zap.String(...)to match 2.8 (upstream usedslog).errorsimport fromservice.go.go build,go vet, andgo test ./pkg/git/...pass.