[jess-lowe]

Description

Resolves out-of-memory (OOM) crashes in combine-to-osv when processing large datasets with populated affected packages/commits.

Root Causes

1. Excessive GCS Writer Memory Allocation: cloud.google.com/go/storage's Writer defaults ChunkSize to 16 MiB. With 64 concurrent upload workers continuously writing small JSON records (~2–10 KiB), this produced gigabytes of buffer allocations per second.
2. Long-lived In-Memory Structures:
   • cve5Files and nvdFiles (~300k strings each) remained reachable on main()'s stack for the entire run.
   • workItems (300k items) remained live in memory while processing.
   • validIDs accumulated 300k strings even when -sync-deletions was false.
3. Reference Range Checking Inconsistency: isReferencesOnly only checked the top-level fields["source"], missing extracted_events with source: "REFERENCES".

Changes

1. Configured ChunkSize: Set wc.ChunkSize = googleapi.MinUploadChunkSize (256 KiB) in gcs.UploadToGCS and writer.uploadIfChanged.
2. Released Memory Early:
   • Nil out cve5Files and nvdFiles immediately after building the work queue.
   • Delete items from workItems map as they are dispatched and nil out the map after dispatch.
   • Only accumulate validIDs if *syncDeletions && *uploadToGCS is true; use an atomic counter validCount for progress and summary logging.
3. Updated isReferencesOnly: Check extracted_events for source: "REFERENCES" (mirroring isCPERange).
4. Cleaned up tests in main_test.go: Added helper mustParseDBSpecific using protojson.Unmarshal to clean up verbose manual structpb.Struct declarations.

Testing

• cd vulnfeeds && go test ./... passed.
• poetry run tools/lint_and_format.sh passed with 0 findings.

[michaelkedar]

this might slow things down considerably?