init and sync produce different symbol counts for the same code (1.4.1)

[jeremypetz]

codegraph 1.4.1, macOS 15 on Apple Silicon, node 22. Swift project, about 250
source files, one package.

codegraph init on a clean checkout finds fewer symbols than the same project
has after every file has been re-parsed one at a time by sync. The files are
byte-identical in both cases.

I ran into this because I build the index in CI as well as locally and the
counts didn't match. Assumed my local one had gone stale, but it's the opposite.
The local one gets built a file at a time by the daemon as I edit, and that path
finds more.

Easiest one to look at is a 994-line test file. init gives it 45 symbols.
Change one byte in the file, sync, and now it's 71. Revert the change, sync
again, still 71.

That file is clean in one direction: 26 symbols that show up after sync and
never after init, and nothing init had that sync dropped. The 26 are all in
lines 796-943. init gets everything up to 795, nothing in that range, then
starts again at 944.

The resume point is what I don't understand. There's a private final class at
905, first method at 913, and init records neither of those, but it does record
the same class's methods from 944 onwards. So it's not skipping the declaration,
it's more like it lost its place and found it again lower down.

Nothing unusual in the skipped range as far as I can tell. No raw strings or
#if. It's private helper funcs, a couple of small enums and three private
classes. The only interpolation is a \(value) inside an XCTAssert message.

Project-wide, on a fresh clone:

codegraph init .                                    5141 symbols
loop over all 236 files, one comment + sync each    5224 symbols

8 of the 236 came out short under init:

+34  26 KB editor view
+26  the 45 KB test file above
+10  17 KB document scanner
 +5  4 KB model file
 +4  6 KB contract test
 +2  33 KB document builder
 +1  editor extension
 +1  65 KB service extension

Two separate fresh clones both gave exactly 5141 and exactly 45 in that test
file, so it's not flaky.

I assumed the workaround would be to run init and then force a re-sync. It
isn't. Changing one file and syncing gets the symbols back. Ten files, same.
Change all 220 Swift files and sync, and you get 5141 again, exactly matching
init. So there seem to be two parse paths with something picking between them on
how many files changed, and the bulk one drops things. Would explain why init is
always affected. I didn't try to find where the cutoff is.

(touch on its own does nothing, so I assume you're hashing content rather than
looking at mtimes.)

The part I couldn't work out

The test file above is the tidy case. For most of the other seven the two
indexes disagree in both directions, and that holds even if I ignore line
numbers and compare on name and kind only.

Worst is the editor view: 74 symbols after the per-file re-parse, 40 after init,
but 71 that only the re-parse has and 37 that only init has. Net is +34 which
lines up, but neither is a subset of the other. I don't have an explanation and
didn't want to guess in a bug report. Might be a second issue. Can dump the full
per-symbol comparison for any of these if it's useful.

Two things I ruled out, in case it saves anyone time. Not visibility: of the
symbols dropped in the clean case 13 are private and 9 internal, and init kept 4
private ones in that same file. Not byte offsets either, the gaps start anywhere
from byte 14 to byte 52797 in files ranging 4 KB to 66 KB, so it isn't a fixed
chunk boundary.

Why this matters to me

I build test-selection data off the index. A symbol that isn't in the graph gets
no tests mapped to it and quietly falls back to running the entire suite.
Nothing fails, it's just slower, and there's no way to tell the data was
incomplete. Recovering those 83 symbols got me 44 targeted test scopes I didn't
have.

More generally I can't trust an index I didn't build incrementally, which rules
out building one in CI.

Repro

git clone <repo> /tmp/repro && cd /tmp/repro
codegraph init .
sqlite3 .codegraph/codegraph.db "SELECT COUNT(*) FROM nodes;"

git ls-files '*.swift' | while read f; do
  printf '\n// reparse\n' >> "$f"
  codegraph sync . >/dev/null 2>&1
done
sqlite3 .codegraph/codegraph.db "SELECT COUNT(*) FROM nodes;"

I haven't got it down to a small standalone file because I don't know what
triggers it. Happy to test a patch or pull specific data out of my index.

[colbymchenry]

Thank you for the unusually precise accounting — the per-file breakdown (45 vs 71 on the 994-line test file, the contiguous 796–943 gap, only the larger files affected, and counts sticking until a file is edited) is what let us find the mechanism. Fixed in #1575; details below, and one ask at the end.

What we found. init parses through a pool of workers with a crash/timeout retry ladder; per-file sync parses one file at a time in a fresh process. On 1.4.1's old Swift grammar — slow and memory-fragile on exactly your larger files — a file can fail its first attempts and be salvaged by a last-resort pass that stores whatever a degraded parse produced: silently, with the failure erased from the summary, under the full file's content hash — so every later sync sees "unchanged" and never repairs it, while your per-edit syncs (fresh process, clean parse) get the full symbol set. That reproduces your exact experience: init < sync, only on the big files, sticky per file until edited. We forced the ladder end-to-end on a public Swift repo and watched init store less and a plain sync silently "find" the difference — the class of divergence you measured. The one piece we couldn't verify without your file is the exact shape of the degraded region (the 796–943 gap with the private final class at 905 dropped but its later methods kept is consistent with the parser's error recovery in a degraded parse).

What's changing in the next release:

• A salvaged file now carries a visible warning ("symbols may be incomplete") instead of being reported as cleanly indexed — an init can no longer quietly disagree with a later re-parse of the same bytes.
• A related retry bug that stored empty results outright on v1.5.0's native extraction path is fixed, and affected rows self-heal on the next sync (v1.5.0 Python parser indexes valid file as 0 symbols; v1.4.1 indexes correctly #1541).
• Independently, v1.5.0 already replaced the Swift grammar and moved Swift to native extraction, which removes most of the original trigger (the old grammar's slow/OOM parses on large files).

The ask: after the next release, rebuild once and re-run your init-vs-sync comparison. If any file still comes out short, please reopen with the file (or share it privately) — with your 45 KB test file in hand we can pin the degraded-region shape exactly.

Closing as fixed-in-next-release; reopen any time if it reproduces.