Derive content reference checksums locally instead of trusting publishers

[allister-beamable]

Closes #4838.

The problem

ContentFile.GetStatus() decided whether content was modified by comparing two values that are not comparable:

• PropertiesChecksum, computed locally by CalculateChecksum over our own canonical serialization, and
• ReferenceContent.checksum, a string lifted verbatim out of the remote manifest.

The second one is whatever the publisher put there. Nothing recomputes or validates it. So the real contract was never "sort your keys" — it was "byte-for-byte reproduce System.Text.Json under one undocumented options object, then SHA1 it." A Game Maker publishing through their own tooling cannot honor that except by accident, and when they do not, every item shows a permanent "modified" badge that survives a Fetch.

Field order is only one of the ways to miss. Others include string escaping (JavaScriptEncoder.Default renders Café & Co as Caf\u00E9 \u0026 Co), raw number text (1.0 and 1 both round-trip as written), and keys differing only in case.

The change

Stop trusting the publisher's checksum. Content files gain a reference object holding what the remote payload hashed to under our canonicalization, together with the platform content version identifying which payload that was:

"reference": { "checksum": "5c1f…", "version": "9ab3…" }

GetStatus() compares against that checksum when the version matches the manifest entry being compared against, and falls back to the old comparison otherwise. Both sides of the comparison then go through the same serializer, so the result no longer depends on how the publisher formatted its JSON.

The version pairing is load-bearing rather than decorative. The platform derives a content version from the payload itself, so it identifies exactly which remote bytes our checksum describes. Without it, a file synced from an older manifest would compare its stale reference against a newer manifest entry, match, and report a genuinely changed remote as up to date — trading a false positive for a false negative, which is much worse. There is a test for exactly that.

The two halves are meaningless apart, so they are never apart. LocalContentReference can only be constructed whole, TryRead yields nothing unless both halves are present, and the JSON nests them under one key, so a half-populated reference is unrepresentable rather than guarded against. Absence is a null, not a pair of empty strings.

The reference is written when we download content, when content we publish becomes the new remote (using the version the platform returns on the save response), and when a file that genuinely matches the target has its manifest reference moved forward. Files without one fall back to the old manifest-checksum comparison, so nothing changes for existing workspaces until their next sync.

Pin the serializer options. GetContentFileSerializationOptions now names Encoder = JavaScriptEncoder.Default explicitly. This is what System.Text.Json was already doing implicitly, so no existing checksum changes — it just can no longer drift with a .NET upgrade, and the comment says why that matters.

Make the key sort a total order. OrderBy is stable over input ordering, so two keys differing only in case previously kept whatever order they arrived in, which is exactly the instability the converter exists to remove. Both sorted converters get an Ordinal tiebreak. In SortedJsonElementConverter the tie was a correctness problem, because its output is hashed. In SortedSnapshotConverter it was not -- snapshot bytes are not hashed -- but an incidental reordering there turns a one-line diff into a two-line one, which defeats the readability the sort was introduced for. No existing checksum changes, since only case-colliding keys reorder.

Deliberately not done

• Numbers are not normalized. Raw text replay is kept. Normalizing risks precision loss on doubles and erases the float-versus-int distinction Unity schemas rely on, and the local-reference design removes most of the exposure anyway, since the local file is written from the downloaded payload through the same serializer. Residual: a Unity edit that rewrites 1 as 1.0 can still read as modified. Covered by a test that documents the behavior as intentional.
• The encoder was not switched to relaxed escaping. It would make content files far more readable in diffs, but it changes every checksum for content containing non-ASCII or & < > ', which means a forced re-sync and skew against teammates on older CLI versions. Worth doing deliberately with a migration, not as a side effect of this fix.
• Snapshot restore does not repopulate the reference. Snapshots record the local checksum at snapshot time, not the reference, so it cannot be recovered from one. Restored files fall back to the legacy comparison until their next sync. Tracked in Snapshot restore should carry the locally derived content reference #4852, which also covers repairing snapshots taken before this change.

Server-side half, not addressed here

Content versions are derived from a hash of unsorted JSON on the platform side as well, so a value-identical republish with different key order mints a new content version, a new stored object, and a new manifest. That means an unstable publisher still causes churn and redundant client downloads even with this fix in place. It is filed separately on the backend tracker; it needs a migration plan because existing stored object keys derive from the current hash, and it is not something the CLI can fix on its own.

Tests

ContentChecksumTest covers the canonical form as a wire format: top-level and nested field order are ignored, case-only key collisions order deterministically, array order is still significant, different values still differ, the pinned encoder still escapes exactly as before, and raw number text is still preserved.

ContentStatusTest covers what GetStatus() trusts: an unreproducible publisher checksum no longer marks value-identical content modified, a remote that moved on since we recorded our reference still reads as modified, genuinely different local properties still read as modified, and a file carrying no reference falls back to the historical comparison. It also pins the on-disk shape — the reference nested under one key, and omitted entirely when absent — and checks that a half-written reference is refused at parse time.

The tiebreak added to SortedSnapshotConverter is not separately covered. No existing test pins snapshot key order, and its output is not hashed, so a regression there would show up as diff churn rather than as wrong status. Say the word if you would rather it were pinned.

Notes for review

• Measured cost of the added hash: roughly 36 µs per item at ~1.9 KB of properties on an M5 Pro, of which SHA1 itself is under 5% — the sorted serialization dominates. It runs once per downloaded item over an already-parsed payload, inside the existing Task.WhenAll, against a sync that issues one CDN request per item. If it ever needs to be cheaper, CalculateChecksum currently serializes to a string and then calls Encoding.UTF8.GetBytes; serializing into a pooled ArrayBufferWriter<byte> and hashing the span measured about 37% faster at that size.
• No runtime or on-device impact. The whole checksum apparatus in ContentObject.cs sits inside #if UNITY_EDITOR, and the CLI is not shipped to devices.

🤖 Generated with Claude Code

[github-actions]

Lightbeam link

[github-actions]

Lightbeam link

[github-actions]

Lightbeam link

[github-actions]

Lightbeam link

[github-actions]

Lightbeam link

[github-actions]

Lightbeam link