Conversation
Add StoragePayloadForSpaceDeriveV1 (deterministic v1 derive builder) and switch CreateSpace, DeriveId and DeriveSpace to the v1 builders. v1 headers embed the acl and settings payloads and validate via byte-compare (needCheckSpaceId=false), matching the one-to-one path already shipping in production. Note: v1 derive produces a different derived space id than v0 for the same keys (the header bytes, hence the CID, differ). Safe on greenfield; existing v0-derived spaces would need an id-continuity strategy before an in-place upgrade. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the SpaceFileProtoVersion enum (Unspecified=0, V2=2) and a fileprotoVersion field to the SpaceHeader proto, and plumb it through SpaceCreatePayload and SpaceDerivePayload into every builder. The field lives in the signed, immutable header, so it is tamper-evident and recovered verbatim on cold restore. Gates the v2 file protocol (filenode v2 + S3-direct + networkSign) per SYN-15. Callers set the value (e.g. the SDK mints spaces at V2); unset stays Unspecified, so legacy any-sync consumers are unaffected. OneToOne spaces stay Unspecified. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the NodeTypeFileV2 node type ("fileV2") and a second, independent chash
container built from the v2 file-node pool at RF=2 (FileV2ReplicationFactor).
New NodeConf methods:
- FileV2Peers() lists the v2 filenodes (mirrors FilePeers).
- FileV2NodeIds(spaceId) returns the RF=2 responsible v2 filenodes for a space.
Unlike NodeIds, the current account is NOT excluded, so the full responsible
pair is visible (e.g. to derive the leader).
The tree (sync) and v1-file containers are untouched. Leader selection and
per-space responsibility consumers are out of scope. Propagated to service,
testconf StubConf and the regenerated mock.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ath (SYN-16) Add FileV2API=6 to the coordinator NodeType proto enum and map it to nodeconf.NodeTypeFileV2 in nodeconfsource. Without this the v2 file type only survived local YAML configs and was dropped on the coordinator-distributed path. The enum->string switch keeps its no-default behavior: an older client that does not know FileV2API (or any future type) silently drops the unknown value while keeping the node's known types, so it stays correctly routed in the tree/file pools and never crashes. New enum values are therefore backward-compatible. Note: the coordinator server (any-sync-coordinator) still needs the reverse string->enum mapping to actually serve fileV2 nodes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Define the filenode v2 (FileV2) broker dRPC contract: a byteless broker that authorizes uploads, signs durable-custody receipts, and issues signed download URLs — never transferring bytes itself. All four RPCs are batch (Upload / RequestSign / RequestDownload / SpaceInfo): each request carries a shared spaceId + repeated items, and each response is a repeated per-item result keyed by the echoed rootCid/spaceId with a flat status enum, so one bad item never fails the whole batch. A two-layer error model keeps whole-RPC failures (auth/malformed/batch-too-large) on the drpc error while per-item business outcomes travel in-body. New proto package filesyncv2 / service FileV2 (route /filesyncv2.FileV2/*), distinct from v1 filesync/File, plus the fileprotov2err registry at error offset 800 (see docs/rpc-error-offsets.md). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document the process-global rpcerr registry as a shared cross-repo namespace: the mechanism, allocation rules (offsets are multiples of 100, local codes 0-99, any-sync core < 1000 / downstream repos >= 1000), the globally reserved low codes, and a full table of offsets in use across the ecosystem (any-sync 100-800, any-sync-node 1000, anytype-push-server 1200, bobrik-clusterctl 1300). Add an ErrGroup doc comment pointing to it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lFilter (SYN-18) Two opt-in, backward-compatible hooks for participants that head-sync a whole space but materialize only selected tree types (filenode-v2 broker): - TreeFullSyncRequest.probe: the responder replies with the tree's root and current heads only, no change bodies. Old responders ignore the flag and stream the full tree. Plumbed via BuildTreeOpts.Probe. - treesyncer.PullFilter: optional extension of the registered TreeSyncer, consulted by objectsync on a head update for a locally-missing tree; head updates already carry the raw root change, so the filter can classify the tree (cleartext changeType) and decline the pull with no extra round trips. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feat(sync): selective-sync hooks — tree probe requests + optional PullFilter (SYN-18)
…SYN-31) Spaces chash-shard across the NodeTypeFileV2 pool (RF=2), and a filenode must reject — and never activate/sync — spaces outside its responsibility set. Give that rejection a dedicated retryable routing signal instead of overloading Forbidden, so client SDKs can re-route to the responsible pair: * whole-RPC ErrCodes.NotResponsible = 4 (registered in fileprotov2err at offset 800+4) — for the single-space RPCs where the entire request is misrouted; * per-item ErrCode.ErrNotResponsible = 6 — for SpaceInfo, whose batch may legally mix spaces owned by different pairs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Downloads start PUBLIC-read: chat-message images must render with zero
broker round-trips, so the client constructs durable object URLs straight
from the rootCid — {publicReadBaseUrl}/blob/{spaceId}/{rootCid} — and GETs
the CDN. The base URL is distributed by the filenode itself (not nodeconf)
via a new node-level Info RPC that clients cache: no spaceId, no
responsibility gate, no auth beyond the connection. InfoRequest is
deliberately empty so future node metadata (capabilities, limits) extends
it without a wire break; an empty publicReadBaseUrl means no public read —
use RequestDownload.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feat(fileprotov2): NotResponsible error codes + node-level Info RPC (SYN-31)
mockConf did not implement the FileV2Peers/FileV2NodeIds methods added to nodeconf.NodeConf on this branch, causing a panic in TestInbox_Fetch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Proto version 13 aligns with the v0.13.X release line and covers the files v2 work (v2 filenodes, chash routing, space header fileprotoVersion). Version 13 is already present in defaultCompatibleVersions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per docs/proto-versioning.md the default compatibility range is self ±1. v0.13.X accepts V-1=12, V=13, V+1=14. Drops bridge v9, which is no longer needed now that v0.12 has shipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…entity One shared account (signing) key signs fileV2 durability receipts for the whole NodeTypeFileV2 fleet (per-node signing was rejected: receipts live in CRDT rows for years and must survive fleet churn). Clients need its public identity — the fileNetworkId — from the network config, so: - nodeconf.Configuration gains FileNetworkId (yaml fileNetworkId, network string encoding, decode with crypto.DecodeNetworkId) - coordinatorproto NetworkConfigurationResponse gains fileNetworkId=5 - nodeconfsource maps it off the proto, so coordinator-refreshed configs carry it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New commonfile/fileproto/filep2p service filesyncp2p.FileP2P, served read-only by clients to LAN peers for direct file fetch: - FileCheck(spaceId, rootCids) -> None/Partial/Full availability - ObjectRead(spaceId, rootCid, offset, length) -> (data, totalSize) A client stores one immutable CARv2 object per file, byte-identical to the S3/public-read object, so a peer is just an ALTERNATE SOURCE of that same object: ObjectRead returns object byte ranges exactly like an HTTP Range GET, and the fetcher's existing seed/verify path is reused. Only full holders serve ObjectRead; FileCheck reports which. Requests are root-scoped (no global block index). Blocks are ciphertext; the fetcher verifies hash==cid. Whole-RPC errors in filep2perr at rpcerr offset 900 (v1=200, v2=800). Route prefix /filesyncp2p.FileP2P/ coexists on one mux. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017Hq5ube4stoD1XM5QAU931
filesyncp2p.FileP2P already declares ErrorOffset=900 (the next free core band); record it in docs/rpc-error-offsets.md, the source of truth for who owns which offset. Adds a filep2perr test that co-registers the neighbouring v1/v2 error groups so a colliding offset would panic at init — a green run proves the 900 band is clear. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017Hq5ube4stoD1XM5QAU931
filep2p: peer-to-peer file transfer protocol (SYN-48)
…storage limits (SYN-19) The v2 file-limits API: brokers pull absolute account-pool bounds per (spaceId, uploader identity) and push per-(space, identity) durable usage reports; the coordinator aggregates the identity total. In-flight never reaches the coordinator; values are absolute and idempotent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoLHZmQFxyhxDkkwJ7Y1jX
SpaceInfo now surfaces the requesting identity's shared account pool (accountLimitBytes / accountUsageBytes) next to the per-space view — files are charged to their uploader, all spaces share one pool. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AoLHZmQFxyhxDkkwJ7Y1jX
…y store - Configuration.Epoch: monotonic version set by the coordinator (0 = pre-epoch) - NetworkConfigurationResponse.epoch in coordinatorproto (+ regen) - nodeconf.Service.ObserveChanges: observers notified on config replacement - nodeconf.HistoryStore + nodeconfstore retains last 100 epochs per network Groundwork for tree-node resharding (see any-sync-node docs/resharding-plan.md). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
feat(coordinatorproto): files-v2 storage limits API — FileLimitsGet / FileUsageReport (SYN-19)
Replaces plaintext spaceId lists in LAN peer discovery with per-space HMAC tokens keyed by a discovery key derived (HKDF) from the space's first ACL read key. Peers learn only the intersection of their space sets; a valid token simultaneously hides the space from non-members and proves the sender's membership, so strangers on the LAN can no longer enumerate spaces, correlate devices across sessions, or poison the local peer table with spaces they don't hold. Request tokens are padded with random fillers to fixed buckets and sorted to hide the true space count; response tokens are keyed by the caller's nonce (challenge-response). Gated by ProtoVersion 13; v1 fallback must be version-gated, never error-triggered. Bench (1000 spaces/peer, 500 shared): ~1.5ms CPU for a full round trip, plus 0.66ms one-time discovery-key derivation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feat(clientspaceproto): SpaceExchangeV2 — token-based LAN space discovery
Add an ephemeral, fire-and-forget, at-most-once publish/subscribe channel scoped to a space, carried over a dedicated PubSub DRPC stream that is fully isolated from the sync engine. commonspace/pubsub: - Wire protocol (pubsubproto): PubSubStream bidi RPC with Subscribe/ Unsubscribe/Publish/Status frames. - Flat + NATS-style wildcard topics (* one segment, > tail) matched by a per-space sublist-shaped trie; reserved acc/.../<accId> self-owned namespace. - ACL-gated subscribe and publish; per-message account-key signatures (length-prefixed, domain-separated) verified by receivers; payloads encrypted with the space ReadKey via a pluggable Crypto. - Node relay with the one-hop rule (client-originated msgs forwarded once to other responsible nodes, relayed msgs never re-forwarded); LAN peers symmetric; echo and duplicate-path suppression via a bounded msgId ring; per-peer publish token bucket. - Serving-side interest keyed by streamId (trie refcounts subscribing streams) so a reconnecting peer's fresh stream keeps its interest and a mid-subscribe close cannot orphan state. - Reconnect watcher (periodic + on-close resync) and peer TTL so a pure subscriber survives stream drops and idle pool GC. - CloseSpace teardown and EvictMember (active drop on ACL removal, strips tags so delivery stops even on an open stream). - Receive path runs cheap filters (interest, membership, ownership, timestamp staleness) before the Ed25519 verify and records dedup only after verify, shedding forged-signature floods and closing a replay window. net/streampool: - NewStreamPool standalone constructor with WithStreamCloseHook (carries streamId) and WithMetric options; CtxStreamId accessor; RemoveTagsById; cross-tag stream dedup in Broadcast. docs/stateless-pubsub: research and design docs.
Add Service.RevalidateMembers(spaceId, isMember) so a relay node can evict every subscriber whose account no longer passes the membership predicate in a single pass on an ACL change, refactoring EvictMember to share the core eviction path.
- WithMetric no longer registers the private pool into the shared single-slot sync metric (RegisterStreamPoolSyncMetric): that slot belongs to the app-level sync streampool, so a second pool there overwrote its OutgoingMsg telemetry and nulled it on Close. WithMetric now only registers the namespaced prometheus gauges — independent observability, no clobber. (F1, medium) - handleSubscribe cap rejection reports only the rejected patterns in the TooManyTopics status, not the whole subscribe batch. (F4) - removeStream always clones tags for its debug log. (F5) - drop the dead streamInterest.peerId field. (F6) - doc: dedup is a FIFO ring (not LRU); rate limiter is per-peer; note that Close blocks on in-flight handlers and that resync coverage is bounded by DialQueueSize. Adds TestCapExceededReportsOnlyRejected. The remaining review items were verified correct (lock ordering, interest+tag rollback, receive-path membership-before-verify, no double-decrement) — no change needed.
All connectable peers negotiate DiffType_V3 since ProtoVersion 7 and the minimum compatible version is 12, so the V2 paths are dead code: - drop app/olddiff and the ldiff DiffContainer (single diff now) - DiffManager keeps only the V3 diff; V2 range requests are rejected - statestorage keeps a single hash (old hash is no longer written) - keep the DiffType negotiation as the future upgrade mechanism, see the upgrade recipe comment on DiffManager
- statestorage: keep mirroring the hash into the legacy "oh" key so a previous release (rollback or mixed-fleet coldsync) doesn't misread NewHash as empty via its legacy-format fallback; transitional, remove once downgrades below this version are out of support - diffmanager: guard FillDiff against empty Heads entries, set all elements with a single variadic Set (one range recalculation instead of N), document why FillDiff and UpdateHeads deliberately use different empty-root conditions, and note the V4 fallback-on-error requirement in the upgrade recipe - add a hash-stability regression test pinned to the golden value produced by the pre-removal dual-diff DiffManager
refactor(headsync): remove old diffsync (DiffType_V2) support
# Conflicts: # coordinator/coordinatorproto/coordinator.pb.go # coordinator/coordinatorproto/coordinator_vtproto.pb.go # coordinator/coordinatorproto/protos/coordinator.proto # coordinator/nodeconfsource/nodeconfsource.go # nodeconf/config.go
# Conflicts: # coordinator/coordinatorproto/coordinator.pb.go # coordinator/coordinatorproto/coordinator_drpc.pb.go # coordinator/coordinatorproto/coordinator_vtproto.pb.go # coordinator/coordinatorproto/protos/coordinator.proto
…eference
Replace docs/stateless-pubsub/{DESIGN,RESEARCH}.md with a single as-built
docs/stateless-pubsub.md: present-tense reference for the shipped v1 plus a
compact background/rationale distilled from the research. Corrects drift
against the code (SyncInterest/RevalidateMembers, NewStreamPool, Unexpected=0
zero value, receive-path ordering) and drops the now-resolved open items.
800 now belongs to filesyncv2 (commonfile/fileproto/fileprotov2) on the v0.13.x base; the any-sync core <1000 offset range is fully allocated, so pubsub takes the next free offset, 1100. Hand-edited the generated descriptor (the pinned protoc-gen plugins are Linux binaries and won't run locally; regenerating with local plugins introduced unrelated drpc/vtproto version drift). Verified the rawDesc varint round-trips to 1100 -> "ErrorOffset". Record the allocation in docs/rpc-error-offsets.md (brought in from v0.13.x) and reconcile its "core < 1000" rule with the now-exhausted core range.
- nodeconf: TestService_ObserveChanges relied on a 10ms sleep for the periodic sync goroutine to fire the first update; under -race it often wasn't scheduled in time. Wait with require.Eventually instead. - webtransport: wt.Server.Serve registers itself in the server's internal WaitGroup from the serve goroutine, which is unordered with Close's refCount.Wait — the documented WaitGroup Add-vs-Wait race, reported by the race detector on fast shutdown. Own the QUIC accept loop and call ServeQUICConn per connection (it never touches refCount), track the goroutines in our own WaitGroup, and have Close wait for them while keeping graceful session close semantics.
Fix race-detector failures: nodeconf observer test and webtransport shutdown
Coverage provided by https://github.com/seriousben/go-patch-cover-action |
ocache runs a load under the context of whichever caller arrived first and shares the result — including the error — with every concurrent waiter. When that first caller goes away mid-load (its request finished, a per-round timeout fired), the load dies with context.Canceled and every waiter inherits an error that has nothing to do with its own, still-live context. The peer pool makes this bite in practice: dials are ocache loads keyed by peerId, so a dial owned by a short-lived background context (e.g. a headsync round that completes mid-dial) poisons the same dial for a concurrent space pull, which then surfaces as a hard "NewSpace: context canceled" — getSpaceStorageFromRemote retries only net.ErrUnableToConnect. Observed as an intermittent client-side Spaces().Get failure right after a space appears in the account index. Fix: load() records whether the load's OWN context was already done when the loadFunc returned (entry.loadAborted) — the precise signal that the load was killed rather than refused. Get retries an aborted load for callers whose own ctx is still alive: the failed entry is already deleted, so the retry starts a fresh load owned by a live context. Bounded by maxLoadRetries against abort storms. Deliberately NOT error-shape based: a loadFunc-internal timeout (transport dial deadline) returns context.DeadlineExceeded with the load ctx alive — that is a verdict about the resource, never retried, so failing dials keep failing in one attempt. Callers whose own ctx is done never retry, and an abandoned load still dies with its owner. One observable change: a Get in flight while the cache closes now reports ErrClosed (the actual reason) instead of the load's raw context.Canceled — the retry observes the closed cache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ocache: retry a load that was killed by another caller's context
feat(payments): add paymentUrl to MembershipV2.Invoice
Decrypt of a wire-controlled blob shorter than NonceBytes panicked with slice bounds out of range instead of returning an error.
…nput crypto: error on ciphertext shorter than GCM nonce in DecryptReuse
deleteFunc reached two detached contexts — MarkTreeDeleted and the settings object AddContent broadcast — so a delete in flight against unreachable nodes never returned and deleteLoop.Close blocked until the app.Close watchdog panicked. Plumb the delete ctx through both, bail out of the queued/bound-child loops once it is cancelled, and bound the Close wait with a timeout and warning so a treemanager that ignores ctx can no longer wedge shutdown.
Two more instances of the deletionmanager hang. concurrentLimiter.Close waited on the WaitGroup with no bound while the scheduled goroutine was inside a peer sync that only checks ctx before it starts. syncAcl broadcast under context.Background while holding the acl lock, which Close then waits for, so an acl record arriving against unreachable nodes wedged Close forever. Bound the limiter wait with a timeout and the close ctx, give syncAcl a cancellable broadcast ctx, and cancel it before Close takes the lock.
actionPool created a cancel func and never called it, so every queued action ran under an effectively uncancellable ctx and kept issuing peer requests after close — it is also the ctx a syncTree holds its lock under, which made settings.Close unrecoverable. app.Start's closeServices closes components whose Run never executed, which deadlocked subscribeClient.Close on a channel only streamWatcher closes and nil-dereferenced actionPool.periodicLoop and streamPool.dial. Guard all three.
ocache.Close waited with context.Background on entries the gc was already closing, and that gc sits in peer.TryClose -> drpc conn Close -> a yamux stream whose peer never sends FIN, which yamux bounds at 5 minutes — past the app close deadline. Give Close one deadline for the whole pass; an entry still held by another closer is left to it instead of blocking shutdown. peer.gc closed those conns while holding p.mu, freezing AcquireDrpcConn for the same peer; close them after the lock is released. Set the yamux stream close timeout from the write timeout, and bound the webtransport dial with DialTimeoutSec, which until now applied only to the accept path.
concurrentLimiter.Close returning on timeout left a waiter parked on the WaitGroup; a later ScheduleRequest panicked with "WaitGroup is reused before previous Wait has returned", and SyncWithPeer stays callable after close. Reject requests once closed. ocache.Close passed its deadline to waitLoad too, where an expired ctx and a finished load are both ready and select picks at random — roughly half the healthy entries returned early and were never closed. Spend the deadline only on entries another closer holds. deleteLoop.Close burned the full timeout when Run never ran.
…ionmanager-deletefunc-not-ctx-cancellable fix: bound component shutdown against unreachable peers (SYN-110)
# Conflicts: # docs/rpc-error-offsets.md
requilence
self-requested a review
July 29, 2026 10:30
# Conflicts: # paymentservice/paymentserviceproto/paymentservice2.pb.go # paymentservice/paymentserviceproto/paymentservice2_vtproto.pb.go # paymentservice/paymentserviceproto/protos/paymentservice2.proto
zarkone
approved these changes
Jul 30, 2026
Storage.Iterate builds per-key callback groups from consecutive runs, but IterateValues scanned unsorted (insertion order), so one key's rows split across groups whenever peers' row batches interleaved — and the split is stable across restarts. Consumers treating a callback as "all rows of this key" silently misbehaved (SYN-116; froze read-marker republish in the SDK, SYN-97). Sorting by id (key+"-"+peerId) makes the grouping group by key, same as IteratePrefix. The diff-loading scans stay unsorted — ldiff.Set is order-independent. Claude-Session: https://claude.ai/code/session_01WCH11UrasWuzpKpgHRepow
…sort keyvaluestorage: sort IterateValues by id so Iterate groups whole keys (SYN-116)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
What type of PR is this? (check all applicable)
Related Tickets & Documents
Mobile & Desktop Screenshots/Recordings
Added tests?
Added to documentation?
[optional] Are there any post-deployment tasks we need to perform?