Atomic shard replacement can invalidate a live mmap on AWS EFS

[aaaaaandrew]

Environment

• Zoekt: 2cb19912a4073e5a9895658b7cb135ee4b35733b
• Linux/arm64
• Go 1.26
• AWS EFS over NFSv4.1
• Indexer and webserver on separate NFS clients

Summary

Zoekt maps shard files using MAP_SHARED and then closes the file descriptor:

zoekt/index/indexfile.go

Lines 55 to 80 in 2cb1991

	// NewIndexFile returns a new index file. The index file takes
	// ownership of the passed in file, and may close it.
	func NewIndexFile(f *os.File) (IndexFile, error) {
	defer f.Close()
	fi, err := f.Stat()
	if err != nil {
	return nil, err
	}
	sz := fi.Size()
	if sz >= math.MaxUint32 {
	return nil, fmt.Errorf("file %s too large: %d", f.Name(), sz)
	}
	r := &mmapedIndexFile{
	name: f.Name(),
	size: uint32(sz),
	}
	rounded := (r.size + 4095) &^ 4095
	r.data, err = unix.Mmap(int(f.Fd()), 0, int(rounded), unix.PROT_READ, unix.MAP_SHARED)
	if err != nil {
	return nil, err
	}
	return r, err

Shard publication atomically renames new files over stable shard paths and
removes obsolete shard paths:

zoekt/index/builder.go

Lines 778 to 803 in 2cb1991

	for tmp, final := range artifactPaths {
	if err := os.Rename(tmp, final); err != nil {
	b.buildError = err
	continue
	}
	delete(toDelete, final)
	}
	b.finishedShards = map[string]string{}
	for p := range toDelete {
	// Don't delete compound shards, set tombstones instead.
	if b.opts.ShardMerging && strings.HasPrefix(filepath.Base(p), "compound-") {
	if !strings.HasSuffix(p, ".zoekt") {
	continue
	}
	err := SetTombstone(p, b.opts.RepositoryDescription.ID)
	b.buildError = err
	continue
	}
	log.Printf("removing old shard file: %s", p)
	if err := os.Remove(p); err != nil {
	b.buildError = err
	}
	}

When publication happens from a different EFS client, replacing the stable path
can remove the last name for the inode backing an existing webserver mmap. A
later cold-page access can then fault and fail the shard search.

The replacement shard itself remains valid and loads normally. This is a shard
publication and mmap-lifetime problem, not malformed index data.

Observed sequence

The failure was observed with this sequence on EFS:

1. A webserver loaded and mapped a valid shard.
2. An indexer on another NFS client renamed a replacement over its stable path.
3. Five seconds later, the webserver faulted while reading its existing mapping.
4. The webserver recovered and the directory watcher loaded the valid
   replacement shortly afterward.

Reopening the shard repaired the failure immediately. Integrity checks on the
replacement file succeeded.

The corresponding shard load path is:

zoekt/search/shards.go

Lines 1258 to 1274 in 2cb1991

	func loadShard(fn string) (zoekt.Searcher, error) {
	f, err := os.Open(fn)
	if err != nil {
	return nil, err
	}
	iFile, err := index.NewIndexFile(f)
	if err != nil {
	return nil, err
	}
	s, err := index.NewSearcher(iFile)
	if err != nil {
	iFile.Close()
	return nil, fmt.Errorf("NewSearcher(%s): %v", fn, err)
	}
	return s, nil

Local reproduction

A minimized harness used two independent Linux NFS clients and a 64 MiB
MAP_SHARED file.

The pre-fix control:

1. Client A opened and mapped the file, matching NewIndexFile.
2. Client A closed the original descriptor.
3. Client B renamed a replacement over the stable path.
4. Client A repeatedly discarded and reread mapped pages.
5. Client A explicitly evicted its page cache.

The reader faulted after 832 iterations:

FAULT iterations=832 error=mmap fault: runtime error:
invalid memory address or nil pointer dereference

This negative control used NFSv3. Linux knfsd NFSv4.1 retained the replaced
open inode and did not reproduce AWS EFS's invalidation behavior, even with
separate client kernels and explicit server and client cache eviction.

The production failure was observed on AWS EFS over NFSv4.1. The local NFSv3
test reproduces the underlying stale mmap mechanism, but is not presented as an
exact local reproduction of EFS behavior.

Mitigation validated locally

The tested publication model uses immutable shard generations:

1. Completed shards move to uniquely named immutable generation paths.
2. The stable .zoekt path is atomically switched as a symlink.
3. Before the first conversion, a legacy regular shard is hard-linked into a
   generation so existing mappings retain a named backing inode.
4. Readers hold shared locks for their mmap lifetime and validate file identity
   after locking.
5. Retired generations are reclaimed only after a grace period and successful
   nonblocking exclusive lock acquisition.

Lifecycle sketch:

reader(stable):
    file = open(stable)
    acquire_shared_lock(file)

    if !same_file(stat(file), stat(stable)):
        close(file)
        retry

    mapping = mmap(file)
    keep file and lock until munmap(mapping)

publish(temp, stable):
    new = move_to_unique_generation(temp)
    old = retain_named_generation(stable)

    acquire_shared_lock(old)
    atomically_replace(stable, symlink(new))
    mark_retired(old)
    release_lock(old)

reclaim(retired):
    wait_until_grace_period_elapsed()

    if !try_exclusive_lock(retired):
        defer

    recheck_retired_is_not_current()
    remove(retired)

Under Linux knfsd NFSv4.1, the fixed path:

• survived 26,771 forced cold-page iterations;
• remained healthy after explicit page-cache eviction;
• caused reclamation to defer while the reader held its shared lock;
• exited successfully;
• then allowed the retired 64 MiB generation to be reclaimed;
• left the current immutable generation readable.

This validates the reader, publication, identity-check, and reclamation
invariants locally. It has not yet been exercised as a controlled unpatched vs.
patched A/B test on AWS EFS.

Relationship to #1106

The field note in #1106 describes the same dead-mmap symptom:

#1106

The active fixes in #1107 and #1108 address malformed-index validation and
error propagation. They do not change shard publication or mmap backing-object
lifetime:

• index: guard posting iterator against overflowing varints #1107
• index: surface corrupt rune and match offsets #1108

Expected behavior

Publishing or retiring a shard must not invalidate mappings held by active
searchers. The inode backing a live mapping must retain a stable name until no
reader can still access it.

[linear]

CU-3197