volumes: concurrent EFS mount/unmount via per-volume locking

[callaway12]

Summary

• Replace the single global sync.RWMutex in AmazonECSVolumePlugin with a short-lived map lock (mapLock) + per-volume mutexes (volLocks), allowing mount/unmount I/O for different volumes to proceed concurrently
• Refactor ECSVolumeDriver.Create/Remove to release the driver lock before the mount/unmount syscall, since the plugin layer already guarantees per-volume serialization
• Add TestConcurrentMountsDifferentVolumes that verifies 5 parallel mounts complete in wall-clock time close to a single mount, not 5x

Motivation

The EFS volume plugin processes all mount requests serially behind a single write lock. A single EFS mount takes 15–72 seconds (TLS tunnel + NFS4 negotiation). When N concurrent ECS tasks each require 2+ EFS mounts, queued requests block for up to 72 × 2N seconds, exceeding Docker's plugin timeout (~60s) and causing:

CannotCreateContainerError: error while checking if volume exists in driver
"amazon-ecs-volume-plugin": Post "http://plugin.moby.localhost/VolumeDriver.Get":
context deadline exceeded

This is tracked in aws/containers-roadmap#2319.

Production impact: On a c6i.8xlarge running concurrent ECS tasks (EC2 launch type, awsvpc mode) with multiple EFS volumes per task, we observed 37 task failures over 3 months directly caused by this serialization, with failure frequency accelerating as instance uptime increased.

Design

Operation	Before	After
Mount()	Global write lock held for entire mount I/O	Global RLock for map lookup → per-volume Mutex for mount I/O
Unmount()	Global write lock held for entire unmount I/O	Global RLock for map lookup → per-volume Mutex for unmount I/O
Create()	Global write lock (metadata only — fast)	Global write lock (unchanged, no I/O)
Remove()	Global write lock	Global write lock (unchanged, infrequent)
List/Get/Path	Global read lock	Global read lock (unchanged)

Invariants preserved:

• Operations on the same volume remain serialized (per-volume mutex)
• The volumes map is protected during all reads/writes
• StateManager.save() has its own mutex for file persistence
• types.Volume.AddMount/RemoveMount is called under the per-volume lock (as documented: "caller is responsible for holding any locks")

Test plan

• All existing unit tests pass with updated lock field initialization
• New TestConcurrentMountsDifferentVolumes: 5 volumes with 50ms simulated mount latency complete in <200ms (proves concurrency), would take ≥250ms if serial
• Integration test on ECS instance with concurrent EFS-backed tasks

Fixes aws/containers-roadmap#2319

[amogh09]

Looks like we are assuming that this method call will only get a pre-created lock and not create it. Which is true but that is not clear from reading this code in isolation. I suggest two separate methods - getVolLock() and createVolLock() so it's clear which one needs RLock and which one needs Lock.

[callaway12]

Good catch. I've refactored this into three explicit methods to make the intent clear:

• getVolLock(name) — read-only lookup, used under mapLock.RLock() in Mount/Unmount
• createVolLock(name) — write-only, used in Create/LoadState which already hold mapLock.Lock()
• getOrCreateVolLock(name) — double-checked locking pattern for the fallback path:
  1. Try RLock → fast path if lock exists
  2. Upgrade to Lock → re-check, create only if still nil

This eliminates the original race (map write under RLock) and makes each call site's contract explicit.

[amogh09]

Can we have a test that tests that concurrent calls to Mount() leads to a single mount call in the driver?

[callaway12]

Added TestConcurrentMountsSameVolume. It fires 5 concurrent Mount() calls for the same volume and asserts:

• driver.Create() is called exactly once (.Times(1))
• All 5 mounts are recorded (len(vol.Mounts) == 5)
• Each individual mount ID has a reference count of 1

[amogh09]

Wall clock based assertions usually end up flaking in some way. How about forcing overlap in the mock function? Something like -

var entered sync.WaitGroup
entered.Add(numVolumes)

slowDriver.EXPECT().Create(gomock.Any()).DoAndReturn(func(...) error {
    entered.Done()       // signal "I'm here"
    entered.Wait()       // block until all are here
    return nil
}).Times(numVolumes)

[callaway12]

Agreed — replaced the wall-clock assertion with a sync.WaitGroup barrier pattern:

var entered sync.WaitGroup
entered.Add(numVolumes)
mockDriver.EXPECT().Create(gomock.Any()).DoAndReturn(func(r *driver.CreateRequest) error {
    entered.Done()  // signal arrival
    entered.Wait()  // block until all goroutines are inside Create()
    return nil
}).Times(numVolumes)

This deterministically proves all mount calls overlap inside the driver, with no timing dependency.

[amogh09]

Unfortunately our CI doesn't use go test -race on ecs-init module. Can you please run the new unit test locally with -race? It is failing for me.

[amogh09]

I have updated our CI to run ecs-init unit tests with -race. Can you please rebase against dev branch?

[callaway12]

Thanks for the review! Rebased onto latest dev and addressed all comments. Here's a summary of the changes:

Review feedback addressed:

1. Split getOrCreateVolLock() into 3 methods (getVolLock, createVolLock, getOrCreateVolLock) with double-checked locking to eliminate the race of writing the map under RLock
2. Added TestConcurrentMountsSameVolume — verifies driver.Create() is called exactly once when 5 goroutines mount the same volume concurrently
3. Replaced wall-clock assertion with sync.WaitGroup barrier to deterministically prove concurrent overlap

Additional race conditions fixed (found via go test -race):
4. StateManager race — recordVolume() was mutating VolState.Volumes map outside the lock, then save() acquired the lock and called json.MarshalIndent which reads the map concurrently. Fixed by extending the lock scope to cover both map mutation and serialization
5. Remove() coordination — Remove() now acquires the per-volume lock before proceeding, which drains any in-flight Mount/Unmount operations. It also performs slow I/O (IsMounted, volDriver.Remove) outside mapLock to avoid blocking other volumes
6. seelog format bug — Fixed Errorf("Error saving state of volume %s", r.Name, err) (err silently dropped) → Errorf("Error saving state of volume %s: %v", r.Name, err)

All tests pass with go test -race -count=20.