Repository navigation
Description
What happens
A Worker hydrates blobs and calls back into the Dex Server named by its own FlowServiceAddress, not the Server that dispatched the call. When two Dex Servers on one host each record a Flow's Worker target, and those targets end up served by one Worker, that Worker serves calls from a Server it does not belong to. It then answers with a misleading error, or with no error at all.
Observed on Server v1.4.1 with Go SDK v1.5.0, on two local Dex environments on one machine, each with its own Temporal and blob store:
- A client started a Flow on Server A with an explicit Worker target
host.docker.internal:25051. That target was the Worker port of environment B. Server A stored the Flow's large initial Attributes in its own blob store, as expected (service/api/service.go:177-198). - Later, environment B's Worker listened on that port. Every RPC read of the Flow through Server A was dispatched to B's Worker (
common/rpc/invoke.go:81-85). - B's Worker hydrated the blob IDs through its own Flow service, Server B (
dex/worker.go:47-49,119-139;dex/worker_service.go:368-373). Server B'sLoadBlobsfound no such objects and omitted them (service.go:1188-1191). - The SDK failed with
dex: LoadBlobs omitted blob "p0|11/…"(dex/hydration.go:173-179), which Server A reported asFailedPrecondition/ worker API error.
The blobs were intact in Server A's store the whole time. Server B's log showed the GetObject … NoSuchKey reads for A's Flow.
Here the hydration failure happened to stop the wrong Worker from running the handler. A Flow whose values are all small and inline would hydrate nothing, and B's Worker would silently run A's handlers against A's state, with B's dependencies.
Suggested fix
Let a Worker refuse a call from a Server it does not serve:
- The dispatching Server sends its identity in the Worker request context, for example its namespace plus a configured deployment ID or a blob-store fingerprint.
- At startup the Worker reads the identity of its own Flow service.
- A mismatch fails the call with a clear
FailedPrecondition, such as "Worker serves Dex ; the call came from Dex ", before any hydration or handler runs.
This needs a protocol field in the Worker request, so the Server and the SDKs change together.
Activity
longquanzheng commentedon Nov 9, 2026
server-zd supports pull Workers only; a Worker receives tasks over its own stream to one server, so it cannot serve calls from another server. Closing as obsolete (Go Server removed in #1059).