Skip to content

A Worker serves calls from a Dex Server it does not belong to #601

Closed as not planned

Description

@longquanzheng

What happens

A Worker hydrates blobs and calls back into the Dex Server named by its own FlowServiceAddress, not the Server that dispatched the call. When two Dex Servers on one host each record a Flow's Worker target, and those targets end up served by one Worker, that Worker serves calls from a Server it does not belong to. It then answers with a misleading error, or with no error at all.

Observed on Server v1.4.1 with Go SDK v1.5.0, on two local Dex environments on one machine, each with its own Temporal and blob store:

  1. A client started a Flow on Server A with an explicit Worker target host.docker.internal:25051. That target was the Worker port of environment B. Server A stored the Flow's large initial Attributes in its own blob store, as expected (service/api/service.go:177-198).
  2. Later, environment B's Worker listened on that port. Every RPC read of the Flow through Server A was dispatched to B's Worker (common/rpc/invoke.go:81-85).
  3. B's Worker hydrated the blob IDs through its own Flow service, Server B (dex/worker.go:47-49,119-139; dex/worker_service.go:368-373). Server B's LoadBlobs found no such objects and omitted them (service.go:1188-1191).
  4. The SDK failed with dex: LoadBlobs omitted blob "p0|11/…" (dex/hydration.go:173-179), which Server A reported as FailedPrecondition / worker API error.

The blobs were intact in Server A's store the whole time. Server B's log showed the GetObject … NoSuchKey reads for A's Flow.

Here the hydration failure happened to stop the wrong Worker from running the handler. A Flow whose values are all small and inline would hydrate nothing, and B's Worker would silently run A's handlers against A's state, with B's dependencies.

Suggested fix

Let a Worker refuse a call from a Server it does not serve:

  • The dispatching Server sends its identity in the Worker request context, for example its namespace plus a configured deployment ID or a blob-store fingerprint.
  • At startup the Worker reads the identity of its own Flow service.
  • A mismatch fails the call with a clear FailedPrecondition, such as "Worker serves Dex ; the call came from Dex ", before any hydration or handler runs.

This needs a protocol field in the Worker request, so the Server and the SDKs change together.

Activity

  1. added
    type:bugWrong behavior in released or main code
    P1Do next; important for an upcoming release
    needs-decisionNeeds an owner decision before implementation
    on Oct 26, 2026
  2. longquanzheng commented on Nov 9, 2026

    @longquanzheng
    ContributorAuthor

    server-zd supports pull Workers only; a Worker receives tasks over its own stream to one server, so it cannot serve calls from another server. Closing as obsolete (Go Server removed in #1059).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Do next; important for an upcoming releasearea:sdk-goComponent: sdk-goneeds-decisionNeeds an owner decision before implementationsize:MA few daystype:bugWrong behavior in released or main code

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions