[ Feedback ] Topology behavior, failure propagation, replicas, and one flaky test

[myfear]

Thanks for publishing Flamme and inviting people to test it.

I built a small Quarkus release-gate application with four Flamme components and took it for a testdrive.

release-gateway -> candidate-validator -> risk-scorer -> release-decider
        ^                                                    |
        +---------------- CompletableFuture reply -----------+

The core idea worked well. I ran the complete graph in one JVM, then moved only
risk-scorer into another process through configuration. The component
interfaces and implementations stayed unchanged. Flamme switched the boundary
from its local broker to NATS and Protocol Buffers as advertised.

This issue collects the behavior I observed around that successful path. It
covers several areas, so I am happy to split it into smaller issues if that is
easier to track.

The demo is here: https://github.com/myfear/the-main-thread/tree/main/flamme-release-gate

Test environment

• Flamme commit:
  8afdaf6e8b59bc3b443750cf099971593ddb66c9
• Flamme version: 1.0.0-SNAPSHOT
• Quarkus: 3.34.1
• Java: Oracle GraalVM 21.0.2
• Maven: 3.9.16
• macOS 26.5.2 on Apple Silicon
• Podman: 5.8.2
• NATS: nats:2.14.1-alpine

What worked

The local topology completed the full request and reply path through the
in-process broker.

The split topology also worked. I configured risk-scorer as remote in the API
process and left only that component local in a worker process. One request
produced the following component logs:

node=api component=candidate-validator release=release-split
node=worker-a component=risk-scorer release=release-split score=27
node=api component=release-decider release=release-split approved=true

The caller received HTTP 200 in about 140 ms. The worker identity in the
response confirmed that the payload crossed the process boundary.

Summary of findings

Area	Observed behavior	Suggested direction
Local-only startup	Flamme requires a NATS connection even when every component is local	Initialize NATS only when the resolved topology has a remote edge
Component failures	An implementation exception becomes a gateway timeout	Send a structured error reply and complete the gateway future exceptionally
Broker loss	Failed publishes are swallowed and the caller eventually times out	Report publish failures, expose transport health, and fail affected requests promptly
Worker replicas	Every replica processes every message	Make broadcast and queue-group semantics explicit configuration choices
Reflection	Package-private component types compile but fail on first invocation	Validate accessibility during Quarkus augmentation
Payload boundary	Local routing shares the map while remote routing rebuilds declared keys only	Document the semantic difference and consider build-time key validation
Delivery contract	Core NATS currently gives the remote path an at-most-once boundary	State the delivery contract next to topology documentation
CI stability	PublishSubscribeLocalTest can read probe state before the fixture publishes it	Release the test latch after writing all observed state

1. A local-only topology still requires NATS

I left every service at the default remote=false setting and started the
application without NATS. Startup failed:

ERROR [com.amadeus.flamme.runtime.ConnectionInitializer]
there was an error connecting to NATS

Caused by: com.amadeus.flamme.runtime.errors.NatsConnectionError:
there was an error connecting to NATS

Caused by: java.io.IOException:
Unable to connect to NATS servers: [nats://localhost:4222]

ConnectionInitializer opens the NATS connection for every application
startup. This means a fully local graph still depends on broker infrastructure,
although no event crosses the process boundary.

I would initialize the transport only when the resolved graph has at least one
remote edge. An explicit property such as flamme.nats.enabled=false could also
help. If that property is disabled while the topology contains a remote route,
startup should fail with a message that names the affected service or subject.

2. Component exceptions become timeouts

I configured risk-scorer as a remote component and made its implementation
throw:

throw new IllegalStateException("forced risk scorer failure");

The worker logged:

error invoking com.themainthread.releasegate.RiskScorer

The caller then waited for the configured three-second reply timeout:

500 - Internal Server Error
java.util.concurrent.TimeoutException
HTTP 500 in 3.008518s

Handler.buildServiceHandler catches FlammeImplRuntimeError and logs it, but
does not publish a reply. The reply-side codec already supports
decodeError(...), which looks like most of the receiving path is present.

I would publish a structured error envelope to replyTo when component
resolution, invocation, or result handling fails. It could contain:

• A stable error code
• The component or service name
• The request correlation ID
• A safe message

I would not put an arbitrary Java stack trace on the wire. The local process can
keep the stack trace in its logs.

Completing the gateway future exceptionally would also let an HTTP or messaging
adapter distinguish a component failure from a timeout.

3. Broker loss hides failed publishes

I stopped NATS while the split topology was running and sent another request.
The local validator ran, but the remote scorer never received the event. The
caller got the same three-second TimeoutException.

The NATS client logged connection failures while it tried to reconnect.
Broker.publishToNats and Broker.forwardReplyToNats catch Throwable without
logging it or completing an affected reply future.

This makes a rejected publish, an unavailable worker, and a slow component look
the same to the caller.

Possible improvements:

• Log failed publishes with the subject and correlation ID
• Expose NATS connection state through a Quarkus health check
• Complete a known gateway request exceptionally when its publish cannot be
  accepted
• Add metrics for publish failures, reconnects, timeouts, and late replies

4. Worker replicas receive duplicate work

I started two processes with the same remote risk-scorer configuration. One
request was processed by both:

node=worker-a component=risk-scorer release=release-replicas score=31
node=worker-b component=risk-scorer release=release-replicas score=31

The API also ran the downstream decider twice:

node=api component=release-decider release=release-replicas approved=true
node=api component=release-decider release=release-replicas approved=true

NatsTransportClient uses the plain Dispatcher.subscribe(subject, handler)
method, so this is normal Core NATS broadcast behavior. It is useful for event
listeners, but worker-style components often need competing consumers.

It would help if a service could choose its replica semantics explicitly. For
example:

flamme.services.risk-scorer.delivery-mode=queue
flamme.services.risk-scorer.queue-group=release-risk

Broadcast should remain available because both models are valid.

5. Inaccessible component types fail at runtime

My first component interfaces and implementations were package-private. The
project compiled, but the first invocation failed with:

error invoking com.themainthread.releasegate.CandidateValidator

Making the interface and implementation classes public fixed the problem.
Handler invokes the interface method through reflection, so inaccessible
types eventually surface as an invocation error and then as the timeout
described above.

Flamme already validates component method signatures during augmentation. The
same build step could validate the accessibility of the interface, method, and
implementation. A build failure naming the exact inaccessible type would make
this much easier to diagnose.

6. Local and remote payloads have different semantics

The local broker passes the same Map<String, Message> object to subscribers.
The remote path serializes the map and reconstructs only the keys declared in
@MultiPayloadKey.

That creates two important rules for component authors:

1. Treat input maps and protobuf messages as immutable.
2. Declare every key that must survive a remote boundary on the receiving
   component.

These rules should be close to the topology documentation because a graph can
work locally and lose a required key after one service becomes remote.

A later build-time enhancement could validate known downstream payload
requirements against the keys declared for remote decoding.

7. Please document the delivery contract

The current transport uses Core NATS publish and subscribe. There is no durable
stream, acknowledgement, replay, or dead-letter path. A message can disappear
when no subscriber is active or when the broker disconnects at the wrong time.

That gives the current remote path an at-most-once delivery boundary. I think
this is a reasonable lightweight default, but it should be stated explicitly so
users can decide where idempotency and retry logic belong.

The transport abstraction leaves room for Kafka or JetStream. Those transports
would add useful options, but they also introduce different guarantees.
Documenting delivery, ordering, retry, and consumer-group semantics per
transport would keep location transparency from hiding operational behavior.

8. PublishSubscribeLocalTest has a race

The GitHub Actions build exposed a race in:

deployment/src/test/java/com/amadeus/flamme/test/PublishSubscribeLocalTest.java
deployment/src/test/java/com/amadeus/flamme/test/fixtures/Components.java

The failing assertion expected the recorded input but read null:

PublishSubscribeLocalTest.nonLeafSubscriberShouldReceiveAndProcessMessage
expected: <{VALUE=value: "Hello"}>
but was: <null>

In the fixture from the pinned commit, countDown() runs before the state read
by the test is stored:

probe.called.countDown();
probe.lastInput.set(input);
// ...
probe.lastOutput.set(result);

The test thread can return from await(...) and read either reference before
the component thread updates it.

Moving the signal after both writes fixes the ordering:

 public Map<String, Message> toUpper(Map<String, Message> input) {
-  probe.called.countDown();
   probe.lastInput.set(input);
   StringValue stringValue = (StringValue) input.get("VALUE");
   Map<String, Message> result = new HashMap<>();
   result.put("VALUE", StringValue.of(stringValue.getValue().toUpperCase()));
   probe.lastOutput.set(result);
+  probe.called.countDown();
   return result;
 }

A successful await(...) then establishes the intended happens-before
relationship for the observed probe state.

Release availability

I vendored the runtime and deployment modules because the project uses
1.0.0-SNAPSHOT and I could not resolve a published release artifact.

A tagged release in Maven Central would make end-to-end examples much easier to
reproduce. A documented snapshot repository and a known-good commit would also
help until the first release is ready.

Thanks again for sharing the project. Hope my feedback helps a little.

[sonOfTheComet-ctrl]

Hello @myfear , thanks a lot for the feedback.
I will create sub-issues to make tracking easier. You indeed made valid points, some of them I was already aware of like 2, 4 and 6.
Some of which I was not.

Some comments I have:

• Regarding 7: I opted for a fire/forget model which is the default with NATS. I am planning to explore other alternatives when I start the implementation of other message brokers with other contracts. But you're right, this should be better documented.
• Regarding 6: In a local-only topology, not specifying multipayloadkeys doesn't cause any errors, but we recommend always doing so because it describes the types this component is expected to receive. So it makes the component easier to read. This should also be better documented.
• Regarding 2: This is a planned feature that we will have before the first release. Any component will be able to short-circuit to the replyTo, if it throws an error.
• I will explore what can be done for 4 and 3.