Skip to content

fix: (rollingops) dynamic cluster ID - #706

Merged
patriciareinoso merged 20 commits into
mainfrom
DPE-11212-rollingops-fix-dynamic-cluster-id
Oct 7, 2026
Merged

patriciareinoso merged 20 commits into
mainfrom
DPE-11212-rollingops-fix-dynamic-cluster-id

Conversation

@patriciareinoso

@patriciareinoso patriciareinoso commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Fix #684

Problem

If the etcd relation is created before the charm has a cluster_id, the etcd backend never gets set up. Once cluster_id becomes available later, the relation-created hook that would normally publish the request has already fired and won't fire again, so etcd never is configured, and the unit stays stuck on the peer backend.

Fix

  • Added EtcdRequiresV1.ensure_request_published(), called in the _EtcdRollingOpsBackend constructor. It writes the request for an existing relation if it hasn't been published yet, using the public dpcharmlibs.interfaces helpers.
  • Fixed an ordering bug: the client certificate must be created before EtcdRequiresV1 is constructed. Previously the cert was created after, so the first published request was always missing it, which etcd rejects with "Missing certificate or prefix"
  • Added a toggle-cluster-id action to the integration test charms to manually reproduce/verify this race.
  • Minor logging cleanup picked up along the way.

Testing

  • Added tests/unit/test_etcd_dynamic_cluster_id.py:
    • request is not published while cluster_id is None
    • request is published on a later event, for an already-existing relation, once cluster_id becomes available
    • the published request includes the client certificate (secret-mtls), guarding against the ordering bug above
    • a request is only ever published once (its request_id, derived from a random salt, stays stable across repeated calls)

Manual test

The fix was manually tested for same-model and cross-model relations

unit-cross-test-0: 16:20:48 INFO unit.cross-test/0.juju-log Shared certificate added to the databag.
unit-cross-test-0: 16:20:48 INFO unit.cross-test/0.juju-log Replaying missed etcd rollingops relation-created handling for cluster_id=/rollingops/default/cluster-12345/ on relation etcd/1.
unit-cross-test-0: 16:21:35 INFO unit.cross-test/0.juju-log etcd:1: resource /rollingops/default/cluster-12345/ created at 2026-10-05 14:21:35.402041
unit-cross-test-0: 16:21:35 INFO juju.worker.uniter.operation ran "etcd-relation-changed" hook (via hook dispatching script: dispatch)
unit-cross-test-0: 16:21:38 WARNING unit.cross-test/0.etcd-relation-changed /var/lib/juju/agents/unit-cross-test-0/charm/venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py:2274: UnsupportedFieldAttributeWarning: The 'default' attribute with value None was provided to the `Field()` function, which has no effect in the context it was used. 'default' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
unit-cross-test-0: 16:21:38 WARNING unit.cross-test/0.etcd-relation-changed   warnings.warn(
unit-cross-test-0: 16:21:38 WARNING unit.cross-test/0.etcd-relation-changed /var/lib/juju/agents/unit-cross-test-0/charm/venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py:2274: UnsupportedFieldAttributeWarning: The 'exclude' attribute with value True was provided to the `Field()` function, which has no effect in the context it was used. 'exclude' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
unit-cross-test-0: 16:21:38 WARNING unit.cross-test/0.etcd-relation-changed   warnings.warn(
unit-cross-test-0: 16:22:03 INFO juju.worker.uniter.operation ran "etcd-relation-changed" hook (via hook dispatching script: dispatch)
unit-cross-test-0: 16:22:23 WARNING unit.cross-test/0.restart /var/lib/juju/agents/unit-cross-test-0/charm/venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py:2274: UnsupportedFieldAttributeWarning: The 'default' attribute with value None was provided to the `Field()` function, which has no effect in the context it was used. 'default' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
unit-cross-test-0: 16:22:23 WARNING unit.cross-test/0.restart   warnings.warn(
unit-cross-test-0: 16:22:23 WARNING unit.cross-test/0.restart /var/lib/juju/agents/unit-cross-test-0/charm/venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py:2274: UnsupportedFieldAttributeWarning: The 'exclude' attribute with value True was provided to the `Field()` function, which has no effect in the context it was used. 'exclude' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
unit-cross-test-0: 16:22:23 WARNING unit.cross-test/0.restart   warnings.warn(
unit-cross-test-0: 16:22:23 INFO unit.cross-test/0.juju-log etcd backend is available. Switching to etcd backend.
unit-cross-test-0: 16:22:23 INFO unit.cross-test/0.juju-log Operation _restart added to the peer queue.
unit-cross-test-0: 16:22:23 INFO unit.cross-test/0.juju-log Operation _restart added to the etcd queue.
unit-cross-test-0: 16:22:23 INFO unit.cross-test/0.juju-log Missing PID or invalid PID found in etcd worker state.
unit-cross-test-0: 16:22:23 INFO unit.cross-test/0.juju-log Started etcd-rollingops-async-worker process with PID 18590
unit-cross-test-0: 16:22:34 INFO unit.cross-test/0.juju-log Executing rollingop on etcd backend.
unit-cross-test-0: 16:22:34 INFO unit.cross-test/0.juju-log Executing callback_id=_restart, attempt=0
unit-cross-test-0: 16:22:34 INFO unit.cross-test/0.juju-log Starting restart operation
unit-cross-test-0: 16:22:34 INFO unit.cross-test/0.juju-log Finished _restart. Lock will be released.
unit-cross-test-0: 16:22:34 INFO unit.cross-test/0.juju-log Execution mirrored to peer relation.
``

@patriciareinoso
patriciareinoso marked this pull request as ready for review October 5, 2026 14:53
@patriciareinoso
patriciareinoso requested a review from a team as a code owner October 5, 2026 14:53
@patriciareinoso
patriciareinoso marked this pull request as draft October 6, 2026 10:19
@patriciareinoso
patriciareinoso marked this pull request as ready for review October 6, 2026 15:54
Comment thread rollingops/src/charmlibs/rollingops/_etcd/_relations.py
@patriciareinoso
patriciareinoso merged commit a90ed58 into main Oct 7, 2026
28 of 29 checks passed
@patriciareinoso
patriciareinoso deleted the DPE-11212-rollingops-fix-dynamic-cluster-id branch October 7, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

(rollingops): etcd is not set up if dynamic cluster ID is provided after the etcd relation is created

3 participants