[Hardik K (hardikkaurani)]

Summary

• Adds serialization support for set, frozenset, and collections.abc.Set in _encode_custom_object, with deterministic ordering for sortable elements.
• Extends _coerce_generic in JsonDataConverter to recursively reconstruct generic set[T] and frozenset[T] containers, including dataclass fields.
• Updates can_reconstruct to validate element types for generic set containers.
• Adds unit tests covering primitive sets, frozensets, dataclasses with set fields, round-trip serialization and deserialization, and reconstruction checks.
• Updates CHANGELOG.md under the Unreleased section.

Validation

• All tests in test_serialization.py and test_data_converter.py passed successfully.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Hardik K (hardikkaurani)]

Hi Bernd Verst (@berndverst),

I've opened this PR to add native support for set and frozenset generic containers in JsonDataConverter.

Previously, serializing objects or dataclasses containing sets raised TypeError: Object of type 'set' is not JSON serializable. On deserialization, fields typed as set[T] or frozenset[T] remained regular lists instead of being converted to the expected container types.

This PR addresses both issues by adding deterministic serialization and recursive coercion for generic set containers, along with comprehensive test coverage.

Whenever you have a chance, could you please review it? Thank you for your time!

[Copilot]

🟡 Changes recommended

Set serialization can produce nondeterministic payloads, and automatic type discovery skips primitive-element set annotations.

2 open issues

• Ensure deterministic serialization order for unordered set elements · New
• Preserve set annotations during type discovery and activity execution · New

_{Review effort: Balanced}

[andystaples]

Requesting changes for the replay-compatibility and deterministic-serialization issues noted inline.

Existing persisted orchestration history is a compatibility contract. An SDK upgrade must not change how an already-running instance reconstructs historical payloads in a way that changes its orchestration decisions. Keeping the JSON readable is necessary, but it is not enough: the same history must still replay with the same behavior. If new reconstruction semantics would change that behavior, they need an explicit opt-in or versioned compatibility path that preserves the old behavior for existing instances, plus regression coverage using pre-change histories.

The legacy marker handling remains intact, and histories that do not exercise the newly recognized set annotations appear unaffected. However, this change is not universally safe for in-flight orchestration upgrades as written. The set encoder also needs deterministic ordering for supported custom objects rather than falling back to set iteration order.

Please also fix the failing pyright CI check. The current typecheck_core-3.10 log reports five errors in the added code: partially unknown return/argument types at serialization.py:324 and :326, and an unknown item type at :489. Please resolve the collection/element typing and get this check passing.

[Hardik K (hardikkaurani)]

andystaples just doing !

[andystaples]

Thanks. The PR was closed earlier, and GH records show it was me, but I didn't do it deliberately, so I re-opened it

[Hardik K (hardikkaurani)]

Hi andystaples, thank you for the detailed review and feedback!

All requested changes and reviewer findings have been addressed:

1. Replay Compatibility & Opt-in Generic Set Reconstruction

• Generic set coercion is now gated behind an explicit opt-in parameter: enable_generic_sets: bool = False on JsonDataConverter (with enable_set_coercion supported as an alias).
• By default (enable_generic_sets=False), existing instances and historical histories retain their exact replay behavior: arrays deserializing under set/frozenset annotations remain raw lists, preserving original order, duplicates, and list lengths without triggering NonDeterminismError.
• Added regression test test_existing_instance_replay_compatibility_preserves_list validating that {"tags": ["b", "a", "b"]} remains a 3-element list ["b", "a", "b"] with default settings and only coerces to {"a", "b"} when opted in.

2. Deterministic Set Serialization Across Processes & Hash Seeds

• Removed the list(o) fallback that leaked hash-dependent iteration orders for unorderable items (such as unordered frozen dataclasses, enums, etc.).
• When sorted(items) fails with a TypeError, elements are deterministically sorted using a composite key derived from their recursive JSON representation, qualified type name, and repr:

  (json.dumps(item, default=_encode_custom_object, sort_keys=True), type(item).__qualname__, repr(item))

• Added cross-process regression test test_set_serialization_deterministic_across_hash_seeds running separate Python processes across PYTHONHASHSEED=1, 2, and 999 to ensure identical payload ordering.

3. Type Discovery & Activity Execution Support

• Updated _can_reconstruct so that supported sets with primitive and hashable element annotations (e.g. set[str], frozenset[int]) are recognized when set reconstruction is enabled, while safely rejecting nested/unhashable structures.
• Added regression test test_type_discovery_and_activity_execution_with_sets verifying that activity and orchestrator type discovery preserves set annotations and activity functions receive sets where .add() works as expected.

4. Pyright & Test Suite Verification

• Resolved all strict pyright typing issues (pyright durabletask/serialization.py reports 0 errors, 0 warnings).
• Full unit test suite passes cleanly (1261 passed, 15 skipped).

[andystaples]

The default-off reconstruction flag addresses the original existing-history replay regression, and primitive-set discovery now works when opted in. There is still one blocking determinism case: nested frozensets take the native sorting path without raising, but that comparison does not provide a total order. Details and a reproduction are inline.

[andystaples]

[P1] Use a deterministic total ordering even when native sorting succeeds

The new fallback fixes the non-orderable dataclass case, but sorted(items) can succeed without producing a deterministic order. Frozensets compare by subset relationships, not a total order, so disjoint frozensets keep hash-dependent iteration order and never reach the fallback.

For example:

JsonDataConverter().serialize({
    frozenset({"alpha"}),
    frozenset({"beta"}),
    frozenset({"gamma"}),
})

In separate processes, I get:

• PYTHONHASHSEED=1: [["alpha"], ["gamma"], ["beta"]]
• PYTHONHASHSEED=2: [["gamma"], ["alpha"], ["beta"]]

This still violates the deterministic-serialization contract across worker processes and replay. Please use a deterministic total ordering based on recursively encoded values rather than relying on native sorting merely because it does not raise, and extend the cross-process regression to nested frozensets.

[Hardik K (hardikkaurani)]

Thank you for identifying this edge case, andystaples.

I have updated the set serialization ordering in durabletask/serialization.py to use a dedicated deterministic total-ordering sort key (_deterministic_set_sort_key) based on recursively encoded values rather than relying on native sorting to succeed. This ensures disjoint and nested frozenset values establish a deterministic total order across processes and replays, while preserving numerical ordering for numeric sets.

I also updated and extended test_set_serialization_deterministic_across_hash_seeds in tests/durabletask/test_serialization.py to exercise disjoint frozensets (Case A), nested frozensets across multiple levels (Case B), and unorderable frozen dataclasses (Case C) across multiple PYTHONHASHSEED values (0, 1, 2, 42, 999) using portable repository-relative path resolution.

Validation Results

• pytest tests/durabletask/test_serialization.py: 56 passed
• pytest tests/durabletask/test_data_converter.py: 35 passed
• pytest tests/durabletask/test_data_converter_roundtrip.py: 48 passed
• pytest tests/durabletask/test_type_discovery.py: 37 passed
• nox -s lint: passed (flake8 clean, 0 issues)
• Strict pyright durabletask/serialization.py: 0 errors, 0 warnings
• nox -s functions_unit-3.13: 406 passed

The fix is pushed to fix/serialization-generic-sets in commit 6c924e5.

[Hardik K (hardikkaurani)]

Hi andystaples,

Thank you for the detailed review.

I've addressed the remaining deterministic-serialization issue by using recursively encoded ordering keys for unordered set elements, including nested frozenset values. The implementation no longer relies on native sorting merely because it completes without raising an exception.

I've added cross-process regression coverage to verify that the serialized payload remains identical across multiple PYTHONHASHSEED values. I also rechecked the existing replay-compatibility defaults, opt-in set reconstruction, type discovery, and typing checks.

Validation

• Nested-frozenset regression: verified in-process and cross-process; disjoint frozensets ([["alpha"], ["beta"], ["gamma"]]) and multi-level nested frozensets ([[["a", "b"], ["c"]], [["d"]]]) produce identical deterministic JSON.
• Cross-process deterministic serialization: test_set_serialization_deterministic_across_hash_seeds passed across PYTHONHASHSEED values 0, 1, 2, 42, and 999 using portable repository-relative path resolution.
• Serialization/data-converter tests:
  • pytest tests/durabletask/test_serialization.py: 56 passed
  • pytest tests/durabletask/test_data_converter.py: 35 passed
  • pytest tests/durabletask/test_data_converter_roundtrip.py: 48 passed
  • pytest tests/durabletask/test_type_discovery.py: 37 passed
• Pyright and other checks:
  • Strict pyright durabletask/serialization.py: 0 errors, 0 warnings
  • nox -s lint: passed (flake8 clean, 0 issues across all targets)
  • nox -s functions_unit-3.13: 406 passed
• CI: CLA check (license/cla) passed; external fork workflows queued/pending maintainer review.

The changes have been pushed to the existing PR branch fix/serialization-generic-sets in commit 6c924e5. I appreciate your feedback and would be grateful if you could review the updated implementation.