[CORE-16775] cluster_link: unsupported-feature policy for Schema Registry config sync

[bartoszpiekny-redpanda]

Extends unsupported_schema_feature_policy (FAIL default / REMOVE) from the
schema-fetch path to the config path, so a migration from a
Confluent-compatible source also handles config-borne features Redpanda cannot
store: compatibilityGroup, default/override metadata & rule sets. The tolerant
config parser now returns structured config_info::unsupported (a JSON pointer

• type per field, mirroring source_schema_read::unsupported), which
  read_config carries through as a source_config_read to the mirroring task
  instead of dropping it. The mirroring task applies the policy per config target
  (subject / context / global).

REMOVE syncs only the supported projection (compatibilityLevel), increments
unsupported_features_removed — which the proto documents as covering "schemas
or schema configs" — and logs the dropped fields to the broker log; the count is
taken only after the config write lands, so a failed write does not report
features as removed. FAIL (the default) treats an unsupported config as a
per-item failure: it logs the offending fields, counts an error, and skips that
subject's config write while the rest of the sync proceeds; the task stays
active, and only global errors (source unavailable) fail-fast. A null-valued
config field is treated as absent. This mirrors the schema-fetch path's per-item
handling exactly.

Note: follow-up to #31047 (schema-fetch path), same ticket. These config
governance fields are the config-borne half the RFC lists as unsupported. No
proto/model/converter/rpk changes: the config contract, the counter, and rpk shadow status rendering already exist.

Tests: reconciler and mirroring-task unit tests cover config REMOVE (strip +
count; no count on a failed config write) and FAIL (count + skip the offender,
sync the rest, task stays active), plus config diagnostics parsing (structured
JSON pointer + type, null-as-absent). Four Confluent-source e2e cases (schema
and config, each under FAIL and REMOVE) exercise the policy end-to-end against a
real Confluent Schema Registry via ducktape.

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v26.1.x
• v25.3.x
• v25.2.x

Release Notes

*none

[Copilot]

Pull request overview

Extends the existing Schema Registry “unsupported feature” policy (FAIL/REMOVE) from schema bodies to Schema Registry config sync, so Confluent-only config governance fields are either rejected per-item (FAIL) or stripped-and-counted after successful writes (REMOVE).

Changes:

• Teach the SR rest_client config parser to surface unmodeled non-null fields as structured unsupported_feature entries and allow missing/null compatibilityLevel.
• Plumb config unsupported diagnostics through the source reader (source_config_read) to the mirroring task, and apply FAIL/REMOVE consistently on the config write path.
• Add unit/integration/e2e coverage for unsupported schema + config behavior under both policies.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/rptest/tests/cluster_linking_schema_registry_sync_test.py	Adds e2e cases for unsupported schema fields and unsupported config governance fields under FAIL/REMOVE.
src/v/pandaproxy/schema_registry/rest_client/tests/parse_test.cc	Updates parse tests to validate structured unsupported and missing/null compatibilityLevel behavior.
src/v/pandaproxy/schema_registry/rest_client/tests/client_test.cc	Updates rest_client tests to assert unsupported instead of unknown_fields.
src/v/pandaproxy/schema_registry/rest_client/tests/client_integration_test.cc	Updates integration assertions to use unsupported for config reads.
src/v/pandaproxy/schema_registry/rest_client/parse.h	Updates parse_config contract documentation (missing/null level allowed; unsupported surfaced).
src/v/pandaproxy/schema_registry/rest_client/parse.cc	Implements config unsupported-feature collection and missing/null compatibility handling.
src/v/pandaproxy/schema_registry/rest_client/config.h	Changes config_info to optional<level> + structured unsupported.
src/v/pandaproxy/schema_registry/rest_client/client.h	Updates API docs to reflect unsupported reporting for config reads.
src/v/cluster_link/schema_registry_sync/unavailable_source_reader.h	Updates interface return type for config reads to source_config_read.
src/v/cluster_link/schema_registry_sync/unavailable_source_reader.cc	Implements updated config read signature for unavailable reader.
src/v/cluster_link/schema_registry_sync/tests/sr_sync_test_fixtures.h	Extends fake source/destination test fixtures to inject unsupported config fields and config write failures.
src/v/cluster_link/schema_registry_sync/tests/mirroring_task_test.cc	Adds unit tests covering FAIL/REMOVE behavior on config sync, including “no count on failed write”.
src/v/cluster_link/schema_registry_sync/tests/http_source_reader_test.cc	Adds test for governance-only configs surfacing unsupported fields while having no compatibility override.
src/v/cluster_link/schema_registry_sync/source_reader.h	Introduces source_config_read (compatibility + unsupported diagnostics).
src/v/cluster_link/schema_registry_sync/mirroring_task.h	Adds helpers to apply FAIL/REMOVE policy on config sync path.
src/v/cluster_link/schema_registry_sync/mirroring_task.cc	Applies policy when syncing config: FAIL skips write; REMOVE counts after successful change.
src/v/cluster_link/schema_registry_sync/http_source_reader.h	Updates config read signature to return source_config_read.
src/v/cluster_link/schema_registry_sync/http_source_reader.cc	Carries unsupported config features through and allows governance-only configs to parse as “no override”.

[bartoszpiekny-redpanda]

Force-pushed: 016983b..bd94392 — on a duplicate compatibilityLevel key the null branch now clears the earlier value (last-wins), with a test pinning both duplicate orders.

[vbotbuildovich]

CI test results

test results on build#87004

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FLAKY(PASS)	EndToEndCloudTopicsTest	test_write	{"storage_mode": "cloud"}	integration	https://buildkite.com/redpanda/redpanda/builds/87004#019f5b29-854d-4eab-8cbb-1ce1630fbc24	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0000, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=EndToEndCloudTopicsTest&test_method=test_write

[pgellert]

What are your thoughts on this bit? I'm tempted to say that we should count (potentially over-count) dropped fields, even if we did not end up changing the compatibility level, so that a response like {"compatibilityGroup": "app.major.version"} that you show in the ducktape doesn't hide that we silently ignored the compatibilityGroup field.

[bartoszpiekny-redpanda]

I think it's reasonable. I didn't want to have very verbose logging but there is second side of that which is indeed hiding ignoring fields. Let me change it.

[bartoszpiekny-redpanda]

Force-pushed: bd94392..0eb842d — REMOVE now counts unsupported config fields once per completed sync (no longer gated on the write applying a change), so a governance-only config is not silently ignored; pinned by remove_policy_counts_unsupported_config_no_level, the per-sync recount assertions, and a strengthened ducktape predicate.