Support enum columns in transformers - #1147
Draft
kvch wants to merge 2 commits into
Draft
Conversation
A transformation rule on a user-defined enum column was rejected at startup with "does not support pg data type: <enum> with OID: <n>", unless it used a transformer declaring AllDataTypes. Problem: An enum's OID is assigned by the database, so it reaches no case of the transformer type compatibility switch, whose fallback branch knows only citext and hstore by name. The name alone cannot identify an enum, so the parser had no way to decide what such a column accepts. Cause: Two further problems sat behind that one. greenmask_choice is the only transformer that can be constrained to values an enum actually has, but its choices parameter is required, and it returned the chosen label as []byte regardless of the input type, which reaches postgres hex encoded and is rejected with "invalid input value for enum". Change: - Mapper.EnumForOID resolves an OID to the enum it names and returns its labels. Postgres reports a domain's base type in the row description, so a domain over an enum arrives already resolved; an array of an enum has its own OID, matches nothing, and is left to the existing type check to reject by name. Results are cached, a nil value being a cached negative. - A new EnumDataType, declared only by greenmask_choice. Enums are deliberately not mapped to StringDataType: greenmask_string and masking would then pass validation and emit a label the enum does not have, failing per row against the target instead of at startup. - The parser resolves the enum before building the transformer, so a greenmask_choice rule that omits choices gets the enum's labels, using the hook that already injects postgres_url for pg_anonymizer. Choices listed explicitly are checked against the labels. An explicitly empty list is left alone for the builder to reject, since it is usually a template that rendered nothing rather than a request for every label. - greenmask_choice returns a string when the input was a string. The defaulted labels are otherwise an invisible config value the operator never wrote, so each one is logged at startup, and combining them with the deterministic generator raises a warning: that generator uses no secret, and an enum publishes its whole label set to anyone who can read the target, so the mapping inverts by trying all of them. Verification: - make lint: 0 issues - make test: passes - PGSTREAM_INTEGRATION_TESTS=true go test ./pkg/stream/integration ./pkg/wal/processor/transformer ./internal/postgres: all pass - Test_SnapshotToPostgres_EnumColumnTransformer asserts the exact rows a deterministic run must produce, computed by running the transformer the parser builds, over both writer paths. Membership in the label set would have passed even with the transformer skipped, since the source values are themselves labels - TestPostgresTransformerParser_ParseAndValidate_enumColumns covers the wiring against a mocked catalog: defaulting, an explicit subset, an invalid label, a rejected transformer, and a failing lookup. Swallowing the EnumForOID error, which no test caught before, now fails it - TestPostgresTransformerParser_EnumColumns_Integration covers the same against a real catalog, plus domains and enum arrays - TestMapper_EnumForOID asserts the cache is used, not just populated Caveats: The labels are read once, at startup, so a replicated ALTER TYPE ... RENAME VALUE is picked up only on restart. Constraints narrowing the column further, such as a CHECK on a domain over the enum, are not visible in the row description and are not honoured. Both are documented, along with the requirement for a source Postgres URL. Emitting the chosen value as a string changes what the Kafka, webhook and search targets carry for every greenmask_choice column, not only enum ones: they JSON encoded the previous []byte as base64. This matches what greenmask_string already did and is documented, but consumers that decoded base64 need updating. Generated by an unattended AFK agent run. Not reviewed by a human. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KyBQAf6EhgQTrY3WccNcwf
kvch
added
ai-review
kvch
marked this pull request as draft
September 3, 2026 18:37
CoverageTotal: 61.4% (+0.2% vs Coverage in packages changed by this PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please provide a brief description of the changes in this PR.
Related Issue(s)
Type of Change
Please select the relevant option(s):
Changes Made
Testing
Checklist
Additional Notes