[PLUGIN-1931] Add defaults for Time and Range Partitioning in BQ source plugin - #1583
Merged
Merged
Conversation
adrikagupta
force-pushed
the
fix-bq-source-partitioning
branch
2 times, most recently
from
September 24, 2025 13:55
1d9e162 to
21de8fc
Compare
adrikagupta
marked this pull request as ready for review
September 24, 2025 14:09
adrikagupta
force-pushed
the
fix-bq-source-partitioning
branch
from
September 24, 2025 14:14
21de8fc to
1d4aa48
Compare
adrikagupta
force-pushed
the
fix-bq-source-partitioning
branch
from
September 25, 2025 09:14
1d4aa48 to
85633f7
Compare
itsankit-google
approved these changes
Sep 25, 2025
itsankit-google
left a comment
Contributor
There was a problem hiding this comment.
one small comment, otherwise LGTM
This was referenced Oct 1, 2025
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If we pass a filter input in BQ source plugin and keep partitionFromDate and partitionToDate values as empty for a table where requirePartitionFilter is set as true, then the preview/deployment run will fail because since requirePartitionFilter is true, the query requires a mandatory “where” clause on partitioning column.
Different types of partitioning for BQ are listed in https://cloud.google.com/bigquery/docs/partitioned-tables .
To fix we are adding defaults for range, time unit and ingestion partitioning which would be used if requirePartitionFilter is set as true and no partitioning inputs are passed. The default would be "parititioning-column is NOT NULL OR parititioning-column is NULL" to get the entire data.
A few things were noted when considering these defaults:
Also, we have added limit and order by fields in UI since filter field is only meant for WHERE clause.
Testing: