feat(bigquery): Support DATE-type event timestamp columns#6362
Open
Jwrede wants to merge 9 commits intofeast-dev:masterfrom
Open
feat(bigquery): Support DATE-type event timestamp columns#6362Jwrede wants to merge 9 commits intofeast-dev:masterfrom
Jwrede wants to merge 9 commits intofeast-dev:masterfrom
Conversation
When the event_timestamp column in BigQuery is a DATE type, the generated SQL wraps comparison values in TIMESTAMP(), causing a type mismatch error. This adds a timestamp_field_type parameter to BigQuerySource that, when set to "DATE", generates DATE() comparisons instead. Closes feast-dev#2530 (part 2) Signed-off-by: Jonathan Wrede <wrede.jonathan00@gmail.com>
Jwrede
requested review from
HaoXuAI,
nquinn408 and
shuchu
and removed request for
a team
The proto files were regenerated with protobuf 6.31.1 / grpcio-tools 1.80.0, which imports runtime_version -- a module that does not exist in protobuf 4.25.x used by the project. Revert generated code to 4.25.1 format while keeping the new timestamp_field_type field. Signed-off-by: Jonathan Wrede <wrede.jonathan00@gmail.com>
Mypy infers str from the ternary expression; annotate with the exact Literal union so the call to get_timestamp_filter_sql passes type checking. Signed-off-by: Jonathan Wrede <wrede.jonathan00@gmail.com>
franciscojavierarceo
approved these changes
May 3, 2026
…text Callers that do not use DATE-typed timestamp fields (e.g. Spark offline store tests) should not be forced to pass timestamp_field_type. Adding a default keeps the new field backward-compatible. Signed-off-by: Jonathan Wrede <wrede.jonathan00@gmail.com>
…feat/bq-date-timestamp-type
A default value on timestamp_field_type breaks the SparkFeatureViewQueryContext subclass because its non-default fields (min_date_partition, max_date_partition) would follow a field with a default. Instead, keep it required and update the Spark test to pass it. Signed-off-by: Jonathan Wrede <wrede.jonathan00@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
When the
event_timestampcolumn in BigQuery is aDATEtype (notTIMESTAMP), the generated SQL wraps comparison values inTIMESTAMP(), causing a type mismatch error. This makes DATE-partitioned summary tables unusable without creating views or duplicate tables.This PR adds an optional
timestamp_field_typeparameter toBigQuerySource. When set to"DATE", SQL generation usesDATE('YYYY-MM-DD')comparisons instead ofTIMESTAMP('...'), both in direct queries (pull_latest_from_table_or_query,pull_all_from_table_or_query) and in the point-in-time join Jinja template.Usage:
Changes:
timestamp_field_typestring field (field 28) toDataSourceDataSourcebase class: addtimestamp_field_typeattribute, equality check, and__init__parameterBigQuerySource: wiretimestamp_field_typethrough__init__,from_proto, and_to_proto_implget_timestamp_filter_sql(): add"date_func"cast style that generatesDATE('YYYY-MM-DD')BigQueryOfflineStore: select cast style based ontimestamp_field_typeDATE()comparisons for DATE-type timestamp fieldsFeatureViewQueryContext: propagatetimestamp_field_typeto template contextBackward-compatible: when
timestamp_field_typeis unset, behavior is unchanged.Which issue(s) this PR fixes:
Fixes #2530 (part 2 -- DATE type event_timestamp support; part 1 was addressed by #6076)
How to test:
4 new tests added:
test_pull_latest_date_type_timestamp_field-- verifiesDATE()cast in pull_latesttest_pull_all_date_type_timestamp_field-- verifiesDATE()cast in pull_alltest_pull_latest_date_type_with_partition_column-- DATE type combined with partition pruningtest_bigquery_source_date_type_proto_roundtrip-- proto serialization roundtrip