APP-1086: fix BigQuery nanosecond timestamp cast failure in edr_cast_as_timestamp#1003
APP-1086: fix BigQuery nanosecond timestamp cast failure in edr_cast_as_timestamp#1003MikaKerman wants to merge 1 commit intomasterfrom
Conversation
…cond precision Co-Authored-By: mika@elementary-data.com <mika.kerman@gmail.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
👋 @MikaKerman |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughA new BigQuery-specific macro ChangesTimestamp Casting Macro
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Summary
Adds a BigQuery-specific override for
edr_cast_as_timestampthat truncates sub-microsecond fractional digits before casting toTIMESTAMP.BigQuery's
TIMESTAMPonly supports microsecond precision (6 fractional digits). Some runtimes (e.g. dbt-fusion) write nanosecond-precision strings like2026-04-03T10:50:50.961498756Zinto Elementary'sdbt_run_resultscolumns (execute_started_at,execute_completed_at,compile_started_at,compile_completed_at). The defaultcast(field as timestamp)then fails withDatabase Error: Invalid timestamp: '...'and blocksedr send-report/edr reportuntil the bad rows age out.The new
bigquery__edr_cast_as_timestampmacro wraps the value inregexp_replace(..., r'(\.\d{6})\d+', r'\1')before the cast, truncating any extra fractional digits to exactly 6. Strings with ≤ 6 fractional digits are unaffected. This follows the same dispatch pattern as the existingdremio__edr_cast_as_timestamptruncation.bigquery__edr_cast_as_datechains throughedr_cast_as_timestamp, so the fix applies there transparently as well.Linear: APP-1086
Review & Testing Checklist for Human
dbt_run_resultswith at least one row whereexecute_completed_athas 9 fractional digits (e.g.2026-04-03T10:50:50.961498756Z) and confirmedr send-report/edr reportno longer fails on thedays_backfilter.edr_cast_as_timestamp.Notes
try_castwhich handles this; Dremio already truncates to millisecond precision.Link to Devin session: https://app.devin.ai/sessions/3a3eac19b8924571820addc7d6b6a877
Requested by: @MikaKerman
Summary by CodeRabbit