Skip to content

APP-1086: fix BigQuery nanosecond timestamp cast failure in edr_cast_as_timestamp#1003

Open
MikaKerman wants to merge 1 commit intomasterfrom
app-1086-fix-bigquery-nanosecond-timestamp-cast-failure-in
Open

APP-1086: fix BigQuery nanosecond timestamp cast failure in edr_cast_as_timestamp#1003
MikaKerman wants to merge 1 commit intomasterfrom
app-1086-fix-bigquery-nanosecond-timestamp-cast-failure-in

Conversation

@MikaKerman
Copy link
Copy Markdown
Contributor

@MikaKerman MikaKerman commented May 6, 2026

Summary

Adds a BigQuery-specific override for edr_cast_as_timestamp that truncates sub-microsecond fractional digits before casting to TIMESTAMP.

BigQuery's TIMESTAMP only supports microsecond precision (6 fractional digits). Some runtimes (e.g. dbt-fusion) write nanosecond-precision strings like 2026-04-03T10:50:50.961498756Z into Elementary's dbt_run_results columns (execute_started_at, execute_completed_at, compile_started_at, compile_completed_at). The default cast(field as timestamp) then fails with Database Error: Invalid timestamp: '...' and blocks edr send-report / edr report until the bad rows age out.

The new bigquery__edr_cast_as_timestamp macro wraps the value in regexp_replace(..., r'(\.\d{6})\d+', r'\1') before the cast, truncating any extra fractional digits to exactly 6. Strings with ≤ 6 fractional digits are unaffected. This follows the same dispatch pattern as the existing dremio__edr_cast_as_timestamp truncation. bigquery__edr_cast_as_date chains through edr_cast_as_timestamp, so the fix applies there transparently as well.

Linear: APP-1086

Review & Testing Checklist for Human

  • On a BigQuery target, seed dbt_run_results with at least one row where execute_completed_at has 9 fractional digits (e.g. 2026-04-03T10:50:50.961498756Z) and confirm edr send-report / edr report no longer fails on the days_back filter.
  • Confirm existing rows with ≤ 6 fractional digits (or no fractional part) still cast correctly and produce identical results to before this change.
  • Run BigQuery integration tests to verify no regression in anomaly detection / monitoring queries that go through edr_cast_as_timestamp.

Notes

  • No version bump or release work is included here per the ticket assignee.
  • Postgres/Snowflake/Databricks/Spark are unaffected (native nanosecond support or string parsing); Athena/Trino already use try_cast which handles this; Dremio already truncates to millisecond precision.

Link to Devin session: https://app.devin.ai/sessions/3a3eac19b8924571820addc7d6b6a877
Requested by: @MikaKerman

Summary by CodeRabbit

  • New Features
    • Improved timestamp precision handling for BigQuery, with enhanced support for truncating sub-microsecond digits before casting to ensure proper data integrity.

…cond precision

Co-Authored-By: mika@elementary-data.com <mika.kerman@gmail.com>
@devin-ai-integration
Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@linear
Copy link
Copy Markdown

linear Bot commented May 6, 2026

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 6, 2026

👋 @MikaKerman
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in the elementary repository.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 6, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 56ef4698-c008-4fb7-9c6f-f1d47781564e

📥 Commits

Reviewing files that changed from the base of the PR and between 91d04fd and 6b65aa5.

📒 Files selected for processing (1)
  • macros/utils/data_types/cast_column.sql

📝 Walkthrough

Walkthrough

A new BigQuery-specific macro bigquery__edr_cast_as_timestamp is added to truncate sub-microsecond digits from timestamps before casting to the standard timestamp type via string manipulation and regex replacement.

Changes

Timestamp Casting Macro

Layer / File(s) Summary
BigQuery Timestamp Precision
macros/utils/data_types/cast_column.sql
New macro bigquery__edr_cast_as_timestamp(timestamp_field) casts timestamp to string, truncates sub-microsecond precision using regex, and casts back to internal timestamp type to handle BigQuery timestamp precision requirements.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

🐰 A macro hops in, swift and clean,
BigQuery timestamps, now pristine!
Sub-microseconds trimmed with care,
Regex magic through the air! ✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly describes the main change: adding a BigQuery-specific fix for nanosecond timestamp casting in the edr_cast_as_timestamp macro, which is the primary objective of this changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch app-1086-fix-bigquery-nanosecond-timestamp-cast-failure-in

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant