Push partition-eligible filters into parent CTEs#2163
Merged
Conversation
✅ Deploy Preview for thriving-cassata-78ae72 canceled.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related issues in v3 measures SQL generation cause filters to be applied at the wrong scope.
Example 1: Set-op CTE bodies skipped
A transform
events_unifiedwith aUNION ALLbody:Requesting a metric on
events_unifiedwith filterevents_unified.status = 'completed'used to produce:Both source tables (
orders_liveandorders_archive) get fully scanned because the pushdown bails on set-op bodies on either side of the union all.Example 2: Filter silently dropped
A transform
events_windowedthat exposes a column via an inlineCROSS JOINin its source SQL:Requesting filter
window_dim.window_size = '7d'would silently drop the filter. The blanketparent_pushdown_activeflag assumed the filter had been pushed intoevents_windowed, so the outerWHEREcopy was dropped, but the actual pushdown failed because the column lives inside an opaque source body.This results in no
window_sizefilter anywhere in the SQL.Fix
For set-op CTEs, we do a per-arm pushdown. When a CTE body is a
UNION/INTERSECT/EXCEPTchain,_resolve_pushdown_filters_for_ctenow walks every arm and rewrites the filter against each arm's projection. This is atomic per filter, so if any arm can't accept the rewrite, the filter is skipped entirely for that CTE.Example 1 above now produces:
We also do per-filter pushdown tracking, so the silent-drop case is gone. The filter that wasn't successfully consumed by the parent CTE stays in the outer WHERE clause, and
_apply_outer_where_atomsroutes it throughinject_filter_into_select, which handles the outer-join safety.Test Plan
make checkpassesmake testshows 100% unit test coverageDeployment Plan