fix(tesseract): multi_stage reduce_by no-ops on type:time dimensions#10855
Open
tlangton3 wants to merge 3 commits into
Open
fix(tesseract): multi_stage reduce_by no-ops on type:time dimensions#10855tlangton3 wants to merge 3 commits into
tlangton3 wants to merge 3 commits into
Conversation
`multi_stage: true` measures (rank, window aggregates) configured with `reduce_by: [<time_dim>]` silently produced wrong results when the consumer query specified a granularity (e.g. `granularity: day`). The projected dim resolved as a TimeDimensionSymbol whose `full_name` carried a granularity suffix, while the reduce_by entry resolved to a base DimensionSymbol with bare name — the string-equality compare in `has_member_in_reference_chain` failed, the time dim stayed in PARTITION BY, and every row collapsed to a one-row partition with rank() = 1. Fix: unwrap any TimeDimensionSymbol to its base DimensionSymbol on both sides of the comparison in `member_partition_by_logical`. Symmetric unwrapping is required because `group_by` entries can themselves be TimeDimensionSymbols (e.g. `group_by: [<time_dim>.month]`); unwrapping only the projected dim would break the symmetric-granularity match and silently drop the time dim from PARTITION BY in cases like `test_group_by_with_time`. Adds a regression test (`test_reduce_by_time_dim`) using the existing multi-stage fixture: a sum measure with `reduce_by: [orders.created_at]` queried with `granularity: month`. Without the fix the snapshot shows per-(status, month) sums; with the fix every row for a given status shows the same per-status grand total across all months. Fixes cube-js#10854.
Replace member_partition_by_logical's separate reduce_by/group_by matching with a single matches_partition_member helper that branches on whether the measure-level entry (m) carries an explicit granularity: - m has explicit granularity (e.g. <time_dim>.quarter): match strictly on full_name. group_by: [<time_dim>.quarter] keeps only the _quarter projection in PARTITION BY; reduce_by: [<time_dim>.month] excludes only the _month projection. - m is bare (plain DimensionSymbol, or TimeDimensionSymbol with granularity = None): unwrap both sides to base. reduce_by: [<time_dim>] excludes the time dim regardless of the projection's granularity suffix — the original cube-js#10854 case. This replaces the symmetric base_of() unwrap on the group_by site, which over-matched when a query projected multiple granularities of the same time dim and degenerated window aggregates to per-row sums. CI surfaced the regression against BigECommerce's "multi-stage group by time dimension" test (both postgres-driver and mssql-driver runs). Adds test_group_by_time_dim_multiple_granularities mirroring the BigECommerce shape against cubesqlplanner's testcontainers Postgres harness — without the fix, sums collapse per-(month, week); with the fix, they correctly aggregate per-month repeated across weeks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
multi_stage: truemeasures (rank, window aggregates) configured withreduce_by: [<time_dim>]silently produce wrong results when the consumer query specifies a granularity — every row reportsrank() = 1(or, for sums, the per-row value rather than the cross-time aggregate). Because the emitted SQL is well-formed, the bug bypasses validation and surfaces only as broken BI dashboards (a "rank by gross profit across all days for this district" tile showing every district at rank 1).The workaround is to query the time dimension without an explicit granularity — sidestepping the suffix mismatch entirely. That constrains every consumer to query at the raw timestamp grain, eliminating natural day/week/month groupings on the time dim that's being reduced by, which is a real loss of expressiveness.
Root cause is a string-equality mismatch in
has_member_in_reference_chain: the projected time dim resolves as aTimeDimensionSymbolwhosefull_namecarries a granularity suffix (_day/_month/etc., defaulting to_daywhen no granularity is set — seeTimeDimensionSymbol::new), while thereduce_byentry resolves to a baseDimensionSymbolwith the bare name. The compare never matched, so the time dim was never excluded fromPARTITION BYand the window collapsed to one-row partitions.Originally surfaced against BigQuery in production; the fix is backend-agnostic (lives in the Tesseract planner before SQL is emitted) and the included regression tests run against the existing testcontainers Postgres harness. Fixes #10854.
Changes
rust/cube/cubesqlplanner/cubesqlplanner/src/planner/planners/multi_stage/member_query_planner.rs: introduce a singlematches_partition_member(d, m)helper used at both thereduce_byandgroup_byfilter sites inmember_partition_by_logical. The helper branches on whethermcarries an explicit granularity (see Implementation Details).rust/cube/cubesqlplanner/cubesqlplanner/src/test_fixtures/schemas/yaml_files/common/integration_multi_stage.yaml: add anamount_reduce_created_atmeasure (sum,reduce_by: [orders.created_at]) to the existingorderscube — fixture for the canonical regression test.rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/multi_stage/reduce_by.rs(+ snapshot): addtest_reduce_by_time_dimexercising the new measure withgranularity: month.rust/cube/cubesqlplanner/cubesqlplanner/src/tests/integration/multi_stage/group_by.rs(+ snapshot): addtest_group_by_time_dim_multiple_granularitiesmirroring the BigECommerce multi-stage shape (group_by single granularity, query projecting multiple granularities of the same time dim).Implementation Details
The two operators have semantically distinct intents that depend on the granularity of the
reduce_by/group_byentry — not on which operator it is.matches_partition_member(d, m)captures this with one rule, applied uniformly at both call sites:mcarries an explicit granularity (resolves to aTimeDimensionSymbolwithhas_granularity() == true— i.e. the user wrote<time_dim>.month/.quarter/ etc.), match strictly onfull_name. Agroup_by: [<time_dim>.quarter]matches only the_quarterprojection, not sibling granularities. Symmetrically, areduce_by: [<time_dim>.month]excludes only the_monthprojection.mis bare (plainDimensionSymbolfromreduce_by: [<time_dim>], orTimeDimensionSymbolwithgranularity = None), unwrap both sides to their base symbol and compare. This is the fix for the granularity-suffix mismatch from Tesseract: multi-stage reduce_by silently no-ops against type:time dimensions #10854: barereduce_by: [<time_dim>]matches and excludes any granularity of the projection.TimeDimensionSymbol::full_namecarries a granularity suffix that defaults to"_day"when no granularity is supplied (seeTimeDimensionSymbol::new); without the base unwrap, the bare entry'sfull_namecould never match a granularity-suffixed projection.An earlier draft of this patch applied symmetric
base_of(...)unwrap to both call sites unconditionally. That was too permissive on thegroup_byside: when a query projected multiple granularities of the same time dim, both projections unwrapped to the same base, both matched the explicit-granularitygroup_byentry, both stayed inPARTITION BY, and the window aggregate degenerated to per-row sums. CI caught this against BigECommerce'smulti-stage group by time dimensiontest in both postgres-driver and mssql-driver runs.Testing
cargo test --features integration-postgres --lib 'tests::integration'→ 421 passed, 0 failed (testcontainers Postgres).cargo test --lib -p cubesqlplanner→ 901 passed, 0 failed.cargo fmt -p cubesqlplanner -- --check→ clean.test_reduce_by_time_dim(new) — barereduce_by: [created_at]+granularity: month. Verified to fail (per-(status, month) sums) without the fix, passes (cross-month per-status totals: completed=1400, pending=650, cancelled=200) with the fix.test_group_by_time_dim_multiple_granularities(new) —group_by: [status, created_at.month]+ query projecting bothmonthandweek. Verified to fail (per-(month, week) sums) under the symmetric-unwrap-on-group_by version, passes (per-month sums 500/750/1000 repeated across weeks) under the unified helper.test_group_by_with_time(single-granularity projection) stays green.