Fix: MemTable LIMIT ignored with reordered projections#21177
Fix: MemTable LIMIT ignored with reordered projections#21177RamakrishnaChilaka wants to merge 1 commit intoapache:mainfrom
Conversation
neilconway
left a comment
There was a problem hiding this comment.
Thanks for this fix!
This fix is fine, although I find the overall code structure here pretty fragile. At present, we will lose sort_information and show_sizes in this scenario (not sure offhand if that is problematic today), and we could potentially lose more metadata in the future.
699092a to
95613af
Compare
95613af to
c63fe9d
Compare
Thank you for the review @neilconway! I've updated the approach to clone self and only mutate the projection-related fields, so all metadata (fetch, sort_information, show_sizes, and any future fields) is preserved automatically. Sort information is re-projected via project_orderings() to remap column indices to the new schema (the old code was dropping it entirely via try_new()). This should make it future-proof against new fields being added to MemorySourceConfig. Please let me know your thoughts on this approach. |
Which issue does this PR close?
Rationale for this change
When
SELECTprojects columns in a different order than the table schema (e.g.,SELECT col_c, col_bvs schema ordercol_a, col_b, col_c), theLIMITclause is silently ignored — all rows are returned instead of the requested limit.The physical optimizer runs
LimitPushdown(which correctly setsfetch=NonMemorySourceConfig) beforeProjectionPushdown. When the projection is pushed down,try_swapping_with_projection()creates a newMemorySourceConfigviatry_new(), which resetsfetchtoNone, silently dropping the limit.What changes are included in this PR?
One-line fix in
MemorySourceConfig::try_swapping_with_projectionto preserveself.fetchon the newly created config:Are these changes tested?
Yes.
try_swapping_with_projection_preserves_fetch): directly verifiesfetchis preserved after projection pushdown. Confirmed it fails without the fix (NonevsSome(5)).limit.slt, tablet21176): end-to-end regression test forSELECT col_c, col_b FROM t LIMIT 5with reverse column order.Are there any user-facing changes?
No API changes. This is a correctness fix —
LIMITnow works correctly regardless ofSELECTcolumn order against in-memory tables.