Skip to content

fix(spark): spark sql union recursion#466

Open
ChaconneLuo wants to merge 1 commit intoDTStack:mainfrom
ChaconneLuo:bugfix/spark_sql_union_recursion
Open

fix(spark): spark sql union recursion#466
ChaconneLuo wants to merge 1 commit intoDTStack:mainfrom
ChaconneLuo:bugfix/spark_sql_union_recursion

Conversation

@ChaconneLuo
Copy link

现象:在spark sql中,如果在with中遇到大量union all,queryTerm中会由于包含存在重复的规则,预测路径出现指数级增长,导致validate卡死。

复现SQL:

WITH t1 AS (
    SELECT 1
    UNION ALL SELECT 2
    UNION ALL SELECT 3
    UNION ALL SELECT 4
    UNION ALL SELECT 5
    UNION ALL SELECT 6
    UNION ALL SELECT 7
    UNION ALL SELECT 8
    UNION ALL SELECT 9
    UNION ALL SELECT 10
    UNION ALL SELECT 11
    UNION ALL SELECT 12
    UNION ALL SELECT 13
    UNION ALL SELECT 14
    UNION ALL SELECT 15
    UNION ALL SELECT 16
    UNION ALL SELECT 17
    UNION ALL SELECT 18
    UNION ALL SELECT 19
    UNION ALL SELECT 20
    UNION ALL SELECT 21
    UNION ALL SELECT 22
    UNION ALL SELECT 23
    UNION ALL SELECT 24
    UNION ALL SELECT 25
    UNION ALL SELECT 26
    UNION ALL SELECT 27
	UNION ALL SELECT 28
	UNION ALL SELECT 29
	UNION ALL SELECT 30
), //  extra  comma
SELECT * FROM t1

单测结果:
image

@ChaconneLuo ChaconneLuo force-pushed the bugfix/spark_sql_union_recursion branch from c9187a7 to a98e4d9 Compare March 18, 2026 08:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant