Skip to content

[Bug] Query Paimon Catalog Error #60112

@tcsll

Description

@tcsll

doris 版本: 3.1.1
背景:
我通过doris catalog 查询 paimon 的数据偶发报错:

java.lang.RuntimeException: The specified scan snapshotId 669 is out of available snapshotId range [709, 813].
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:289) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.splitFragments(NereidsPlanner.java:453) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.distribute(NereidsPlanner.java:539) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.lambda$plan$0(NereidsPlanner.java:153) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.planWithLock(NereidsPlanner.java:228) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:149) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:856) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:631) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:593) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:578) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:371) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:252) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:229) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:257) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:437) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.lang.IllegalArgumentException: The specified scan snapshotId 669 is out of available snapshotId range [709, 813].
        at org.apache.paimon.utils.Preconditions.checkArgument(Preconditions.java:149) ~[paimon-common-1.1.1.jar:1.1.1]
        at org.apache.paimon.table.source.snapshot.StaticFromSnapshotStartingScanner.configure(StaticFromSnapshotStartingScanner.java:59) ~[paimon-core-1.1.1.jar:1.1.1]
        at org.apache.paimon.table.source.snapshot.ReadPlanStartingScanner.scan(ReadPlanStartingScanner.java:41) ~[paimon-core-1.1.1.jar:1.1.1]
        at org.apache.paimon.table.source.DataTableBatchScan.plan(DataTableBatchScan.java:77) ~[paimon-core-1.1.1.jar:1.1.1]
        at org.apache.doris.datasource.paimon.source.PaimonScanNode.getPaimonSplitFromAPI(PaimonScanNode.java:409) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.paimon.source.PaimonScanNode.getSplits(PaimonScanNode.java:274) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.createScanRangeLocations(FileQueryScanNode.java:374) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.doFinalize(FileQueryScanNode.java:216) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.FileQueryScanNode.finalizeForNereids(FileQueryScanNode.java:207) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:287) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 18 more

然后再次查询就不会出现了,是因为当前的session缓存了paimon的snapshotId,然后paimon本身做了snapshotId的合并演进,然后会出现这种查询报错snapshotId不在范围内的情况吗?

我看3.1.2 版本有说了改进 Paimon 时间旅行(Time Travel)功能,是这块修复的内容吗?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions