Scan performance regression after DataFusion 53

[comphead]

Describe the bug

There is observable performance degradation for native_datafusion scan operations after #3574.
It was bunch of crates upgraded datafusion, arrow, parquet, opendal, object_store that directly involved into scan.

Currently I can see degradation on HDFS only, the scan task for typical test went from 3 minutes per task to 5 minutes

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[0lai0]

take

[0lai0]

I found that get_ranges in HDFS ObjectStore implementation was reading all byte ranges sequentially inside a single blocking task (one open, then a loop of read_range calls). Parquet tends to issue many scattered ranges per file, so that pattern stacks HDFS latency.

[comphead]

I found that get_ranges in HDFS ObjectStore implementation was reading all byte ranges sequentially inside a single blocking task (one open, then a loop of read_range calls). Parquet tends to issue many scattered ranges per file, so that pattern stacks HDFS latency.

hey @0lai0 that was also my suspicion, I'm not sure if we can do a fix in Comet(without copying lots of code ), rather create a PR in opendal and refer to specific commit. I'm planning to look into this on Monday, but appreciate if you can find a solution faster