What is the problem the feature request solves?
When Comet has a native ShuffleWriter and a native child plan, batches get created in native code, then imported into JVM, then exported back to native. The JVM side does not need to access the data so the import/export is unnecessary overhead.
I would like to explore a simpler approach where a simple pointer (for the whole batch) is passed back and forth without any additional processing.
Describe the potential solution
No response
Additional context
No response
What is the problem the feature request solves?
When Comet has a native ShuffleWriter and a native child plan, batches get created in native code, then imported into JVM, then exported back to native. The JVM side does not need to access the data so the import/export is unnecessary overhead.
I would like to explore a simpler approach where a simple pointer (for the whole batch) is passed back and forth without any additional processing.
Describe the potential solution
No response
Additional context
No response