Async avro reader optimization

[ariel-miculas]

Note we have also been trying to separate the IO from the decoding in Parquet -- see https://docs.rs/parquet/58.0.0/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html

Perhaps we could move Avro to that model too rather than implementing the async stuff first

Originally posted by @alamb in #9632 (comment)

Issue:
The async avro reader reads all the data upfront, even though the avro file format is serial, thus decoding and data fetching could happen in parallel (like datafusion's json scan, for example).

One potential solution: use an async stream, as presented in #9632

[ariel-miculas]

CC: @EmilyMatt @mzabaluev

[mzabaluev]

Note we have also been trying to separate the IO from the decoding in Parquet -- see https://docs.rs/parquet/58.0.0/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html

Perhaps we could move Avro to that model too rather than implementing the async stuff first

There is already a push decoder implementation shared by the sync and async reader implementations. The differences and duplication mainly arise from the I/O model: for the sync reader it's stictly sequential, while the async reader is designed around ranged GET requests provided by object_store, with an abstraction to also support generic AsyncRead + AsyncSeek.

[alamb]

Shall we close this ticket then?

[mzabaluev]

I don't know if the description is insufficient, but #9632 introduces an optimization: rather than reading the entire requested file range in memory at once and then parsing the file, the reader can request a stream and parse one chunk at a time as the file reader yields them.

[ariel-miculas]

rather than reading the entire requested file range in memory at once

@alamb Note that the above is also a bug in itself: if there's not enough memory to read the entire file range, the existing AsyncFileReader will crash. So #9632 is not only about performance optimization, but it also fixes an important issue.

[ariel-miculas]

@alamb any concerns about the proposed changes or about the underlying issue that we're trying to solve?