Skip to content

Add Parquet caching for DataFusion metrics scans#6389

Draft
alexanderbianchi wants to merge 2 commits intoquickwit-oss:mainfrom
alexanderbianchi:bianchi/parquet-byte-range-cache
Draft

Add Parquet caching for DataFusion metrics scans#6389
alexanderbianchi wants to merge 2 commits intoquickwit-oss:mainfrom
alexanderbianchi:bianchi/parquet-byte-range-cache

Conversation

@alexanderbianchi
Copy link
Copy Markdown
Collaborator

Summary

Adds Parquet IO caching for DataFusion metrics scans.

This PR includes the footer metadata cache work from #6388 and the follow-up DataFusion-local parquet byte-range cache:

  • installs DataFusion's CachedParquetFileReaderFactory for parquet footer metadata reuse;
  • adds a DataFusion object-store cache path backed by Quickwit's in-memory MemorySizedCache;
  • scopes cache keys by storage URI so equivalent object paths in different buckets/authorities do not collide;
  • keeps DataFusion cache sizing behind the datafusion serve setup with a 4 GiB default and QW_DATAFUSION_PARQUET_RANGE_CACHE_CAPACITY override;
  • documents the path to a future Foyer-backed block/range cache in docs/internals/parquet-byte-range-cache-option2.md.

Validation

  • cargo check -p quickwit-serve --features datafusion
  • cargo test -p quickwit-datafusion
  • cargo test -p quickwit-storage --features testsuite
  • cargo check -p quickwit-lambda-server
  • git diff --check

@alexanderbianchi alexanderbianchi force-pushed the bianchi/parquet-byte-range-cache branch from 993f854 to 3b85d88 Compare May 6, 2026 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant