Add the new readahead_v2 cache

[googlyrahman]

Adds the new readahead_v2 cache specifically for GCSFS.

The Problem: The existing readahead cache minimizes network connections by over-fetching (e.g., fetching 10MB when 5MB is requested). However, it stores this data as a single, contiguous bytes object. Serving a read request requires slicing this object, which in Python triggers a memory copy. For large block sizes (e.g., 100MB or 500MB), this copying operation becomes CPU-intensive and blocks the asyncio event loop, degrading performance.

The Solution: The new readahead_v2 cache implementation stores fetched chunks as separate bytes objects rather than concatenating them. When a read request matches a stored chunk, the cache returns a direct reference to the object. This "zero-copy" approach eliminates the overhead of slicing and prevents CPU/event loop blocking, ensuring efficient memory usage when the request size aligns with the block size.

The cache is currently being integrated into the system only when EXPERIMENTAL environment variable is set, so basically as an opt-in feature. We plan to make it default in subsequent release.

NOTE: Although we could optimize memory usage further, we chose to maintain the exact semantics of the original readahead cache. This ensures stability for unlikely but valid read patterns.

[martindurant]

There is clearly some thought and effort going into this implementation. Might I suggest that we try to keep to memoryview objects instead of bytes? They certainly allow zero-copy referencing and slicing. Unfortunately, the fsspec APIs generally return bytes in all cases, so I wonder whether we can make a memoryview-backed bytes-like object.

This is connected to the effort in Jamie-Chang/aiointerpreters#8 to bring true parallelism to fsspec: memoryview objects are zero-cost passing between interpreters, the only python object this applies to.

[googlyrahman]

@martindurant, Thanks for taking an early look, and leaving your view.

I actually did consider a memoryview solution before arriving at this one, but my understanding is that it isn't performance-improving in this context. The main reason is exactly what you noted: "Unfortunately, the fsspec APIs generally return bytes in all cases." That is the core issue. We receive data in bytes from aiohttp (it allocates bytes internally even if we pass a buffer). We could convert that to a memoryview without hitting the CPU, but the moment we serve read calls to customers, we can't return the memoryview; we have to return bytes to the user to maintain backward compatibility and API contract, Doing so (memoryview.to_bytes()) creates a fresh copy, which hits the CPU and hampers performance.

This particular solution avoids creating copies and blocking the CPU. It stores those 5MB aiohttp responses as-is and serves the user a reference rather than creating a new copy. This works because Python bytes are immutable, so even if we pass the reference, the user cannot change it.

As I understand it, to use memoryview effectively, the fsspec API contract would need to change, which likely isn't feasible at the moment or maybe a long term effort where users would need to migrate their workload from bytes to memoryview, This solution maintains the backward compatibility, and improves performance. With that said, please let me know if I missed anything or if there is a different way to approach this.

[googlyrahman]

/gcbrun

[googlyrahman]

/gcbrun

[googlyrahman]

/gcbrun

[ankitaluthra1]

/gcbrun

[martindurant]

Yeah, I did some reading around and indeed I see no way to ingest memoryviews directly out of aiohttp (or any other network client). This seems like a mistake to me! Maybe someone should implement the buffers in rust to give true zero-copy behaviour...

[googlyrahman]

Yeah that's correct, but that's only half of the problem. Even if we hypothetically achieved zero-copy ingestion using memoryview internally, My understanding says it would still be less efficient for a cache under the current API constraints. Since fsspec mandates returning bytes, a memoryview approach would force us to run .tobytes() on every cache hit to satisfy the contract. This means we would pay the CPU copy 'tax' repeatedly every time the user reads the same chunk, or there is cache hit.

By contrast, the readahead_v2 approach leverages the immutability of bytes. We pay the allocation tax exactly once (during the initial download). All subsequent reads whether immediate or from cache are just reference passing, which is truly zero-copy and zero-CPU. This also avoids a memory spike: with memoryview, serving a read would temporarily require 2x RAM (the internal buffer + the new bytes copy). With the current approach, the cache and the user share the exact same memory object.

To completely solve the problem for any request size, we would need two things:

• Support for reading directly into a memoryview in aiohttp (or the network client) with zero copy. (Current not exists)
• An update to the fsspec contract to return memoryview (which would be a breaking or backward incompatible change)

Until then, the proposed solution optimizes performance specifically for cases where request_size == block_size.

Let me know if you have any questions, or if you have a better approach to this I would love to hear it!

[Jamie-Chang]

Not particularly useful for you right now, but It might interest you that in Python 3.15 there will be a zero copy way to get bytes out of a bytearray https://docs.python.org/3.15/library/stdtypes.html#bytearray.take_bytes

[googlyrahman]

Addressed comments.

[googlyrahman]

Added the before vs after comparision as well in the description for regional as well as zonal. Please take a look once you've time :)

[ankitaluthra1]

/gcbrun

[ankitaluthra1]

/gcbrun

[ankitaluthra1]

/gcbrun

[ankitaluthra1]

/gcbrun

[googlyrahman]

Had to rebase, and force-push it as it was having merge conflicts with fsspec/main

[ankitaluthra1]

/gcbrun