Skip to content

Add mixed pattern in gcsfs microbenchmarks#822

Open
googlyrahman wants to merge 1 commit into
fsspec:mainfrom
ankitaluthra1:prob-read
Open

Add mixed pattern in gcsfs microbenchmarks#822
googlyrahman wants to merge 1 commit into
fsspec:mainfrom
ankitaluthra1:prob-read

Conversation

@googlyrahman
Copy link
Copy Markdown
Collaborator

@googlyrahman googlyrahman commented Apr 23, 2026

The read benchmarks in our microbenchmarks are either (100% sequential 0% seek) or (0% sequential 100% seek) with fixed io size. These are often not correct representation of real world scenarios.

Real world scenarios are mixed of sequential, and seeks with a variable of io size. The following PR add this feature in microbenchmarks to correctly measure the typical real world performance.

Sample output after this PR

| Bucket Type | Group | Pattern | Seq Prob | Files | Folders | Threads | Processes | Depth | Target Type | File Size (MiB) | Chunk Size (MiB) | Min Chunk (MiB) | Max Chunk (MiB) | Block Size (MiB) | Mean Latency (s) | Mean Throughput (MiB/s) | Max CPU (%) | Max Memory (MiB) |
+-------------+-------+---------+----------+-------+---------+---------+-----------+-------+-------------+-----------------+------------------+-----------------+-----------------+------------------+------------------+-------------------------+-------------+------------------+
|   regional  |  read |   seq   |   N/A    |   1   |   N/A   |    1    |     1     |  N/A  |     N/A     |     1024.00     |      16.00       |       N/A       |       N/A       |       5.00       |      30.00       |          143.47         |     0.47    |      224.88      |
|   regional  |  read |  mixed  |   0.3    |   1   |   N/A   |    1    |     1     |  N/A  |     N/A     |     1024.00     |       N/A        |       0.06      |     1024.00     |       5.00       |      30.00       |          328.90         |     1.04    |     2868.44      |
|   regional  |  read |  mixed  |   0.5    |   1   |   N/A   |    1    |     1     |  N/A  |     N/A     |     1024.00     |       N/A        |       0.06      |     1024.00     |       5.00       |      30.00       |          249.38         |     1.14    |     2934.86      |
+-------------+-------+---------+----------+-------+---------+---------+-----------+-------+-------------+-----------------+------------------+-----------------+-----------------+------------------+------------------+-------------------------+-------------+------------------+

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 23, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.43%. Comparing base (5596df9) to head (e5386e8).
⚠️ Report is 12 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #822      +/-   ##
==========================================
+ Coverage   79.55%   81.43%   +1.88%     
==========================================
  Files          16       16              
  Lines        3042     3076      +34     
==========================================
+ Hits         2420     2505      +85     
+ Misses        622      571      -51     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@googlyrahman googlyrahman force-pushed the prob-read branch 3 times, most recently from 1419abf to 10676f9 Compare April 23, 2026 07:39
@googlyrahman googlyrahman marked this pull request as ready for review April 23, 2026 08:16
@googlyrahman googlyrahman requested a review from jasha26 April 23, 2026 08:16
Comment thread gcsfs/tests/perf/microbenchmarks/read/test_read.py Outdated
@ankitaluthra1
Copy link
Copy Markdown
Collaborator

Can you please fix the test failure so its ready for merge

@googlyrahman
Copy link
Copy Markdown
Collaborator Author

Aah this needs to rebased with main to fix the failure, let me do it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants