Read trades from derived table#2589
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
How to use the Graphite Merge QueueAdd the label Raindex-queue to this PR to add it to the merge queue. You must have a Graphite account in order to use the merge queue. Sign up using this link. An organization admin has enabled the Graphite Merge Queue in this repository. Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue. This stack of pull requests is managed by Graphite. Learn more about stacking. |
3124557 to
055e02d
Compare
1abc55d to
aac028c
Compare
|
Folded into #2587 so the derived-trades local DB schema generation lands atomically. |
## Motivation Trade-list reads should use a pipeline-maintained local DB read model instead of reconstructing trades from source event tables on every request. This keeps producer bootstrap DBs and client incremental syncs on the same schema generation and avoids landing partial local DB protocol changes on `main`. ## Solution - Add `derived_trades` as the derived/read-model table for local DB trades. - Maintain `derived_trades` in the shared local DB apply pipeline in the same transaction as source event/state writes. - Rebuild derived rows by sync window while allowing order resolution to look back to older active `AddOrderV3` events. - Include the table in required-table checks, clear/reset paths, bootstrap export/import, and schema versioning. - Update trade local DB read/count queries to read from `derived_trades` instead of reconstructing from event tables. - Cover maintenance, incremental sync, bootstrap/export, clear/reset, and fetch equivalence with tests. ## Review note This was folded from the previous split stack (#2588, #2589, #2590) so the local DB schema generation lands atomically. ## Checks - `COMMIT_SHA=local nix develop -c cargo test -p raindex_app_settings --lib` - `COMMIT_SHA=local nix develop -c cargo test -p raindex_common local_db::query --lib` - `COMMIT_SHA=local nix develop -c cargo fmt --all` - `git diff --check` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a pipeline-maintained "derived trades" table with multiple indexes and an upsert refresh to speed and simplify trade queries. * Queries and exports now read from the derived-trades read model for faster, more consistent results. * **Chores** * Database schema version bumped to 4; local DBs must be migrated/reinitialized to match the new schema. <!-- end of auto-generated comment: release notes by coderabbit.ai -->

Chained PRs
Depends on #2588. Stack order: #2587 -> #2588 -> #2589 -> #2590
Motivation
The slow trade endpoints come from reconstructing trades from event tables on every read. Once the sync pipeline maintains
derived_trades, the SDK read path should query that table directly.Solution
fetch_tradesandfetch_trades_countto read fromderived_trades.Local performance check
On the locally synced Base ST0x DB for
0xe522cB4a5fCb2eb31a52Ff41a4653d85A4fd7C9D, USDC matched 9,813 derived rows. The SDK local DB path averaged about 8.7ms for page + exact count over 100 iterations.Checks
COMMIT_SHA=local nix develop -c cargo test -p raindex_common local_db::query --libCOMMIT_SHA=local nix develop -c cargo test -p raindex_common local_db::pipeline::adapters::apply --libCOMMIT_SHA=local nix develop -c cargo test -p raindex_common local_db::export --libCOMMIT_SHA=local nix develop -c cargo test -p raindex_common raindex_client::trades --lib