Skip to content

Add derived trades local DB read model#2587

Merged
graphite-app[bot] merged 1 commit into
mainfrom
arda/derived-trades-schema
May 26, 2026
Merged

Add derived trades local DB read model#2587
graphite-app[bot] merged 1 commit into
mainfrom
arda/derived-trades-schema

Conversation

@findolor
Copy link
Copy Markdown
Collaborator

@findolor findolor commented May 22, 2026

Motivation

Trade-list reads should use a pipeline-maintained local DB read model instead of reconstructing trades from source event tables on every request. This keeps producer bootstrap DBs and client incremental syncs on the same schema generation and avoids landing partial local DB protocol changes on main.

Solution

  • Add derived_trades as the derived/read-model table for local DB trades.
  • Maintain derived_trades in the shared local DB apply pipeline in the same transaction as source event/state writes.
  • Rebuild derived rows by sync window while allowing order resolution to look back to older active AddOrderV3 events.
  • Include the table in required-table checks, clear/reset paths, bootstrap export/import, and schema versioning.
  • Update trade local DB read/count queries to read from derived_trades instead of reconstructing from event tables.
  • Cover maintenance, incremental sync, bootstrap/export, clear/reset, and fetch equivalence with tests.

Review note

This was folded from the previous split stack (#2588, #2589, #2590) so the local DB schema generation lands atomically.

Checks

  • COMMIT_SHA=local nix develop -c cargo test -p raindex_app_settings --lib
  • COMMIT_SHA=local nix develop -c cargo test -p raindex_common local_db::query --lib
  • COMMIT_SHA=local nix develop -c cargo fmt --all
  • git diff --check

Summary by CodeRabbit

Release Notes

  • New Features

    • Added a precomputed derived trades table with multiple optimized database indexes to enhance the speed and efficiency of trade-related queries.
  • Chores

    • Updated database schema version from 4 to 5; automatic migration of existing user data will occur during the next synchronization cycle.
    • Updated the registry URL endpoint to reference the latest available rain.strategies resource.

Review Change Stack

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 22, 2026

📝 Walkthrough

Walkthrough

This PR introduces a derived_trades read-model table to the local database schema, bumps the schema version to 5, implements a pipeline-based upsert mechanism to populate it, integrates the refresh into the apply pipeline, and refactors all trade-fetch queries to use the precomputed table instead of building trades on the fly via expensive multi-CTE unions and joins.

Changes

Derived Trades Read-Model Introduction and Query Refactoring

Layer / File(s) Summary
Derived trades table definition, registration, and schema version bump
crates/common/src/local_db/query/create_tables/query.sql, crates/common/src/local_db/query/create_tables/mod.rs, crates/settings/src/local_db_manifest.rs, crates/settings/src/remote/manifest.rs
Creates derived_trades table with composite primary key (chain_id, raindex_address, trade_id) and seven supporting indexes. Adds table to REQUIRED_TABLES constant. Bumps DB_SCHEMA_VERSION from 4 to 5. Updates all test YAML fixtures across local and remote manifest tests to schema version 5.
Table lifecycle operations in clear/drop flows
crates/common/src/local_db/query/clear_raindex_data/query.sql, crates/common/src/local_db/query/clear_tables/query.sql
Extends clear-raindex-data transaction to delete scoped derived_trades rows by (chain_id, raindex_address). Extends clear-tables to drop derived_trades table.
Upsert batch builder and SQL implementation
crates/common/src/local_db/query/upsert_derived_trades/mod.rs, crates/common/src/local_db/query/upsert_derived_trades/query.sql, crates/common/src/local_db/query/mod.rs
Exports upsert_derived_trades_batch function that generates a 2-statement batch: window-scoped delete then deterministic insert. The insert derives take trades and clear-alice/bob trades from order events, maps vault/token metadata via order_ios, computes trade_id from transaction hash + log index + side suffix, and joins vault_balance_changes to populate running balances. Includes unit tests and non-wasm SQLite integration tests.
Apply pipeline integration of derived trades refresh
crates/common/src/local_db/pipeline/adapters/apply.rs
Calls upsert_derived_trades_batch during vault balance update window when start_block <= target_block. Updates tests to verify derived-trades refresh statements are emitted and ordered correctly (after decoded events and derived-vault-deltas, before target watermark).
Owner trades query refactoring to use derived_trades
crates/common/src/local_db/query/fetch_owner_trades_common.rs, crates/common/src/local_db/query/fetch_owner_trades/mod.rs, crates/common/src/local_db/query/fetch_owner_trades/query.sql, crates/common/src/local_db/query/fetch_owner_trades_count/mod.rs, crates/common/src/local_db/query/fetch_owner_trades_count/query.sql
Refactors both queries to use filtered_trades CTE from derived_trades. Removes side-specific clause constants and unifies filtering on tws.order_owner. Updates common-filter helper to bind parameters directly without returning intermediate struct. Updates tests to assert new tws.* column references.
Order trades query refactoring to use derived_trades
crates/common/src/local_db/query/fetch_order_trades/mod.rs, crates/common/src/local_db/query/fetch_order_trades/query.sql, crates/common/src/local_db/query/fetch_order_trades_count/mod.rs, crates/common/src/local_db/query/fetch_order_trades_count/query.sql
Refactors both queries to select from derived_trades using filtered_trades CTE. Updates timestamp clause formatting and column references.
Unified fetch_trades query builder and SQL refactoring
crates/common/src/local_db/query/fetch_trades/mod.rs, crates/common/src/local_db/query/fetch_trades/query.sql, crates/common/src/local_db/query/fetch_trades/count_query.sql
Extracts shared build_fetch_trades_stmt_from_template helper for unified filtering across chain_id, raindex_address, and taker lists. Introduces separate COUNT_QUERY_TEMPLATE. Updates all queries to use filtered_trades CTE from derived_trades.
Trades by transaction query refactoring
crates/common/src/local_db/query/fetch_trades_by_tx/mod.rs, crates/common/src/local_db/query/fetch_trades_by_tx/query.sql
Simplifies builder and query to use filtered_trades CTE from derived_trades. Unifies filter clause placeholders for chain_id and raindex_address.
Test fixtures and infrastructure updates
crates/common/src/local_db/export.rs, crates/common/src/local_db/pipeline/runner/remotes.rs, crates/common/src/raindex_client/trades/mod.rs
Updates export test fixtures to seed and dump derived_trades rows. Updates remote manifest test fixtures and imports. Extends WASM test SQL callback to recognize FROM derived_trades as a trade query. Updates webapp registry URL constant.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

The PR is substantial and involves multiple layers of refactoring across the trade-query architecture. While changes are cohesive in purpose (migrate to precomputed trades table), they span numerous files with heterogeneous edit patterns: table/index definitions, complex SQL CTEs, parameter-binding logic refactors, test updates, and fixture changes. The upsert SQL logic and query refactoring patterns require careful verification across different query builders, and the pipeline integration ordering requires tracing statement sequences through apply.rs tests.

Possibly related PRs

  • rainlanguage/raindex#2592: Adds derived_vault_deltas refresh to the apply pipeline; this PR's derived_trades refresh must execute after that step, so the two PRs are sequenced in the apply batch.
  • rainlanguage/raindex#2572: Extends fetch_trades to support batched order_hashes filtering; this PR refactors fetch_trades to use derived_trades, so the query builder changes overlap.
  • rainlanguage/raindex#2570: Adds token-filter bindings to fetch_trades; this PR refactors the same query builder, so changes overlap at the parameter-binding level.

Suggested reviewers

  • hardyjosh
  • 0xgleb
  • JuaniRios

Poem

🐰 A table derived now holds the trades,
No more unions, joins, and cascades;
The pipeline refreshes block by block,
While queries read fast from the precomputed stock.
One truth maintained, all queries sing—
Efficiency wins when you cache the thing!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 63.38% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: introducing a new derived trades local DB read model to replace reconstructed trades queries.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch arda/derived-trades-schema

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

packages/webapp/src/lib/constants.ts

Oops! Something went wrong! :(

ESLint: 8.57.1

Error: ESLint configuration in --config is invalid:

  • Unexpected top-level property "__esModule".

    at ConfigValidator.validateConfigSchema (/node_modules/@eslint/eslintrc/dist/eslintrc.cjs:2177:19)
    at ConfigArrayFactory._normalizeConfigData (/node_modules/@eslint/eslintrc/dist/eslintrc.cjs:3019:19)
    at ConfigArrayFactory._loadConfigData (/node_modules/@eslint/eslintrc/dist/eslintrc.cjs:2984:21)
    at ConfigArrayFactory.loadFile (/node_modules/@eslint/eslintrc/dist/eslintrc.cjs:2850:40)
    at createCLIConfigArray (/node_modules/@eslint/eslintrc/dist/eslintrc.cjs:3660:35)
    at new CascadingConfigArrayFactory (/node_modules/@eslint/eslintrc/dist/eslintrc.cjs:3735:29)
    at new CLIEngine (/node_modules/eslint/lib/cli-engine/cli-engine.js:617:36)
    at new ESLint (/node_modules/eslint/lib/eslint/eslint.js:430:27)
    at Object.execute (/node_modules/eslint/lib/cli.js:410:24)
    at async main (/node_modules/eslint/bin/eslint.js:152:22)


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Collaborator Author

findolor commented May 22, 2026


How to use the Graphite Merge Queue

Add the label Raindex-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

@findolor findolor marked this pull request as ready for review May 22, 2026 11:00
@findolor findolor self-assigned this May 22, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/settings/src/local_db_manifest.rs`:
- Line 12: Tests in this file still assert and use the old db-schema-version (3)
after updating the constant DB_SCHEMA_VERSION to 4, causing
parse_manifest_header/parse_manifest_doc to fail; update all fixtures and
assertions to use the new schema version or, better, make fixtures driven by the
DB_SCHEMA_VERSION constant (e.g., use the constant when constructing test
manifest headers and replace hard-coded `db-schema-version: 3` and `sv == 3`
checks with DB_SCHEMA_VERSION or a helper that emits the current
DB_SCHEMA_VERSION) so parse_manifest_header/parse_manifest_doc and sv assertions
stay in sync.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: fe951c46-c297-4b06-911f-ef16e953f9f4

📥 Commits

Reviewing files that changed from the base of the PR and between 29b7822 and 46756c6.

📒 Files selected for processing (5)
  • crates/common/src/local_db/query/clear_raindex_data/query.sql
  • crates/common/src/local_db/query/clear_tables/query.sql
  • crates/common/src/local_db/query/create_tables/mod.rs
  • crates/common/src/local_db/query/create_tables/query.sql
  • crates/settings/src/local_db_manifest.rs

Comment thread crates/settings/src/local_db_manifest.rs Outdated
@findolor findolor force-pushed the arda/derived-trades-schema branch from 46756c6 to 381599c Compare May 23, 2026 13:46
@findolor findolor changed the title Add derived trades table schema Add derived trades local DB read model May 23, 2026
@findolor findolor force-pushed the arda/derived-trades-schema branch from 2334e5b to 710fddf Compare May 23, 2026 19:50
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/settings/src/remote/manifest.rs (1)

212-272: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Reject mixed schema generations when fetching multiple manifests.

These tests all move to db-schema-version: 4, but fetch_multiple_manifests still just folds results into a map, so different URLs can return different db_schema_versions without tripping an error. That undermines the goal of keeping bootstrap/import sources and incremental sync on the same local-DB protocol, and the existing InconsistentManifestVersions path is never exercised.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/settings/src/remote/manifest.rs` around lines 212 - 272,
fetch_multiple_manifests currently collects per-URL manifests into a map without
checking db-schema-version consistency, allowing mixed schema generations;
update the fetch_multiple_manifests function to validate that all fetched
Manifest instances share the same db_schema_version (the
Manifest.db_schema_version field) and if any differ return the existing
InconsistentManifestVersions error (or a clear error variant) instead of folding
them silently into the map, ensuring the check runs after parsing each manifest
and before returning the map so callers never get mixed-schema results.
🧹 Nitpick comments (1)
crates/common/src/local_db/query/fetch_trades_by_tx/mod.rs (1)

49-88: ⚡ Quick win

Add one test that binds both list clauses at once.

The refactor only exercises chain IDs alone and raindex addresses alone. A case with both populated is the one that proves the second bind_list_clause starts numbering at the correct offset.

🧪 Suggested test shape
+    #[test]
+    fn builds_with_chain_ids_and_raindex_address_filters() {
+        let tx_hash = b256!("0x00000000000000000000000000000000000000000000000000000000deadbeef");
+        let ob = address!("0x2f209e5b67a33b8fe96e28f24628df6da301c8eb");
+        let stmt = build_fetch_trades_by_tx_stmt(&FetchTradesByTxArgs {
+            chain_ids: vec![137, 1],
+            raindex_addresses: vec![ob],
+            tx_hash,
+        })
+        .unwrap();
+
+        assert_eq!(stmt.params.len(), 4);
+        assert!(stmt.sql.contains("tws.chain_id IN (?2, ?3)"));
+        assert!(stmt.sql.contains("tws.raindex_address IN (?4)"));
+    }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/common/src/local_db/query/fetch_trades_by_tx/mod.rs` around lines 49 -
88, Add a new unit test that calls build_fetch_trades_by_tx_stmt with
FetchTradesByTxArgs where both chain_ids and raindex_addresses are non-empty
(e.g., two chain IDs and two raindex addresses) to verify bind_list_clause
numbering continues across multiple list clauses; in the new test (name it
something like builds_with_chain_ids_and_raindex_addresses) assert stmt.params
length equals 1 + num_chain_ids + num_raindex_addresses, assert params[0] is the
tx_hash Text, assert the subsequent params match the chain IDs (SqlValue::U64)
followed by the raindex addresses (SqlValue::Text(hex::encode_prefixed(...))),
and assert the generated SQL contains the expected IN clauses for both chain IDs
and raindex addresses (e.g., "tws.chain_id IN (?2, ?3)" and "tws.raindex_address
IN (?4, ?5)") and does not contain the CHAIN_IDS_CLAUSE or RAINDEXES_CLAUSE
placeholders.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/common/src/local_db/query/create_tables/query.sql`:
- Around line 384-395: The current set of indexes on derived_trades adds
predicates (input_token, output_token, order_owner, transaction_sender,
order_hash, transaction_hash) before the time columns which prevents efficient
use of chain/raindex/time-only queries; add a plain time-prefixed index for the
main trade-list path by creating an index (e.g., idx_derived_trades_time) on
derived_trades(chain_id, raindex_address, block_timestamp DESC, block_number
DESC, log_index DESC) alongside the other CREATE INDEX statements so
fetch_trades/query.sql can use the chain+raindex+time filter without
scanning/sorting the whole table.

In `@crates/common/src/local_db/query/upsert_derived_trades/query.sql`:
- Around line 78-104: The query's JOIN/NOT EXISTS uses order_events lookups by
oe.order_owner and oe.order_nonce (symbols: JOIN order_events oe,
oe.order_owner, oe.order_nonce, NOT EXISTS (SELECT 1 FROM order_events newer
...)), which will cause full scans for active owners; add a supporting b-tree
index to order_events to cover (chain_id, raindex_address, order_owner,
order_nonce, event_type, block_number, log_index) (suggested index name:
idx_order_events_owner_nonce_time) in create_tables/query.sql so those equality
and range predicates use the index and avoid scanning all historical orders.

---

Outside diff comments:
In `@crates/settings/src/remote/manifest.rs`:
- Around line 212-272: fetch_multiple_manifests currently collects per-URL
manifests into a map without checking db-schema-version consistency, allowing
mixed schema generations; update the fetch_multiple_manifests function to
validate that all fetched Manifest instances share the same db_schema_version
(the Manifest.db_schema_version field) and if any differ return the existing
InconsistentManifestVersions error (or a clear error variant) instead of folding
them silently into the map, ensuring the check runs after parsing each manifest
and before returning the map so callers never get mixed-schema results.

---

Nitpick comments:
In `@crates/common/src/local_db/query/fetch_trades_by_tx/mod.rs`:
- Around line 49-88: Add a new unit test that calls
build_fetch_trades_by_tx_stmt with FetchTradesByTxArgs where both chain_ids and
raindex_addresses are non-empty (e.g., two chain IDs and two raindex addresses)
to verify bind_list_clause numbering continues across multiple list clauses; in
the new test (name it something like
builds_with_chain_ids_and_raindex_addresses) assert stmt.params length equals 1
+ num_chain_ids + num_raindex_addresses, assert params[0] is the tx_hash Text,
assert the subsequent params match the chain IDs (SqlValue::U64) followed by the
raindex addresses (SqlValue::Text(hex::encode_prefixed(...))), and assert the
generated SQL contains the expected IN clauses for both chain IDs and raindex
addresses (e.g., "tws.chain_id IN (?2, ?3)" and "tws.raindex_address IN (?4,
?5)") and does not contain the CHAIN_IDS_CLAUSE or RAINDEXES_CLAUSE
placeholders.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 07ef357b-717f-459d-b2d9-e5b990860d2e

📥 Commits

Reviewing files that changed from the base of the PR and between 46756c6 and 710fddf.

📒 Files selected for processing (27)
  • crates/cli/src/commands/local_db/pipeline/runner/manifest.rs
  • crates/common/src/local_db/export.rs
  • crates/common/src/local_db/pipeline/adapters/apply.rs
  • crates/common/src/local_db/query/clear_raindex_data/query.sql
  • crates/common/src/local_db/query/clear_tables/query.sql
  • crates/common/src/local_db/query/create_tables/mod.rs
  • crates/common/src/local_db/query/create_tables/query.sql
  • crates/common/src/local_db/query/fetch_order_trades/mod.rs
  • crates/common/src/local_db/query/fetch_order_trades/query.sql
  • crates/common/src/local_db/query/fetch_order_trades_count/mod.rs
  • crates/common/src/local_db/query/fetch_order_trades_count/query.sql
  • crates/common/src/local_db/query/fetch_owner_trades/mod.rs
  • crates/common/src/local_db/query/fetch_owner_trades/query.sql
  • crates/common/src/local_db/query/fetch_owner_trades_common.rs
  • crates/common/src/local_db/query/fetch_owner_trades_count/mod.rs
  • crates/common/src/local_db/query/fetch_owner_trades_count/query.sql
  • crates/common/src/local_db/query/fetch_trades/count_query.sql
  • crates/common/src/local_db/query/fetch_trades/mod.rs
  • crates/common/src/local_db/query/fetch_trades/query.sql
  • crates/common/src/local_db/query/fetch_trades_by_tx/mod.rs
  • crates/common/src/local_db/query/fetch_trades_by_tx/query.sql
  • crates/common/src/local_db/query/mod.rs
  • crates/common/src/local_db/query/upsert_derived_trades/mod.rs
  • crates/common/src/local_db/query/upsert_derived_trades/query.sql
  • crates/common/src/raindex_client/trades/mod.rs
  • crates/settings/src/local_db_manifest.rs
  • crates/settings/src/remote/manifest.rs
✅ Files skipped from review due to trivial changes (2)
  • crates/common/src/local_db/query/clear_tables/query.sql
  • crates/common/src/local_db/query/fetch_order_trades/mod.rs

Comment thread crates/common/src/local_db/query/create_tables/query.sql
Comment thread crates/common/src/local_db/query/upsert_derived_trades/query.sql
@findolor findolor force-pushed the arda/derived-trades-schema branch from 710fddf to cce428b Compare May 24, 2026 14:19
@findolor findolor changed the base branch from main to graphite-base/2587 May 24, 2026 18:33
@findolor findolor force-pushed the arda/derived-trades-schema branch from cce428b to 79565b3 Compare May 24, 2026 18:33
@findolor findolor changed the base branch from graphite-base/2587 to local-db-manifest-urls May 24, 2026 18:33
@findolor findolor force-pushed the arda/derived-trades-schema branch from 79565b3 to efde380 Compare May 25, 2026 14:49
@graphite-app graphite-app Bot changed the base branch from local-db-manifest-urls to graphite-base/2587 May 26, 2026 06:13
@graphite-app graphite-app Bot force-pushed the graphite-base/2587 branch from 3f8691a to 7e32a38 Compare May 26, 2026 07:41
@graphite-app graphite-app Bot force-pushed the arda/derived-trades-schema branch from efde380 to 282ec63 Compare May 26, 2026 07:41
@graphite-app graphite-app Bot changed the base branch from graphite-base/2587 to main May 26, 2026 07:41
@graphite-app graphite-app Bot force-pushed the arda/derived-trades-schema branch from 282ec63 to e1eb035 Compare May 26, 2026 07:42
findolor added a commit to rainlanguage/rain.local-db.remote that referenced this pull request May 26, 2026
## Depends on

- #8
- rainlanguage/raindex#2587

## Motivation

We want to test the deployment-id local DB remote flow with the raindex changes from rainlanguage/raindex#2587 before deploying a named sync instance.

That upstream PR adds a `derived_trades` local DB read model and bumps the local DB schema to version 4. Testing it through this repository lets the remote deployment workflow build and install the CLI from the exact raindex commit that contains those schema changes.

## Solution

- bump `lib/raindex` from `9b33a3c4eca44726b4602f1a2d4046982584ef93` to `cce428bf874443f55280705d8d2da157ef9921ab`
- use the latest commit from rainlanguage/raindex#2587: `cce428bf874443f55280705d8d2da157ef9921ab`
- include the upstream derived-trades changes needed for the local DB schema version 4 deployment test

Relevant upstream raindex PR context:
- PR: rainlanguage/raindex#2587
- Title: Add derived trades local DB read model
- Branch: `arda/derived-trades-schema`
- Latest commit: `cce428bf874443f55280705d8d2da157ef9921ab`
- Main behavior: maintain `derived_trades` during local DB sync and read trade lists from that derived table

## Checks

By submitting this for review, I am confirming I have done the following:
- [x] made this PR as small as possible
- [ ] unit-tested any new functionality
- [x] linked any relevant issues or PRs
- [ ] included screenshots (if this involves a front-end change)

Validated locally with:
- `git submodule status lib/raindex`
- `git diff --stat feature/deployment-id-ops..HEAD`

Not run:
- no deploy was run from this branch
- no local raindex build was run
@findolor findolor force-pushed the arda/derived-trades-schema branch from d091973 to da40522 Compare May 26, 2026 16:17
@graphite-app
Copy link
Copy Markdown

graphite-app Bot commented May 26, 2026

Merge activity

## Motivation

Trade-list reads should use a pipeline-maintained local DB read model instead of reconstructing trades from source event tables on every request. This keeps producer bootstrap DBs and client incremental syncs on the same schema generation and avoids landing partial local DB protocol changes on `main`.

## Solution

- Add `derived_trades` as the derived/read-model table for local DB trades.
- Maintain `derived_trades` in the shared local DB apply pipeline in the same transaction as source event/state writes.
- Rebuild derived rows by sync window while allowing order resolution to look back to older active `AddOrderV3` events.
- Include the table in required-table checks, clear/reset paths, bootstrap export/import, and schema versioning.
- Update trade local DB read/count queries to read from `derived_trades` instead of reconstructing from event tables.
- Cover maintenance, incremental sync, bootstrap/export, clear/reset, and fetch equivalence with tests.

## Review note

This was folded from the previous split stack (#2588, #2589, #2590) so the local DB schema generation lands atomically.

## Checks

- `COMMIT_SHA=local nix develop -c cargo test -p raindex_app_settings --lib`
- `COMMIT_SHA=local nix develop -c cargo test -p raindex_common local_db::query --lib`
- `COMMIT_SHA=local nix develop -c cargo fmt --all`
- `git diff --check`

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

* **New Features**
  * Added a pipeline-maintained "derived trades" table with multiple indexes and an upsert refresh to speed and simplify trade queries.
  * Queries and exports now read from the derived-trades read model for faster, more consistent results.

* **Chores**
  * Database schema version bumped to 4; local DBs must be migrated/reinitialized to match the new schema.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
@graphite-app graphite-app Bot force-pushed the arda/derived-trades-schema branch from da24ba2 to d6777c9 Compare May 26, 2026 18:11
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (3)
crates/common/src/local_db/query/upsert_derived_trades/mod.rs (1)

161-168: 💤 Low value

Potential overflow in u64 to i64 cast.

The conversion u as i64 at Line 165 can overflow for values > i64::MAX. While block numbers won't realistically exceed this limit, this is test-only code and the pattern is acceptable for integration tests.

Defensive alternative
 SqlValue::U64(u) => rusqlite::types::Value::Integer(
-    u as i64
+    i64::try_from(u).expect("u64 value exceeds i64 range")
 ),
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/common/src/local_db/query/upsert_derived_trades/mod.rs` around lines
161 - 168, The match arm in sqlvalue_to_rusqlite for SqlValue::U64 does a direct
cast (u as i64) which can overflow; update the SqlValue::U64 branch in function
sqlvalue_to_rusqlite to defensively handle values > i64::MAX (e.g., check if u
<= i64::MAX and cast, otherwise clamp to i64::MAX or choose a safe fallback) so
we never perform an unchecked u64-to-i64 cast that could overflow at runtime.
crates/common/src/local_db/query/fetch_owner_trades_count/mod.rs (1)

10-11: ⚡ Quick win

Inconsistent timestamp column qualification.

The timestamp body strings use unqualified block_timestamp while the corresponding code in fetch_order_trades_count/mod.rs (lines 18-20) uses tws.block_timestamp. Though the query will work since there's only one table, qualifying with tws. maintains consistency across the codebase.

♻️ Suggested fix for consistency
-const START_TS_BODY: &str = "\nAND block_timestamp >= {param}\n";
-const END_TS_BODY: &str = "\nAND block_timestamp <= {param}\n";
+const START_TS_BODY: &str = "\nAND tws.block_timestamp >= {param}\n";
+const END_TS_BODY: &str = "\nAND tws.block_timestamp <= {param}\n";
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/common/src/local_db/query/fetch_owner_trades_count/mod.rs` around
lines 10 - 11, The constants START_TS_BODY and END_TS_BODY use an unqualified
column name `block_timestamp`; update these string constants to use the
qualified column `tws.block_timestamp` (i.e., replace `block_timestamp` with
`tws.block_timestamp`) so they match the style used in fetch_order_trades_count
(and keep consistency across queries that reference the `tws` alias).
crates/common/src/local_db/query/fetch_trades/mod.rs (1)

62-63: ⚡ Quick win

Simplify statement initialization to use the provided template directly.

The statement is created from QUERY_TEMPLATE but immediately overwritten with template. This creates unnecessary overhead.

♻️ Suggested fix
-    let mut stmt = SqlStatement::new(QUERY_TEMPLATE);
-    stmt.sql = template.to_string();
+    let mut stmt = SqlStatement::new(template);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/common/src/local_db/query/fetch_trades/mod.rs` around lines 62 - 63,
Replace the two-step initialization that creates SqlStatement from
QUERY_TEMPLATE then overwrites stmt.sql with template by initializing it
directly from template; specifically, stop constructing with QUERY_TEMPLATE and
instead create the SqlStatement using template (e.g., call
SqlStatement::new(template.to_string()) or otherwise pass template into the
constructor) so you no longer set stmt.sql after construction (remove usage of
QUERY_TEMPLATE and the stmt.sql assignment).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/webapp/src/lib/constants.ts`:
- Around line 1-2: The PR is missing the required built-webapp screenshot for
frontend changes; attach a screenshot of the built webapp that shows the effect
of your change (e.g., capture the app page after building and running the site)
to the PR so reviewers can verify UI output; reference the frontend change in
packages/webapp/src/lib/constants.ts (REGISTRY_URL) in the screenshot or PR
description, and add the image file to the PR (upload to the PR conversation or
include under the repo's doc/artifacts folder and link it) before merging.

---

Nitpick comments:
In `@crates/common/src/local_db/query/fetch_owner_trades_count/mod.rs`:
- Around line 10-11: The constants START_TS_BODY and END_TS_BODY use an
unqualified column name `block_timestamp`; update these string constants to use
the qualified column `tws.block_timestamp` (i.e., replace `block_timestamp` with
`tws.block_timestamp`) so they match the style used in fetch_order_trades_count
(and keep consistency across queries that reference the `tws` alias).

In `@crates/common/src/local_db/query/fetch_trades/mod.rs`:
- Around line 62-63: Replace the two-step initialization that creates
SqlStatement from QUERY_TEMPLATE then overwrites stmt.sql with template by
initializing it directly from template; specifically, stop constructing with
QUERY_TEMPLATE and instead create the SqlStatement using template (e.g., call
SqlStatement::new(template.to_string()) or otherwise pass template into the
constructor) so you no longer set stmt.sql after construction (remove usage of
QUERY_TEMPLATE and the stmt.sql assignment).

In `@crates/common/src/local_db/query/upsert_derived_trades/mod.rs`:
- Around line 161-168: The match arm in sqlvalue_to_rusqlite for SqlValue::U64
does a direct cast (u as i64) which can overflow; update the SqlValue::U64
branch in function sqlvalue_to_rusqlite to defensively handle values > i64::MAX
(e.g., check if u <= i64::MAX and cast, otherwise clamp to i64::MAX or choose a
safe fallback) so we never perform an unchecked u64-to-i64 cast that could
overflow at runtime.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d7f3b189-7381-4536-8d56-a763f8f6b780

📥 Commits

Reviewing files that changed from the base of the PR and between e1eb035 and d6777c9.

📒 Files selected for processing (28)
  • crates/common/src/local_db/export.rs
  • crates/common/src/local_db/pipeline/adapters/apply.rs
  • crates/common/src/local_db/pipeline/runner/remotes.rs
  • crates/common/src/local_db/query/clear_raindex_data/query.sql
  • crates/common/src/local_db/query/clear_tables/query.sql
  • crates/common/src/local_db/query/create_tables/mod.rs
  • crates/common/src/local_db/query/create_tables/query.sql
  • crates/common/src/local_db/query/fetch_order_trades/mod.rs
  • crates/common/src/local_db/query/fetch_order_trades/query.sql
  • crates/common/src/local_db/query/fetch_order_trades_count/mod.rs
  • crates/common/src/local_db/query/fetch_order_trades_count/query.sql
  • crates/common/src/local_db/query/fetch_owner_trades/mod.rs
  • crates/common/src/local_db/query/fetch_owner_trades/query.sql
  • crates/common/src/local_db/query/fetch_owner_trades_common.rs
  • crates/common/src/local_db/query/fetch_owner_trades_count/mod.rs
  • crates/common/src/local_db/query/fetch_owner_trades_count/query.sql
  • crates/common/src/local_db/query/fetch_trades/count_query.sql
  • crates/common/src/local_db/query/fetch_trades/mod.rs
  • crates/common/src/local_db/query/fetch_trades/query.sql
  • crates/common/src/local_db/query/fetch_trades_by_tx/mod.rs
  • crates/common/src/local_db/query/fetch_trades_by_tx/query.sql
  • crates/common/src/local_db/query/mod.rs
  • crates/common/src/local_db/query/upsert_derived_trades/mod.rs
  • crates/common/src/local_db/query/upsert_derived_trades/query.sql
  • crates/common/src/raindex_client/trades/mod.rs
  • crates/settings/src/local_db_manifest.rs
  • crates/settings/src/remote/manifest.rs
  • packages/webapp/src/lib/constants.ts
✅ Files skipped from review due to trivial changes (2)
  • crates/common/src/local_db/query/fetch_order_trades/mod.rs
  • crates/settings/src/remote/manifest.rs

Comment on lines 1 to +2
export const REGISTRY_URL =
"https://raw.githubusercontent.com/rainlanguage/rain.strategies/1a4cd6e3f30b3a66d30501743d4282054d5c66fd/registry";
"https://raw.githubusercontent.com/rainlanguage/rain.strategies/96007c4d5a06cad390b67112a55f8a892dad29fc/registry";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Missing required frontend screenshot artifact.

Please attach a screenshot of the built webapp for this frontend change before merge.

As per coding guidelines, "packages/{webapp,ui-components}/**/*.{svelte,ts,tsx,js,jsx}: When modifying frontend code or functionality, you MUST provide a screenshot of the built webapp reflecting your change."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/webapp/src/lib/constants.ts` around lines 1 - 2, The PR is missing
the required built-webapp screenshot for frontend changes; attach a screenshot
of the built webapp that shows the effect of your change (e.g., capture the app
page after building and running the site) to the PR so reviewers can verify UI
output; reference the frontend change in packages/webapp/src/lib/constants.ts
(REGISTRY_URL) in the screenshot or PR description, and add the image file to
the PR (upload to the PR conversation or include under the repo's doc/artifacts
folder and link it) before merging.

@graphite-app graphite-app Bot merged commit d6777c9 into main May 26, 2026
18 checks passed
@github-actions
Copy link
Copy Markdown
Contributor

@coderabbitai assess this PR size classification for the totality of the PR with the following criterias and report it in your comment:

S/M/L PR Classification Guidelines:

This guide helps classify merged pull requests by effort and complexity rather than just line count. The goal is to assess the difficulty and scope of changes after they have been completed.

Small (S)

Characteristics:

  • Simple bug fixes, typos, or minor refactoring
  • Single-purpose changes affecting 1-2 files
  • Documentation updates
  • Configuration tweaks
  • Changes that require minimal context to review

Review Effort: Would have taken 5-10 minutes

Examples:

  • Fix typo in variable name
  • Update README with new instructions
  • Adjust configuration values
  • Simple one-line bug fixes
  • Import statement cleanup

Medium (M)

Characteristics:

  • Feature additions or enhancements
  • Refactoring that touches multiple files but maintains existing behavior
  • Breaking changes with backward compatibility
  • Changes requiring some domain knowledge to review

Review Effort: Would have taken 15-30 minutes

Examples:

  • Add new feature or component
  • Refactor common utility functions
  • Update dependencies with minor breaking changes
  • Add new component with tests
  • Performance optimizations
  • More complex bug fixes

Large (L)

Characteristics:

  • Major feature implementations
  • Breaking changes or API redesigns
  • Complex refactoring across multiple modules
  • New architectural patterns or significant design changes
  • Changes requiring deep context and multiple review rounds

Review Effort: Would have taken 45+ minutes

Examples:

  • Complete new feature with frontend/backend changes
  • Protocol upgrades or breaking changes
  • Major architectural refactoring
  • Framework or technology upgrades

Additional Factors to Consider

When deciding between sizes, also consider:

  • Test coverage impact: More comprehensive test changes lean toward larger classification
  • Risk level: Changes to critical systems bump up a size category
  • Team familiarity: Novel patterns or technologies increase complexity

Notes:

  • the assessment must be for the totality of the PR, that means comparing the base branch to the last commit of the PR
  • the assessment output must be exactly one of: S, M or L (single-line comment) in format of: SIZE={S/M/L}
  • do not include any additional text, only the size classification
  • your assessment comment must not include tips or additional sections
  • do NOT tag me or anyone else on your comment

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 26, 2026

Tip

For best results, initiate chat on the files or code changes.

SIZE=L

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants