feat(stellar): add streaming scan and fetch APIs#34
Conversation
|
@xJeffx23 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
|
The streaming scan + fetch generator API additions are what we want for issue #9. Rebase needed before merge, your branch is off pre-develop main so the current diff would undo work in #13/#14/#16. git fetch origin
git rebase origin/develop
git push --force-with-leaseKeep the streaming additions in |
|
The new commit ( Could you do the actual rebase? Quick version: git fetch origin
git rebase origin/develop
# resolve conflicts in src/chains/stellar/* by keeping develop's versions
# resolve in test/chains/stellar/announcements.test.ts by keeping develop's
# your streaming additions in src/chains/stellar/scan.ts and the new tests in scan.test.ts should layer in cleanly
git push --force-with-leaseAfter that the diff should only show your streaming additions, no reverts. Thanks @xJeffx23. |
|
Hi @truthixify ! Yes, absolutely. I had an unexpected situation come up and had to step away before I could finish pushing the remaining commits, so the PR is currently incomplete. Sorry for the delay. I'll push the updated commits in the next few minutes. |
617672e to
8ac15ac
Compare
8ac15ac to
4c4259f
Compare
Summary
scanAnnouncementsStream— async generator, O(window) memory, configurable window (default 64), clean cancellation viafinally/iter.return()fetchAnnouncementsStream— wraps paginated Soroban RPC, yields per-page, never accumulates full setfetchAnnouncementsrefactored as thin wrapper over streaming versionscanAnnouncementsmarked@deprecated, points consumers to stream APIUsage
Tests
Follow-up: Spectre connector migration
Spectre agents currently use the deprecated scanAnnouncements path. Once this PR merges, the connector must be updated:
// Before (deprecated — loads full array into memory)
const matches = await scanAnnouncements(await fetchAnnouncements('stellar'), v, K, m);
// After (streaming — O(window) memory, TEE-safe)
for await (const match of scanAnnouncementsStream(fetchAnnouncementsStream('stellar'), v, K, m, { window: 64 })) {
handleMatch(match);
}
At current announcement growth rate the deprecated path will OOM the TEE within ~6 months of mainnet. Migration must happen before launch.
Closes #9