Skip to content

feat(stellar): add streaming scan and fetch APIs#34

Open
xJeffx23 wants to merge 1 commit into
wraith-protocol:developfrom
xJeffx23:feat/stellar-scan-stream
Open

feat(stellar): add streaming scan and fetch APIs#34
xJeffx23 wants to merge 1 commit into
wraith-protocol:developfrom
xJeffx23:feat/stellar-scan-stream

Conversation

@xJeffx23
Copy link
Copy Markdown

@xJeffx23 xJeffx23 commented May 29, 2026

Summary

  • Adds scanAnnouncementsStream — async generator, O(window) memory, configurable window (default 64), clean cancellation via finally/iter.return()
  • Adds fetchAnnouncementsStream — wraps paginated Soroban RPC, yields per-page, never accumulates full set
  • fetchAnnouncements refactored as thin wrapper over streaming version
  • scanAnnouncements marked @deprecated, points consumers to stream API

Usage

for await (const match of scanAnnouncementsStream(fetchAnnouncementsStream('stellar'), viewKey, spendPub, spendScalar)) {
  notify(match)
}

Tests

  • 13 tests for scanAnnouncementsStream: match, filter, cancellation + source cleanup, window=1, window=200, memory benchmark
  • 8 tests for fetchAnnouncementsStream: pagination, cursor, probe error, early return, URL override, cancellation
  • Memory benchmark: 100k announcements < 10× memory of 1k ✅

Follow-up: Spectre connector migration

Spectre agents currently use the deprecated scanAnnouncements path. Once this PR merges, the connector must be updated:

// Before (deprecated — loads full array into memory)
const matches = await scanAnnouncements(await fetchAnnouncements('stellar'), v, K, m);

// After (streaming — O(window) memory, TEE-safe)
for await (const match of scanAnnouncementsStream(fetchAnnouncementsStream('stellar'), v, K, m, { window: 64 })) {
handleMatch(match);
}

At current announcement growth rate the deprecated path will OOM the TEE within ~6 months of mainnet. Migration must happen before launch.

Closes #9

@drips-wave
Copy link
Copy Markdown

drips-wave Bot commented May 29, 2026

@xJeffx23 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@truthixify truthixify changed the base branch from main to develop June 1, 2026 15:32
@truthixify
Copy link
Copy Markdown
Contributor

The streaming scan + fetch generator API additions are what we want for issue #9. Rebase needed before merge, your branch is off pre-develop main so the current diff would undo work in #13/#14/#16.

git fetch origin
git rebase origin/develop
git push --force-with-lease

Keep the streaming additions in src/chains/stellar/scan.ts and your new tests; resolve conflicts by keeping develop's versions for files you didn't intend to change. Thanks @xJeffx23.

@truthixify
Copy link
Copy Markdown
Contributor

The new commit (style: format stellar streaming test files with prettier) is just a formatting pass, but the rebase against develop didn't actually happen. Merge base is still f383081 which is before #13, #14, #16, and #37 all landed.

Could you do the actual rebase? Quick version:

git fetch origin
git rebase origin/develop
# resolve conflicts in src/chains/stellar/* by keeping develop's versions
# resolve in test/chains/stellar/announcements.test.ts by keeping develop's
# your streaming additions in src/chains/stellar/scan.ts and the new tests in scan.test.ts should layer in cleanly
git push --force-with-lease

After that the diff should only show your streaming additions, no reverts. Thanks @xJeffx23.

@xJeffx23
Copy link
Copy Markdown
Author

xJeffx23 commented Jun 2, 2026

Hi @truthixify ! Yes, absolutely.

I had an unexpected situation come up and had to step away before I could finish pushing the remaining commits, so the PR is currently incomplete.

Sorry for the delay. I'll push the updated commits in the next few minutes.

@xJeffx23 xJeffx23 force-pushed the feat/stellar-scan-stream branch from 617672e to 8ac15ac Compare June 2, 2026 01:22
@xJeffx23 xJeffx23 force-pushed the feat/stellar-scan-stream branch from 8ac15ac to 4c4259f Compare June 2, 2026 01:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory-efficient streaming scan for large announcement sets

2 participants