Memory-efficient streaming scan for large announcement sets

[truthixify]

Labels: Stellar Wave, stellar, perf, feature, drips, help-wanted
Tier: M (2–4 days)
Type: perf / feature

Context

scanAnnouncements currently takes a full array of announcements as input. For Spectre agents that have been running for months, this array can grow into the hundreds of thousands. Loading it all into memory is wasteful and breaks the agent's TEE memory budget.

We want a streaming API that can scan as the announcement stream arrives, never holding more than a small window in memory.

Scope

Add a new export scanAnnouncementsStream (Stellar first; pattern repeated per chain later):

async function* scanAnnouncementsStream(
  source: AsyncIterable<Announcement>,
  viewingKey: ...,
  spendingPubKey: ...,
  spendingKey: ...,
): AsyncGenerator<MatchedAnnouncement> { ... }

It must:

1. Process announcements one at a time from the async iterable source.
2. Yield matches as they're found (not in a batched array at the end).
3. Keep peak memory bounded by O(window) where window is configurable (default 64).
4. Support cancellation — if the consumer breaks out of the for-await loop, the stream stops cleanly.

Source providers

In the same PR, ship one source provider for Stellar:

async function* fetchAnnouncementsStream(network: Network, opts?): AsyncIterable<Announcement>

This wraps the existing paginated fetchAnnouncements and yields per page. Together they let a consumer write:

for await (const match of scanAnnouncementsStream(fetchAnnouncementsStream('stellar'), v, K, m)) {
  notify(match)
}

Acceptance criteria

• Streaming scan implementation with tests.
• Streaming fetch implementation with tests.
• Memory benchmark: scanning 100k announcements with streaming must use less than 10× the memory of scanning 1k. (Numbers tunable but the property must hold.)
• Existing scanAnnouncements becomes a thin wrapper over the streaming version (or deprecated cleanly).
• Spectre follow-up issue filed describing the connector migration.

Why this matters

The TEE has tight memory limits. Spectre's current Stellar scan path will hit them within ~6 months of real production usage. We need to fix this before mainnet, not after.

Files

• sdk/src/chains/stellar/scan.ts
• sdk/src/chains/stellar/announcements.ts

[SYLVIANNORUKA]

@SYLVIANNORUKA has applied to work on this issue as part of the Stellar Wave Program's 5th wave.

Hello , Can i work on this pls? I will Implement a memory-efficient streaming scan API for large announcement sets using async iterators to process data incrementally without loading full datasets into memory. Add streaming fetch support for Stellar announcements with real-time match yielding and clean cancellation handling. Improve SDK performance and scalability for long-running agents operating within constrained execution environments.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @SYLVIANNORUKA to this issue.

[TheDEV111]

@TheDEV111 has applied to work on this issue as part of the Stellar Wave Program's 5th wave.

Hi, follwoing the completion of issue 3 I would like to contribute more to your project if you let me work on this repo

ℹ️ Repo Maintainers: To accept this application, review their application or assign @TheDEV111 to this issue.

[xJeffx23]

@xJeffx23 has applied to work on this issue as part of the Stellar Wave Program's 5th wave.

Hi! My name is Jefferson Calderón.

I’m an open-source contributor with 1+ year of experience contributing to open source projects. I have experience working with frontend/backend workflows, Git/GitHub, TypeScript, blockchain-related applications, and Stellar ecosystem projects.

I reviewed this issue and I’m interested in working on it. I understand the goal is to add a memory-efficient streaming API for scanning large announcement sets, using async iterables/generators so announcements can be processed one at a time and matches can be yielded as they are found.

My first step would be to review the current scanAnnouncements and fetchAnnouncements implementations, then design scanAnnouncementsStream and fetchAnnouncementsStream in a way that keeps the existing API compatible while improving memory usage. I’d also add tests for streaming behavior, cancellation, and query/fetch flow, plus a memory benchmark for large announcement sets.

Could I work on this issue? I’d be happy to contribute to this project.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @xJeffx23 to this issue.

[drips-wave]

Congratulations, @xJeffx23! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on June 2, 2026.

🧑‍💻 @xJeffx23: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊

[xJeffx23]

Thank you so much for the opportunity. I’ll get started on them in a few moments 🫡🚀