fix(routing/http): cap records after filtering

[lidel]

Problem

The Delegated Routing server applied the records limit before filter-addrs/filter-protocols ran, so records the filters dropped shrank /routing/v1/providers and /routing/v1/peers responses below the configured limit.

This Fix

The server now calls the delegate FindProviders/FindPeers with a limit of 0 (unbounded), reads the result iterator lazily, and applies the cap itself after filtering. A delegate that used the limit to end its walk early will now end it on Close instead.

See it in action in:

• feat: add records-limit flags for json and ndjson someguy#150

Changes

• routing/http/types/iter: add Limit, an iterator that caps another iterator at a fixed number of values
• routing/http/server: apply the records limit post-filter; pass 0 to the delegate; update tests and DelegatedRouter docs

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.21%. Comparing base (4ea1540) to head (7fa3c6c).

@@            Coverage Diff             @@
##             main    #1157      +/-   ##
==========================================
+ Coverage   63.15%   63.21%   +0.05%     
==========================================
  Files         267      268       +1     
  Lines       26867    26884      +17     
==========================================
+ Hits        16969    16994      +25     
+ Misses       8173     8163      -10     
- Partials     1725     1727       +2     

Files with missing lines	Coverage Δ
routing/http/server/server.go	77.83% <100.00%> (+1.67%)	⬆️
routing/http/types/iter/limit.go	100.00% <100.00%> (ø)

... and 12 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[guillaumemichel]

This seems like a better approach than ipfs/someguy#149 👍🏻

Would it make sense to use FindProvidersAsync() instead of FindProviders()? It is the more standard way to find provider (go-libp2p interface), and gives caller more control.

FindProvidersAsync(ctx, cid, 0) would allow to cancel the request (cancel context) after we are happy with the results (e.g enough providers with addresses), so that we don't solicit the content router more than strictly needed.

[lidel]

Hm.. the goal here (cancel once we have enough, don't over-ask the router) is right, and it's already what this PR does.

@guillaumemichel lmk if i misunderstood, but the DelegatedRouter.FindProviders is slightly different from go-libp2p interface:

• go-libp2p ContentDiscovery.FindProvidersAsync returns <-chan peer.AddrInfo a value channel, no error channel
• boxo's DelegatedRouter is a separate interface, returning iter.ResultIter[types.Record] (richer type than peer.AddrInfo, and is lazy: the server pulls record by record, stops once iter.Limit is satisfied, and the deferred Close() cancels the rest
  • In feat: add records-limit flags for json and ndjson someguy#150 that chain is real: peerChanIter.Close() cancels the context passed to FindProvidersAsync. So FindProvidersAsync is already used, but at the layer where it fits best: the DHT boundary inside someguy's libp2pRouter.

Is the idea to lift FindProvidersAsync to DelegatedRouter? If so, it wouldn't add control we lack, and it has costs:

• <-chan peer.AddrInfo has no slot for per-item errors; iter.Result does.
• a channel forces a goroutine even for synchronous delegates. boxo's own HTTP-delegated client is a goroutine-free streaming decoder today; a channel interface would spawn a pump goroutine per request.
• ApplyFiltersToIter, Limit, and the ndjson writer are all iter-based, so a channel would be converted right back into an iterator.

So I'd keep FindProviders as-is. I did expand the DelegatedRouter doc comment to spell out the lazy + Close() contract, so implementers know to stream results and stop work on Close.

[guillaumemichel]

Alright, I was missing a piece of the puzzle in someguy.

Everything looks good, just 2 nits inline

[guillaumemichel]

provIter could be unconditionally closed after iter.ReadAllResults(), which would cancel request on router faster. No need to wait for response to be sent.

Suggested change

	defer provIter.Close()
	filteredIter := filters.ApplyFiltersToIter(provIter, filterAddrs, filterProtocols)
	providers, err := iter.ReadAllResults(filteredIter)
	var limitedIter iter.ResultIter[types.Record] = iter.Limit(filteredIter, recordsLimit)
	providers, err := iter.ReadAllResults(limitedIter)
	filteredIter := filters.ApplyFiltersToIter(provIter, filterAddrs, filterProtocols)
	var limitedIter iter.ResultIter[types.Record] = iter.Limit(filteredIter, recordsLimit)
	providers, err := iter.ReadAllResults(limitedIter)
	provIter.Close()

[guillaumemichel]

Or even better to close limitedIter?

[guillaumemichel]

Same reasoning as above, no need to keep query longer than required.

Suggested change

	defer peersIter.Close()
	peersIter = filters.ApplyFiltersToPeerRecordIter(peersIter, filterAddrs, filterProtocols)
	filteredIter := filters.ApplyFiltersToPeerRecordIter(peersIter, filterAddrs, filterProtocols)
	var limitedIter iter.ResultIter[*types.PeerRecord] = iter.Limit(filteredIter, recordsLimit)
	peers, err := iter.ReadAllResults(peersIter)
	peers, err := iter.ReadAllResults(limitedIter)
	filteredIter := filters.ApplyFiltersToPeerRecordIter(peersIter, filterAddrs, filterProtocols)
	var limitedIter iter.ResultIter[*types.PeerRecord] = iter.Limit(filteredIter, recordsLimit)
	peers, err := iter.ReadAllResults(limitedIter)
	peersIter.Close()

[guillaumemichel]

Or even better to close limitedIter?