LO:TECH response involving failed symbols

[hacheigriega]

Explanation of Changes

Explanation of the three commits in order:

1. Apply changes from chore: always give back prices even when one symbol fails #130 to the lo-tech module. Also, move cleanup upon timeout error from the call sites to the shared cache.
2. Lo-tech module now handles acknowledgement messages. The price feed map is now updated only after receiving ack messages, preventing updates when a subscription request fails.
3. Lo-tech module also handles subscription failure errors. This helps us clean up the price feed IDs we add for every subscription request.
4. Price fetch is now done concurrently so that timeouts do not accumulate across multiple invalid symbols.

Testing

$ curl http://localhost:5384/proxy/lotech/NVDA,WTIN6 | jq

[
  {
    "type": "PRICE",
    "symbol": "NVDA",
    "ingress_ts": 1779818875945310,
    "publish_ts": null,
    "transaction_ts": 1779818875860000,
    "price": 213.496,
    "spread": 0.01,
    "__sedaHasPrice": true
  },
  {
    "symbol": "WTIN6",
    "__sedaHasPrice": false
  }
]

[Thomasvdam]

Is this correct? It reads a little weird that a message from lo-tech->proxy updates a map that tracks requests from client->proxy

[hacheigriega]

So with this change we update the price feed subscription map and the last request map upon receiving ack to avoid updating these maps for invalid symbols that we have failed to subscribe for. Previously, we were tracking last request times for all the invalid symbols we receive from clients, even though there were no active price feeds for those.

[Thomasvdam]

Ah check, so if I understand it correctly the flow now is:

• Request for symbol X
• If new symbol, create new feedId (monotonically increasing counter) and store in priceFeedIds
• Receive ack for an ID
• Check if priceFeedIds contains the ID
• If yes, make the pricefeed 'official' by adding mapping entry in priceFeeds
• And mark it as the first request by setting a timestamp in lastRequestToPriceFeed

If this is correct I find it a little hard to get this from the code, but I also don't have an idea on how to express it differently.

I do have one question; what happens to priceFeedIds that never receive an ack? Is that a potential source of an unbounded map that doesn't get cleaned?

[hacheigriega]

Yes and yes, unfortunately. The ack message doesn't contain the symbol, so there is no way to avoid storing a symbol in priceFeedIds every time we send a subscription request.

But now that I think about it, we could handle this issue by handling the subscription failure errors we receive from the upstream. I will try that