Spec for streaming inference

[drewoldag]

This PR puts together a spec file for "streaming inference". The gist is that we could load the model into memory once and process batches of data as they arrive with the expectation that there are potentially large gaps between the batches arriving.

Two additional things worth calling out that aren't mentioned in the spec:

1. It probably makes sense for there to be a timeout of some kind that would release the model from memory if a batch hasn't arrived in the last N seconds
2. We could probably turn infer into a special case of infer_streaming such that infer simply calls infer_streaming under the hood and provides all the data to it, like it normally would.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.94%. Comparing base (8e9231f) to head (d0fdac6).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #930   +/-   ##
=======================================
  Coverage   62.94%   62.94%           
=======================================
  Files          71       71           
  Lines        7357     7357           
=======================================
  Hits         4631     4631           
  Misses       2726     2726           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.