[JS & Python] Add cancellation and structured error handling to live audio streaming

[rui-ren]

Summary

Adds cancellation and structured error handling to the JavaScript and Python SDKs' live audio transcription clients, closing two cross-language API parity gaps from the realtime-audio analysis.

What's new

Structured errors

Language	Surface
JS	New CoreError class code + isTransient from CoreErrorResponse exposed on every error thrown by start / pushLoop / stop. Re-exported from foundry-local-sdk.
Python	FoundryLocalException messages still parse via existing CoreErrorResponse.try_parse; sample now demonstrates how to use it for transient-error recovery.

Cancellation

Language	Surface
JS	audioClient.createLiveTranscriptionSession(signal?: AbortSignal) set once on the session, applies to every subsequent start / append / stop / getTranscriptionStream call. Mirrors C# CancellationToken shape.
Python	audio_client.create_live_transcription_session(cancel_event: Optional[threading.Event] = None) same pattern. Backpressured append() and idle get_transcription_stream() poll a documented _CANCEL_POLL_INTERVAL (100 ms) when cancellation is configured. Zero overhead on the fast path.

Both APIs are intentionally symmetric: cancellation is configured once on the constructor, not threaded through every method. Aborted append() does NOT enqueue the chunk (no risk of late delivery to native core). Aborted session also best-effort releases the native handle so it doesn't leak.

Files changed

File	Change
sdk/js/src/openai/liveAudioTranscriptionTypes.ts	New CoreError class + wrapCoreError() helper
sdk/js/src/openai/liveAudioTranscriptionClient.ts	AbortSignal plumbing, abort-aware AsyncQueue.write(), listener-leak fixes, native-handle release on abort
sdk/js/src/openai/audioClient.ts	createLiveTranscriptionSession(signal?)
sdk/js/src/index.ts	Re-exports CoreError
sdk/js/test/openai/liveAudioTranscription.test.ts	+CoreError tests, +AbortSignal listener-cleanup test (Proxy-wrapped to actually count listeners), +non-Error signal.reason test
samples/js/live-audio-transcription/app.js	Demonstrates CoreError + AbortSignal graceful shutdown
sdk/python/src/openai/audio_client.py	create_live_transcription_session(cancel_event=...)
sdk/python/src/openai/live_audio_transcription_client.py	cancel_event plumbing, _CANCEL_POLL_INTERVAL constant
sdk/python/test/openai/test_live_audio_transcription.py	+cancellation tests covering pre-set cancel, backpressure unblocking, generator clean exit
samples/python/live-audio-transcription/src/app.py	Demonstrates cancel_event + CoreErrorResponse.try_parse

Tests

• JS: 19/19 live-audio tests pass
• Python: 25/25 live-audio tests pass
• E2E synthetic-audio test still skips gracefully when the Nemotron model is not cached
• Both samples syntax-verified

Backwards compatibility

100% every new parameter is optional. Existing callers see no behavior change.

Out of scope (parity report items not in this PR)

• Missing id field in C++/Rust LiveAudioTranscriptionResponse addressed in [Rust & Python] Add id field to LiveAudioTranscriptionResponse #687 / [C++] Fix Nemotron-ASR of LiveAudioTranscriptionResponse #689.
• C++ polling vs. async streams different SDK; separate refactor.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	May 13, 2026 6:40pm

[Copilot]

Pull request overview

This PR improves the JS live audio transcription streaming API by adding (1) structured native-core error surfacing via a new CoreError type, and (2) optional cancellation via AbortSignal across streaming operations to close cross-language parity gaps.

Changes:

• Introduces CoreError + wrapCoreError() to parse and surface code / isTransient from native-core error payloads.
• Adds optional AbortSignal support to start(), append(), stop(), and getTranscriptionStream().
• Re-exports CoreError from the JS SDK entrypoint and adds unit tests for structured/unstructured wrapping behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
sdk/js/src/openai/liveAudioTranscriptionTypes.ts	Adds CoreError and wrapCoreError() to provide structured error handling for live audio streaming.
sdk/js/src/openai/liveAudioTranscriptionClient.ts	Adds AbortSignal support and switches several error paths to throw CoreError instead of plain Error.
sdk/js/src/index.ts	Re-exports CoreError from the public package entrypoint.
sdk/js/test/openai/liveAudioTranscription.test.ts	Adds unit tests covering CoreError wrapping behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kunal-vaishnavi]

Can you show how these changes would work in the JS example and how they are different than the below error handling?

Foundry-Local/samples/js/live-audio-transcription/app.js

Lines 69 to 72 in 58bba93

	} catch (err) {
	if (err.name !== 'AbortError') {
	console.error('Stream error:', err.message);
	}

[rui-ren]

Great question the existing catch block is fine for the happy path but throws away every signal we now expose. Pushed 83e8dc4 to the sample (samples/js/live-audio-transcription/app.js) so the diff is concrete:

Before (the lines you linked):

} catch (err) {
    if (err.name !== 'AbortError') {
        console.error('Stream error:', err.message);
    }
}

This treats every non-abort error the same and loses code + isTransient.

After:

} catch (err) {
    if (err.name === 'AbortError') return;
    if (err instanceof CoreError) {
        if (err.isTransient) {
            console.warn(` Transient ASR error (\): \. Continuing...`);
            return;  // recoverable  don't kill the app on a momentary native stall
        }
        console.error(` Stream error [\]: \`);
        return;
    }
    console.error(' Stream error:', err.message);
}

So the user-visible difference is:

• Transient blips (e.g., native-side recoverable hiccup) no longer silently log-and-die the loop continues, which is what a customer building a long-running transcription UI actually wants.
• Hard errors include the machine-readable code, so customers can route on it (telemetry, retry policy, UI message) instead of regex-matching err.message.

For AbortSignal, I also wired it through to show what it buys:

const shutdown = new AbortController();
await session.start(shutdown.signal);
for await (const r of session.getTranscriptionStream(shutdown.signal)) { ... }
await session.append(pcm, shutdown.signal);

process.on('SIGINT', async () => {
    shutdown.abort();              // unblocks any in-flight append/iterator
    if (audioInput) audioInput.quit();
    await session.stop();           // now drain-and-stop without racing
    ...
});

Previously, if session.append() was backpressured (waiting for native-core to drain), Ctrl+C had to wait for stop() to finish draining which could take seconds. With shutdown.abort(), the in-flight append resolves immediately with AbortError, the pump exits, and we shut down cleanly.

Let me know if you'd like the sample changes split out into a separate PR.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[rui-ren]

Both points addressed in df4260c.

JS session-level signal (no more passing it everywhere)

const shutdown = new AbortController();
const session = audioClient.createLiveTranscriptionSession({ signal: shutdown.signal });

await session.start();           // signal applies automatically
await session.append(pcm);       // signal applies automatically
for await (const r of session.getTranscriptionStream()) { ... }
await session.stop();

process.on('SIGINT', () => shutdown.abort());

Per-call signals still work as overrides if both session-level and per-call are set, they compose via AbortSignal.any so EITHER aborting cancels. Sample (samples/js/live-audio-transcription/app.js) now uses this one-line pattern.

Python cancellation parity (closes the gap from the analysis doc)

cancel = threading.Event()
signal.signal(signal.SIGINT, lambda *_: cancel.set())

session = audio_client.create_live_transcription_session(cancel_event=cancel)
session.start()
session.append(pcm)
for result in session.get_transcription_stream():
    ...
session.stop()

Same composition rule: each method also accepts an optional per-call cancel_event; either being set cancels. Backpressured append() and idle get_transcription_stream() poll on a 100 ms timeout when cancellation is configured; zero overhead on the fast path with no cancel sources. An aborted append() does NOT enqueue the chunk (no late delivery to native core).

Tests: JS 20/20, Python 26/26 (+4 new Python cancellation tests covering pre-set cancel, backpressure unblocking, and clean generator exit).

[kunal-vaishnavi]

Can we revert the session API changes to pass in cancel_event since we already have a cancel_event in the session constructor? We want JS and Python to have similar APIs for start, stop, append, etc.

[rui-ren]

Done in eee2cbe. Reverted the per-call signal / cancel_event parameters on both JS and Python so cancellation is configured once on the constructor and the method signatures stay symmetric.

JS start(), append(), stop(), getTranscriptionStream() now take no abort parameter; they read this.sessionSignal directly. Removed the resolveSignal / AbortSignal.any composition machinery.

Python same: start(), append(), stop(), get_transcription_stream() take no cancel_event. _is_cancelled() simplified to check only the session-level event.

Final API now matches across both languages:

const session = audioClient.createLiveTranscriptionSession({ signal: shutdown.signal });
await session.start();
await session.append(pcm);
for await (const r of session.getTranscriptionStream()) { ... }
await session.stop();

session = audio_client.create_live_transcription_session(cancel_event=cancel)
session.start()
session.append(pcm)
for r in session.get_transcription_stream(): ...
session.stop()

Tests: JS 19/19, Python 25/25 (-2 tests for the dropped per-call paths; backpressure-unblock test still validates the session-level cancel works).

[kunal-vaishnavi]

Instead of creating a new class to represent the parameters passed into the session, can we instead make an optional parameter for signal in createLiveTranscriptionSession? Then, we can remove the need for the LiveAudioTranscriptionSessionOptions class. The new optional parameter for signal can behave similar to a CancellationToken parameter.

[rui-ren]

sure, I will update the cancellation pattern. Thanks

[rui-ren]

Done in 6ab38c4. Dropped LiveAudioTranscriptionSessionOptions and made signal a plain optional parameter on createLiveTranscriptionSession, matching C#'s CancellationToken shape:

const shutdown = new AbortController();
const session = audioClient.createLiveTranscriptionSession(shutdown.signal);

(was createLiveTranscriptionSession({ signal: shutdown.signal }))

JS 19/19 tests pass; sample updated.

[kunal-vaishnavi]

Same question as here but for the Python example

[rui-ren]

Done in 143a8b6. Updated samples/python/live-audio-transcription/src/app.py with the same pattern as the JS sample concrete diff so the API value is visible.

Before: the sample created the session with no cancellation token and used a parallel local stop_event for the mic loop, which couldn't reach inside the SDK:

session = audio_client.create_live_transcription_session()
...
stop_event = threading.Event()
def shutdown(*_):
    stop_event.set()  # only the mic loop sees this
    session.stop()    # has to drain everything

After: one event drives the whole shutdown path SIGINT just sets it:

shutdown_event = threading.Event()
session = audio_client.create_live_transcription_session(cancel_event=shutdown_event)
...
def shutdown(*_):
    shutdown_event.set()
    #  this single call:
    #   - aborts any in-flight session.append() blocked on backpressure
    #     (FoundryLocalException, no late delivery to native core)
    #   - ends session.get_transcription_stream() iteration cleanly
    #   - short-circuits session.stop()'s drain wait
    #   - exits the mic capture loop
    session.stop()

I also added CoreErrorResponse.try_parse in the read loop to inspect structured error metadata (code + is_transient) transient blips no longer kill long-running sessions, mirroring the JS CoreError demo.

Tests: 25/25 pass; sample syntax-verified.

[baijumeswani]

Is my understanding correct that shutdown and session.stop do the same thing but shutdown is similar to session.stop(hard=true) so we avoid draining the queue? Should we use the same API to represent that, or do we need separate handlers for shutdown vs session.stop()?

[rui-ren]

They're not the same thing — they're orthogonal:

• shutdown.abort() cancels in-flight calls (e.g., append() blocked on backpressure, getTranscriptionStream() waiting for
  results).
• session.stop() signals end-of-audio to native core, drains the queue, and releases the native handle.

[baijumeswani]

Is it expected for FoundryLocalCore to throw transient errors? What do transient errors mean? If we detect transient errors, and there is no action that the application/user should take during such errors, can we catch them before they reach the application layer and continue as necessary?

[rui-ren]

You're right the transient branch was speculative (no documented list of transient codes from FLC core today, no SDK-internal retry policy yet). Pushed dfc9dee that drops the transient handling from both the JS and Python samples.

What changed:

• Removed the if (err.isTransient) ... branch from both samples
• Added a NOTE comment in each sample saying SDK-internal retry on transient errors is future work
• Samples still demonstrate the value of the structured error type customers can log/route on LiveAudioStreamError.code (JS) or CoreErrorResponse.try_parse(...).code (Python) instead of regex-matching err.message

What stayed:

• LiveAudioStreamError.isTransient and CoreErrorResponse.is_transient are still on the SDK API surface (no breaking change). Advanced users can still inspect them. We just don't lead the typical sample reader to expect recovery behavior the SDK doesn't deliver.

When the SDK gets a real retry policy + a documented set of transient codes, we can re-introduce internal handling and the field becomes meaningful end-to-end. Happy to file a follow-up issue tracking that work let me know.

We have decided that we don't need to merge this PR fix for the FL 1.1 release. We can wait to get this PR right before merging.