Support OpenAI Realtime Whisper STT

[toubatbrian]

Summary

This PR adds realtime OpenAI STT support for gpt-realtime-whisper in openai.STT.

openai.STT now defaults to gpt-realtime-whisper with useRealtime: true. Because this model does not support OpenAI server-side turn_detection, callers must provide a VAD instance so the plugin can commit the audio buffer at end-of-speech.

Usage

const vad = await silero.VAD.load();

const session = new voice.AgentSession({
  vad,
  stt: new openai.STT({
    model: 'gpt-realtime-whisper',
    vad,
  }),
  // llm, tts, ...
});

If turnDetection is provided with gpt-realtime-whisper, the plugin warns and ignores it by normalizing to null.

Notes

• gpt-realtime-whisper uses the OpenAI Realtime transcription WebSocket path.
• WebSocket server events are parsed with Zod for type safety.
• useRealtime: false still uses the previous batch whisper-1 path.
• Tests cover realtime defaults, batch opt-out, required VAD behavior, and model-specific turnDetection handling.

[changeset-bot]

🦋 Changeset detected

Latest commit: 2ef35f0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 31 packages

Name	Type
@livekit/agents	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hedra	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch
@livekit/agents-plugin-xai	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[tinalenguyen]

should we match python's default of gpt-4o-mini-transcribe or standardize across both?

[toubatbrian]

Maybe let's update the default on both side since 'gpt-realtime-whisper' just came out recently and should be better than previous model?

[toubatbrian]

One thing to note is that 'gpt-realtime-whisper' does not support server-side VAD like previous model does. So I have to add a VAD option just like the way we did to mistral STT. Happy to make the same PR on python side once we merged this.

[tinalenguyen]

oh i didn't know it was new, yeah we should make that the default on both sides if it's better, sounds good!

[tinalenguyen]

tested it and it lgtm! one note is that in the mistral stt plugin, if a vad is not passed then one will be created by default, i think we could do that too. wdyt?

[toubatbrian]

tested it and it lgtm! one note is that in the mistral stt plugin, if a vad is not passed then one will be created by default, i think we could do that too. wdyt?

Sounds good! I'll merge this and create a follow up PR, just to make things separate and cleaner