Frequent server_error from OpenAI Realtime API during active sessions

[sagorbrur]

Bug Description

We are frequently encountering session failures while using the OpenAI Realtime model (gpt-realtime) in our voice agent pipeline. The error occurs during active realtime sessions and is logged as a realtime_model_error with an underlying server_error returned by the OpenAI Realtime API.

Error Log (Sample)

Session error occurred: 
type='realtime_model_error'
label='livekit.plugins.openai.realtime.realtime_model.RealtimeModel'
error=APIError('OpenAI Realtime API response failed with error type: server_error')
recoverable=True

We are using the following way with different types of tools in agents.py

 turn_detection = build_turn_detection(vad_type="semantic_vad", eagerness="high")
                logger.info(f"Using VAD type: {turn_detection.type} for OpenAI Realtime model")

model = openai.realtime.RealtimeModel(
    model=llm_config.model_name,
    modalities=["text"],  # Force text output only for hybrid mode
    input_audio_transcription=input_audio_transcription,
    turn_detection=turn_detection,
)

Expected Behavior

It should work properly

Reproduction Steps

Using OpenAI Realtime Model should raise this error

Operating System

ubuntu 22.04

Models Used

Deepgram, OpenAI, ElevenLab

Package Versions

livekit-agents==1.3.10

Session/Room/Call IDs

No response

Proposed Solution

Additional Context

No response

Screenshots and Recordings

No response

[longcw]

you should be able to see the error details in RealtimeModelError.error.body, can you share more about that

[sagorbrur]

Hi @longcw ,
I didn't find this log from livekit-agents. Where should I search this?

you should be able to see the error details in RealtimeModelError.error.body, can you share more about that

[longcw]

@sagorbrur it's emitted by the AgentSession, maybe add a error handler to print the details with session.on("error", ...)

[sagorbrur]

HI @longcw ,
According to the livekit OpenAI Realtime default behaviour, if OpenAI Realtime fails, it should retry, right?
But in our cases, no retry happens after this error. Should we explicitly mention in realtime plugin integration parameter that it should retry?

Session error occurred: type='realtime_model_error' timestamp=1773046091.197551 label='livekit.plugins.openai.realtime.realtime_model.RealtimeModel' error=APIError('OpenAI Realtime API response failed with error type: server_error', body=Error(code=None, type='server_error', message='The server had an error while processing your request. Sorry about that! Please contact us through our help center at help.openai.com if the error persists. (include session ID in your message: sess_DHQZVHkJT8QwlFxOjy5wd). We recommend you retry your request.'), retryable=True) recoverable=True

[longcw]

it will restart the connection and session if the connection is closed or an error raised, but such single error message returned from the server will be ignored.

if you want to restart the connection on some error messages, you can listen to the error event of agent.realtime_llm_session and raise the exception from there.