feat: add realtime chat support with OpenAI and ElevenLabs adapters#300
Open
feat: add realtime chat support with OpenAI and ElevenLabs adapters#300
Conversation
# Conflicts: # packages/typescript/ai-openai/src/index.ts # pnpm-lock.yaml
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a provider-agnostic realtime voice chat system: shared realtime types and token helpers, a RealtimeClient, OpenAI and ElevenLabs adapters (token + client), a React useRealtimeChat hook, demo UI/components, and package manifests/docs for the new realtime feature. Changes
Sequence Diagram(s)sequenceDiagram
actor User
participant Browser as Browser (React)
participant Server as Server (token)
participant OpenAI as OpenAI Realtime
participant WebRTC as WebRTC
User->>Browser: open /realtime
Browser->>Server: request realtime token (provider, agentId?)
Server->>OpenAI: create session / get client_secret
OpenAI-->>Server: session + client_secret
Server-->>Browser: RealtimeToken (token, expiresAt, config)
Browser->>WebRTC: create RTCPeerConnection & offer
WebRTC->>OpenAI: send offer (via OpenAI endpoint)
OpenAI-->>WebRTC: answer
Browser->>WebRTC: attach mic, start audio capture
OpenAI->>Browser: audio + event stream (transcripts, tool calls)
Browser->>Browser: RealtimeClient maps events -> React UI
sequenceDiagram
actor User
participant Browser as Browser (React)
participant Server as Server (token)
participant ElevenLabs as ElevenLabs API/SDK
participant SDK as ElevenLabs SDK
User->>Browser: select ElevenLabs + agentId
Browser->>Server: request signed URL token
Server->>ElevenLabs: GET signed_url (agent_id, API key)
ElevenLabs-->>Server: signed_url
Server-->>Browser: RealtimeToken (signed_url, expiresAt)
Browser->>SDK: initialize Conversation with signed_url
SDK->>ElevenLabs: open WS/WebRTC
ElevenLabs-->>SDK: connected + events
SDK-->>Browser: onModeChange / transcript / message
Browser->>Browser: RealtimeClient updates UI
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Poem
🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Co-Authored-By: Warp <agent@warp.dev>
|
| Command | Status | Duration | Result |
|---|---|---|---|
nx affected --targets=test:sherif,test:knip,tes... |
❌ Failed | 3m 50s | View ↗ |
nx run-many --targets=build --exclude=examples/** |
✅ Succeeded | 1m 20s | View ↗ |
☁️ Nx Cloud last updated this comment at 2026-03-07 15:20:51 UTC
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 17
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
🟡 Minor comments (10)
packages/typescript/ai-elevenlabs/README.md-48-62 (1)
48-62:⚠️ Potential issue | 🟡 Minor
messagesis destructured but never used in the React example.
messagesis included in the destructuring on Line 48 but does not appear in the JSX. Either remove it to avoid misleading readers about the API surface, or add a minimal rendering example (e.g., a message list) to demonstrate its intended use.📝 Option A — remove unused destructuring
- const { status, mode, messages, connect, disconnect } = useRealtimeChat({ + const { status, mode, connect, disconnect } = useRealtimeChat({🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/README.md` around lines 48 - 62, The example destructures messages from useRealtimeChat (symbols: messages, useRealtimeChat, elevenlabsRealtime, connect, disconnect, status, mode) but never uses it; either remove messages from the destructuring to avoid implying it's used, or add a minimal rendering of it (e.g., map over messages to output a list of message.text or message.content) inside the JSX so the example demonstrates its intended use—update the return JSX accordingly and keep connect/disconnect and status/mode logic unchanged.packages/typescript/ai-elevenlabs/README.md-7-9 (1)
7-9:⚠️ Potential issue | 🟡 Minor
@tanstack/ai-reactis missing from the installation instructions.The React section (Lines 44–63) imports
useRealtimeChatfrom@tanstack/ai-react, but that package is not listed in thenpm installcommand. Users following the README sequentially will encounter a missing module error.📝 Proposed fix
-npm install `@tanstack/ai-elevenlabs` `@tanstack/ai` `@tanstack/ai-client` +npm install `@tanstack/ai-elevenlabs` `@tanstack/ai` `@tanstack/ai-client` `@tanstack/ai-react`🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/README.md` around lines 7 - 9, Update the installation command in README.md to include the missing React package: add `@tanstack/ai-react` to the npm install list so the React example that imports useRealtimeChat from `@tanstack/ai-react` will not fail; locate the npm install snippet in the README (the block that currently lists `@tanstack/ai-elevenlabs`, `@tanstack/ai`, `@tanstack/ai-client`) and append `@tanstack/ai-react` to that list.packages/typescript/ai-elevenlabs/package.json-39-39 (1)
39-39:⚠️ Potential issue | 🟡 Minor
test:libscript will run Vitest in watch mode — should bevitest run.The
ai-openaipackage uses"vitest run"for itstest:libscript, which exits after one pass. Using bare"vitest"here defaults to watch mode and will hang in CI.Proposed fix
- "test:lib": "vitest", + "test:lib": "vitest run",🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/package.json` at line 39, Update the package.json "test:lib" npm script so it runs Vitest in single-run mode instead of watch mode; locate the "test:lib" script entry and change its value from "vitest" to "vitest run" so CI will exit after one pass (match the ai-openai package pattern).packages/typescript/ai-elevenlabs/package.json-50-54 (1)
50-54:⚠️ Potential issue | 🟡 MinorMissing
vitedev dependency.The
buildscript usesvite build, butviteis not listed indevDependencies. Other packages likeai-openaiinclude it. Without it, the build will fail ifviteisn't hoisted from a sibling.Proposed fix
"devDependencies": { "@tanstack/ai": "workspace:*", "@tanstack/ai-client": "workspace:*", - "@vitest/coverage-v8": "4.0.14" + "@vitest/coverage-v8": "4.0.14", + "vite": "^7.2.7" }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/package.json` around lines 50 - 54, package.json's build script calls "vite build" but "vite" is missing from devDependencies; add "vite" to the devDependencies section (use the same version string used in sibling packages like ai-openai to keep hoisting consistent), then reinstall so "vite" is available for the build script; update package.json's "devDependencies" to include "vite" and run your package manager to restore node_modules.packages/typescript/ai-elevenlabs/src/realtime/token.ts-81-82 (1)
81-82:⚠️ Potential issue | 🟡 MinorValidate
signed_urlbefore using it — silentundefinedtoken if the response shape changes.
response.json()returnsany; if the ElevenLabs API returns a 2xx with an unexpected shape (field rename, error envelope, rate-limit wrapper),signedUrlisundefinedand propagates astoken: undefinedwithout an immediate, readable error.🔧 Proposed fix
- const data = await response.json() - const signedUrl = data.signed_url as string + const data = await response.json() + const signedUrl: string | undefined = data?.signed_url + if (!signedUrl) { + throw new Error( + `ElevenLabs signed URL response missing 'signed_url' field: ${JSON.stringify(data)}`, + ) + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/src/realtime/token.ts` around lines 81 - 82, Validate that data.signed_url is present and a non-empty string before assigning to signedUrl and using it as token: after const data = await response.json() check typeof data?.signed_url === 'string' && data.signed_url.trim() !== '' and if not, throw or return a clear error (include response.status and JSON body) so callers don't receive a silent undefined token; update the code path that builds the token to rely on this validated signedUrl variable.docs/architecture/realtime-chat.md-19-63 (1)
19-63:⚠️ Potential issue | 🟡 MinorAdd a language specifier to the fenced code block.
The ASCII architecture diagram code block lacks a language identifier, which triggers a markdownlint warning (MD040). Use
textorplaintextto satisfy the linter.-``` +```text ┌─────────────────────────────────────────────────────────────────┐🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/architecture/realtime-chat.md` around lines 19 - 63, The fenced ASCII diagram block (the triple backticks before the diagram showing "React Application", "RealtimeClient", "Provider Adapters", etc.) is missing a language specifier and triggers MD040; update the opening fence from ``` to ```text (or ```plaintext) so the diagram is treated as plain text by the linter and the MD040 warning is resolved.packages/typescript/ai-openai/src/realtime/token.ts-1-7 (1)
1-7:⚠️ Potential issue | 🟡 MinorFix import order per ESLint rule.
ESLint reports that
../utilsshould be imported before the type import of@tanstack/ai.Proposed fix
+import { getOpenAIApiKeyFromEnv } from '../utils' import type { RealtimeToken, RealtimeTokenAdapter, Tool } from '@tanstack/ai' -import { getOpenAIApiKeyFromEnv } from '../utils' import type {🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/token.ts` around lines 1 - 7, Reorder the top-level imports so that the runtime import getOpenAIApiKeyFromEnv from '../utils' appears before the type-only import from '@tanstack/ai'; specifically move the import of getOpenAIApiKeyFromEnv to come before the import type { RealtimeToken, RealtimeTokenAdapter, Tool } line in this file (token.ts) and keep the OpenAIRealtimeModel/OpenAIRealtimeSessionResponse/OpenAIRealtimeTokenOptions type imports unchanged.examples/ts-react-chat/src/routes/api.realtime-token.ts-5-11 (1)
5-11:⚠️ Potential issue | 🟡 MinorFix import ordering per ESLint rules.
ESLint reports two issues: members in the
@/lib/realtime-toolsimport should be sorted alphabetically, and thezodimport should precede@/lib/realtime-tools.Proposed fix
+import * as z from 'zod' import { getCurrentTimeToolDef, getWeatherToolDef, - setReminderToolDef, searchKnowledgeToolDef, + setReminderToolDef, } from '@/lib/realtime-tools' -import * as z from 'zod'🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/ts-react-chat/src/routes/api.realtime-token.ts` around lines 5 - 11, Move the external import for zod above the local module and alphabetize the named exports from '@/lib/realtime-tools'; specifically place "import * as z from 'zod'" before the other import and reorder the members to: getCurrentTimeToolDef, getWeatherToolDef, searchKnowledgeToolDef, setReminderToolDef so the import list is sorted per ESLint.packages/typescript/ai-openai/src/realtime/adapter.ts-124-130 (1)
124-130:⚠️ Potential issue | 🟡 MinorRemove debug
console.logstatements before merging.Multiple
console.logcalls are left in (lines 125, 128, 178, 180, 514). These will pollute the browser console for every user of the library. Gate them behind thedebugoption or remove them entirely.Also applies to: 178-180, 512-514
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 124 - 130, Remove or gate the stray console.log calls in the realtime adapter: locate the pc.ontrack handler and other places where console.log is used (e.g., the lines printing '[Realtime] ontrack event:', output analyser creation, and the other occurrences around the 178-180 and 512-514 regions) and either delete those console.log statements or wrap them with the library's debug flag (e.g., if (options?.debug) { ... }) or use the existing logger; ensure references like pc.ontrack, setupOutputAudioAnalysis and outputAnalyser remain unchanged and only the logging behavior is modified.packages/typescript/ai-openai/src/realtime/adapter.ts-378-405 (1)
378-405:⚠️ Potential issue | 🟡 Minor
disconnectdoes not null out analyser/source nodes, risking use-after-close.After
audioContext.close()at line 400,inputAnalyser,outputAnalyser,inputSource, andoutputSourcestill reference closed nodes. IfgetAudioVisualization()getters are called after disconnect (e.g., due to a pending animation frame), calling methods on these closed nodes could throw.Null out audio nodes on disconnect
if (audioContext) { await audioContext.close() audioContext = null + inputAnalyser = null + outputAnalyser = null + inputSource = null + outputSource = null }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 378 - 405, The disconnect() method leaves references to closed audio nodes which can cause use-after-close errors; after awaiting audioContext.close() in disconnect(), explicitly set inputAnalyser, outputAnalyser, inputSource, and outputSource to null (and any related visualization state used by getAudioVisualization()) and ensure any pending animation frames/handlers that may reference them are cancelled or guarded so subsequent calls cannot invoke methods on closed nodes; update the disconnect() implementation to null out these symbols (inputAnalyser, outputAnalyser, inputSource, outputSource, and any visualizer handles) immediately after closing audioContext.
🧹 Nitpick comments (14)
packages/typescript/ai-elevenlabs/README.md (1)
65-72: Consider documenting browser/microphone prerequisites.The package targets realtime voice conversations over WebRTC/WebSocket, which requires microphone access and a browser environment. The Requirements section currently only covers the ElevenLabs account and agent ID; a note about browser microphone permissions (and that this is a browser-only API) would help developers avoid unexpected failures, especially in SSR contexts.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/README.md` around lines 65 - 72, Update the README's Requirements section (near the "## Environment Variables" and "## Requirements" headings) to include a short note that this package is browser-only and requires microphone access/permissions for realtime WebRTC/WebSocket voice conversations; mention that ELEVENLABS_API_KEY is for server-side token generation and warn about SSR contexts where microphone/browser APIs are unavailable to prevent unexpected runtime failures.packages/typescript/ai-elevenlabs/vite.config.ts (2)
18-18:'node_modules/'incoverage.excludeis redundant.The v8 coverage provider already excludes
node_modulesby default.♻️ Proposed cleanup
exclude: [ - 'node_modules/', 'dist/',🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/vite.config.ts` at line 18, Remove the redundant 'node_modules/' entry from the coverage.exclude array in vite.config.ts (the array used to configure the v8 coverage provider); locate the coverage config (symbol: coverage.exclude) and delete that string element so the default v8 behavior remains and the exclusion list is not duplicated.
8-8:dir: './'is redundant.
'./'is the default when the config file lives at the project root. Removing it avoids confusion about whether it's intentionally overriding a parent-level setting.♻️ Proposed cleanup
test: { name: packageJson.name, - dir: './', watch: false,🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/vite.config.ts` at line 8, Remove the redundant property "dir: './'" from the Vite config (in packages/typescript/ai-elevenlabs/vite.config.ts) — locate the object that contains dir: './' and delete that key so the config uses the default project-root directory behavior; no other changes needed.packages/typescript/ai-client/src/realtime-client.ts (2)
314-336: Token refresh could spin ifgetTokenreturns already-near-expired tokens.If
getToken()consistently returns tokens withexpiresAtless thanTOKEN_REFRESH_BUFFER_MSin the future,refreshInis0andscheduleTokenRefresh→refreshToken→scheduleTokenRefreshcreates a tight loop of immediate refreshes.Consider adding a minimum delay or a retry-backoff to guard against this.
Proposed fix
+ const MIN_REFRESH_INTERVAL_MS = 5_000 + private scheduleTokenRefresh(): void { if (!this.token) return const timeUntilExpiry = this.token.expiresAt - Date.now() - const refreshIn = Math.max(0, timeUntilExpiry - TOKEN_REFRESH_BUFFER_MS) + const refreshIn = Math.max(MIN_REFRESH_INTERVAL_MS, timeUntilExpiry - TOKEN_REFRESH_BUFFER_MS) this.tokenRefreshTimeout = setTimeout(() => { this.refreshToken() }, refreshIn) }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-client/src/realtime-client.ts` around lines 314 - 336, The token refresh can spin if getToken() returns tokens already expiring within TOKEN_REFRESH_BUFFER_MS; modify scheduleTokenRefresh() and refreshToken() to enforce a minimum delay/backoff: when computing refreshIn (currently using token.expiresAt - Date.now() - TOKEN_REFRESH_BUFFER_MS) clamp it with a MIN_REFRESH_INTERVAL_MS (e.g., a few seconds) and a MAX_REFRESH_INTERVAL_MS if desired, and when refreshToken() catches an error or receives a token that would schedule immediate refresh (refreshIn <= 0), schedule the next attempt using an increasing retryBackoff (exponential or capped linear) stored on the instance (reset on success). Update uses of tokenRefreshTimeout, scheduleTokenRefresh, refreshToken, TOKEN_REFRESH_BUFFER_MS, and getToken to implement this guard.
63-63: Remove debugconsole.logstatements from library code.Lines 63, 105, and 254 contain debug logging that will appear in every consumer's console. Library packages should either omit these or use a conditional debug logger.
Proposed fix
- console.log(`[RealtimeClient #${this.instanceId}] Created`) ... - console.log(`[RealtimeClient #${this.instanceId}] Connection established:`, !!this.connection) ... - console.log(`[RealtimeClient #${this.instanceId}] audio getter, connection:`, !!this.connection)Also applies to: 105-105, 254-254
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-client/src/realtime-client.ts` at line 63, Remove the stray debug console.log statements from the RealtimeClient class (the creation log that prints `[RealtimeClient #${this.instanceId}] Created` and the other console.log calls around the same class at the other locations). Replace them with either no-op removal or routed debug logging using the module's logger/debug facility (e.g., use an existing logger on the class or a conditional debug utility tied to env/flag) so library consumers don't get unsolicited console output; update references inside the RealtimeClient constructor/initialization and the methods where those console.logs appear to call the chosen logger (or remove entirely) instead of console.log.examples/ts-react-chat/package.json (1)
20-20: Dependency is out of alphabetical order.
@tanstack/ai-elevenlabsshould be placed between@tanstack/ai-clientand@tanstack/ai-geminito maintain the sorted order of the other@tanstackdependencies.Proposed fix
"@tanstack/ai-client": "workspace:*", + "@tanstack/ai-elevenlabs": "workspace:*", "@tanstack/ai-gemini": "workspace:*", "@tanstack/ai-grok": "workspace:*", "@tanstack/ai-ollama": "workspace:*", "@tanstack/ai-openai": "workspace:*", - "@tanstack/ai-elevenlabs": "workspace:*", "@tanstack/ai-openrouter": "workspace:*",🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/ts-react-chat/package.json` at line 20, The dependency "@tanstack/ai-elevenlabs" is out of alphabetical order in package.json; move the line with "@tanstack/ai-elevenlabs": "workspace:*" so it appears between "@tanstack/ai-client" and "@tanstack/ai-gemini" to keep all "@tanstack" entries sorted; update the dependencies block in package.json accordingly and run a quick yarn/npm install to ensure lockfile consistency.packages/typescript/ai/src/index.ts (1)
82-104: Consider a dedicated/realtimesubpath export to keep the main barrel lean.All 20 realtime symbols (including the runtime
realtimeTokenfunction) are added to the root entry point. Consumers who never use realtime features still have to parse/consider these exports; tree-shakers help for values but some bundlers treat the whole barrel as one chunk. A@tanstack/ai/realtimesubpath (viapackage.jsonexports) would let realtime-only code be cleanly separated, consistent with the existing/adapterspattern in the monorepo.Based on learnings: "Applies to
packages/typescript/*/src/index.ts: Export tree-shakeable adapters with clear subpath exports inpackage.json(e.g.,@tanstack/ai/adapters,@tanstack/ai-openai/adapters) to minimize bundle size."🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai/src/index.ts` around lines 82 - 104, Remove the realtime exports from the root barrel (the list starting with realtimeToken and the Realtime* types in packages/typescript/ai/src/index.ts) and re-export them from a dedicated subpath; create a new entry re-export file (e.g., packages/typescript/ai/src/realtime/index.ts) that exports realtimeToken and all Realtime* types, then update package.json "exports" to add a "./realtime" subpath pointing to the built realtime entry so consumers can import "@tanstack/ai/realtime"; ensure the build output includes the new realtime entry so the new subpath resolves at runtime.examples/ts-react-chat/src/routes/realtime.tsx (1)
134-138: Variableerrorshadows the outererrorfromuseRealtimeChat.The
errordeclared in the catch block at line 136 shadows the destructurederrorfromuseRealtimeChatat line 116. This works due to block scoping but harms readability. Consider renaming toerrBodyorerrorData.Proposed fix
if (!response.ok) { - const error = await response.json() - throw new Error(error.error || 'Failed to get token') + const errBody = await response.json() + throw new Error(errBody.error || 'Failed to get token') }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/ts-react-chat/src/routes/realtime.tsx` around lines 134 - 138, The response error variable shadows the outer `error` from `useRealtimeChat`; in the fetch failure branch inside the token retrieval logic rename the local `error` to a non-conflicting name like `errBody` or `errorData` (where `const error = await response.json()` is declared) and update the subsequent throw to use that new name (e.g., `throw new Error(errorData.error || 'Failed to get token')`) so `useRealtimeChat`'s `error` remains unshadowed and readability is preserved.packages/typescript/ai-elevenlabs/src/realtime/adapter.ts (2)
58-59: Shared mutableUint8Arrayfallback buffers can be corrupted by callers.
emptyFrequencyDataandemptyTimeDomainDataare allocated once and returned by reference fromgetAudioVisualization(). If any consumer writes to the returned array (e.g., applying a transform in-place), subsequent callers will see the mutated data. The same pattern exists in the OpenAI adapter.A low-risk concern for now, but worth noting. A defensive approach would be to return a new copy each time, or document that the returned arrays must not be mutated.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/src/realtime/adapter.ts` around lines 58 - 59, The shared fallback buffers emptyFrequencyData and emptyTimeDomainData used by getAudioVisualization() are returned by reference and can be mutated by callers; change getAudioVisualization() to return fresh copies instead of the shared Uint8Array instances (e.g., construct a new Uint8Array from the fallback buffer or use .slice()) so each caller receives an independent buffer, and apply the same defensive copying approach in the OpenAI adapter where the same pattern exists.
148-168: Stub methods silently degrade — consider throwing or returning a sentinel to surface unsupported operations.
sendText,sendToolResult, andupdateSessionallconsole.warnbut otherwise succeed silently. A caller using the genericRealtimeConnectioninterface has no programmatic way to know the operation was a no-op. If these are genuinely unsupported, throwing a clear error (or returning afalse/Promise.reject) would prevent subtle bugs where the caller assumes the action succeeded.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/src/realtime/adapter.ts` around lines 148 - 168, These stub methods (sendText, sendToolResult, updateSession) currently only console.warn and silently succeed; update them to surface unsupported operations to callers by throwing a clear Error (e.g., new Error('ElevenLabs realtime does not support sendText')) or returning a rejected Promise with that Error to match the method's async contract; ensure the behavior is consistent with the RealtimeConnection interface (adjust signatures to return Promise.reject if these are async) and include descriptive messages that reference the operation name so callers can programmatically detect failures.packages/typescript/ai-react/src/use-realtime-chat.ts (2)
200-206:setVADModeonly updates local state — the TODO for propagating to the session is still open.Calling
setVADModewhile connected will show the new mode in the UI but won't actually change the server-side VAD behavior, leading to a confusing discrepancy.Would you like me to open an issue to track implementing the session update propagation?
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 200 - 206, setVADMode currently only calls setVADModeState and doesn't propagate the change to the active session, causing UI/server mismatch; modify the useCallback to include relevant dependencies (e.g., session and isConnected) and, when connected, call the session API to update the server-side config (for example session.updateConfig or session.send/update method to set vadMode to newMode) before/after calling setVADModeState, handling errors appropriately so the server and UI stay in sync.
110-114: Explicitly capture and unsubscribe fromonStateChangesubscription for better cleanup handling.The return value from
onStateChange()is discarded at lines 110-114. While the current code is safe becausedestroy()clears all state change callbacks on unmount, explicitly storing and calling the unsubscribe function provides more robust cleanup:const unsubscribe = clientRef.current.onStateChange((state) => { setPendingUserTranscript(state.pendingUserTranscript) setPendingAssistantTranscript(state.pendingAssistantTranscript) })Then clean it up in the unmount effect or store it for explicit cleanup. This makes the subscription lifecycle explicit and less dependent on the internal implementation of
destroy().🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 110 - 114, The onStateChange subscription return value is discarded; explicitly capture and unsubscribe it to ensure proper cleanup: when calling clientRef.current.onStateChange(...) assign the returned unsubscribe function to a variable (e.g., unsubscribe) and then call that unsubscribe during the component cleanup (or when calling destroy()), ensuring you still update setPendingUserTranscript and setPendingAssistantTranscript from the callback and not rely solely on clientRef.current.destroy() to clear callbacks.packages/typescript/ai-openai/src/realtime/adapter.ts (2)
516-537:calculateLevelallocates a newUint8Array(fftSize)on every invocation — hot-path allocation concern.
calculateLevelis called via theinputLevelandoutputLevelgetters, which the React hook polls on every animation frame (~60 fps). Each call allocates anew Uint8Array(2048), creating ~240 KB/s of short-lived garbage (2 analyzers × 2048 bytes × 60 fps). The frequency and time-domain methods (lines 550–576) have the same pattern.Consider pre-allocating reusable buffers:
Pre-allocate buffers for visualization
+ // Pre-allocated buffers for visualization (avoid GC pressure in hot path) + const inputLevelBuffer = new Uint8Array(2048) + const outputLevelBuffer = new Uint8Array(2048) + function calculateLevel(analyser: AnalyserNode): number { - const data = new Uint8Array(analyser.fftSize) + const data = analyser === inputAnalyser ? inputLevelBuffer : outputLevelBuffer analyser.getByteTimeDomainData(data)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 516 - 537, The calculateLevel function is allocating a new Uint8Array on every frame; fix by pre-allocating and reusing buffers per AnalyserNode instead of creating new arrays each call: add a module-scoped Map<AnalyserNode, Uint8Array> (or attach a Symbol-keyed property) to cache a buffer for each analyser, on first use create Uint8Array(analyser.fftSize) and store it, then in calculateLevel (and the frequency/time-domain helper methods used by the inputLevel/outputLevel getters) retrieve and reuse that cached buffer for analyser.getByteTimeDomainData / getByteFrequencyData to eliminate per-frame allocations. Ensure buffer size matches analyser.fftSize if it can change (recreate and replace cache entry when size differs).
260-299:response.donehandler doesn't handle missingresponse.outputgracefully.At line 267,
response.outputis cast toArray<...>without a null check. The fallbackoutput || []on line 277 only guards the iteration, but ifresponseitself is undefined/null (e.g., malformed server event), line 266 will throw. Also, if there arefunction_calltype items in output, they're silently skipped — consider whether tool-call parts should be included in the message.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 260 - 299, The handler for 'response.done' accesses event.response.output without guarding against a missing response; update the block around currentMode/emit/currentMessageId to safely handle missing or non-array data by null-checking event.response (or defaulting to an empty object) and validating response.output as an array before iterating (e.g., const response = event.response ?? {}; const output = Array.isArray(response.output) ? response.output : []), and while extracting parts extend the item-type handling to also capture 'function_call' (or tool-call) items into message.parts (e.g., add a branch for item.type === 'function_call' that pushes a part with a clear type/name/arguments) so tool calls aren't silently skipped; leave emit('message_complete', { message }) and currentMessageId reset logic unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@examples/ts-react-chat/src/routes/realtime.tsx`:
- Line 108: The adapter variable (created with openaiRealtime() or
elevenlabsRealtime()) is recreated on every render; memoize it (e.g., with
React's useMemo or a useRef + effect) so it only changes when provider changes,
and ensure the RealtimeClient that consumes it is recreated or reconnected when
that memoized adapter changes. Locate references to adapter, provider,
openaiRealtime, elevenlabsRealtime, and the RealtimeClient instantiation and
update them so the adapter is stable across renders and is replaced only when
provider changes, triggering a clean disconnect/reconnect of the RealtimeClient.
In `@packages/typescript/ai-client/src/realtime-client.ts`:
- Around line 1-8: There are two separate type imports from the same module
(`@tanstack/ai`) — one importing AudioVisualization, RealtimeMessage,
RealtimeMode, RealtimeStatus, RealtimeToken and another importing AnyClientTool
— which triggers import/no-duplicates; fix it by merging them into a single
type-only import (e.g., import type { AudioVisualization, RealtimeMessage,
RealtimeMode, RealtimeStatus, RealtimeToken, AnyClientTool } from
'@tanstack/ai') and remove the duplicate import statement so all types come from
one import declaration.
- Around line 383-402: The handler registered in the
this.connection.on('tool_call', ...) callback currently ignores requests when
this.clientTools.get(toolName) returns undefined, causing the provider to wait
forever; update the callback so that if the tool is not found (i.e., tool is
falsy or lacks an execute method) you immediately call
this.connection?.sendToolResult(toolCallId, JSON.stringify({ error: `tool
"${toolName}" not registered` })) (and optionally log via processLogger or
this.logger) before returning, otherwise proceed to call tool.execute as before;
ensure you reference toolCallId, toolName, this.clientTools.get, and
sendToolResult in the change.
- Around line 281-284: The destroy() method currently calls async disconnect()
without awaiting, causing stateChangeCallbacks.clear() to run before disconnect
completes; change destroy to await the disconnect call (e.g., make destroy async
and await this.disconnect()) so updateState and any final `'idle'` notifications
run before clearing; if you cannot change the API to async, instead chain the
Promise (return this.disconnect().finally(() =>
this.stateChangeCallbacks.clear())) or at minimum call
this.disconnect().catch(() => {}).finally(() =>
this.stateChangeCallbacks.clear()) to avoid unhandled rejections and ensure
cleanup happens after disconnect; reference methods: destroy(), disconnect(),
updateState, stateChangeCallbacks, and connection.disconnect().
- Around line 163-169: In startListening(), await the Promise returned by
this.connection.startAudioCapture() and only call this.updateState({ mode:
'listening' }) after the await succeeds; if startAudioCapture() rejects, catch
the error (e.g., in a try/catch), log or surface it via your logger or
this.updateState to reflect failure, and avoid flipping the mode to 'listening'
on error — modify the startListening() implementation to use async/await with
try/catch around this.connection.startAudioCapture(), referencing
startListening(), this.connection.startAudioCapture(), and this.updateState().
In `@packages/typescript/ai-client/src/realtime-types.ts`:
- Around line 1-11: Merge the two imports from '@tanstack/ai' into a single
import statement: include AudioVisualization, RealtimeEvent,
RealtimeEventHandler, RealtimeMessage, RealtimeMode, RealtimeSessionConfig,
RealtimeStatus, RealtimeToken and AnyClientTool together in one import to
resolve the import/no-duplicates ESLint error (locate the current separate
imports in realtime-types.ts and combine them into one).
- Around line 21-72: Change all interface method signatures in RealtimeAdapter
and RealtimeConnection to function property signatures (e.g., connect: (token:
RealtimeToken) => Promise<RealtimeConnection>) instead of shorthand method
forms; update every lifecycle, audio I/O, text input, tool result, session
management, events, and audio visualization members (disconnect,
startAudioCapture, stopAudioCapture, sendText, sendToolResult, updateSession,
interrupt, on, getAudioVisualization) to the form methodName: (args...) =>
ReturnType so the types are checked as function properties and satisfy
`@typescript-eslint/method-signature-style`.
In `@packages/typescript/ai-elevenlabs/src/realtime/adapter.ts`:
- Around line 79-125: The initial emits inside the Conversation.startSession
callbacks (specifically the onConnect handler that calls emit('status_change', {
status: 'connected' }) and emit('mode_change', { mode: 'listening' })) run
synchronously during await and get dropped before the caller can register
handlers; wrap those initial emit calls in queueMicrotask() (or equivalent
microtask deferral) inside the onConnect callback so the connection object is
returned and handlers can be attached before the events fire; update the
onConnect handler in the Conversation.startSession options to defer the
emit(...) calls via queueMicrotask while leaving other callbacks
(onDisconnect/onModeChange/onMessage/onError) unchanged.
In `@packages/typescript/ai-elevenlabs/src/realtime/token.ts`:
- Around line 9-29: getElevenLabsApiKey currently reads from window.env which
can leak secrets if bundled to the browser; remove the browser fallback so
getElevenLabsApiKey only reads server-safe sources (process.env or explicit
injection), and update elevenlabsRealtimeToken to accept an explicit apiKey
(e.g., via ElevenLabsRealtimeTokenOptions) or use a safe runtime global like
globalThis.ELEVENLABS_API_KEY for edge runtimes; ensure getElevenLabsApiKey
throws the same clear error when no server-side key is provided and update
callers (elevenlabsRealtimeToken) to pass the api key explicitly rather than
relying on window.env.
In `@packages/typescript/ai-openai/src/index.ts`:
- Around line 103-121: The package is missing dedicated subpath exports for the
realtime adapter (and other adapters) so tree-shaking fails; add subpath export
entries in package.json for "./realtime" (and similarly for "./text",
"./summarize", "./image", "./video", "./tts", "./transcription") that map
import/require/types to the built outputs (e.g., "import":
"./dist/esm/realtime/index.js", "require": "./dist/cjs/realtime/index.js",
"types": "./dist/types/realtime/index.d.ts") so the top-level exported symbols
openaiRealtimeToken, openaiRealtime and the exported types OpenAIRealtimeVoice,
OpenAIRealtimeModel, OpenAIRealtimeTokenOptions, OpenAIRealtimeOptions,
OpenAITurnDetection, OpenAISemanticVADConfig, OpenAIServerVADConfig resolve via
the "./realtime" subpath and become tree-shakeable.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 132-180: Wrap the SDP fetch/answer negotiation (the fetch to
OPENAI_REALTIME_URL, response handling, and pc.setRemoteDescription) in a
try/catch/finally so that if any of fetch, awaiting the response, or
pc.setRemoteDescription fails you explicitly clean up resources: stop all tracks
on localStream (localStream.getTracks().forEach(t => t.stop())), close the
RTCPeerConnection instance pc (pc.close()), close the dataChannel if open
(dataChannel.close()), and tear down any input audio analysis state created by
setupInputAudioAnalysis/inputAnalyser (undo whatever setupInputAudioAnalysis
does or call a new cleanup function). Ensure
setupInputAudioAnalysis(localStream) and the console logs only run after
successful negotiation (or move setup into the try after successful
setRemoteDescription), and rethrow the original error after cleanup so callers
still receive the failure.
In `@packages/typescript/ai-openai/src/realtime/types.ts`:
- Around line 19-23: Replace the restricted preview-only union in
OpenAIRealtimeModel with the new production names and a flexible string pattern:
update the OpenAIRealtimeModel type to include 'gpt-realtime' and
'gpt-realtime-mini' (instead of the four preview names) and add the permissive
fallback (string & {}) so unknown/future model identifiers are accepted while
preserving IDE autocomplete for known models; modify the type definition for
OpenAIRealtimeModel accordingly.
In `@packages/typescript/ai-react/src/realtime-types.ts`:
- Around line 1-8: There are two separate imports from '@tanstack/ai' causing an
import/no-duplicates ESLint error; merge AnyClientTool into the existing import
that brings in RealtimeMessage, RealtimeMode, RealtimeStatus, and RealtimeToken
so all types from '@tanstack/ai' are imported in one statement (leave the
separate import type { RealtimeAdapter } from '@tanstack/ai-client' as-is).
Update the import that references RealtimeMessage, RealtimeMode, RealtimeStatus,
RealtimeToken to also include AnyClientTool.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts`:
- Around line 120-139: The updateLevels loop in useEffect calls
setInputLevel/setOutputLevel every requestAnimationFrame causing ~60fps
re-renders; modify updateLevels to throttle updates to ~15fps (or ~66ms) and
only call setInputLevel/setOutputLevel when the new value differs by a small
threshold (e.g., delta > 0.01) to avoid unnecessary renders. Concretely, inside
the updateLevels function (which uses animationFrameRef and clientRef), track
the last update timestamp (or frame count) and only perform the React state
updates when enough time has elapsed, and compute a small difference check
against previousLevel refs before calling setInputLevel/setOutputLevel; keep the
existing cleanup of animationFrameRef and the status === 'connected' start
condition.
- Around line 74-115: The RealtimeClient is being created with getToken,
adapter, and tools captured once (optionsRef.current.getToken,
optionsRef.current.adapter, optionsRef.current.tools) which freezes stale
values; change the RealtimeClient construction so those fields are passed as
thin wrapper functions that read from optionsRef.current at invocation time
(e.g., getToken: (...args) => optionsRef.current.getToken?.(...args), adapter:
(...args) => optionsRef.current.adapter?.(...args) or adapter: () =>
optionsRef.current.adapter, and similarly for tools) so the client uses
up-to-date values from optionsRef when methods are called; keep existing
callbacks (onMessage/onError/etc.), clientRef.current and the onStateChange
subscription unchanged.
In `@packages/typescript/ai/src/realtime/types.ts`:
- Around line 56-61: Convert all shorthand interface method signatures to
function-property form: in RealtimeTokenAdapter change "generateToken():
Promise<RealtimeToken>" to a property-style signature like "generateToken: () =>
Promise<RealtimeToken>"; do the same for every shorthand method in the
AudioVisualization interface (replace e.g. "start(): void" with "start: () =>
void", "stop(): void" with "stop: () => void", etc.), ensuring return types and
parameter lists are preserved exactly.
---
Nitpick comments:
In `@examples/ts-react-chat/package.json`:
- Line 20: The dependency "@tanstack/ai-elevenlabs" is out of alphabetical order
in package.json; move the line with "@tanstack/ai-elevenlabs": "workspace:*" so
it appears between "@tanstack/ai-client" and "@tanstack/ai-gemini" to keep all
"@tanstack" entries sorted; update the dependencies block in package.json
accordingly and run a quick yarn/npm install to ensure lockfile consistency.
In `@examples/ts-react-chat/src/routes/realtime.tsx`:
- Around line 134-138: The response error variable shadows the outer `error`
from `useRealtimeChat`; in the fetch failure branch inside the token retrieval
logic rename the local `error` to a non-conflicting name like `errBody` or
`errorData` (where `const error = await response.json()` is declared) and update
the subsequent throw to use that new name (e.g., `throw new
Error(errorData.error || 'Failed to get token')`) so `useRealtimeChat`'s `error`
remains unshadowed and readability is preserved.
In `@packages/typescript/ai-client/src/realtime-client.ts`:
- Around line 314-336: The token refresh can spin if getToken() returns tokens
already expiring within TOKEN_REFRESH_BUFFER_MS; modify scheduleTokenRefresh()
and refreshToken() to enforce a minimum delay/backoff: when computing refreshIn
(currently using token.expiresAt - Date.now() - TOKEN_REFRESH_BUFFER_MS) clamp
it with a MIN_REFRESH_INTERVAL_MS (e.g., a few seconds) and a
MAX_REFRESH_INTERVAL_MS if desired, and when refreshToken() catches an error or
receives a token that would schedule immediate refresh (refreshIn <= 0),
schedule the next attempt using an increasing retryBackoff (exponential or
capped linear) stored on the instance (reset on success). Update uses of
tokenRefreshTimeout, scheduleTokenRefresh, refreshToken,
TOKEN_REFRESH_BUFFER_MS, and getToken to implement this guard.
- Line 63: Remove the stray debug console.log statements from the RealtimeClient
class (the creation log that prints `[RealtimeClient #${this.instanceId}]
Created` and the other console.log calls around the same class at the other
locations). Replace them with either no-op removal or routed debug logging using
the module's logger/debug facility (e.g., use an existing logger on the class or
a conditional debug utility tied to env/flag) so library consumers don't get
unsolicited console output; update references inside the RealtimeClient
constructor/initialization and the methods where those console.logs appear to
call the chosen logger (or remove entirely) instead of console.log.
In `@packages/typescript/ai-elevenlabs/README.md`:
- Around line 65-72: Update the README's Requirements section (near the "##
Environment Variables" and "## Requirements" headings) to include a short note
that this package is browser-only and requires microphone access/permissions for
realtime WebRTC/WebSocket voice conversations; mention that ELEVENLABS_API_KEY
is for server-side token generation and warn about SSR contexts where
microphone/browser APIs are unavailable to prevent unexpected runtime failures.
In `@packages/typescript/ai-elevenlabs/src/realtime/adapter.ts`:
- Around line 58-59: The shared fallback buffers emptyFrequencyData and
emptyTimeDomainData used by getAudioVisualization() are returned by reference
and can be mutated by callers; change getAudioVisualization() to return fresh
copies instead of the shared Uint8Array instances (e.g., construct a new
Uint8Array from the fallback buffer or use .slice()) so each caller receives an
independent buffer, and apply the same defensive copying approach in the OpenAI
adapter where the same pattern exists.
- Around line 148-168: These stub methods (sendText, sendToolResult,
updateSession) currently only console.warn and silently succeed; update them to
surface unsupported operations to callers by throwing a clear Error (e.g., new
Error('ElevenLabs realtime does not support sendText')) or returning a rejected
Promise with that Error to match the method's async contract; ensure the
behavior is consistent with the RealtimeConnection interface (adjust signatures
to return Promise.reject if these are async) and include descriptive messages
that reference the operation name so callers can programmatically detect
failures.
In `@packages/typescript/ai-elevenlabs/vite.config.ts`:
- Line 18: Remove the redundant 'node_modules/' entry from the coverage.exclude
array in vite.config.ts (the array used to configure the v8 coverage provider);
locate the coverage config (symbol: coverage.exclude) and delete that string
element so the default v8 behavior remains and the exclusion list is not
duplicated.
- Line 8: Remove the redundant property "dir: './'" from the Vite config (in
packages/typescript/ai-elevenlabs/vite.config.ts) — locate the object that
contains dir: './' and delete that key so the config uses the default
project-root directory behavior; no other changes needed.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 516-537: The calculateLevel function is allocating a new
Uint8Array on every frame; fix by pre-allocating and reusing buffers per
AnalyserNode instead of creating new arrays each call: add a module-scoped
Map<AnalyserNode, Uint8Array> (or attach a Symbol-keyed property) to cache a
buffer for each analyser, on first use create Uint8Array(analyser.fftSize) and
store it, then in calculateLevel (and the frequency/time-domain helper methods
used by the inputLevel/outputLevel getters) retrieve and reuse that cached
buffer for analyser.getByteTimeDomainData / getByteFrequencyData to eliminate
per-frame allocations. Ensure buffer size matches analyser.fftSize if it can
change (recreate and replace cache entry when size differs).
- Around line 260-299: The handler for 'response.done' accesses
event.response.output without guarding against a missing response; update the
block around currentMode/emit/currentMessageId to safely handle missing or
non-array data by null-checking event.response (or defaulting to an empty
object) and validating response.output as an array before iterating (e.g., const
response = event.response ?? {}; const output = Array.isArray(response.output) ?
response.output : []), and while extracting parts extend the item-type handling
to also capture 'function_call' (or tool-call) items into message.parts (e.g.,
add a branch for item.type === 'function_call' that pushes a part with a clear
type/name/arguments) so tool calls aren't silently skipped; leave
emit('message_complete', { message }) and currentMessageId reset logic
unchanged.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts`:
- Around line 200-206: setVADMode currently only calls setVADModeState and
doesn't propagate the change to the active session, causing UI/server mismatch;
modify the useCallback to include relevant dependencies (e.g., session and
isConnected) and, when connected, call the session API to update the server-side
config (for example session.updateConfig or session.send/update method to set
vadMode to newMode) before/after calling setVADModeState, handling errors
appropriately so the server and UI stay in sync.
- Around line 110-114: The onStateChange subscription return value is discarded;
explicitly capture and unsubscribe it to ensure proper cleanup: when calling
clientRef.current.onStateChange(...) assign the returned unsubscribe function to
a variable (e.g., unsubscribe) and then call that unsubscribe during the
component cleanup (or when calling destroy()), ensuring you still update
setPendingUserTranscript and setPendingAssistantTranscript from the callback and
not rely solely on clientRef.current.destroy() to clear callbacks.
In `@packages/typescript/ai/src/index.ts`:
- Around line 82-104: Remove the realtime exports from the root barrel (the list
starting with realtimeToken and the Realtime* types in
packages/typescript/ai/src/index.ts) and re-export them from a dedicated
subpath; create a new entry re-export file (e.g.,
packages/typescript/ai/src/realtime/index.ts) that exports realtimeToken and all
Realtime* types, then update package.json "exports" to add a "./realtime"
subpath pointing to the built realtime entry so consumers can import
"@tanstack/ai/realtime"; ensure the build output includes the new realtime entry
so the new subpath resolves at runtime.
Comment on lines
+96
to
+122
| try { | ||
| // Fetch token from server | ||
| this.token = await this.options.getToken() | ||
|
|
||
| // Schedule token refresh | ||
| this.scheduleTokenRefresh() | ||
|
|
||
| // Connect via adapter | ||
| this.connection = await this.options.adapter.connect(this.token) | ||
| console.log(`[RealtimeClient #${this.instanceId}] Connection established:`, !!this.connection) | ||
|
|
||
| // Subscribe to connection events | ||
| this.subscribeToConnectionEvents() | ||
|
|
||
| // Start audio capture if configured | ||
| if (this.options.autoCapture) { | ||
| await this.connection.startAudioCapture() | ||
| } | ||
|
|
||
| this.updateState({ status: 'connected', mode: 'listening' }) | ||
| this.options.onConnect?.() | ||
| } catch (error) { | ||
| const err = error instanceof Error ? error : new Error(String(error)) | ||
| this.updateState({ status: 'error', error: err }) | ||
| this.options.onError?.(err) | ||
| throw err | ||
| } |
Contributor
There was a problem hiding this comment.
Partial failure in connect(): if startAudioCapture throws, the connection is left open.
When autoCapture is enabled and startAudioCapture() fails (e.g., mic permission denied), the catch block sets status to 'error' but doesn't disconnect the already-established connection. This leaks the open WebRTC/WebSocket session.
Proposed fix — disconnect on partial failure
} catch (error) {
const err = error instanceof Error ? error : new Error(String(error))
+ // Clean up partial connection
+ if (this.connection) {
+ await this.connection.disconnect().catch(() => {})
+ this.connection = null
+ }
this.updateState({ status: 'error', error: err })
this.options.onError?.(err)
throw err
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| try { | |
| // Fetch token from server | |
| this.token = await this.options.getToken() | |
| // Schedule token refresh | |
| this.scheduleTokenRefresh() | |
| // Connect via adapter | |
| this.connection = await this.options.adapter.connect(this.token) | |
| console.log(`[RealtimeClient #${this.instanceId}] Connection established:`, !!this.connection) | |
| // Subscribe to connection events | |
| this.subscribeToConnectionEvents() | |
| // Start audio capture if configured | |
| if (this.options.autoCapture) { | |
| await this.connection.startAudioCapture() | |
| } | |
| this.updateState({ status: 'connected', mode: 'listening' }) | |
| this.options.onConnect?.() | |
| } catch (error) { | |
| const err = error instanceof Error ? error : new Error(String(error)) | |
| this.updateState({ status: 'error', error: err }) | |
| this.options.onError?.(err) | |
| throw err | |
| } | |
| try { | |
| // Fetch token from server | |
| this.token = await this.options.getToken() | |
| // Schedule token refresh | |
| this.scheduleTokenRefresh() | |
| // Connect via adapter | |
| this.connection = await this.options.adapter.connect(this.token) | |
| console.log(`[RealtimeClient #${this.instanceId}] Connection established:`, !!this.connection) | |
| // Subscribe to connection events | |
| this.subscribeToConnectionEvents() | |
| // Start audio capture if configured | |
| if (this.options.autoCapture) { | |
| await this.connection.startAudioCapture() | |
| } | |
| this.updateState({ status: 'connected', mode: 'listening' }) | |
| this.options.onConnect?.() | |
| } catch (error) { | |
| const err = error instanceof Error ? error : new Error(String(error)) | |
| // Clean up partial connection | |
| if (this.connection) { | |
| await this.connection.disconnect().catch(() => {}) | |
| this.connection = null | |
| } | |
| this.updateState({ status: 'error', error: err }) | |
| this.options.onError?.(err) | |
| throw err | |
| } |
Comment on lines
+163
to
+169
| startListening(): void { | ||
| if (!this.connection || this.state.status !== 'connected') { | ||
| return | ||
| } | ||
| this.connection.startAudioCapture() | ||
| this.updateState({ mode: 'listening' }) | ||
| } |
Contributor
There was a problem hiding this comment.
startListening() ignores the Promise from startAudioCapture().
startAudioCapture() returns Promise<void> (e.g., requesting mic permissions). The unawaited call means permission failures or errors are silently swallowed, and mode is set to 'listening' even if capture fails.
Proposed fix
- startListening(): void {
+ async startListening(): Promise<void> {
if (!this.connection || this.state.status !== 'connected') {
return
}
- this.connection.startAudioCapture()
- this.updateState({ mode: 'listening' })
+ try {
+ await this.connection.startAudioCapture()
+ this.updateState({ mode: 'listening' })
+ } catch (error) {
+ const err = error instanceof Error ? error : new Error(String(error))
+ this.updateState({ error: err })
+ this.options.onError?.(err)
+ }
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-client/src/realtime-client.ts` around lines 163 - 169,
In startListening(), await the Promise returned by
this.connection.startAudioCapture() and only call this.updateState({ mode:
'listening' }) after the await succeeds; if startAudioCapture() rejects, catch
the error (e.g., in a try/catch), log or surface it via your logger or
this.updateState to reflect failure, and avoid flipping the mode to 'listening'
on error — modify the startListening() implementation to use async/await with
try/catch around this.connection.startAudioCapture(), referencing
startListening(), this.connection.startAudioCapture(), and this.updateState().
Comment on lines
+281
to
+284
| destroy(): void { | ||
| this.disconnect() | ||
| this.stateChangeCallbacks.clear() | ||
| } |
Contributor
There was a problem hiding this comment.
destroy() does not await disconnect() — final state update and cleanup race.
disconnect() is async (awaits connection.disconnect() and then calls updateState). Since destroy() is synchronous and doesn't await the returned Promise, stateChangeCallbacks.clear() on line 283 executes immediately — before disconnect() finishes. This means:
- State-change listeners never receive the final
'idle'status fromdisconnect(). - If
connection.disconnect()rejects, it becomes an unhandled Promise rejection.
Proposed fix — make destroy async, or guard the clear
- destroy(): void {
- this.disconnect()
- this.stateChangeCallbacks.clear()
+ async destroy(): Promise<void> {
+ await this.disconnect()
+ this.stateChangeCallbacks.clear()
}If callers cannot await (e.g., React cleanup), chain instead:
- destroy(): void {
- this.disconnect()
- this.stateChangeCallbacks.clear()
+ destroy(): void {
+ this.disconnect().catch(() => {}).finally(() => {
+ this.stateChangeCallbacks.clear()
+ })
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-client/src/realtime-client.ts` around lines 281 - 284,
The destroy() method currently calls async disconnect() without awaiting,
causing stateChangeCallbacks.clear() to run before disconnect completes; change
destroy to await the disconnect call (e.g., make destroy async and await
this.disconnect()) so updateState and any final `'idle'` notifications run
before clearing; if you cannot change the API to async, instead chain the
Promise (return this.disconnect().finally(() =>
this.stateChangeCallbacks.clear())) or at minimum call
this.disconnect().catch(() => {}).finally(() =>
this.stateChangeCallbacks.clear()) to avoid unhandled rejections and ensure
cleanup happens after disconnect; reference methods: destroy(), disconnect(),
updateState, stateChangeCallbacks, and connection.disconnect().
Comment on lines
+19
to
+23
| export type OpenAIRealtimeModel = | ||
| | 'gpt-4o-realtime-preview' | ||
| | 'gpt-4o-realtime-preview-2024-10-01' | ||
| | 'gpt-4o-mini-realtime-preview' | ||
| | 'gpt-4o-mini-realtime-preview-2024-12-17' |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
OpenAI realtime API models 2025 2026 latest gpt-4o-realtime
💡 Result:
As of February 19, 2026, OpenAI’s latest Realtime API models are:
gpt-realtime(general availability realtime model; text+audio over WebRTC/WebSocket/SIP) [1]gpt-realtime-mini(lower-cost realtime model; text+audio over WebRTC/WebSocket/SIP) [2]
The older GPT‑4o realtime naming you mentioned is not gpt-4o-realtime; it exists as the preview line:
gpt-4o-realtime-preview(and dated snapshots like...-2025-06-03,...-2024-12-17) [3]gpt-4o-mini-realtime-preview[4]
But these preview models are scheduled for shutdown on 2026-03-24, with recommended replacements:
gpt-4o-realtime-preview*→gpt-realtimegpt-4o-mini-realtime-preview→gpt-realtime-mini[4]
Sources:
[1] gpt-realtime model docs [1]
[2] gpt-realtime-mini model docs [2]
[3] gpt-4o-realtime-preview model docs [3]
[4] OpenAI API deprecations notice (shutdown/replacements) [4]
🏁 Script executed:
cat packages/typescript/ai-openai/src/realtime/types.tsRepository: TanStack/ai
Length of output: 3083
Update realtime model type to include production models and allow flexible versioning — preview models are being deprecated.
The OpenAIRealtimeModel union is restricted to four preview models that OpenAI is deprecating on 2026-03-24. Replace them with the new production models (gpt-realtime, gpt-realtime-mini) and adopt a flexible pattern to support future updates without library changes:
Suggested fix
export type OpenAIRealtimeModel =
+ | 'gpt-realtime'
+ | 'gpt-realtime-mini'
| 'gpt-4o-realtime-preview'
| 'gpt-4o-realtime-preview-2024-10-01'
| 'gpt-4o-mini-realtime-preview'
| 'gpt-4o-mini-realtime-preview-2024-12-17'
+ | (string & {})The | (string & {}) pattern preserves IDE autocomplete for known models while allowing arbitrary model identifiers for rapid API updates (same pattern used in the FAL adapter for 600+ models).
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-openai/src/realtime/types.ts` around lines 19 - 23,
Replace the restricted preview-only union in OpenAIRealtimeModel with the new
production names and a flexible string pattern: update the OpenAIRealtimeModel
type to include 'gpt-realtime' and 'gpt-realtime-mini' (instead of the four
preview names) and add the permissive fallback (string & {}) so unknown/future
model identifiers are accepted while preserving IDE autocomplete for known
models; modify the type definition for OpenAIRealtimeModel accordingly.
Comment on lines
+74
to
+115
| if (!clientRef.current) { | ||
| clientRef.current = new RealtimeClient({ | ||
| getToken: optionsRef.current.getToken, | ||
| adapter: optionsRef.current.adapter, | ||
| tools: optionsRef.current.tools, | ||
| autoPlayback: optionsRef.current.autoPlayback, | ||
| autoCapture: optionsRef.current.autoCapture, | ||
| vadMode: optionsRef.current.vadMode, | ||
| onStatusChange: (newStatus) => { | ||
| setStatus(newStatus) | ||
| }, | ||
| onModeChange: (newMode) => { | ||
| setMode(newMode) | ||
| optionsRef.current.onModeChange?.(newMode) | ||
| }, | ||
| onMessage: (message) => { | ||
| setMessages((prev) => [...prev, message]) | ||
| optionsRef.current.onMessage?.(message) | ||
| }, | ||
| onError: (err) => { | ||
| setError(err) | ||
| optionsRef.current.onError?.(err) | ||
| }, | ||
| onConnect: () => { | ||
| setError(null) | ||
| optionsRef.current.onConnect?.() | ||
| }, | ||
| onDisconnect: () => { | ||
| optionsRef.current.onDisconnect?.() | ||
| }, | ||
| onInterrupted: () => { | ||
| setPendingAssistantTranscript(null) | ||
| optionsRef.current.onInterrupted?.() | ||
| }, | ||
| }) | ||
|
|
||
| // Subscribe to state changes for transcripts | ||
| clientRef.current.onStateChange((state) => { | ||
| setPendingUserTranscript(state.pendingUserTranscript) | ||
| setPendingAssistantTranscript(state.pendingAssistantTranscript) | ||
| }) | ||
| } |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
find . -type f -name "*realtime-client*" | head -20Repository: TanStack/ai
Length of output: 109
🏁 Script executed:
rg -l "class RealtimeClient" --type=ts --type=jsRepository: TanStack/ai
Length of output: 107
🏁 Script executed:
rg -n "class RealtimeClient" -A 50 --type=ts --type=jsRepository: TanStack/ai
Length of output: 4418
🏁 Script executed:
rg -n "async connect" -A 30 packages/typescript/ai-client/src/realtime-client.tsRepository: TanStack/ai
Length of output: 1166
🏁 Script executed:
rg -n "getToken" -B 2 -A 5 packages/typescript/ai-client/src/realtime-client.tsRepository: TanStack/ai
Length of output: 944
🏁 Script executed:
cat -n packages/typescript/ai-react/src/use-realtime-chat.ts | head -150Repository: TanStack/ai
Length of output: 5823
🏁 Script executed:
rg -n "optionsRef" packages/typescript/ai-react/src/use-realtime-chat.tsRepository: TanStack/ai
Length of output: 711
🏁 Script executed:
rg -n "clientTools" packages/typescript/ai-client/src/realtime-client.ts | head -20Repository: TanStack/ai
Length of output: 248
🏁 Script executed:
cat -n packages/typescript/ai-client/src/realtime-client.ts | sed -n '200,250p'Repository: TanStack/ai
Length of output: 1749
🏁 Script executed:
rg -n "onClientToolExecute\|sendToolResult" packages/typescript/ai-client/src/realtime-client.ts -A 5Repository: TanStack/ai
Length of output: 37
🏁 Script executed:
cat -n packages/typescript/ai-client/src/realtime-client.ts | sed -n '375,395p'Repository: TanStack/ai
Length of output: 904
getToken, adapter, and tools are captured by value at client creation time and never updated.
Lines 76–78 pass optionsRef.current.getToken, optionsRef.current.adapter, and optionsRef.current.tools to the RealtimeClient constructor. These are evaluated once (when clientRef.current is first set) and frozen for the lifetime of the hook. If the parent component changes getToken (e.g., after auth refresh) or adapter or tools, the client will still use the stale values.
The event callbacks correctly read from optionsRef.current at invocation time — getToken, adapter, and tools should follow the same pattern:
Proposed fix
clientRef.current = new RealtimeClient({
- getToken: optionsRef.current.getToken,
- adapter: optionsRef.current.adapter,
- tools: optionsRef.current.tools,
+ getToken: (...args) => optionsRef.current.getToken(...args),
+ adapter: optionsRef.current.adapter, // adapter is likely stable, but worth noting
+ tools: optionsRef.current.tools, // same concern if tools change📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if (!clientRef.current) { | |
| clientRef.current = new RealtimeClient({ | |
| getToken: optionsRef.current.getToken, | |
| adapter: optionsRef.current.adapter, | |
| tools: optionsRef.current.tools, | |
| autoPlayback: optionsRef.current.autoPlayback, | |
| autoCapture: optionsRef.current.autoCapture, | |
| vadMode: optionsRef.current.vadMode, | |
| onStatusChange: (newStatus) => { | |
| setStatus(newStatus) | |
| }, | |
| onModeChange: (newMode) => { | |
| setMode(newMode) | |
| optionsRef.current.onModeChange?.(newMode) | |
| }, | |
| onMessage: (message) => { | |
| setMessages((prev) => [...prev, message]) | |
| optionsRef.current.onMessage?.(message) | |
| }, | |
| onError: (err) => { | |
| setError(err) | |
| optionsRef.current.onError?.(err) | |
| }, | |
| onConnect: () => { | |
| setError(null) | |
| optionsRef.current.onConnect?.() | |
| }, | |
| onDisconnect: () => { | |
| optionsRef.current.onDisconnect?.() | |
| }, | |
| onInterrupted: () => { | |
| setPendingAssistantTranscript(null) | |
| optionsRef.current.onInterrupted?.() | |
| }, | |
| }) | |
| // Subscribe to state changes for transcripts | |
| clientRef.current.onStateChange((state) => { | |
| setPendingUserTranscript(state.pendingUserTranscript) | |
| setPendingAssistantTranscript(state.pendingAssistantTranscript) | |
| }) | |
| } | |
| if (!clientRef.current) { | |
| clientRef.current = new RealtimeClient({ | |
| getToken: (...args) => optionsRef.current.getToken(...args), | |
| adapter: optionsRef.current.adapter, // adapter is likely stable, but worth noting | |
| tools: optionsRef.current.tools, // same concern if tools change | |
| autoPlayback: optionsRef.current.autoPlayback, | |
| autoCapture: optionsRef.current.autoCapture, | |
| vadMode: optionsRef.current.vadMode, | |
| onStatusChange: (newStatus) => { | |
| setStatus(newStatus) | |
| }, | |
| onModeChange: (newMode) => { | |
| setMode(newMode) | |
| optionsRef.current.onModeChange?.(newMode) | |
| }, | |
| onMessage: (message) => { | |
| setMessages((prev) => [...prev, message]) | |
| optionsRef.current.onMessage?.(message) | |
| }, | |
| onError: (err) => { | |
| setError(err) | |
| optionsRef.current.onError?.(err) | |
| }, | |
| onConnect: () => { | |
| setError(null) | |
| optionsRef.current.onConnect?.() | |
| }, | |
| onDisconnect: () => { | |
| optionsRef.current.onDisconnect?.() | |
| }, | |
| onInterrupted: () => { | |
| setPendingAssistantTranscript(null) | |
| optionsRef.current.onInterrupted?.() | |
| }, | |
| }) | |
| // Subscribe to state changes for transcripts | |
| clientRef.current.onStateChange((state) => { | |
| setPendingUserTranscript(state.pendingUserTranscript) | |
| setPendingAssistantTranscript(state.pendingAssistantTranscript) | |
| }) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 74 - 115,
The RealtimeClient is being created with getToken, adapter, and tools captured
once (optionsRef.current.getToken, optionsRef.current.adapter,
optionsRef.current.tools) which freezes stale values; change the RealtimeClient
construction so those fields are passed as thin wrapper functions that read from
optionsRef.current at invocation time (e.g., getToken: (...args) =>
optionsRef.current.getToken?.(...args), adapter: (...args) =>
optionsRef.current.adapter?.(...args) or adapter: () =>
optionsRef.current.adapter, and similarly for tools) so the client uses
up-to-date values from optionsRef when methods are called; keep existing
callbacks (onMessage/onError/etc.), clientRef.current and the onStateChange
subscription unchanged.
Comment on lines
+120
to
+139
| useEffect(() => { | ||
| function updateLevels() { | ||
| if (clientRef.current?.audio) { | ||
| setInputLevel(clientRef.current.audio.inputLevel) | ||
| setOutputLevel(clientRef.current.audio.outputLevel) | ||
| } | ||
| animationFrameRef.current = requestAnimationFrame(updateLevels) | ||
| } | ||
|
|
||
| if (status === 'connected') { | ||
| updateLevels() | ||
| } | ||
|
|
||
| return () => { | ||
| if (animationFrameRef.current) { | ||
| cancelAnimationFrame(animationFrameRef.current) | ||
| animationFrameRef.current = null | ||
| } | ||
| } | ||
| }, [status]) |
Contributor
There was a problem hiding this comment.
Animation frame loop triggers React state updates at ~60 fps, causing excessive re-renders.
setInputLevel / setOutputLevel are called on every requestAnimationFrame tick (~60 fps). Each call triggers a React re-render of the component and all its children, even when the audio levels haven't meaningfully changed. For a real-time audio UI this will cause significant performance degradation.
Consider one of:
- Use refs for levels and let the visualization consumer read them directly (e.g., via a getter or a ref-forwarding API).
- Throttle updates to ~10-15 fps, which is visually sufficient for level meters.
- Only update state when the value changes beyond a threshold.
Option 2: Throttle to ~15 fps
useEffect(() => {
+ let lastUpdate = 0
function updateLevels() {
- if (clientRef.current?.audio) {
- setInputLevel(clientRef.current.audio.inputLevel)
- setOutputLevel(clientRef.current.audio.outputLevel)
+ const now = performance.now()
+ if (now - lastUpdate > 66 && clientRef.current?.audio) { // ~15fps
+ setInputLevel(clientRef.current.audio.inputLevel)
+ setOutputLevel(clientRef.current.audio.outputLevel)
+ lastUpdate = now
}
animationFrameRef.current = requestAnimationFrame(updateLevels)
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 120 -
139, The updateLevels loop in useEffect calls setInputLevel/setOutputLevel every
requestAnimationFrame causing ~60fps re-renders; modify updateLevels to throttle
updates to ~15fps (or ~66ms) and only call setInputLevel/setOutputLevel when the
new value differs by a small threshold (e.g., delta > 0.01) to avoid unnecessary
renders. Concretely, inside the updateLevels function (which uses
animationFrameRef and clientRef), track the last update timestamp (or frame
count) and only perform the React state updates when enough time has elapsed,
and compute a small difference check against previousLevel refs before calling
setInputLevel/setOutputLevel; keep the existing cleanup of animationFrameRef and
the status === 'connected' start condition.
@tanstack/ai
@tanstack/ai-anthropic
@tanstack/ai-client
@tanstack/ai-devtools-core
@tanstack/ai-elevenlabs
@tanstack/ai-fal
@tanstack/ai-gemini
@tanstack/ai-grok
@tanstack/ai-groq
@tanstack/ai-ollama
@tanstack/ai-openai
@tanstack/ai-openrouter
@tanstack/ai-preact
@tanstack/ai-react
@tanstack/ai-react-ui
@tanstack/ai-solid
@tanstack/ai-solid-ui
@tanstack/ai-svelte
@tanstack/ai-vue
@tanstack/ai-vue-ui
@tanstack/preact-ai-devtools
@tanstack/react-ai-devtools
@tanstack/solid-ai-devtools
commit: |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 13
🧹 Nitpick comments (6)
packages/typescript/ai-elevenlabs/package.json (2)
46-49:peerDependenciesuseworkspace:^whiledevDependenciesuseworkspace:*for the same packagesBoth
@tanstack/aiand@tanstack/ai-clientappear inpeerDependenciesasworkspace:^(lines 47–48) and indevDependenciesasworkspace:*(lines 51–52). The inconsistency is confusing and deviates from the established convention.♻️ Proposed fix
"peerDependencies": { - "@tanstack/ai": "workspace:^", - "@tanstack/ai-client": "workspace:^" + "@tanstack/ai": "workspace:*", + "@tanstack/ai-client": "workspace:*" },Based on learnings: "Use
workspace:*protocol for internal package dependencies in package.json (e.g.,"tanstack/ai": "workspace:*")"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/package.json` around lines 46 - 49, Update the peerDependencies entries in package.json so they use the same workspace protocol as devDependencies: change "@tanstack/ai": "workspace:^" and "@tanstack/ai-client": "workspace:^" to use "workspace:*" under the peerDependencies block so the package names and protocols are consistent with devDependencies and the repo convention.
23-28: Consider adding a"default"fallback condition to the exports mapThe current exports only declare
"types"and"import". Bundlers and tools that do not recognise the"import"condition (e.g., certain Jest or CJS-first configurations) have no fallback and will fail to resolve the package.♻️ Proposed fix
"exports": { ".": { "types": "./dist/esm/index.d.ts", - "import": "./dist/esm/index.js" + "import": "./dist/esm/index.js", + "default": "./dist/esm/index.js" } },🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/package.json` around lines 23 - 28, The exports map currently only provides "types" and the "import" condition which can break resolution in CJS-first runtimes; update the package.json "exports" object to include a "default" fallback entry that points to the CommonJS bundle (e.g., the package's CJS runtime entry such as the compiled ./dist CJS file) so tools that don't support the "import" condition can resolve the package; ensure the "types" key remains and that the "default" path matches your distributed CJS output file.packages/typescript/ai-client/src/realtime-client.ts (1)
63-63: Debugconsole.logcalls should be removed from library code.Lines 63, 105–108, and 257–260 contain
[RealtimeClient #...]debug logs. These will appear in every consumer's production console.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-client/src/realtime-client.ts` at line 63, Remove the debug console.log calls that emit `[RealtimeClient #${this.instanceId}]` from the RealtimeClient class (the constructor and other locations that currently log at lines where instanceId is used), replacing them with either no-op or a proper logger call at debug/trace level if a logging facility exists; specifically eliminate the console.log statements in the RealtimeClient that print instance creation and other internal state so library consumers don't get production console output, or switch them to use the project's logger (e.g., logger.debug) if consistent logging is required.packages/typescript/ai-openai/src/realtime/adapter.ts (1)
175-180: Severalconsole.logdebug statements should be removed before release.Lines 122–126 (
[Realtime] ontrack event), 175–180 ([Realtime] Setting up input audio analysis), and 519–525 ([Realtime] getAudioVisualization called) are debug artifacts that will pollute consumers' production consoles. ThegetAudioVisualizationgetter is called on every animation frame, so line 519–525 will log at ~60 fps when audio is active.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 175 - 180, Remove the debug console.log statements that were left in the realtime adapter: delete the logs emitted inside the ontrack handler (the "[Realtime] ontrack event" messages), remove the "[Realtime] Setting up input audio analysis" and "[Realtime] Input analyser created:" logs around the call to setupInputAudioAnalysis(localStream) and inputAnalyser, and remove any logging inside the getAudioVisualization getter (which is called every animation frame). Keep behavior unchanged—only remove the console.log calls so production consoles are not spammed.examples/ts-react-chat/src/lib/realtime-tools.ts (1)
116-139:setTimeoutinsidesetReminderClientis untracked — leaks if the component unmounts.
setTimeoutstores no reference and is never cancelled. If the page/component is torn down before the timer fires, thealertwill still trigger. For a demo this is acceptable, but callers should be aware.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/ts-react-chat/src/lib/realtime-tools.ts` around lines 116 - 139, The setReminderClient implementation uses setTimeout without storing or exposing the timer id, which can leak and still call alert after a component/page unmounts; modify the function (the setReminderClient created from setReminderToolDef) to capture the timeout id returned by setTimeout, return alongside the success payload a cancel/clear function or token (e.g., { cancel: () => clearTimeout(id) }) so callers can cancel the scheduled reminder, and update any docs/comments to advise callers to call that cancel method (or call clearTimeout with the stored id) during unmount/cleanup to prevent the alert from firing after teardown.packages/typescript/ai/src/realtime/types.ts (1)
210-243: Consider derivingRealtimeEventfromRealtimeEventPayloadsto keep them in sync.
RealtimeEventis a manually maintained string union that must exactly mirror the keys ofRealtimeEventPayloads. Deriving one from the other eliminates the risk of them drifting apart.Proposed fix
-export type RealtimeEvent = - | 'status_change' - | 'mode_change' - | 'transcript' - | 'audio_chunk' - | 'tool_call' - | 'message_complete' - | 'interrupted' - | 'error' - /** * Event payloads for realtime events */ export interface RealtimeEventPayloads { status_change: { status: RealtimeStatus } mode_change: { mode: RealtimeMode } transcript: { role: 'user' | 'assistant' transcript: string isFinal: boolean } audio_chunk: { data: ArrayBuffer; sampleRate: number } tool_call: { toolCallId: string; toolName: string; input: unknown } message_complete: { message: RealtimeMessage } interrupted: { messageId?: string } error: { error: Error } } +/** + * Events emitted by the realtime connection + */ +export type RealtimeEvent = keyof RealtimeEventPayloads🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai/src/realtime/types.ts` around lines 210 - 243, Replace the manually-maintained string union RealtimeEvent with a derived union from the keys of RealtimeEventPayloads to keep them in sync: change the declaration of RealtimeEvent to be based on keyof RealtimeEventPayloads (ensuring it still exports the same literal union type), and keep RealtimeEventHandler<TEvent extends RealtimeEvent> using the mapped lookup RealtimeEventPayloads[TEvent] as before so handlers continue to type-check against the payloads.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@examples/ts-react-chat/src/routes/api.realtime-token.ts`:
- Around line 5-11: Reorder the imports to satisfy ESLint import-order rules:
move the `import * as z from 'zod'` line above the `@/lib/realtime-tools` import
group, and within the named imports from `@/lib/realtime-tools` ensure
`searchKnowledgeToolDef` appears alphabetically before `setReminderToolDef` (so
the named list reads e.g. getCurrentTimeToolDef, getWeatherToolDef,
searchKnowledgeToolDef, setReminderToolDef).
In `@examples/ts-react-chat/src/routes/realtime.tsx`:
- Around line 136-140: The error handling uses response.json() which will throw
if the server returns non-JSON; modify the error path where you check if
(!response.ok) to safely attempt response.json() inside a try/catch and fall
back to await response.text() if parsing fails, then throw a new Error that
includes response.status (and/or statusText) plus the parsed error message or
raw text; keep the successful return as return response.json(). Ensure you
reference the same response variable in this block so the thrown error contains
meaningful HTTP status and body content instead of a parse exception.
In `@packages/typescript/ai-client/src/realtime-client.ts`:
- Around line 89-92: The connect() method can be invoked concurrently causing
two fetches/connections; protect it by making the status check atomic: when
entering connect() check this.state.status and if not 'connected' immediately
set this.state.status = 'connecting' (or acquire a simple mutex/Promise lock)
before any async work so other callers will see 'connecting' and return/wait;
ensure any early returns or errors restore status to 'idle' and that successful
completion sets status = 'connected'. Update the connect() implementation to use
this new status transition (or an explicit lock) around token fetch and socket
open to prevent parallel connect attempts.
- Around line 379-385: The assistant transcript handler currently replaces
pendingAssistantTranscript on each non-final delta, losing earlier tokens;
modify the logic in the transcript event handler (the block that calls
updateState and touches pendingAssistantTranscript) to accumulate chunks instead
of overwriting: when isFinal is false, set pendingAssistantTranscript to
(existing pendingAssistantTranscript || '') + transcript (handling
null/undefined), and when isFinal is true clear pendingAssistantTranscript (or
let message_complete handle finalization) so the transcript grows across deltas;
ensure you reference updateState and pendingAssistantTranscript in the change.
In `@packages/typescript/ai-elevenlabs/package.json`:
- Around line 43-45: Update the deprecated dependency "@11labs/client" in
package.json to the maintained "@elevenlabs/client" (pin to a modern release
like "^0.14.0"), and replace all import/require occurrences of "@11labs/client"
in the codebase with "@elevenlabs/client"; also audit call sites for SDK
breaking changes (notably replace any use of the removed prompt.tools field with
prompt.tool_ids and update any changed method/option names per the ElevenLabs
changelog) so the package compiles and tests pass.
In `@packages/typescript/ai-elevenlabs/README.md`:
- Around line 7-9: The install command in the README is missing the React
package required by the example; update the npm install line to include
`@tanstack/ai-react` so the React example that imports useRealtimeChat (and other
React hooks from `@tanstack/ai-react`) works correctly — ensure the install
command adds `@tanstack/ai-react` alongside `@tanstack/ai-elevenlabs`, `@tanstack/ai`,
and `@tanstack/ai-client`.
In `@packages/typescript/ai-elevenlabs/src/realtime/token.ts`:
- Around line 64-66: The URL built for the fetch call to
`${ELEVENLABS_API_URL}/convai/conversation/get_signed_url?agent_id=${agentId}`
uses raw agentId which can contain characters that break the URL; fix this in
the token generation logic by URL-encoding agentId (e.g., via encodeURIComponent
or equivalent) when constructing the query string so the fetch to the
get_signed_url endpoint always targets the correct resource while keeping
ELEVENLABS_API_URL and the rest of the path unchanged.
- Around line 81-82: The code unsafely casts data.signed_url to string
(signedUrl = data.signed_url as string) without validation; update the
token-fetching logic (the response/json handling and variable signedUrl) to
validate that data.signed_url exists and is a non-empty string (e.g., typeof
data.signed_url === 'string' && data.signed_url.trim() !== '') and if not, throw
or return a clear error (with context) instead of silently casting, so
downstream callers of the token/signedUrl get a guaranteed valid value or an
actionable error.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 497-502: interrupt() is emitting 'interrupted' optimistically then
the server's conversation.item.truncated handler also emits 'interrupted',
causing duplicate notifications; introduce a clientInterruptPending boolean that
interrupt() sets to true before emitting (and include currentMessageId), then in
the conversation.item.truncated handler check clientInterruptPending and
suppress the emit('interrupted', ...) for the same messageId if the flag is set,
finally clear clientInterruptPending when the server event arrives (or on other
flows like response.cancel acknowledgement) so future interruptions behave
normally; reference interrupt(), the conversation.item.truncated handler,
currentMessageId, and emit('interrupted', ...) when adding the flag logic.
- Around line 116-118: The onerror handler for dataChannel currently stringifies
the Event, producing "[object Event]"; instead, extract a useful message from
the event (handle RTCErrorEvent by checking event.error?.message or event.error,
fallback to event.message or event.type) and include that in the emitted Error,
or attach the original event as the Error's cause/metadata; update the
dataChannel.onerror callback (the code that calls emit('error', { error: new
Error(`Data channel error: ${error}`) }) ) to build the Error with the extracted
message and/or include the original event as the cause so logs contain
actionable info.
In `@packages/typescript/ai-openai/src/realtime/token.ts`:
- Around line 114-118: The code assumes sessionData.tools is an array and calls
sessionData.tools.map which will throw if tools is null/undefined; update the
logic that builds tools (the variable named tools and the sessionData.tools.map
usage in token.ts) to first coerce or guard the value (e.g., treat
non-array/null as an empty array using a nullish/Array.isArray check) and then
map to the Tool shape so tools is always an Array<Tool> (empty when no tools
provided).
In `@packages/typescript/ai-react/src/use-realtime-chat.ts`:
- Around line 153-157: The cleanup effect currently calls
clientRef.current?.destroy() but leaves clientRef.current set, so after React
StrictMode unmount/remount the destroyed client exists and onStateChange
subscriptions aren't re-registered causing
pendingUserTranscript/pendingAssistantTranscript to never update; update the
cleanup in the useEffect to call clientRef.current?.destroy() and then null out
clientRef.current (or set to undefined) so subsequent mounts detect no client
and recreate it and re-subscribe via onStateChange.
- Around line 13-15: The fallback arrays emptyFrequencyData and
emptyTimeDomainData are sized at 128 but must match the adapter's FFT output
(frequencyBinCount = 1024 and time domain length = 2048) to avoid consumers
seeing length changes; update emptyFrequencyData to new Uint8Array(1024) and
emptyTimeDomainData to new Uint8Array(2048). Keep the time-domain default values
(e.g., fill(128)) as before so that getInputFrequencyData() returns the
same-shaped arrays whether connected or not.
---
Duplicate comments:
In `@examples/ts-react-chat/src/routes/realtime.tsx`:
- Around line 108-109: The adapter is recreated on every render because it
directly calls openaiRealtime() or elevenlabsRealtime(); wrap the adapter
creation in useMemo so it returns a stable reference based on provider. Replace
the direct assignment to adapter with a memoized value using React's useMemo and
key the memo on provider (referencing adapter, openaiRealtime, and
elevenlabsRealtime) so the hook consuming adapter receives a stable object
between renders.
In `@packages/typescript/ai-client/src/realtime-client.ts`:
- Around line 96-126: The connect() flow can leave this.connection open if
this.options.autoCapture is true and connection.startAudioCapture() throws;
update the catch branch in connect() to perform cleanup: if this.connection
exists call its disconnect() (or close) and null out this.connection,
unsubscribe/cleanup any connection event handlers created by
subscribeToConnectionEvents(), and cancel any scheduled token refresh started by
scheduleTokenRefresh(); ensure you still set state to error, call
this.options.onError, and rethrow the error so the connection/permission
indicator is not left active when startAudioCapture fails.
- Around line 163-172: The startListening function calls
connection.startAudioCapture() without awaiting it so any Promise rejection
(e.g., mic permission denied) is swallowed and updateState({ mode: 'listening'
}) runs regardless; change startListening to await
this.connection.startAudioCapture(), wrap the await in try/catch, only call
this.updateState({ mode: 'listening' }) on success, and on error log or
propagate the error (e.g., via processLogger or this.updateState to an error
status) so failures are not silently ignored; reference the startListening
method and connection.startAudioCapture() and updateState() calls when making
the change.
- Around line 393-409: When this.clientTools.get(toolName) returns undefined (or
the retrieved tool lacks an execute method) the provider never responds; update
the branch around the retrieval of tool (this.clientTools.get(toolName)) to send
a failure tool result back using this.connection?.sendToolResult(toolCallId,
JSON.stringify({ error: ... })) including the toolName and a clear message
(e.g., "tool not registered" or "tool has no execute method") so callers aren’t
left waiting; keep the existing try/catch for actual execution errors in execute
and only short-circuit with the error response when the tool is missing or
invalid.
- Around line 287-290: The destroy() implementation calls the async disconnect()
without awaiting it, causing stateChangeCallbacks.clear() to run too early and
any rejection from connection.disconnect() to be an unhandled promise; change
destroy() to be async (or return a Promise) and await this.disconnect(), then
clear stateChangeCallbacks after the await, and wrap the await in a try/catch to
handle/log any error from connection.disconnect() (reference methods: destroy(),
disconnect(), and the stateChangeCallbacks collection).
In `@packages/typescript/ai-elevenlabs/src/realtime/token.ts`:
- Around line 9-29: The getElevenLabsApiKey function currently falls back to
reading window.env, which can leak the server API key if this server-only module
is bundled client-side; remove the browser fallback so getElevenLabsApiKey only
reads process.env.ELEVENLABS_API_KEY (and throws if missing) and delete the
block that checks/casts window and window.env; keep the existing error throw
path and ensure the function is only using the process check (symbol:
getElevenLabsApiKey).
In `@packages/typescript/ai-openai/src/index.ts`:
- Around line 103-118: The realtime adapter exports (openaiRealtimeToken,
openaiRealtime and the OpenAIRealtime* types) were added but the package-level
subpath exports for the realtime folder are missing, breaking tree-shaking and
subpath imports; update the package.json "exports" to add a "./realtime" subpath
(and corresponding "./realtime/*" if needed) pointing to the built JS and type
entry files so consumers can import openaiRealtimeToken/openaiRealtime and the
OpenAIRealtime types directly and tree-shake unused code; ensure the paths map
to the compiled output and type declaration files consistent with how other
adapter subpaths are declared.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 150-172: Wrap the SDP exchange and remote-description steps (the
fetch to OPENAI_REALTIME_URL, sdpResponse.ok handling, and
pc.setRemoteDescription) in a try/catch (or try/catch/finally) so any failure
cleans up resources: if an error occurs, stop all tracks on localStream (for
each track call stop()), close dataChannel if it exists (dataChannel.close()),
and close the RTCPeerConnection (await pc.close()); also null out or clear
references as needed to avoid dangling state. Ensure checks for undefined/null
on localStream, dataChannel, and pc before calling stop/close, and rethrow the
original error after cleanup so callers still see the failure.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts`:
- Around line 87-89: The client is being constructed with stale snapshots of
getToken, adapter, and tools; instead pass through optionsRef to always read the
latest values. Update the client construction (where getToken, adapter, tools
are set) to use wrappers that call optionsRef.current (e.g., getToken: (...args)
=> optionsRef.current.getToken?.(...args), and similarly resolve adapter and
tools via optionsRef.current when invoked) so the realtime client uses
up-to-date references; keep the unique names getToken, adapter, tools and
optionsRef to locate and modify the assignment.
- Around line 131-150: The animation loop in updateLevels currently calls
setInputLevel/setOutputLevel on every requestAnimationFrame (via
animationFrameRef) causing excessive re-renders; modify updateLevels (used with
clientRef.current.audio and status === 'connected') to only call
setInputLevel/setOutputLevel at a lower rate (e.g., throttle to ~15 fps using a
lastUpdateRef timestamp) or only when the level delta exceeds a small threshold
(e.g., Math.abs(new - prev) > 0.01), and keep scheduling requestAnimationFrame
as before; ensure you read and update a ref for last input/output values (and
last update time) so you avoid state updates when values haven't meaningfully
changed.
---
Nitpick comments:
In `@examples/ts-react-chat/src/lib/realtime-tools.ts`:
- Around line 116-139: The setReminderClient implementation uses setTimeout
without storing or exposing the timer id, which can leak and still call alert
after a component/page unmounts; modify the function (the setReminderClient
created from setReminderToolDef) to capture the timeout id returned by
setTimeout, return alongside the success payload a cancel/clear function or
token (e.g., { cancel: () => clearTimeout(id) }) so callers can cancel the
scheduled reminder, and update any docs/comments to advise callers to call that
cancel method (or call clearTimeout with the stored id) during unmount/cleanup
to prevent the alert from firing after teardown.
In `@packages/typescript/ai-client/src/realtime-client.ts`:
- Line 63: Remove the debug console.log calls that emit `[RealtimeClient
#${this.instanceId}]` from the RealtimeClient class (the constructor and other
locations that currently log at lines where instanceId is used), replacing them
with either no-op or a proper logger call at debug/trace level if a logging
facility exists; specifically eliminate the console.log statements in the
RealtimeClient that print instance creation and other internal state so library
consumers don't get production console output, or switch them to use the
project's logger (e.g., logger.debug) if consistent logging is required.
In `@packages/typescript/ai-elevenlabs/package.json`:
- Around line 46-49: Update the peerDependencies entries in package.json so they
use the same workspace protocol as devDependencies: change "@tanstack/ai":
"workspace:^" and "@tanstack/ai-client": "workspace:^" to use "workspace:*"
under the peerDependencies block so the package names and protocols are
consistent with devDependencies and the repo convention.
- Around line 23-28: The exports map currently only provides "types" and the
"import" condition which can break resolution in CJS-first runtimes; update the
package.json "exports" object to include a "default" fallback entry that points
to the CommonJS bundle (e.g., the package's CJS runtime entry such as the
compiled ./dist CJS file) so tools that don't support the "import" condition can
resolve the package; ensure the "types" key remains and that the "default" path
matches your distributed CJS output file.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 175-180: Remove the debug console.log statements that were left in
the realtime adapter: delete the logs emitted inside the ontrack handler (the
"[Realtime] ontrack event" messages), remove the "[Realtime] Setting up input
audio analysis" and "[Realtime] Input analyser created:" logs around the call to
setupInputAudioAnalysis(localStream) and inputAnalyser, and remove any logging
inside the getAudioVisualization getter (which is called every animation frame).
Keep behavior unchanged—only remove the console.log calls so production consoles
are not spammed.
In `@packages/typescript/ai/src/realtime/types.ts`:
- Around line 210-243: Replace the manually-maintained string union
RealtimeEvent with a derived union from the keys of RealtimeEventPayloads to
keep them in sync: change the declaration of RealtimeEvent to be based on keyof
RealtimeEventPayloads (ensuring it still exports the same literal union type),
and keep RealtimeEventHandler<TEvent extends RealtimeEvent> using the mapped
lookup RealtimeEventPayloads[TEvent] as before so handlers continue to
type-check against the payloads.
Comment on lines
+5
to
+11
| import { | ||
| getCurrentTimeToolDef, | ||
| getWeatherToolDef, | ||
| setReminderToolDef, | ||
| searchKnowledgeToolDef, | ||
| } from '@/lib/realtime-tools' | ||
| import * as z from 'zod' |
Contributor
There was a problem hiding this comment.
ESLint reports two import-ordering violations.
Static analysis flags:
searchKnowledgeToolDef(line 9) should appear alphabetically beforesetReminderToolDef.- The
zodimport (line 11) should come before the@/lib/realtime-toolsimport (lines 5–10).
✏️ Proposed fix
+import * as z from 'zod'
import { createFileRoute } from '@tanstack/react-router'
import { realtimeToken } from '@tanstack/ai'
import { openaiRealtimeToken } from '@tanstack/ai-openai'
import { elevenlabsRealtimeToken } from '@tanstack/ai-elevenlabs'
import {
getCurrentTimeToolDef,
getWeatherToolDef,
+ searchKnowledgeToolDef,
setReminderToolDef,
- searchKnowledgeToolDef,
} from '@/lib/realtime-tools'
-import * as z from 'zod'🧰 Tools
🪛 ESLint
[error] 9-9: Member 'searchKnowledgeToolDef' of the import declaration should be sorted alphabetically.
(sort-imports)
[error] 11-11: zod import should occur before import of @/lib/realtime-tools
(import/order)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@examples/ts-react-chat/src/routes/api.realtime-token.ts` around lines 5 - 11,
Reorder the imports to satisfy ESLint import-order rules: move the `import * as
z from 'zod'` line above the `@/lib/realtime-tools` import group, and within the
named imports from `@/lib/realtime-tools` ensure `searchKnowledgeToolDef`
appears alphabetically before `setReminderToolDef` (so the named list reads e.g.
getCurrentTimeToolDef, getWeatherToolDef, searchKnowledgeToolDef,
setReminderToolDef).
Comment on lines
+89
to
+92
| async connect(): Promise<void> { | ||
| if (this.state.status === 'connected') { | ||
| return | ||
| } |
Contributor
There was a problem hiding this comment.
Concurrent connect() calls are not guarded — two callers can both proceed past the 'connected' check.
The guard only returns early when status === 'connected'; concurrent invocations starting from 'idle' both pass through, fetch two tokens, and open two connections.
🛡️ Proposed fix
- if (this.state.status === 'connected') {
+ if (this.state.status === 'connected' || this.state.status === 'connecting') {
return
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-client/src/realtime-client.ts` around lines 89 - 92,
The connect() method can be invoked concurrently causing two
fetches/connections; protect it by making the status check atomic: when entering
connect() check this.state.status and if not 'connected' immediately set
this.state.status = 'connecting' (or acquire a simple mutex/Promise lock) before
any async work so other callers will see 'connecting' and return/wait; ensure
any early returns or errors restore status to 'idle' and that successful
completion sets status = 'connected'. Update the connect() implementation to use
this new status transition (or an explicit lock) around token fetch and socket
open to prevent parallel connect attempts.
Comment on lines
+379
to
+385
| } else { | ||
| // Assistant transcripts - just update pending, message_complete handles final | ||
| this.updateState({ | ||
| pendingAssistantTranscript: isFinal ? null : transcript, | ||
| }) | ||
| } | ||
| }), |
Contributor
There was a problem hiding this comment.
Streaming assistant transcript overwrites on each delta instead of accumulating.
The OpenAI adapter emits individual delta chunks via the transcript event (isFinal: false). The current code replaces pendingAssistantTranscript with each new chunk rather than concatenating, so users see only the last word/token rather than the growing sentence.
🛡️ Proposed fix
} else {
// Assistant transcripts - just update pending, message_complete handles final
this.updateState({
- pendingAssistantTranscript: isFinal ? null : transcript,
+ pendingAssistantTranscript: isFinal
+ ? null
+ : (this.state.pendingAssistantTranscript ?? '') + transcript,
})
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| } else { | |
| // Assistant transcripts - just update pending, message_complete handles final | |
| this.updateState({ | |
| pendingAssistantTranscript: isFinal ? null : transcript, | |
| }) | |
| } | |
| }), | |
| } else { | |
| // Assistant transcripts - just update pending, message_complete handles final | |
| this.updateState({ | |
| pendingAssistantTranscript: isFinal | |
| ? null | |
| : (this.state.pendingAssistantTranscript ?? '') + transcript, | |
| }) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-client/src/realtime-client.ts` around lines 379 - 385,
The assistant transcript handler currently replaces pendingAssistantTranscript
on each non-final delta, losing earlier tokens; modify the logic in the
transcript event handler (the block that calls updateState and touches
pendingAssistantTranscript) to accumulate chunks instead of overwriting: when
isFinal is false, set pendingAssistantTranscript to (existing
pendingAssistantTranscript || '') + transcript (handling null/undefined), and
when isFinal is true clear pendingAssistantTranscript (or let message_complete
handle finalization) so the transcript grows across deltas; ensure you reference
updateState and pendingAssistantTranscript in the change.
Comment on lines
+43
to
+45
| "dependencies": { | ||
| "@11labs/client": "^0.2.0" | ||
| }, |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
@elevenlabs/client npm latest version changelog
💡 Result:
Latest @elevenlabs/client version: 0.14.0 (released 2026-02-04). [1][2]
Changelog (v0.14.0):
- Reduced audio chunk length from 250ms → 100ms for lower-latency agent conversations. [1][3]
- Normalized
textOnlyoption handling between the top-level option and the overrides object (top-level takes precedence if both are provided). [1][3] - Dependency update:
@elevenlabs/types@0.5.0. [1]
Sources:
[1] GitHub releases (elevenlabs/packages) (github.com)
[2] jsDelivr package page (jsdelivr.com)
[3] ElevenLabs docs changelog (elevenlabs.io)
Citations:
- 1: https://github.com/elevenlabs/packages/releases
- 2: https://www.jsdelivr.com/package/npm/%40elevenlabs/client?utm_source=openai
- 3: https://elevenlabs.io/docs/changelog
🌐 Web query:
@11labs/client npm package status deprecated
💡 Result:
Yes — @11labs/client is deprecated on npm. The npm page shows the deprecation notice with the author message: “This package is no longer maintained. Please use @elevenlabs/client for the latest version.” [1]
🌐 Web query:
@elevenlabs/client migration guide from @11labs/client breaking changes
💡 Result:
Migration: @11labs/client → @elevenlabs/client
1) Swap the package
npm uninstall `@11labs/client`
npm install `@elevenlabs/client`@11labs/client is deprecated and explicitly points to @elevenlabs/client as the maintained replacement. [1]
2) Update imports
Before
import { Conversation } from "@11labs/client";After
import { Conversation } from "@elevenlabs/client";The core usage pattern remains:
const conversation = await Conversation.startSession({ agentId: "...", connectionType: "webrtc" });[2]
3) Re-check “client tools” / agent tool wiring (API breaking change, if you manage agents)
If you’re configuring ConvAI agent tools via the HTTP API, the legacy agent field prompt.tools was removed in July 2025; you must use prompt.tool_ids (and prompt.built_in_tools for built-ins). [3]
4) Review SDK release notes for behavioral changes
ElevenLabs publishes SDK releases in their changelog (look for “ConvAI packages” entries mentioning @elevenlabs/client). [4]
Sources: [1] [2] [3] [4]
@11labs/client is deprecated — migrate to @elevenlabs/client
@11labs/client is deprecated on npm with an explicit notice directing users to @elevenlabs/client. The actively maintained package is at v0.14.0, released 2026-02-04. Using the stale @11labs/client@^0.2.0 pins consumers to an unmaintained dependency and misses significant feature and stability improvements from 12+ versions of releases.
Note: If agent tool configuration is involved, be aware that the legacy prompt.tools API field was removed in July 2025 and must be replaced with prompt.tool_ids (see ElevenLabs SDK changelog for ConvAI packages).
📦 Proposed fix
"dependencies": {
- "@11labs/client": "^0.2.0"
+ "@elevenlabs/client": "^0.14.0"
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| "dependencies": { | |
| "@11labs/client": "^0.2.0" | |
| }, | |
| "dependencies": { | |
| "@elevenlabs/client": "^0.14.0" | |
| }, |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-elevenlabs/package.json` around lines 43 - 45, Update
the deprecated dependency "@11labs/client" in package.json to the maintained
"@elevenlabs/client" (pin to a modern release like "^0.14.0"), and replace all
import/require occurrences of "@11labs/client" in the codebase with
"@elevenlabs/client"; also audit call sites for SDK breaking changes (notably
replace any use of the removed prompt.tools field with prompt.tool_ids and
update any changed method/option names per the ElevenLabs changelog) so the
package compiles and tests pass.
Comment on lines
+116
to
+118
| dataChannel.onerror = (error) => { | ||
| emit('error', { error: new Error(`Data channel error: ${error}`) }) | ||
| } |
Contributor
There was a problem hiding this comment.
dataChannel.onerror passes an Event object to new Error(...) — produces "[object Event]" in the error message.
The onerror handler receives an Event (or RTCErrorEvent), not an Error. Template-literal interpolation of a DOM event yields [object Event], making the error message useless.
🛡️ Proposed fix
- dataChannel.onerror = (error) => {
- emit('error', { error: new Error(`Data channel error: ${error}`) })
+ dataChannel.onerror = (event) => {
+ const rtcError = event as RTCErrorEvent
+ const message = rtcError.error?.message ?? 'Data channel error'
+ emit('error', { error: new Error(message) })
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| dataChannel.onerror = (error) => { | |
| emit('error', { error: new Error(`Data channel error: ${error}`) }) | |
| } | |
| dataChannel.onerror = (event) => { | |
| const rtcError = event as RTCErrorEvent | |
| const message = rtcError.error?.message ?? 'Data channel error' | |
| emit('error', { error: new Error(message) }) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 116 -
118, The onerror handler for dataChannel currently stringifies the Event,
producing "[object Event]"; instead, extract a useful message from the event
(handle RTCErrorEvent by checking event.error?.message or event.error, fallback
to event.message or event.type) and include that in the emitted Error, or attach
the original event as the Error's cause/metadata; update the dataChannel.onerror
callback (the code that calls emit('error', { error: new Error(`Data channel
error: ${error}`) }) ) to build the Error with the extracted message and/or
include the original event as the cause so logs contain actionable info.
Comment on lines
+497
to
+502
| interrupt() { | ||
| sendEvent({ type: 'response.cancel' }) | ||
| currentMode = 'listening' | ||
| emit('mode_change', { mode: 'listening' }) | ||
| emit('interrupted', { messageId: currentMessageId ?? undefined }) | ||
| }, |
Contributor
There was a problem hiding this comment.
interrupt() and the conversation.item.truncated server event both emit 'interrupted' for the same message.
interrupt() immediately emits 'interrupted' (line 501), and later the server sends conversation.item.truncated which also emits 'interrupted' (line 309). The RealtimeClient listener runs twice for the same interruption: the message is marked interrupted: true twice and onInterrupted fires twice. Consider only emitting optimistically in interrupt() and suppressing the duplicate from conversation.item.truncated when a client-side interrupt was already issued (e.g., track a clientInterruptPending flag).
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 497 -
502, interrupt() is emitting 'interrupted' optimistically then the server's
conversation.item.truncated handler also emits 'interrupted', causing duplicate
notifications; introduce a clientInterruptPending boolean that interrupt() sets
to true before emitting (and include currentMessageId), then in the
conversation.item.truncated handler check clientInterruptPending and suppress
the emit('interrupted', ...) for the same messageId if the flag is set, finally
clear clientInterruptPending when the server event arrives (or on other flows
like response.cancel acknowledgement) so future interruptions behave normally;
reference interrupt(), the conversation.item.truncated handler,
currentMessageId, and emit('interrupted', ...) when adding the flag logic.
Comment on lines
+13
to
+15
| // Empty frequency data for when client is not connected | ||
| const emptyFrequencyData = new Uint8Array(128) | ||
| const emptyTimeDomainData = new Uint8Array(128).fill(128) |
Contributor
There was a problem hiding this comment.
Fallback empty-array sizes (128) don't match the adapter's actual output sizes (1024 / 2048).
When the client is not connected, getInputFrequencyData() returns a 128-element array; when connected, the OpenAI adapter returns 1024 elements (frequencyBinCount = fftSize/2). Visualization consumers that assume a fixed array length will misbehave when switching between the disconnected fallback and live data.
🛡️ Proposed fix — match the adapter's fftSize
-const emptyFrequencyData = new Uint8Array(128)
-const emptyTimeDomainData = new Uint8Array(128).fill(128)
+const emptyFrequencyData = new Uint8Array(1024) // frequencyBinCount = fftSize/2 = 1024
+const emptyTimeDomainData = new Uint8Array(2048).fill(128) // fftSize = 2048🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 13 - 15,
The fallback arrays emptyFrequencyData and emptyTimeDomainData are sized at 128
but must match the adapter's FFT output (frequencyBinCount = 1024 and time
domain length = 2048) to avoid consumers seeing length changes; update
emptyFrequencyData to new Uint8Array(1024) and emptyTimeDomainData to new
Uint8Array(2048). Keep the time-domain default values (e.g., fill(128)) as
before so that getInputFrequencyData() returns the same-shaped arrays whether
connected or not.
Comment on lines
+153
to
+157
| useEffect(() => { | ||
| return () => { | ||
| clientRef.current?.destroy() | ||
| } | ||
| }, []) |
Contributor
There was a problem hiding this comment.
React StrictMode: destroy() clears stateChangeCallbacks but clientRef is not nulled — the subscription is permanently lost.
In React 18 Strict Mode the cleanup effect runs, calling destroy() which clears stateChangeCallbacks. On re-mount, if (!clientRef.current) evaluates to false (the destroyed client still exists), so the onStateChange subscription is never re-established. pendingUserTranscript / pendingAssistantTranscript will never update for the rest of that session.
🛡️ Proposed fix
return () => {
clientRef.current?.destroy()
+ clientRef.current = null
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| useEffect(() => { | |
| return () => { | |
| clientRef.current?.destroy() | |
| } | |
| }, []) | |
| useEffect(() => { | |
| return () => { | |
| clientRef.current?.destroy() | |
| clientRef.current = null | |
| } | |
| }, []) |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 153 -
157, The cleanup effect currently calls clientRef.current?.destroy() but leaves
clientRef.current set, so after React StrictMode unmount/remount the destroyed
client exists and onStateChange subscriptions aren't re-registered causing
pendingUserTranscript/pendingAssistantTranscript to never update; update the
cleanup in the useEffect to call clientRef.current?.destroy() and then null out
clientRef.current (or set to undefined) so subsequent mounts detect no client
and recreate it and re-subscribe via onStateChange.
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
packages/typescript/ai-openai/package.json (1)
15-20: Add subpath export for the realtime adapter to enable tree-shaking.The PR adds a new realtime adapter at
src/realtime/adapter.tswith proper module structure (realtime/index.tsexists), but theexportssection only exposes the root entry point. Since realtime exports are currently re-exported from the main index.ts, they cannot be tree-shaken. Consider adding a dedicated subpath export to follow the tree-shakeable adapter pattern:Suggested package.json change
"exports": { ".": { "types": "./dist/esm/index.d.ts", "import": "./dist/esm/index.js" }, + "./realtime": { + "types": "./dist/esm/realtime/index.d.ts", + "import": "./dist/esm/realtime/index.js" + } },🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/package.json` around lines 15 - 20, The package exports only the root entry and needs a subpath export for the realtime adapter so it can be imported/tree-shaken directly; add a new export key for the realtime subpath (e.g., "./realtime") that points to the built ESM JS and d.ts outputs for src/realtime/adapter.ts (and/or realtime/index.ts) so consumers can import the adapter via the subpath instead of through the main index; update the "exports" object to include that subpath with "types" -> dist/esm/realtime/index.d.ts (or adapter d.ts) and "import" -> dist/esm/realtime/index.js (or adapter.js).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@packages/typescript/ai-openai/package.json`:
- Around line 15-20: The package exports only the root entry and needs a subpath
export for the realtime adapter so it can be imported/tree-shaken directly; add
a new export key for the realtime subpath (e.g., "./realtime") that points to
the built ESM JS and d.ts outputs for src/realtime/adapter.ts (and/or
realtime/index.ts) so consumers can import the adapter via the subpath instead
of through the main index; update the "exports" object to include that subpath
with "types" -> dist/esm/realtime/index.d.ts (or adapter d.ts) and "import" ->
dist/esm/realtime/index.js (or adapter.js).
AlemTuzlak
reviewed
Mar 2, 2026
Comment on lines
+59
to
+84
| adapter: openaiRealtimeToken({ | ||
| model: 'gpt-4o-realtime-preview', | ||
| voice: 'alloy', | ||
| instructions: `You are a helpful, friendly voice assistant with access to several tools. | ||
|
|
||
| You can: | ||
| - Tell the user the current time and date (getCurrentTime) | ||
| - Get weather information for any location (getWeather) | ||
| - Set reminders for the user (setReminder) | ||
| - Search a knowledge base for information (searchKnowledge) | ||
|
|
||
| Keep your responses concise and conversational since this is a voice interface. | ||
| When using tools, briefly explain what you're doing and then share the results naturally. | ||
| Be friendly and engaging!`, | ||
| turnDetection: { | ||
| type: 'server_vad', | ||
| threshold: 0.5, | ||
| prefix_padding_ms: 300, | ||
| silence_duration_ms: 500, | ||
| }, | ||
| inputAudioTranscription: { | ||
| model: 'whisper-1', | ||
| }, | ||
| tools, | ||
| toolChoice: 'auto', | ||
| }), |
Contributor
There was a problem hiding this comment.
why is everything inside the openaiRealtimeToken opposed to the token? feels like a different convnetion to chat
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
♻️ Duplicate comments (3)
packages/typescript/ai-openai/src/realtime/adapter.ts (3)
103-123:⚠️ Potential issue | 🟠 MajorMake
dataChannelReadyfail fast instead of hanging forever.Line 618 awaits a promise that only resolves on
onopen. If the channel errors or closes before opening,connect()never settles, and Line 122 currently reduces the failure to"[object Event]", so callers get neither a rejection nor a useful error. Reject ononerror/onclose(ideally with a timeout) and extract the RTC error message.🛠️ Suggested fix
- const dataChannelReady = new Promise<void>((resolve) => { - dataChannel!.onopen = () => { - flushPendingEvents() - emit('status_change', { status: 'connected' as RealtimeStatus }) - resolve() - } - }) + let rejectDataChannelReady: ((error: Error) => void) | null = null + const dataChannelReady = new Promise<void>((resolve, reject) => { + rejectDataChannelReady = reject + dataChannel!.onopen = () => { + flushPendingEvents() + emit('status_change', { status: 'connected' as RealtimeStatus }) + resolve() + } + dataChannel!.onclose = () => { + reject(new Error('Realtime data channel closed before opening')) + } + }) @@ - dataChannel.onerror = (error) => { - emit('error', { error: new Error(`Data channel error: ${error}`) }) + dataChannel.onerror = (event) => { + const rtcError = event as RTCErrorEvent + const message = + rtcError.error?.message ?? event.type ?? 'Realtime data channel error' + const error = new Error(message, { cause: event }) + emit('error', { error }) + if (dataChannel?.readyState !== 'open') { + rejectDataChannelReady?.(error) + } } @@ - await dataChannelReady + await Promise.race([ + dataChannelReady, + new Promise<never>((_, reject) => + setTimeout( + () => reject(new Error('Timed out waiting for realtime data channel')), + 10_000, + ), + ), + ])Also applies to: 616-618
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 103 - 123, The dataChannelReady promise only resolves on dataChannel.onopen and can hang if the channel errors or closes; update the logic in the dataChannelReady construction so it also rejects on dataChannel.onerror and dataChannel.onclose (and optionally after a short timeout) with a meaningful Error that extracts the RTC error/event details (instead of producing "[object Event]"), so callers of connect() receive a rejection; adjust handlers referenced (dataChannel.onopen, dataChannel.onerror, dataChannel.onclose, and the connect() flow that awaits dataChannelReady) to ensure proper rejection and cleanup.
306-308:⚠️ Potential issue | 🟡 MinorSuppress the duplicate
'interrupted'emission.
interrupt()emits'interrupted'immediately at Line 525, andconversation.item.truncatedemits it again at Line 307 for the same response. Subscribers will see duplicate interruption callbacks and duplicate state transitions.Also applies to: 521-526
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 306 - 308, The handler for the 'conversation.item.truncated' event is emitting 'interrupted' a second time for the same response (duplicate of interrupt()), so add a guard to avoid duplicate emissions: detect if the currentMessageId has already been marked interrupted (e.g., track a Set like interruptedMessageIds or a boolean flag on the response context) and only call emit('interrupted', { messageId: currentMessageId }) when that ID has not yet been emitted; update both the interrupt() function (which currently emits at the time of interruption) and the 'conversation.item.truncated' case to check the same guard before emitting to ensure subscribers receive a single interruption callback per message.
153-175:⚠️ Potential issue | 🔴 CriticalClean up media/WebRTC resources when SDP negotiation fails.
After Line 145 has added microphone tracks, any failure in
createOffer(),setLocalDescription(),fetch(), orsetRemoteDescription()escapesconnect()without stopping tracks or closing the peer connection/data channel. That leaks the session and can leave the browser mic indicator on.🛠️ Suggested fix
+ const cleanupFailedConnection = () => { + localStream?.getTracks().forEach((track) => track.stop()) + localStream = null + dataChannel?.close() + pc.close() + } + - const offer = await pc.createOffer() - await pc.setLocalDescription(offer) - - // Send SDP to OpenAI and get answer - const sdpResponse = await fetch(`${OPENAI_REALTIME_URL}?model=${model}`, { - method: 'POST', - headers: { - Authorization: `Bearer ${token.token}`, - 'Content-Type': 'application/sdp', - }, - body: offer.sdp, - }) - - if (!sdpResponse.ok) { - const errorText = await sdpResponse.text() - throw new Error( - `Failed to establish WebRTC connection: ${sdpResponse.status} - ${errorText}`, - ) - } - - const answerSdp = await sdpResponse.text() - await pc.setRemoteDescription({ type: 'answer', sdp: answerSdp }) + try { + const offer = await pc.createOffer() + await pc.setLocalDescription(offer) + + const sdpResponse = await fetch(`${OPENAI_REALTIME_URL}?model=${model}`, { + method: 'POST', + headers: { + Authorization: `Bearer ${token.token}`, + 'Content-Type': 'application/sdp', + }, + body: offer.sdp, + }) + + if (!sdpResponse.ok) { + const errorText = await sdpResponse.text() + throw new Error( + `Failed to establish WebRTC connection: ${sdpResponse.status} - ${errorText}`, + ) + } + + const answerSdp = await sdpResponse.text() + await pc.setRemoteDescription({ type: 'answer', sdp: answerSdp }) + } catch (error) { + cleanupFailedConnection() + throw error + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 153 - 175, The SDP negotiation code can throw before cleanup runs and leaks media/RTC resources; wrap the offer/answer/fetch/setRemoteDescription sequence in a try/catch (or try/finally) inside the connect function so that on any error you stop all tracks on the local MediaStream (e.g. localStream.getTracks().forEach(t => t.stop())), close the RTCPeerConnection (pc.close()) and close the data channel if opened (dataChannel.close()), then rethrow the error; ensure these cleanup steps reference the existing symbols pc, dataChannel, and localStream so resources are always released when createOffer(), setLocalDescription(), fetch(...), or setRemoteDescription(...) fail.
🧹 Nitpick comments (3)
docs/architecture/realtime-chat.md (2)
287-287: Minor grammar: hyphenate compound modifier."JSON Schema compliant" should be "JSON Schema-compliant" when used as an adjective.
Proposed fix
-Tool schemas are converted from Zod (or any Standard JSON Schema compliant library) to JSON Schema using `convertSchemaToJsonSchema()` before being sent to the provider. +Tool schemas are converted from Zod (or any Standard JSON Schema-compliant library) to JSON Schema using `convertSchemaToJsonSchema()` before being sent to the provider.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/architecture/realtime-chat.md` at line 287, The phrase "JSON Schema compliant" in the sentence referencing convertSchemaToJsonSchema() should be hyphenated as "JSON Schema-compliant" when used as an adjective; update the text to read: "Tool schemas are converted from Zod (or any JSON Schema-compliant library) to JSON Schema using convertSchemaToJsonSchema() before being sent to the provider."
19-19: Add language identifier to fenced code block.The ASCII diagram code block should have a language identifier for consistent rendering.
Proposed fix
-``` +```text ┌─────────────────────────────────────────────────────────────────┐🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/architecture/realtime-chat.md` at line 19, The fenced ASCII diagram code block is missing a language identifier; update the opening fence from ``` to ```text so the diagram uses a plain-text language tag for consistent rendering (locate the ASCII diagram fenced block and change its opening fence to "```text").packages/typescript/ai-client/src/realtime-client.ts (1)
310-332: Token refresh doesn't notify the active connection.When the token is refreshed (line 323), the new token is stored but the active connection continues using the old credentials. The comment on lines 325-326 acknowledges this, but reconnection or token update isn't implemented.
Consider adding a mechanism to pass the refreshed token to the adapter or trigger reconnection when the token expires.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-client/src/realtime-client.ts` around lines 310 - 332, The token is refreshed in refreshToken() but the active connection isn't updated; after assigning this.token and calling scheduleTokenRefresh(), notify the adapter or force a reconnection so the live connection uses the new credentials. Modify refreshToken() to, after this.token = await this.options.getToken(), call a token-update or reconnect hook on the connection layer (e.g., this.adapter.updateToken?.(this.token) or if unavailable call the existing reconnect/connect methods such as this.disconnect(); this.connect();), and ensure errors still flow through updateState and this.options.onError.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/architecture/realtime-chat.md`:
- Around line 82-86: Update the example that configures the realtime adapter to
use the current production model name instead of the deprecated
'gpt-4o-realtime-preview'; locate the openaiRealtimeToken(...) call (the
adapter: openaiRealtimeToken block) and replace the model value with the current
production model string provided by the platform, ensuring the example reflects
the supported model name.
In `@examples/ts-react-chat/src/lib/use-realtime.ts`:
- Around line 19-24: The code is using the deprecated model string
'gpt-4o-realtime-preview' in the openaiRealtimeToken call; update the model to
the supported name 'gpt-realtime' inside the openaiRealtimeToken options (the
call site is the realtimeToken wrapper around openaiRealtimeToken in
use-realtime.ts) so the realtimeToken({ adapter: openaiRealtimeToken({ model:
... }) }) uses 'gpt-realtime' instead of the preview model.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 132-146: connect() is eagerly calling
navigator.mediaDevices.getUserMedia and adding audio tracks to the
RTCPeerConnection (localStream and pc.addTrack), which forces mic permission for
text-only sessions and contradicts startAudioCapture(); change this by either
deferring getUserMedia until startAudioCapture() is called, or (simpler) keep
the current getUserMedia call but immediately disable each added AudioTrack
(track.enabled = false) after pc.addTrack so the microphone stays inactive until
startAudioCapture() sets track.enabled = true; update startAudioCapture() /
stopAudioCapture() to enable/disable those tracks on the existing localStream
and ensure getUserMedia is only created once (reuse localStream) if deferring
acquisition.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts`:
- Around line 220-227: setVADMode currently only calls setVADModeState and never
propagates the change to the connected session; add a way to update the live
session config and call it from setVADMode. Implement an updateSession(...)
method on RealtimeClient that delegates to connection.updateSession(...) (or
expose connection.updateSession directly), then update the setVADMode callback
to call that new RealtimeClient.updateSession({ vadMode: newMode }) when a
client/connection is active (fall back to only setVADModeState if not
connected). Ensure you reference the RealtimeClient instance used in this hook
and keep setVADModeState for local state updates.
---
Duplicate comments:
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 103-123: The dataChannelReady promise only resolves on
dataChannel.onopen and can hang if the channel errors or closes; update the
logic in the dataChannelReady construction so it also rejects on
dataChannel.onerror and dataChannel.onclose (and optionally after a short
timeout) with a meaningful Error that extracts the RTC error/event details
(instead of producing "[object Event]"), so callers of connect() receive a
rejection; adjust handlers referenced (dataChannel.onopen, dataChannel.onerror,
dataChannel.onclose, and the connect() flow that awaits dataChannelReady) to
ensure proper rejection and cleanup.
- Around line 306-308: The handler for the 'conversation.item.truncated' event
is emitting 'interrupted' a second time for the same response (duplicate of
interrupt()), so add a guard to avoid duplicate emissions: detect if the
currentMessageId has already been marked interrupted (e.g., track a Set like
interruptedMessageIds or a boolean flag on the response context) and only call
emit('interrupted', { messageId: currentMessageId }) when that ID has not yet
been emitted; update both the interrupt() function (which currently emits at the
time of interruption) and the 'conversation.item.truncated' case to check the
same guard before emitting to ensure subscribers receive a single interruption
callback per message.
- Around line 153-175: The SDP negotiation code can throw before cleanup runs
and leaks media/RTC resources; wrap the offer/answer/fetch/setRemoteDescription
sequence in a try/catch (or try/finally) inside the connect function so that on
any error you stop all tracks on the local MediaStream (e.g.
localStream.getTracks().forEach(t => t.stop())), close the RTCPeerConnection
(pc.close()) and close the data channel if opened (dataChannel.close()), then
rethrow the error; ensure these cleanup steps reference the existing symbols pc,
dataChannel, and localStream so resources are always released when
createOffer(), setLocalDescription(), fetch(...), or setRemoteDescription(...)
fail.
---
Nitpick comments:
In `@docs/architecture/realtime-chat.md`:
- Line 287: The phrase "JSON Schema compliant" in the sentence referencing
convertSchemaToJsonSchema() should be hyphenated as "JSON Schema-compliant" when
used as an adjective; update the text to read: "Tool schemas are converted from
Zod (or any JSON Schema-compliant library) to JSON Schema using
convertSchemaToJsonSchema() before being sent to the provider."
- Line 19: The fenced ASCII diagram code block is missing a language identifier;
update the opening fence from ``` to ```text so the diagram uses a plain-text
language tag for consistent rendering (locate the ASCII diagram fenced block and
change its opening fence to "```text").
In `@packages/typescript/ai-client/src/realtime-client.ts`:
- Around line 310-332: The token is refreshed in refreshToken() but the active
connection isn't updated; after assigning this.token and calling
scheduleTokenRefresh(), notify the adapter or force a reconnection so the live
connection uses the new credentials. Modify refreshToken() to, after this.token
= await this.options.getToken(), call a token-update or reconnect hook on the
connection layer (e.g., this.adapter.updateToken?.(this.token) or if unavailable
call the existing reconnect/connect methods such as this.disconnect();
this.connect();), and ensure errors still flow through updateState and
this.options.onError.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 15f0d068-2bdc-439c-8e30-8179dcd638ff
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (13)
docs/architecture/realtime-chat.mdexamples/ts-react-chat/src/components/AudioSparkline.tsxexamples/ts-react-chat/src/lib/use-realtime.tsexamples/ts-react-chat/src/routeTree.gen.tsexamples/ts-react-chat/src/routes/realtime.tsxpackages/typescript/ai-client/src/realtime-client.tspackages/typescript/ai-client/src/realtime-types.tspackages/typescript/ai-openai/src/realtime/adapter.tspackages/typescript/ai-openai/src/realtime/token.tspackages/typescript/ai-openai/src/realtime/types.tspackages/typescript/ai-react/src/realtime-types.tspackages/typescript/ai-react/src/use-realtime-chat.tspackages/typescript/ai/src/realtime/types.ts
🚧 Files skipped from review as they are similar to previous changes (3)
- examples/ts-react-chat/src/routes/realtime.tsx
- packages/typescript/ai-openai/src/realtime/token.ts
- packages/typescript/ai/src/realtime/types.ts
docs/architecture/realtime-chat.md
Outdated
Comment on lines
+82
to
+86
| adapter: openaiRealtimeToken({ | ||
| model: 'gpt-4o-realtime-preview', | ||
| }), | ||
| }) | ||
| ``` |
Contributor
There was a problem hiding this comment.
Documentation uses deprecated model name.
The example shows 'gpt-4o-realtime-preview' which is being deprecated. Update to reflect the current production model when available.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/architecture/realtime-chat.md` around lines 82 - 86, Update the example
that configures the realtime adapter to use the current production model name
instead of the deprecated 'gpt-4o-realtime-preview'; locate the
openaiRealtimeToken(...) call (the adapter: openaiRealtimeToken block) and
replace the model value with the current production model string provided by the
platform, ensuring the example reflects the supported model name.
Comment on lines
+19
to
+24
| if (data.provider === 'openai') { | ||
| return realtimeToken({ | ||
| adapter: openaiRealtimeToken({ | ||
| model: 'gpt-4o-realtime-preview', | ||
| }), | ||
| }) |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, examine the file to confirm the model being used
cat -n examples/ts-react-chat/src/lib/use-realtime.ts | head -30Repository: TanStack/ai
Length of output: 1272
🌐 Web query:
OpenAI gpt-realtime model availability deprecation 2026
💡 Result:
- Current availability (as of Mar 6, 2026):
gpt-realtimeis generally available in the OpenAI Realtime API (audio + text real-time over WebRTC/WebSocket/SIP), with an aliasgpt-realtimeand a published snapshot (e.g.,gpt-realtime-2025-08-28). (developers.openai.com) - Deprecations affecting “realtime” models in 2026: OpenAI deprecated the older
gpt-4o-realtime-previewfamily and scheduled it for shutdown on 2026-02-27, withgpt-realtimeas the recommended replacement. (platform.openai.com) - Is
gpt-realtimeitself deprecated in 2026? I found no OpenAI notice indicating a 2026 deprecation/shutdown forgpt-realtimeitself—only for the older preview realtime models. (developers.openai.com)
Citations:
- 1: https://developers.openai.com/api/docs/models/gpt-realtime?utm_source=openai
- 2: https://platform.openai.com/docs/deprecations/2023-03-20-codex-models%23.doc?utm_source=openai
- 3: https://developers.openai.com/api/docs/models/gpt-realtime?utm_source=openai
🌐 Web query:
OpenAI gpt-4o-realtime-preview model status
💡 Result:
gpt-4o-realtime-preview is deprecated (preview-only) and is scheduled to be shut down on March 24, 2026 (along with gpt-4o-realtime-preview-2025-06-03 and gpt-4o-realtime-preview-2024-12-17). The recommended replacement is gpt-realtime. [1]
As of today (March 6, 2026), that shutdown date is still in the future, so the model may still be callable, but you should migrate now to avoid disruption. [1][2]
References:
[1] OpenAI API Deprecations (gpt-4o-realtime-preview models shutdown schedule) (platform.openai.com)
[2] OpenAI blog: introducing gpt-realtime (GA replacement model) (openai.com)
Citations:
- 1: https://platform.openai.com/docs/deprecations
- 2: https://openai.com/blog/introducing-gpt-realtime/
🏁 Script executed:
# Check if there are other uses of gpt-4o-realtime-preview in the codebase
rg 'gpt-4o-realtime-preview' --type ts --type tsx --type js --type jsxRepository: TanStack/ai
Length of output: 82
🏁 Script executed:
# Search for the deprecated model string across the codebase
rg 'gpt-4o-realtime-preview'Repository: TanStack/ai
Length of output: 908
Use gpt-realtime instead of the deprecated preview model.
gpt-4o-realtime-preview is scheduled for shutdown on 2026-03-24. Replace with gpt-realtime, which is now generally available and the recommended model for the OpenAI Realtime API.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@examples/ts-react-chat/src/lib/use-realtime.ts` around lines 19 - 24, The
code is using the deprecated model string 'gpt-4o-realtime-preview' in the
openaiRealtimeToken call; update the model to the supported name 'gpt-realtime'
inside the openaiRealtimeToken options (the call site is the realtimeToken
wrapper around openaiRealtimeToken in use-realtime.ts) so the realtimeToken({
adapter: openaiRealtimeToken({ model: ... }) }) uses 'gpt-realtime' instead of
the preview model.
Comment on lines
+132
to
+146
| // IMPORTANT: Request microphone access and add audio track BEFORE creating offer | ||
| // OpenAI's Realtime API requires an audio track in the SDP offer | ||
| try { | ||
| localStream = await navigator.mediaDevices.getUserMedia({ | ||
| audio: { | ||
| echoCancellation: true, | ||
| noiseSuppression: true, | ||
| sampleRate: 24000, | ||
| }, | ||
| }) | ||
|
|
||
| // Add audio track to peer connection | ||
| for (const track of localStream.getAudioTracks()) { | ||
| pc.addTrack(track, localStream) | ||
| } |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
cat -n packages/typescript/ai-openai/src/realtime/adapter.ts | sed -n '120,160p'Repository: TanStack/ai
Length of output: 1576
🏁 Script executed:
cat -n packages/typescript/ai-openai/src/realtime/adapter.ts | sed -n '410,460p'Repository: TanStack/ai
Length of output: 1710
🏁 Script executed:
wc -l packages/typescript/ai-openai/src/realtime/adapter.tsRepository: TanStack/ai
Length of output: 112
🏁 Script executed:
cat -n packages/typescript/ai-openai/src/realtime/adapter.ts | sed -n '140,160p'Repository: TanStack/ai
Length of output: 812
🏁 Script executed:
rg "track\.enabled" packages/typescript/ai-openai/src/realtime/adapter.ts -A 2 -B 2Repository: TanStack/ai
Length of output: 326
Eager microphone acquisition undermines the startAudioCapture() / stopAudioCapture() API contract.
Lines 135–145 request microphone access and add live audio tracks during connect(), forcing all sessions—including text-only flows—to require microphone permission at connection time. However, startAudioCapture() (lines 424–431) only toggles track.enabled, not capture initiation; it does not actually define the capture boundary.
This creates two problems:
- Text-only callers receive unexpected microphone permission prompts.
- The public capture control API misleads developers about when audio capture begins.
Fix: Either disable tracks immediately after adding them (lines 144–145), then enable in startAudioCapture(); or defer getUserMedia() until the caller explicitly invokes startAudioCapture(). The former is simpler if OpenAI's Realtime API requires an audio m-line in the SDP offer; the latter better respects user privacy.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 132 -
146, connect() is eagerly calling navigator.mediaDevices.getUserMedia and adding
audio tracks to the RTCPeerConnection (localStream and pc.addTrack), which
forces mic permission for text-only sessions and contradicts
startAudioCapture(); change this by either deferring getUserMedia until
startAudioCapture() is called, or (simpler) keep the current getUserMedia call
but immediately disable each added AudioTrack (track.enabled = false) after
pc.addTrack so the microphone stays inactive until startAudioCapture() sets
track.enabled = true; update startAudioCapture() / stopAudioCapture() to
enable/disable those tracks on the existing localStream and ensure getUserMedia
is only created once (reuse localStream) if deferring acquisition.
Comment on lines
+220
to
+227
| // VAD mode control | ||
| const setVADMode = useCallback( | ||
| (newMode: 'server' | 'semantic' | 'manual') => { | ||
| setVADModeState(newMode) | ||
| // TODO: Update session config if connected | ||
| }, | ||
| [], | ||
| ) |
Contributor
There was a problem hiding this comment.
setVADMode only updates local state — session configuration is not propagated.
The TODO comment (line 224) indicates this is incomplete. The connected session won't receive the VAD mode change since RealtimeClient doesn't expose a method to update session config post-connection.
Proposed approach
Either:
- Add an
updateSession()method toRealtimeClientthat delegates toconnection.updateSession() - Or document that
vadModecan only be set before connecting
const setVADMode = useCallback(
(newMode: 'server' | 'semantic' | 'manual') => {
setVADModeState(newMode)
- // TODO: Update session config if connected
+ // VAD mode can only be changed before connecting
+ // To change during a session, disconnect and reconnect with new mode
},
[],
)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-react/src/use-realtime-chat.ts` around lines 220 -
227, setVADMode currently only calls setVADModeState and never propagates the
change to the connected session; add a way to update the live session config and
call it from setVADMode. Implement an updateSession(...) method on
RealtimeClient that delegates to connection.updateSession(...) (or expose
connection.updateSession directly), then update the setVADMode callback to call
that new RealtimeClient.updateSession({ vadMode: newMode }) when a
client/connection is active (fall back to only setVADModeState if not
connected). Ensure you reference the RealtimeClient instance used in this hook
and keep setVADModeState for local state updates.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
♻️ Duplicate comments (5)
packages/typescript/ai-openai/src/realtime/adapter.ts (3)
121-123:⚠️ Potential issue | 🟡 MinorBuild the emitted error from the event, not
${error}.
RTCDataChannel.onerrorreceives an event object, so this currently emitsData channel error: [object Event]. Extract theRTCErrorEventdetails or at least fall back toevent.type.Suggested fix
- dataChannel.onerror = (error) => { - emit('error', { error: new Error(`Data channel error: ${error}`) }) + dataChannel.onerror = (event) => { + const rtcError = event as RTCErrorEvent + const message = + rtcError.error?.message ?? + (event as ErrorEvent).message ?? + `Data channel error (${event.type})` + emit('error', { error: new Error(message) }) }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 121 - 123, The current dataChannel.onerror handler constructs an Error from the event object, producing "Data channel error: [object Event]"; update the handler for dataChannel.onerror to build the emitted Error from the actual event details: treat the callback parameter as an RTCErrorEvent (or Event) and extract the underlying error (e.g., evt.error or evt.error?.message/name) and/or fall back to evt.type and evt.message, then call emit('error', { error: new Error(`Data channel error: ${extractedDetail}`) }) so the emitted error contains meaningful information; locate the dataChannel.onerror assignment in adapter.ts and replace the stringified event use with the extracted fields.
153-178:⚠️ Potential issue | 🔴 CriticalClean up WebRTC resources if SDP negotiation fails.
Once
getUserMedia()succeeds, any failure increateOffer(),fetch(), reading the answer, orsetRemoteDescription()leaves the mic track active and the peer connection/data channel open. This block needs a try/catch cleanup path before rethrowing.Suggested shape
+ function cleanupNegotiationFailure() { + if (localStream) { + for (const track of localStream.getTracks()) { + track.stop() + } + localStream = null + } + dataChannel?.close() + dataChannel = null + pc.close() + } + - const offer = await pc.createOffer() - await pc.setLocalDescription(offer) - - const sdpResponse = await fetch(`${OPENAI_REALTIME_URL}?model=${model}`, { - method: 'POST', - headers: { - Authorization: `Bearer ${token.token}`, - 'Content-Type': 'application/sdp', - }, - body: offer.sdp, - }) - - if (!sdpResponse.ok) { - const errorText = await sdpResponse.text() - throw new Error( - `Failed to establish WebRTC connection: ${sdpResponse.status} - ${errorText}`, - ) - } - - const answerSdp = await sdpResponse.text() - await pc.setRemoteDescription({ type: 'answer', sdp: answerSdp }) - - setupInputAudioAnalysis(localStream) + try { + const offer = await pc.createOffer() + await pc.setLocalDescription(offer) + + const sdpResponse = await fetch(`${OPENAI_REALTIME_URL}?model=${model}`, { + method: 'POST', + headers: { + Authorization: `Bearer ${token.token}`, + 'Content-Type': 'application/sdp', + }, + body: offer.sdp, + }) + + if (!sdpResponse.ok) { + const errorText = await sdpResponse.text() + throw new Error( + `Failed to establish WebRTC connection: ${sdpResponse.status} - ${errorText}`, + ) + } + + const answerSdp = await sdpResponse.text() + await pc.setRemoteDescription({ type: 'answer', sdp: answerSdp }) + setupInputAudioAnalysis(localStream) + } catch (error) { + cleanupNegotiationFailure() + throw error + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 153 - 178, The SDP negotiation (createOffer, setLocalDescription, fetch to OPENAI_REALTIME_URL, reading answerSdp, setRemoteDescription) must be wrapped in a try/catch so failures clean up WebRTC resources: on error stop all tracks on localStream, close/cleanup the RTCPeerConnection instance (pc.close()), and close any open DataChannel if created, then rethrow the error; call setupInputAudioAnalysis(localStream) only after successful setRemoteDescription. Locate these symbols: pc, localStream, createOffer, setLocalDescription, fetch, setRemoteDescription, setupInputAudioAnalysis to implement the try/catch and resource cleanup.
132-146:⚠️ Potential issue | 🟠 MajorMicrophone capture starts during
connect(), notstartAudioCapture().The tracks are added enabled by default, so text-only sessions still prompt for mic access and start streaming before the public capture API is called. Disable them immediately after
pc.addTrack(...)or defergetUserMedia()untilstartAudioCapture().Minimal fix if the SDP still needs an audio m-line
for (const track of localStream.getAudioTracks()) { pc.addTrack(track, localStream) + track.enabled = false }Also applies to: 444-460
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 132 - 146, The code calls navigator.mediaDevices.getUserMedia and adds audio tracks inside connect(), causing microphone prompts for text-only sessions; either defer calling navigator.mediaDevices.getUserMedia and adding tracks until startAudioCapture() (move the getUserMedia + pc.addTrack logic from connect() into startAudioCapture()), or if the SDP must include an audio m-line keep the current flow but immediately disable the tracks after adding them by setting each track.enabled = false (and only set enabled = true in startAudioCapture()); apply the same change for the duplicate block around lines 444-460.packages/typescript/ai-elevenlabs/src/realtime/adapter.ts (1)
84-87:⚠️ Potential issue | 🟠 MajorDefer the initial connection emits until listeners can subscribe.
Conversation.startSession()can invokeonConnectbeforeconnect()returns, so these firststatus_change/mode_changeevents are dropped beforeRealtimeClientregisters handlers.Suggested fix
onConnect: () => { - emit('status_change', { status: 'connected' as RealtimeStatus }) - emit('mode_change', { mode: 'listening' }) + queueMicrotask(() => { + emit('status_change', { status: 'connected' as RealtimeStatus }) + emit('mode_change', { mode: 'listening' }) + }) },🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-elevenlabs/src/realtime/adapter.ts` around lines 84 - 87, The initial emits in onConnect are happening synchronously and may fire before RealtimeClient handlers are registered (Conversation.startSession() can call onConnect before connect() returns); change onConnect to defer emitting the initial "status_change" and "mode_change" until after the caller can subscribe (e.g., schedule emit calls using a microtask or next tick mechanism such as Promise.resolve().then(...) or process.nextTick/setImmediate) so that emit('status_change', ...) and emit('mode_change', ...) occur asynchronously; update the onConnect handler in adapter.ts accordingly so RealtimeClient receives those events (referencing onConnect, emit, RealtimeClient, Conversation.startSession, and connect).docs/architecture/realtime-chat.md (1)
82-87:⚠️ Potential issue | 🟡 MinorAvoid hard-coding a preview model in the example.
Preview model names churn quickly, so this snippet will age badly unless it is kept in sync with the currently supported OpenAI realtime model. Either verify the current model before publishing or drop the explicit
modelline from the example.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/architecture/realtime-chat.md` around lines 82 - 87, The example hard-codes a preview model name in getRealtimeTokenFn using openaiRealtimeToken(model: 'gpt-4o-realtime-preview'), which will quickly become outdated; update the snippet to either remove the explicit model argument (letting defaults be used) or replace it with a configurable value (e.g., read from env or a constant) and document verifying the current OpenAI realtime model before publishing so realtimeToken/openaiRealtimeToken are not tied to a brittle literal.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/architecture/realtime-chat.md`:
- Around line 317-327: The event documentation table is missing the
`audio_chunk` event defined in the RealtimeEvent contract; update the adapters
event table to include a row for `audio_chunk` (payload shape like `{ chunk,
timestamp?, isFinal? }` or the exact payload used by RealtimeEvent) and a short
description such as "Raw audio buffer/data chunk for streaming audio input" so
the docs match the public RealtimeEvent type and avoid misleading adapter
authors/consumers.
- Around line 19-68: The fenced diagram block is missing a language tag which
triggers markdownlint; update the opening fence for the ASCII diagram (the
triple backticks that start the diagram block containing the RealtimeClient /
useRealtimeChat diagram) to include the text language specifier (i.e., change
``` to ```text) so the diagram renders the same but satisfies markdownlint.
In `@packages/typescript/ai-elevenlabs/src/realtime/adapter.ts`:
- Around line 162-166: The adapter currently ignores runtime session config in
updateSession(); instead, read the initial overrides (token.overrides) that
contain config.voice and config.instructions and pass them into
Conversation.startSession when the ElevenLabs session is created so the
client-side settings are applied; locate the session creation flow (where
Conversation.startSession is invoked during connect/initialization) and ensure
it merges token.overrides (instructions, voice, tools if present) into the
startSession payload so updateSession() no longer needs to silently drop those
fields.
- Around line 139-147: The startAudioCapture and stopAudioCapture methods only
emit mode_change events but don’t control the ElevenLabs microphone; update
startAudioCapture to call conversation.setMicMuted(false) before/after emitting
'listening' and update stopAudioCapture to call conversation.setMicMuted(true)
before/after emitting 'idle' so the SDK actually pauses/resumes capture; locate
the methods startAudioCapture and stopAudioCapture in the adapter and invoke
conversation.setMicMuted(true/false) (using the same conversation object used
elsewhere in this file) to keep reported mode in sync with real capture state.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 104-110: The dataChannelReady Promise only resolves on onopen, so
change it to also reject (and emit a failure status) if the data channel errors
or closes (and optionally add a timeout) to avoid await hanging; specifically,
augment the Promise created around dataChannel!.onopen in adapter.ts to attach
dataChannel!.onerror and dataChannel!.onclose handlers that call reject(new
Error(...)) and emit('status_change', { status: 'disconnected' | 'failed' }),
and mirror the same change for the other dataChannelReady instance referenced
around lines 639-641 so callers can recover instead of remaining stuck in
"connecting".
---
Duplicate comments:
In `@docs/architecture/realtime-chat.md`:
- Around line 82-87: The example hard-codes a preview model name in
getRealtimeTokenFn using openaiRealtimeToken(model: 'gpt-4o-realtime-preview'),
which will quickly become outdated; update the snippet to either remove the
explicit model argument (letting defaults be used) or replace it with a
configurable value (e.g., read from env or a constant) and document verifying
the current OpenAI realtime model before publishing so
realtimeToken/openaiRealtimeToken are not tied to a brittle literal.
In `@packages/typescript/ai-elevenlabs/src/realtime/adapter.ts`:
- Around line 84-87: The initial emits in onConnect are happening synchronously
and may fire before RealtimeClient handlers are registered
(Conversation.startSession() can call onConnect before connect() returns);
change onConnect to defer emitting the initial "status_change" and "mode_change"
until after the caller can subscribe (e.g., schedule emit calls using a
microtask or next tick mechanism such as Promise.resolve().then(...) or
process.nextTick/setImmediate) so that emit('status_change', ...) and
emit('mode_change', ...) occur asynchronously; update the onConnect handler in
adapter.ts accordingly so RealtimeClient receives those events (referencing
onConnect, emit, RealtimeClient, Conversation.startSession, and connect).
In `@packages/typescript/ai-openai/src/realtime/adapter.ts`:
- Around line 121-123: The current dataChannel.onerror handler constructs an
Error from the event object, producing "Data channel error: [object Event]";
update the handler for dataChannel.onerror to build the emitted Error from the
actual event details: treat the callback parameter as an RTCErrorEvent (or
Event) and extract the underlying error (e.g., evt.error or
evt.error?.message/name) and/or fall back to evt.type and evt.message, then call
emit('error', { error: new Error(`Data channel error: ${extractedDetail}`) }) so
the emitted error contains meaningful information; locate the
dataChannel.onerror assignment in adapter.ts and replace the stringified event
use with the extracted fields.
- Around line 153-178: The SDP negotiation (createOffer, setLocalDescription,
fetch to OPENAI_REALTIME_URL, reading answerSdp, setRemoteDescription) must be
wrapped in a try/catch so failures clean up WebRTC resources: on error stop all
tracks on localStream, close/cleanup the RTCPeerConnection instance
(pc.close()), and close any open DataChannel if created, then rethrow the error;
call setupInputAudioAnalysis(localStream) only after successful
setRemoteDescription. Locate these symbols: pc, localStream, createOffer,
setLocalDescription, fetch, setRemoteDescription, setupInputAudioAnalysis to
implement the try/catch and resource cleanup.
- Around line 132-146: The code calls navigator.mediaDevices.getUserMedia and
adds audio tracks inside connect(), causing microphone prompts for text-only
sessions; either defer calling navigator.mediaDevices.getUserMedia and adding
tracks until startAudioCapture() (move the getUserMedia + pc.addTrack logic from
connect() into startAudioCapture()), or if the SDP must include an audio m-line
keep the current flow but immediately disable the tracks after adding them by
setting each track.enabled = false (and only set enabled = true in
startAudioCapture()); apply the same change for the duplicate block around lines
444-460.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 9a19d69e-3b95-4fe5-8598-54a6e384adfd
📒 Files selected for processing (4)
docs/architecture/realtime-chat.mdexamples/ts-react-chat/src/routes/realtime.tsxpackages/typescript/ai-elevenlabs/src/realtime/adapter.tspackages/typescript/ai-openai/src/realtime/adapter.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- examples/ts-react-chat/src/routes/realtime.tsx
docs/architecture/realtime-chat.md
Outdated
Comment on lines
+19
to
+68
| ``` | ||
| ┌─────────────────────────────────────────────────────────────────┐ | ||
| │ React Application │ | ||
| │ ┌─────────────────────────────────────────────────────────┐ │ | ||
| │ │ useRealtimeChat() │ │ | ||
| │ │ - Connection state (status, mode) │ │ | ||
| │ │ - Messages & transcripts │ │ | ||
| │ │ - Audio visualization (levels, waveforms) │ │ | ||
| │ │ - Control methods (connect, disconnect, interrupt) │ │ | ||
| │ │ - Client-side tool configuration │ │ | ||
| │ └─────────────────────────────────────────────────────────┘ │ | ||
| └─────────────────────────────────────────────────────────────────┘ | ||
| │ | ||
| ▼ | ||
| ┌─────────────────────────────────────────────────────────────────┐ | ||
| │ @tanstack/ai-client │ | ||
| │ ┌─────────────────────────────────────────────────────────┐ │ | ||
| │ │ RealtimeClient │ │ | ||
| │ │ - Connection lifecycle management │ │ | ||
| │ │ - Token refresh scheduling │ │ | ||
| │ │ - Client-side session configuration (tools, voice, etc) │ │ | ||
| │ │ - Event subscription & dispatch │ │ | ||
| │ │ - Tool execution coordination │ │ | ||
| │ └─────────────────────────────────────────────────────────┘ │ | ||
| └─────────────────────────────────────────────────────────────────┘ | ||
| │ | ||
| ▼ | ||
| ┌─────────────────────────────────────────────────────────────────┐ | ||
| │ Provider Adapters │ | ||
| │ ┌──────────────────────┐ ┌──────────────────────┐ │ | ||
| │ │ openaiRealtime() │ │ elevenlabsRealtime() │ │ | ||
| │ │ - WebRTC connection │ │ - SDK wrapper │ │ | ||
| │ │ - Audio I/O │ │ - Signed URL auth │ │ | ||
| │ │ - Event mapping │ │ - Event mapping │ │ | ||
| │ │ - Session updates │ │ │ │ | ||
| │ └──────────────────────┘ └──────────────────────┘ │ | ||
| └─────────────────────────────────────────────────────────────────┘ | ||
| │ | ||
| ▼ | ||
| ┌─────────────────────────────────────────────────────────────────┐ | ||
| │ Server-Side │ | ||
| │ ┌─────────────────────────────────────────────────────────┐ │ | ||
| │ │ Token Generation (Server Function) │ │ | ||
| │ │ - openaiRealtimeToken() - ephemeral client secrets │ │ | ||
| │ │ - elevenlabsRealtimeToken() - signed URLs │ │ | ||
| │ │ (Minimal config: model only — session config is │ │ | ||
| │ │ applied client-side via session.update) │ │ | ||
| │ └─────────────────────────────────────────────────────────┘ │ | ||
| └─────────────────────────────────────────────────────────────────┘ | ||
| ``` |
Contributor
There was a problem hiding this comment.
Add a language to this fenced block.
markdownlint is already flagging the diagram fence here. Use text so the rendering stays the same and the docs remain lint-clean.
Suggested fix
-```
+```text
┌─────────────────────────────────────────────────────────────────┐
│ React Application │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ useRealtimeChat() │ │
@@
└─────────────────────────────────────────────────────────────────┘
-```
+```🧰 Tools
🪛 markdownlint-cli2 (0.21.0)
[warning] 19-19: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/architecture/realtime-chat.md` around lines 19 - 68, The fenced diagram
block is missing a language tag which triggers markdownlint; update the opening
fence for the ASCII diagram (the triple backticks that start the diagram block
containing the RealtimeClient / useRealtimeChat diagram) to include the text
language specifier (i.e., change ``` to ```text) so the diagram renders the same
but satisfies markdownlint.
docs/architecture/realtime-chat.md
Outdated
Comment on lines
+317
to
+327
| Adapters emit standardized events: | ||
|
|
||
| | Event | Payload | Description | | ||
| |-------|---------|-------------| | ||
| | `status_change` | `{ status }` | Connection status changed | | ||
| | `mode_change` | `{ mode }` | Conversation mode changed | | ||
| | `transcript` | `{ role, transcript, isFinal }` | Speech-to-text update | | ||
| | `message_complete` | `{ message }` | Full message received | | ||
| | `tool_call` | `{ toolCallId, toolName, input }` | Tool invocation requested | | ||
| | `interrupted` | `{ messageId? }` | Response was interrupted | | ||
| | `error` | `{ error }` | Error occurred | |
Contributor
There was a problem hiding this comment.
The event table is missing audio_chunk.
RealtimeEvent includes audio_chunk, so this table no longer matches the public contract and will mislead adapter authors or consumers reading the docs.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/architecture/realtime-chat.md` around lines 317 - 327, The event
documentation table is missing the `audio_chunk` event defined in the
RealtimeEvent contract; update the adapters event table to include a row for
`audio_chunk` (payload shape like `{ chunk, timestamp?, isFinal? }` or the exact
payload used by RealtimeEvent) and a short description such as "Raw audio
buffer/data chunk for streaming audio input" so the docs match the public
RealtimeEvent type and avoid misleading adapter authors/consumers.
Comment on lines
+139
to
+147
| async startAudioCapture() { | ||
| // ElevenLabs SDK handles audio capture automatically | ||
| // This is called when the session starts | ||
| emit('mode_change', { mode: 'listening' }) | ||
| }, | ||
|
|
||
| stopAudioCapture() { | ||
| // ElevenLabs SDK handles this | ||
| emit('mode_change', { mode: 'idle' }) |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's examine the adapter.ts file to understand the full context
wc -l packages/typescript/ai-elevenlabs/src/realtime/adapter.tsRepository: TanStack/ai
Length of output: 116
🏁 Script executed:
# Get the full content of the adapter.ts file to understand the context
cat -n packages/typescript/ai-elevanlabs/src/realtime/adapter.tsRepository: TanStack/ai
Length of output: 144
🏁 Script executed:
# Let me check the correct path
find . -name "adapter.ts" -path "*/elevenlabs/*"Repository: TanStack/ai
Length of output: 37
🏁 Script executed:
# Search for the file more broadly
fd adapter.ts | grep elevenlabsRepository: TanStack/ai
Length of output: 112
🏁 Script executed:
# Read the adapter.ts file with correct path
cat -n packages/typescript/ai-elevenlabs/src/realtime/adapter.tsRepository: TanStack/ai
Length of output: 8522
🏁 Script executed:
# Search for ElevenLabs SDK type definitions or documentation
fd -e d.ts -e ts | xargs grep -l "Conversation" | grep elevenlabsRepository: TanStack/ai
Length of output: 271
🏁 Script executed:
# Check if there are any other files in ai-elevenlabs package
find packages/typescript/ai-elevenlabs -type f -name "*.ts" -o -name "*.tsx"Repository: TanStack/ai
Length of output: 376
🏁 Script executed:
# Search for any documentation about ElevenLabs Conversation methods
rg "conversation\." packages/typescript/ai-elevenlabs/src/realtime/adapter.ts -A 3Repository: TanStack/ai
Length of output: 781
🏁 Script executed:
# Search for how other adapters handle audio capture control
find packages/typescript -name "adapter.ts" -type f | head -5Repository: TanStack/ai
Length of output: 347
🏁 Script executed:
# Check how OpenAI adapter handles audio capture
cat -n packages/typescript/ai-openai/src/realtime/adapter.ts | head -200Repository: TanStack/ai
Length of output: 7412
🏁 Script executed:
# Search for pause/resume methods in ElevenLabs SDK types
rg "pause|resume" packages/typescript/ai-elevenlabs/Repository: TanStack/ai
Length of output: 37
🏁 Script executed:
# Check the types file for ElevenLabs to see what methods are available
cat -n packages/typescript/ai-elevenlabs/src/realtime/types.tsRepository: TanStack/ai
Length of output: 1898
🏁 Script executed:
# Search for `@11labs/client` package in the repo to understand what methods are available
rg "@11labs/client" packages/typescript/ai-elevenlabs/Repository: TanStack/ai
Length of output: 351
🏁 Script executed:
# Look for any additional context about how the conversation is used
rg "Conversation\." packages/typescript/ai-elevenlabs/ -B 2 -A 2Repository: TanStack/ai
Length of output: 958
🏁 Script executed:
# Check if there's a package.json to see the version of `@11labs/client`
cat packages/typescript/ai-elevenlabs/package.jsonRepository: TanStack/ai
Length of output: 1336
🏁 Script executed:
# Search for any documentation or comments about ElevenLabs API in the codebase
rg "startSession|endSession|pause|resume|mute|unmute" packages/typescript/ai-elevenlabs/ -B 2 -A 2Repository: TanStack/ai
Length of output: 1378
🏁 Script executed:
# Let's check if there are any tests that show how the adapter is used
find packages/typescript/ai-elevenlabs -name "*.test.*" -o -name "*.spec.*" | xargs cat 2>/dev/nullRepository: TanStack/ai
Length of output: 37
🏁 Script executed:
# Search for any other realtime adapters to see how they handle audio capture
cat -n packages/typescript/ai-openai/src/realtime/adapter.ts | sed -n '350,450p'Repository: TanStack/ai
Length of output: 3731
🏁 Script executed:
# Get more of the OpenAI adapter implementation
cat -n packages/typescript/ai-openai/src/realtime/adapter.ts | sed -n '444,480p'Repository: TanStack/ai
Length of output: 1352
🏁 Script executed:
# Search for stopAudioCapture in OpenAI adapter
rg "stopAudioCapture" packages/typescript/ai-openai/src/realtime/adapter.ts -B 2 -A 10Repository: TanStack/ai
Length of output: 372
🏁 Script executed:
# Let me search for any other references or documentation about ElevenLabs session control
rg "mute|unmute|pause|resume|audioCapture" packages/typescript/ai-elevenlabs/ -iRepository: TanStack/ai
Length of output: 227
🏁 Script executed:
# Check if there's any documentation or README in the elevenlabs package
find packages/typescript/ai-elevenlabs -name "README*" -o -name "CHANGELOG*" -o -name "*.md"Repository: TanStack/ai
Length of output: 98
🏁 Script executed:
# Let's verify by checking what a typical usage pattern looks like in tests or examples
find packages/typescript/ai-elevenlabs -type f \( -name "*.test.*" -o -name "*.example.*" -o -name "index.ts" \)Repository: TanStack/ai
Length of output: 157
🏁 Script executed:
# Check the README for ElevenLabs
cat packages/typescript/ai-elevenlabs/README.mdRepository: TanStack/ai
Length of output: 1719
🏁 Script executed:
# Check the index files for any additional context
cat packages/typescript/ai-elevenlabs/src/index.tsRepository: TanStack/ai
Length of output: 515
🏁 Script executed:
# Let's also check what methods the ElevenLabs Conversation object exposes by looking at how it's used
rg "conversation\." packages/typescript/ai-elevenlabs/src/realtime/adapter.tsRepository: TanStack/ai
Length of output: 355
🌐 Web query:
ElevenLabs @11labs/client SDK v0.2.0 Conversation pause resume audio capture API
💡 Result:
In the official ElevenLabs Agents “client” SDK, the package name is @elevenlabs/client (not @11labs/client). In v0.2.0 it provides start/end, mute, and audio callbacks, but no dedicated “pause/resume” API. [1][2]
Conversation “pause / resume” in v0.2.0
There is no conversation.pause() / conversation.resume() method in the Conversation / VoiceConversation classes. [2][3]
Practical equivalents:
- Pause microphone capture / stop sending user audio:
conversation.setMicMuted(true) - Resume microphone capture:
conversation.setMicMuted(false)[3] - Silence agent playback (not a real pause):
conversation.setVolume({ volume: 0 })then restore volume later. [2]
Audio capture / access in v0.2.0
Agent audio (playback) capture
- Provide the
onAudio(base64Audio: string)callback instartSessionoptions; it receives the agent’s audio as base64-encoded chunks. [2][3]
User audio (microphone) capture
- The SDK captures mic audio internally via
getUserMedia({ audio: true })and an AudioWorklet (Input), then sends it to the server as base64 PCM (user_audio_chunk). It does not expose a public callback to intercept raw user PCM before sending. [3]
Connection types (v0.2.0)
- Supports WebSocket and WebRTC modes (WebRTC was introduced in this release). [1][3]
Sources
[1] @elevenlabs/client@0.2.0 release notes (WebRTC mode) (github.com)
[2] BaseConversation (public API surface: callbacks, setVolume, etc.) (raw.githubusercontent.com)
[3] VoiceConversation (mic capture, setMicMuted, onAudio, internal audio send) (github.com)
Citations:
- 1: https://github.com/elevenlabs/packages/releases/tag/%40elevenlabs%2Fclient%400.2.0
- 2: https://raw.githubusercontent.com/elevenlabs/packages/refs/tags/%40elevenlabs/client%400.2.0/packages/client/src/BaseConversation.ts
- 3: https://github.com/elevenlabs/packages/raw/refs/tags/%40elevenlabs/client%400.2.0/packages/client/src/VoiceConversation.ts
Control microphone capture using setMicMuted() instead of only emitting mode events.
The startAudioCapture() and stopAudioCapture() methods currently only emit mode_change events but don't actually control the ElevenLabs session's microphone. The ElevenLabs SDK v0.2.0 provides conversation.setMicMuted(true/false) to pause/resume microphone capture. Without calling this method, stopAudioCapture() can report 'idle' mode while audio is still being captured upstream.
Call conversation.setMicMuted(false) in startAudioCapture() and conversation.setMicMuted(true) in stopAudioCapture() to synchronize the reported mode with actual audio capture state.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-elevenlabs/src/realtime/adapter.ts` around lines 139 -
147, The startAudioCapture and stopAudioCapture methods only emit mode_change
events but don’t control the ElevenLabs microphone; update startAudioCapture to
call conversation.setMicMuted(false) before/after emitting 'listening' and
update stopAudioCapture to call conversation.setMicMuted(true) before/after
emitting 'idle' so the SDK actually pauses/resumes capture; locate the methods
startAudioCapture and stopAudioCapture in the adapter and invoke
conversation.setMicMuted(true/false) (using the same conversation object used
elsewhere in this file) to keep reported mode in sync with real capture state.
Comment on lines
+104
to
+110
| const dataChannelReady = new Promise<void>((resolve) => { | ||
| dataChannel!.onopen = () => { | ||
| flushPendingEvents() | ||
| emit('status_change', { status: 'connected' as RealtimeStatus }) | ||
| resolve() | ||
| } | ||
| }) |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
wc -l packages/typescript/ai-openai/src/realtime/adapter.tsRepository: TanStack/ai
Length of output: 112
🏁 Script executed:
sed -n '95,120p' packages/typescript/ai-openai/src/realtime/adapter.tsRepository: TanStack/ai
Length of output: 708
🏁 Script executed:
sed -n '630,650p' packages/typescript/ai-openai/src/realtime/adapter.tsRepository: TanStack/ai
Length of output: 362
🏁 Script executed:
sed -n '95,160p' packages/typescript/ai-openai/src/realtime/adapter.tsRepository: TanStack/ai
Length of output: 1942
🏁 Script executed:
rg -A 5 "dataChannel\.on(error|close)" packages/typescript/ai-openai/src/realtime/adapter.tsRepository: TanStack/ai
Length of output: 229
🏁 Script executed:
sed -n '1,50p' packages/typescript/ai-openai/src/realtime/adapter.tsRepository: TanStack/ai
Length of output: 1438
connect() hangs indefinitely if the data channel fails to open or encounters an error.
The dataChannelReady Promise (lines 104-110) only resolves when onopen fires. However, if the data channel encounters an error or never opens, the Promise never settles. While dataChannel.onerror emits an error event, it does not reject the Promise. This causes await dataChannelReady to hang forever, leaving callers stuck in "connecting" with no recovery path.
Also applies to: 639-641
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-openai/src/realtime/adapter.ts` around lines 104 -
110, The dataChannelReady Promise only resolves on onopen, so change it to also
reject (and emit a failure status) if the data channel errors or closes (and
optionally add a timeout) to avoid await hanging; specifically, augment the
Promise created around dataChannel!.onopen in adapter.ts to attach
dataChannel!.onerror and dataChannel!.onclose handlers that call reject(new
Error(...)) and emit('status_change', { status: 'disconnected' | 'failed' }),
and mirror the same change for the other dataChannelReady instance referenced
around lines 639-641 so callers can recover instead of remaining stuck in
"connecting".
# Conflicts: # pnpm-lock.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds realtime voice/text chat capabilities:
@tanstack/ai-client@tanstack/ai-openai@tanstack/ai-elevenlabspackageuseRealtimeChathook in@tanstack/ai-react@tanstack/aiexamples/ts-react-chatCo-Authored-By: Warp agent@warp.dev
Summary by CodeRabbit
New Features
Documentation