Broadcast and Presence API causing channel crashes in self-hosted Realtime

# Bug Report: Broadcast and Presence API causing channel crashes in self-hosted Realtime

## Summary

When using Supabase Realtime v2.63.0 (and earlier versions) in a self-hosted Docker environment with the default tenant configuration, **any broadcast or presence operation causes the channel to crash** with `UndefinedFunctionError: function RealtimeWeb.RealtimeChannel.handle_out/3 is undefined or private`.

This crash terminates the entire channel, breaking `postgres_changes` subscriptions and causing message loss in real-time applications.

## Affected Versions

- ❌ v2.25.35 (tested - bug present)
- ❌ v2.28.32 (tested - schema incompatibility)
- ❌ v2.33.66 (tested - RLIMIT_NOFILE error)
- ❌ **v2.63.0** (tested - bug still present)

## Environment

- **Setup**: Self-hosted Supabase with Docker Compose
- **Realtime version**: `supabase/realtime:v2.63.0`
- **PostgreSQL**: `supabase/postgres:15.8.1.085`
- **Client**: `@supabase/supabase-js` v2.81.0
- **Tenant ID**: `realtime` (default)
- **OS**: Linux (Docker)
- **Node.js**: v22.x

---

## Steps to Reproduce

### 1. Docker Compose Configuration

```yaml
version: '3.8'

services:
  realtime:
    image: supabase/realtime:v2.63.0
    container_name: supabase-realtime
    restart: unless-stopped
    environment:
      PORT: 4000
      DB_HOST: db
      DB_PORT: 5432
      DB_USER: postgres
      DB_PASSWORD: postgres
      DB_NAME: postgres
      DB_AFTER_CONNECT_QUERY: 'SET search_path TO _realtime'
      DB_ENC_KEY: supabaserealtime
      API_JWT_SECRET: super-secret-jwt-token-with-at-least-32-characters-long
      SECRET_KEY_BASE: UpNVntn3cDxHJpq99YMc1T1AQgQpc8kfYTuRgBiYa15BLrx8etQoXz3gZv1/u2oq
      ERL_AFLAGS: -proto_dist inet_tcp
      DNS_NODES: "''"
      RLIMIT_NOFILE: 10000
      APP_NAME: realtime
      SEED_SELF_HOST: "true"
      RUN_JANITOR: "true"
    depends_on:
      db:
        condition: service_healthy
    command: >
      sh -c "/app/bin/migrate && /app/bin/realtime eval 'Realtime.Release.seeds(Realtime.Repo)' && /app/bin/server"
```

### 2. Client Code - Broadcast Example (Triggers Bug)

```typescript
import { createClient } from '@supabase/supabase-js'

const supabase = createClient(
  'http://localhost:54321',
  'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZS1kZW1vIiwicm9sZSI6ImFub24iLCJleHAiOjE5ODM4MTI5OTZ9.CRXP1A7WOeoJeXxjNni43kdQwgnWNReilDMblYTn_I0'
)

// Create channel
const channel = supabase.channel('chat-room:general')

// Subscribe to postgres_changes (THIS WORKS FINE)
channel.on(
  'postgres_changes',
  {
    event: 'INSERT',
    schema: 'public',
    table: 'messages',
  },
  (payload) => {
    console.log('Message received:', payload)
  }
)

// Send broadcast (THIS CRASHES THE ENTIRE CHANNEL)
channel.subscribe((status) => {
  if (status === 'SUBSCRIBED') {
    channel.send({
      type: 'broadcast',
      event: 'typing',
      payload: {
        user_name: 'John',
        timestamp: new Date().toISOString()
      }
    })
  }
})
```

### 3. Client Code - Presence Example (Also Crashes)

```typescript
// Track presence (ALSO CRASHES THE CHANNEL)
channel.track({
  user_name: 'John',
  online_at: new Date().toISOString()
})
```

---

## Expected Behavior

1. Broadcast events should be sent successfully without crashing the channel
2. Presence tracking should work without terminating the channel
3. `postgres_changes` subscriptions should remain active after broadcast/presence operations
4. Multiple clients should be able to exchange broadcast messages reliably

---

## Actual Behavior

### Complete Error Log from Realtime Container

```elixir
12:00:25.226 project=realtime external_id=realtime [error] GenServer #PID<0.3899.0> terminating
** (UndefinedFunctionError) function RealtimeWeb.RealtimeChannel.handle_out/3 is undefined or private
    (realtime 2.63.0) RealtimeWeb.RealtimeChannel.handle_out("broadcast", %{
      "event" => "typing",
      "payload" => %{
        "timestamp" => "2025-11-14T12:00:25.226Z",
        "user_name" => "Usuario_6970"
      },
      "type" => "broadcast"
    }, %Phoenix.Socket{
      assigns: %{
        log_level: :error,
        tenant: "realtime",
        tenant_topic: "realtime:chat-room:general",
        channel_name: "chat-room:general",
        jwt_secret: "iNjicxc4+llvc9wovDvqymwfnj9teWMlyOIbJ8Fh6j2WNU8CIJ2ZgjR6MUIKqSmeDmvpsKLsZ9jgXJmQPpwL8w==",
        access_token: "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
        presence_enabled?: false,
        pg_change_params: [%{
          id: "81c772f8-c151-11f0-8c6d-424f085fba4f",
          params: %{
            "event" => "INSERT",
            "filter" => "room_id=eq.ee45c875-cfc1-4a88-a59b-575fa831d4aa",
            "schema" => "public",
            "table" => "messages"
          },
          channel_pid: #PID<0.3903.0>,
          claims: %{"exp" => 1983812996, "iss" => "supabase-demo", "role" => "anon"}
        }],
        ...
      },
      channel: RealtimeWeb.RealtimeChannel,
      channel_pid: #PID<0.3899.0>,
      endpoint: RealtimeWeb.Endpoint,
      handler: RealtimeWeb.UserSocket,
      id: "user_socket:realtime",
      joined: true,
      topic: "realtime:chat-room:general",
      transport: :websocket,
      ...
    })
Last message: %Phoenix.Socket.Broadcast{
  topic: "realtime:chat-room:general",
  event: "broadcast",
  payload: %{
    "event" => "typing",
    "payload" => %{
      "timestamp" => "2025-11-14T12:00:25.226Z",
      "user_name" => "Usuario_6970"
    },
    "type" => "broadcast"
  }
}
```

### Consequences

1. **Channel crashes immediately**: GenServer terminates on first broadcast
2. **Message loss**: `postgres_changes` subscription is destroyed
3. **Auto-reconnect loop**: Channel reconnects but crashes again on next broadcast
4. **Infinite crash cycle**: Makes broadcast/presence completely unusable

---

## Root Cause Analysis

Based on [this GitHub issue comment](https://github.com/supabase/realtime/issues/863#issuecomment-2028263894), the problem occurs when `tenant_id` is literally named `"realtime"`, causing internal name conflicts in the Elixir/Phoenix application.

### Evidence from Error Logs

The key indicators in the error:

```elixir
tenant: "realtime"                      // ← PROBLEM: Reserved internal name
topic: "realtime:chat-room:general"
tenant_topic: "realtime:chat-room:general"
```

The function `RealtimeWeb.RealtimeChannel.handle_out/3` is being invoked but doesn't exist in the module, suggesting an incorrect code path when the tenant uses the reserved name `"realtime"`.

### Database Tenant Configuration

```sql
-- Query to check tenant configuration
SELECT external_id, name FROM _realtime.tenants;

-- Result showing problematic default:
-- external_id  |     name
-- -------------+--------------
-- realtime     | realtime     ← This is the problem
```

---

## Attempted Workarounds

### ❌ Attempt 1: Change channel name prefix

```typescript
// Tried using different channel prefix
const channel = supabase.channel('chat-room:general')  // instead of 'room:general'

// Result: Bug persists
// Reason: tenant_id is still "realtime" regardless of channel name
```

### ❌ Attempt 2: Upgrade Realtime versions

Tested multiple versions looking for one without the bug:

- **v2.28.32**: Schema incompatibility - missing `enable_authorization` column
  ```
  ERROR 42703 (undefined_column) column t0.enable_authorization does not exist
  ```

- **v2.33.66**: Environment variable error
  ```
  /app/run.sh: line 6: RLIMIT_NOFILE: unbound variable
  ```

- **v2.63.0**: Same `handle_out/3` error persists

### ❌ Attempt 3: Manually change tenant in database

```sql
-- Attempt to rename tenant
BEGIN;
UPDATE _realtime.extensions
SET tenant_external_id = 'realtime-local'
WHERE tenant_external_id = 'realtime';

UPDATE _realtime.tenants
SET external_id = 'realtime-local', name = 'realtime-local'
WHERE external_id = 'realtime';
COMMIT;

-- Result: Breaks system expecting default "realtime" tenant
-- Error: TenantNotFound: Tenant not found: realtime
```

### ✅ Current Workaround (Not Ideal)

**Completely disable broadcast and presence features** - only use `postgres_changes`:

```typescript
// Only use postgres_changes (works perfectly)
channel.on('postgres_changes', { ... }, callback)

// DO NOT use these (they crash):
// channel.send({ type: 'broadcast', ... })      // ❌ CRASHES
// channel.track({ ... })                          // ❌ CRASHES
```

This maintains message delivery but loses all real-time collaboration features.

---

## Impact Assessment

This bug makes **broadcast and presence APIs completely unusable** in self-hosted Supabase with default configuration, breaking:

| Feature | Status | Impact |
|---------|--------|--------|
| Message delivery (`postgres_changes`) | ✅ Works | Core functionality intact |
| Typing indicators (`broadcast`) | ❌ Broken | No typing status |
| Active users (`presence`) | ❌ Broken | Can't track who's online |
| Real-time collaboration | ❌ Broken | No ephemeral state sharing |
| Custom broadcast events | ❌ Broken | All broadcast features unavailable |

### Use Cases Affected

- Chat applications with typing indicators
- Collaborative editing with cursor positions
- Online presence tracking
- Real-time gaming state
- Live cursors/selections
- Any feature requiring ephemeral state broadcast

---

## Proposed Solutions

### Option 1: Fix `handle_out/3` (Recommended)

Add proper handling for broadcast messages in `RealtimeWeb.RealtimeChannel` module:

```elixir
# Add missing function clause in RealtimeWeb.RealtimeChannel
def handle_out("broadcast", payload, socket) do
  # Implement proper broadcast handling
  push(socket, "broadcast", payload)
  {:noreply, socket}
end
```

**Benefits**: Fixes the issue without breaking existing deployments

### Option 2: Prevent Reserved Tenant Names

Add validation during tenant seeding to prevent using `"realtime"`:

```elixir
# In Realtime.Release.seeds/1
defp validate_tenant_name(name) do
  if name == "realtime" do
    raise "Tenant name 'realtime' is reserved. Use a different name like 'realtime-local' or 'my-app'"
  end
end
```

**Benefits**: Prevents the issue from occurring in new deployments

### Option 3: Auto-rename Default Tenant

Change the seeding script to use a safe default name:

```elixir
# Change default tenant from "realtime" to "realtime-default"
@default_tenant_id "realtime-default"  # instead of "realtime"
```

**Benefits**: Fixes the issue for all new deployments automatically

### Option 4: Documentation & Migration Guide

Provide clear documentation and migration steps:

1. Document that `tenant_id` must NOT be "realtime"
2. Provide SQL migration script to safely rename tenant
3. Update example docker-compose files
4. Add troubleshooting guide for this error

**Benefits**: Helps existing users fix their deployments

---

## Additional Context

### What Works (postgres_changes)

```typescript
// This configuration works perfectly - NO crashes
channel
  .on('postgres_changes', {
    event: 'INSERT',
    schema: 'public',
    table: 'messages'
  }, (payload) => {
    console.log('New message:', payload)
  })
  .subscribe()

// Messages are delivered reliably with zero issues
```

### What Breaks (broadcast/presence)

```typescript
// Both of these crash the channel immediately:

// Broadcast
channel.send({
  type: 'broadcast',
  event: 'typing',
  payload: { user: 'john' }
})  // ❌ CRASH

// Presence
channel.track({
  user: 'john',
  status: 'online'
})  // ❌ CRASH
```

---

## Questions for Maintainers

1. Is `tenant_id="realtime"` intentionally reserved? If so, this should be documented.
2. Why does `handle_out/3` not exist for the broadcast message type?
3. What is the recommended way to change tenant_id in self-hosted setups?
4. Are there other reserved tenant names we should avoid?
5. Is there a migration path for existing deployments using "realtime"?

---

## Reproduction Repository

A minimal reproduction repository is available with:
- Complete Docker Compose setup
- TypeScript client code demonstrating the bug
- Step-by-step instructions to reproduce
- Logs showing the crash

Contact me if you need access to the reproduction case.

---

## Request

This bug is **blocking real-time collaboration features** in self-hosted Supabase deployments. Could you please:

1. Confirm if this is a known issue with `tenant_id="realtime"`
2. Provide guidance on safely changing the tenant_id
3. Consider implementing one of the proposed solutions
4. Add validation/warnings if using reserved tenant names

Thank you for your work on Supabase Realtime!

---

## Related Issues

- Possibly related to: https://github.com/supabase/realtime/issues/863
- Similar tenant naming issues discussed in Discord

---

**Note**: This issue only affects self-hosted Supabase with default configuration. Supabase Cloud deployments are not affected as they use different tenant naming.


Feature	Status	Impact
Message delivery (`postgres_changes`)	✅ Works	Core functionality intact
Typing indicators (`broadcast`)	❌ Broken	No typing status
Active users (`presence`)	❌ Broken	Can't track who's online
Real-time collaboration	❌ Broken	No ephemeral state sharing
Custom broadcast events	❌ Broken	All broadcast features unavailable

Uh oh!

Broadcast and Presence API causing channel crashes in self-hosted Realtime #1617

Description

Bug Report: Broadcast and Presence API causing channel crashes in self-hosted Realtime

Summary

Affected Versions

Environment

Steps to Reproduce

1. Docker Compose Configuration

2. Client Code - Broadcast Example (Triggers Bug)

3. Client Code - Presence Example (Also Crashes)

Expected Behavior

Actual Behavior

Complete Error Log from Realtime Container

Consequences

Root Cause Analysis

Evidence from Error Logs

Database Tenant Configuration

Attempted Workarounds

❌ Attempt 1: Change channel name prefix

❌ Attempt 2: Upgrade Realtime versions

❌ Attempt 3: Manually change tenant in database

✅ Current Workaround (Not Ideal)

Impact Assessment

Use Cases Affected

Proposed Solutions

Option 1: Fix handle_out/3 (Recommended)

Option 2: Prevent Reserved Tenant Names

Option 3: Auto-rename Default Tenant

Option 4: Documentation & Migration Guide

Additional Context

What Works (postgres_changes)

What Breaks (broadcast/presence)

Questions for Maintainers

Reproduction Repository

Request

Related Issues

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Option 1: Fix `handle_out/3` (Recommended)