Documentation

REAL_TIME_WEBSOCKET

docs/features/REAL_TIME_WEBSOCKET.md

Real-Time WebSocket System

Owner: Theo Ramos Version: 2.2.0

OwnerLast verifiedNext verification dueLast CI run IDCoverage %Open risks
Theo Ramos2025-12-282026-01-27ci.yml#2191Pending (websocket API + inbox Cypress smoke)Polling fallback untested; Redis adapter scale tests missing

Verification

  • CI jobs: ci.yml API + Cypress (last run: ci.yml#2191)
  • Commands (seeded demo portal; use demo websocket credentials):
    • pnpm test:api --config jest.config.api.cjs --runTestsByPath tests/app/api/websocket/route.test.ts tests/app/api/socket/route.test.ts tests/app/api/socket/metrics/route.test.ts
    • pnpm cypress:run --spec "cypress/e2e/inbox/inbox-interactions.cy.ts","cypress/e2e/inbox/inbox-error-handling.cy.ts"
    • pnpm db:health
  • Test suites + environment:
    • tests/app/api/websocket/route.test.ts, tests/app/api/socket/route.test.ts, tests/app/api/socket/metrics/route.test.ts (CI seeded demo DB; websocket secret from env mock)
    • cypress/e2e/inbox/inbox-interactions.cy.ts, inbox-error-handling.cy.ts (staging demo tenant; exercises realtime message flow)
  • Next verification due: 2026-01-04

Table of Contents

  1. Overview
  2. Architecture
  3. Authentication
  4. Events & Messages
  5. Configuration
  6. Persistence
  7. Client Integration
  8. Deployment
  9. Troubleshooting

Overview

Petunia's real-time WebSocket system provides instant updates for inbox messages, user presence, typing indicators, and conversation events. Built on Socket.IO with Prisma-backed persistence, the system runs as a separate Node.js server on port 3001.

Key Features

  • JWT Authentication - Secure token-based authentication
  • Room-Based Subscriptions - Portal and conversation-level isolation
  • Horizontal Scaling - Redis adapter support for multi-instance deployments
  • Persistent State - Prisma models for presence and typing indicators
  • Automatic Cleanup - Cron-based expiration of stale data
  • Connection Recovery - Automatic reconnection with exponential backoff

Technology Stack

  • Server: Socket.IO 4.8+ on Node.js 20
  • Database: PostgreSQL via Prisma
  • Cache: Redis (optional, for scaling)
  • Protocol: WebSocket with fallback to HTTP long-polling

Architecture

System Diagram

┌─────────────────┐         JWT Token          ┌──────────────────┐
│   Next.js App   │ ────────────────────────> │  /api/auth/      │
│   (port 3000)   │ <──────────────────────── │   ws-token       │
└────────┬────────┘                            └──────────────────┘
         │
         │ WebSocket Connection
         │ (with JWT in auth handshake)
         ▼
┌─────────────────────────────────────────────────────────────┐
│           Realtime Server (port 3001)                        │
│  ┌────────────────────────────────────────────────────────┐ │
│  │  Socket.IO Server                                       │ │
│  │  • JWT Authentication Middleware                        │ │
│  │  • Connection State Recovery (2min window)              │ │
│  │  • CORS Protection                                      │ │
│  └────────────────────────────────────────────────────────┘ │
│                                                               │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │   Presence   │  │    Typing    │  │    Inbox     │      │
│  │   Handler    │  │   Handler    │  │   Handler    │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
│                                                               │
│  ┌────────────────────────────────────────────────────────┐ │
│  │  Cleanup Service (1min interval)                       │ │
│  │  • Expire typing indicators                             │ │
│  │  • Remove stale presence (24h offline)                  │ │
│  └────────────────────────────────────────────────────────┘ │
└───────────────────────┬───────────────────────────────────┘
                        │
                        ▼
           ┌────────────────────────┐
           │   PostgreSQL Database   │
           │  • UserPresence         │
           │  • TypingIndicator      │
           └────────────────────────┘

Redis Pub/Sub Bridge (for webhook → WebSocket):
┌────────────────────────┐     petunia:inbox:events     ┌────────────────────────┐
│  Next.js App           │ ──────────────────────────> │  Realtime Server       │
│  (inbox-realtime.ts)   │                              │  (redis-subscriber.ts) │
│  Publisher             │                              │  Subscriber            │
└────────────────────────┘                              └────────────────────────┘

Optional Redis Adapter (for horizontal scaling):
           ┌────────────────────────┐
           │        Redis            │
           │  (pub/sub for Socket.IO)│
           └────────────────────────┘

Redis Pub/Sub Bridge

The realtime system uses Redis pub/sub to bridge between webhook handlers (running in the Next.js app) and the WebSocket server:

  1. Publisher (lib/services/inbox-realtime.ts):

    • Called by webhook handlers when new messages arrive
    • Called by AI/agent response handlers when replies are sent
    • Publishes to petunia:inbox:events channel
  2. Subscriber (realtime-server/services/redis-subscriber.ts):

    • Listens to petunia:inbox:events channel
    • Dispatches events to Socket.IO for WebSocket broadcast
    • Handles new_message, conversation_update, message_status events

Event Flow:

Webhook → Message stored → broadcastNewMessage() → Redis pub → Socket.IO emit

Reliability Features:

  • Auto-reconnection with exponential backoff (up to 30s)
  • In-memory retry queue (100 events max) for failed publishes
  • Health status tracking for monitoring endpoints
  • Graceful degradation when Redis unavailable

Separate Server Architecture

The realtime-server is a standalone Node.js application in /realtime-server/:

realtime-server/
├── index.ts                    # Main server entry point
├── package.json                # Separate dependencies
├── tsconfig.json               # TypeScript config
├── Dockerfile                  # Production image
├── auth/
│   ├── jwt.ts                  # JWT verification
│   └── permissions.ts          # Event authorization
├── handlers/
│   ├── presence.ts             # Online/offline/away status
│   ├── typing.ts               # Typing indicators
│   ├── inbox.ts                # Message & conversation updates
│   └── analytics.ts            # Usage analytics (future)
└── services/
    ├── prisma.ts               # Database client
    ├── logger.ts               # Structured logging
    ├── metrics.ts              # Metrics collection
    ├── cleanup.ts              # Periodic data cleanup
    └── redis-subscriber.ts     # Redis pub/sub bridge for inbox events

Why Separate?

  • Independent scaling (WebSocket server ≠ HTTP server)
  • Different resource requirements (persistent connections)
  • Isolated dependencies (Socket.IO, Redis adapter)
  • Simplified deployment (can deploy realtime-server separately)

Room Structure

Portal Rooms (portal:{portalId})

  • All users with access to a portal
  • Broadcasts: new conversations, portal-wide events

Conversation Rooms (conversation:{conversationId})

  • Users actively viewing a conversation
  • Broadcasts: new messages, read receipts, status changes

Typing Rooms (conversation:{conversationId}:typing)

  • Users subscribed to typing indicators for a conversation
  • Broadcasts: typing start/stop events

Authentication

Flow Diagram

Client                    Next.js API                Realtime Server
  │                           │                            │
  │  1. Request WS token      │                            │
  │ ────────────────────────> │                            │
  │                           │                            │
  │                           │  2. Get user from session  │
  │                           │  3. Fetch portal access    │
  │                           │  4. Sign JWT (1h expiry)   │
  │                           │                            │
  │  5. Return JWT            │                            │
  │ <──────────────────────── │                            │
  │                           │                            │
  │  6. Connect with token & portalId                      │
  │ ──────────────────────────────────────────────────────>│
  │                           │                            │
  │                           │  7. Verify JWT signature   │
  │                           │  8. Check portal access    │
  │                           │  9. Join portal room       │
  │                           │                            │
  │  10. Connection confirmed │                            │
  │ <──────────────────────────────────────────────────────│

JWT Token Generation

Endpoint: GET /api/auth/ws-token

Source: /app/api/auth/ws-token/route.ts

Requirements:

  • Authenticated user session (NextAuth)
  • Valid portal access via PortalUser records

Token Payload:

{
  id: string;              // User ID
  email?: string;          // User email
  name?: string;           // Display name
  isPlatformAdmin: boolean; // Platform admin flag
  portalIds: string[];     // Array of accessible portal IDs
  iat: number;             // Issued at (Unix timestamp)
  exp: number;             // Expires at (iat + 1 hour)
}

Response:

{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "expiresIn": "1h"
}

Error Codes:

  • 401 - User not authenticated
  • 500 - Server configuration error (missing JWT_SECRET)

Connection Authentication

Client Handshake:

const socket = io('http://localhost:3001', {
  auth: {
    token: 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...',
    portalId: 'portal_123'
  }
});

Server Validation:

  1. Extract token and portalId from handshake
  2. Verify JWT signature using NEXTAUTH_SECRET
  3. Check token expiration
  4. Validate user has access to requested portal
  5. Store user data in socket context
  6. Join portal:{portalId} room

Rejection Reasons:

  • Missing token or portalId
  • Invalid JWT signature
  • Expired token
  • No access to portal
  • Malformed token payload

Token Refresh

Tokens expire after 1 hour. The client hook (useWebSocketToken) automatically refreshes 5 minutes before expiry:

// Auto-refresh at 55 minutes (5min buffer)
const REFRESH_BUFFER_MS = 5 * 60 * 1000;
const refreshIn = Math.max(expiryMs - REFRESH_BUFFER_MS, 60000);

On refresh, the WebSocket connection is maintained. New tokens are fetched silently in the background.


Events & Messages

Presence Events

Set Presence

// Client → Server
socket.emit('presence:update', {
  status: 'online' | 'away' | 'busy'
});

// Server → Clients (broadcast to portal)
socket.on('presence:update', (data: {
  userId: string;
  userName?: string;
  status: 'online' | 'away' | 'busy' | 'offline';
  lastSeen: string; // ISO 8601
}) => {
  // Update UI with user's new status
});

Subscribe to Presence

// Client → Server
socket.emit('presence:subscribe');

// Server → Client (response)
socket.on('presence:list', (data: {
  users: Array<{
    userId: string;
    userName?: string;
    status: 'online' | 'away' | 'busy';
    connectedAt: string;
  }>;
}) => {
  // Show online users in UI
});

Automatic Events:

  • User comes online on connection
  • User goes offline on disconnect
  • Presence updated on status change

Typing Indicators

Start Typing

// Client → Server
socket.emit('typing:start', {
  conversationId: 'conv_123'
});

// Server → Clients (broadcast to conversation room)
socket.on('typing:update', (data: {
  conversationId: string;
  userId: string;
  userName?: string;
  isTyping: boolean;
}) => {
  // Show "User is typing..." indicator
});

Stop Typing

// Client → Server
socket.emit('typing:stop', {
  conversationId: 'conv_123'
});

// Automatically stops after 5 seconds of inactivity

Subscribe to Typing

// Client → Server
socket.emit('typing:subscribe', {
  conversationId: 'conv_123'
});

// Server → Client (current state)
socket.on('typing:list', (data: {
  conversationId: string;
  users: Array<{
    userId: string;
    userName?: string;
    startedAt: string;
  }>;
}) => {
  // Initialize typing state
});

Auto-Expiration:

  • Typing indicators expire after 5 seconds
  • Cleanup service removes expired indicators every minute
  • Client should re-emit typing:start periodically while user is typing

Inbox Events

Subscribe to Inbox

// Client → Server
socket.emit('inbox:subscribe');

// Server → Client (confirmation)
socket.on('inbox:subscribed', (data: {
  portalId: string;
}) => {
  // Subscription confirmed
});

Subscribe to Conversation

// Client → Server
socket.emit('inbox:conversation:subscribe', {
  conversationId: 'conv_123'
});

// Server → Client (confirmation)
socket.on('inbox:conversation:subscribed', (data: {
  conversationId: string;
}) => {
  // Now receiving real-time updates
});

New Message

// Server → Clients (broadcast to conversation)
socket.on('inbox:message:new', (data: {
  conversationId: string;
  messageId: string;
  content: string;
  channel: string;
  senderId: string;
  senderName?: string;
  timestamp: string;
  attachments?: Array<{
    id: string;
    name: string;
    type: string;
    url: string;
  }>;
}) => {
  // Add message to UI
  // Play notification sound
  // Update unread count
});

Conversation Updated

// Server → Clients (broadcast to portal & conversation)
socket.on('inbox:conversation:updated', (data: {
  conversationId: string;
  update: {
    status?: 'open' | 'closed' | 'snoozed';
    assignedTo?: string;
    assignedToName?: string;
    priority?: 'low' | 'normal' | 'high' | 'urgent';
    labels?: string[];
    unreadCount?: number;
    lastMessageAt?: string;
    lastMessagePreview?: string;
  };
}) => {
  // Update conversation in list
});

Read Receipts

// Client → Server
socket.emit('inbox:messages:read', {
  conversationId: 'conv_123',
  messageIds: ['msg_1', 'msg_2']
});

// Server → Clients (broadcast to conversation)
socket.on('inbox:read_receipt', (data: {
  conversationId: string;
  messageIds: string[];
  readBy: string;
  readByName?: string;
  readAt: string;
}) => {
  // Update message read status
  // Show read indicators
});

Message Status

// Server → Clients (broadcast to conversation)
socket.on('inbox:message:status', (data: {
  conversationId: string;
  messageId: string;
  status: 'sending' | 'sent' | 'delivered' | 'read' | 'failed';
  timestamp: string;
  error?: string;
}) => {
  // Update message status icon
  // Show error if failed
});

Error Handling

All handlers emit error events on permission failures:

socket.on('presence:error', (error: { error: string }) => {
  console.error('Presence error:', error);
});

socket.on('typing:error', (error: { error: string }) => {
  console.error('Typing error:', error);
});

socket.on('inbox:error', (error: { error: string }) => {
  console.error('Inbox error:', error);
});

Configuration

Environment Variables

Next.js App (.env.local):

# WebSocket server URL (client-side)
NEXT_PUBLIC_WEBSOCKET_URL=http://localhost:3001

# JWT secret (must match realtime-server)
NEXTAUTH_SECRET=your-secret-here

# Optional: Use demo WebSocket (client-side mock)
NEXT_PUBLIC_WEBSOCKET_TEST_MODE=false

Realtime Server (realtime-server/.env):

# Required: JWT secret (must match NEXTAUTH_SECRET)
JWT_SECRET=your-secret-here
# Or use NEXTAUTH_SECRET for consistency
NEXTAUTH_SECRET=your-secret-here

# Server port
REALTIME_PORT=3001

# CORS allowed origins (comma-separated)
ALLOWED_ORIGINS=http://localhost:3000,https://yourdomain.com

# Optional: Redis for horizontal scaling
REDIS_URL=redis://localhost:6379

# Optional: Log level
LOG_LEVEL=info

Docker Configuration

Dockerfile: /realtime-server/Dockerfile

Multi-stage build:

# Build stage
FROM node:24-alpine AS builder
# ... install deps, build TypeScript

# Production stage
FROM node:24-alpine
# ... copy built files, run as non-root

Key Settings:

  • Non-root user (realtime:1001)
  • Health check every 30 seconds
  • Signal handling via dumb-init
  • Exposed port: 3001

Health Check:

// GET http://localhost:3001
{
  "status": "ok",  // or "degraded" if Redis disconnected
  "service": "petunia-realtime-server",
  "timestamp": "2025-12-04T12:00:00.000Z",
  "connections": 42,
  "redis": {
    "status": "connected",  // connected | connecting | reconnecting | disconnected
    "lastConnectedAt": "2025-12-04T12:00:00.000Z",
    "messagesReceived": 1234,
    "messagesProcessed": 1230,
    "messagesFailed": 4
  }
}

Production Deployment

Docker Compose:

services:
  realtime:
    build: ./realtime-server
    ports:
      - "3001:3001"
    environment:
      - JWT_SECRET=${NEXTAUTH_SECRET}
      - REDIS_URL=${REDIS_URL}
      - ALLOWED_ORIGINS=${ALLOWED_ORIGINS}
    depends_on:
      - postgres
      - redis

Environment-Specific URLs:

  • Development: http://localhost:3001
  • Staging: https://realtime-staging.petunia.gardenpatch.xyz
  • Production: https://realtime.petunia.gardenpatch.xyz

Connection Tuning

Socket.IO Settings (in realtime-server/index.ts):

const io = new SocketIOServer(httpServer, {
  cors: {
    origin: ALLOWED_ORIGINS,
    credentials: true,
  },
  connectionStateRecovery: {
    maxDisconnectionDuration: 2 * 60 * 1000, // 2 minutes
    skipMiddlewares: true,
  },
  pingTimeout: 30000,  // 30 seconds
  pingInterval: 25000, // 25 seconds
});

Client Reconnection (in useWebSocket):

const {
  reconnectAttempt,
  startReconnect,
  stopReconnect,
  canReconnect,
} = useWebSocketReconnect({
  maxAttempts: 5,
  delay: 3000, // Start at 3 seconds
});

Persistence

Database Models

UserPresence (prisma/schema.prisma):

model UserPresence {
  id        String   @id @default(cuid())
  userId    String
  portalId  String?
  status    String   @default("offline") // online, away, busy, offline
  lastSeen  DateTime @default(now())
  metadata  Json?    // Additional data (device, location, etc.)
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt

  User   User    @relation(fields: [userId], references: [id], onDelete: Cascade)
  Portal Portal? @relation(fields: [portalId], references: [id])

  @@unique([userId, portalId])
  @@index([portalId, status])
  @@index([userId])
  @@index([lastSeen])
}

TypingIndicator (prisma/schema.prisma):

model TypingIndicator {
  id             String   @id @default(cuid())
  conversationId String
  userId         String
  expiresAt      DateTime
  createdAt      DateTime @default(now())

  Conversation Conversation @relation(fields: [conversationId], references: [id], onDelete: Cascade)
  User         User         @relation(fields: [userId], references: [id], onDelete: Cascade)

  @@unique([conversationId, userId])
  @@index([expiresAt])
}

Cleanup Service

Source: /realtime-server/services/cleanup.ts

Schedule: Every 1 minute

Tasks:

  1. Expired Typing Indicators

    • Delete records where expiresAt < now()
    • Prevents stale "is typing" indicators
  2. Stale Presence Records

    • Delete offline users not seen in 24 hours
    • Keeps presence table lean

Implementation:

// Cleanup interval (1 minute)
const CLEANUP_INTERVAL_MS = 60_000;

async function cleanupExpiredTypingIndicators() {
  await prisma.typingIndicator.deleteMany({
    where: { expiresAt: { lt: new Date() } }
  });
}

async function cleanupStalePresence() {
  const twentyFourHoursAgo = new Date(Date.now() - 24 * 60 * 60 * 1000);
  await prisma.userPresence.deleteMany({
    where: {
      status: 'offline',
      lastSeen: { lt: twentyFourHoursAgo }
    }
  });
}

Startup & Shutdown:

// Start on server boot
cleanupTimer = startCleanupService();

// Stop on graceful shutdown
process.on('SIGTERM', () => {
  if (cleanupTimer) stopCleanupService(cleanupTimer);
});

Data Retention

Data TypeRetentionCleanup Method
Typing indicators5 secondsAuto-expire via expiresAt + cron
Active presenceWhile connectedUpdated on disconnect
Offline presence24 hoursCron cleanup
Connection metricsIn-memory onlyCleared on disconnect

Client Integration

Hook: useWebSocketToken

Location: /lib/hooks/useWebSocketToken.ts

Purpose: Fetch and auto-refresh JWT tokens for WebSocket authentication

Usage:

const { token, isLoading, error, refresh } = useWebSocketToken();

Features:

  • Fetches token on mount (if authenticated)
  • Auto-refreshes 5 minutes before expiry
  • Returns null if user not authenticated
  • Handles refresh failures gracefully

Example:

function MyComponent() {
  const { token, isLoading } = useWebSocketToken();

  if (isLoading) return <div>Connecting...</div>;
  if (!token) return <div>Authentication required</div>;

  return <WebSocketProvider token={token}>...</WebSocketProvider>;
}

Hook: useWebSocket

Location: /lib/hooks/useWebSocket.ts

Purpose: Low-level WebSocket connection management

Usage:

const {
  isConnected,
  connectionState,
  send,
  reconnect,
  disconnect,
  subscribeToInbox,
  subscribeToConversation,
  emitTyping,
  emitReadReceipt,
} = useWebSocket({
  businessId: portalId,
  token: wsToken,
  userId: user.id,
  onMessage: handleMessage,
  onError: handleError,
});

Connection States:

  • idle - Not connected
  • connecting - Establishing connection
  • connected - Active connection
  • reconnecting - Attempting to reconnect
  • disconnected - Intentionally disconnected
  • error - Connection failed

Methods:

// Subscribe to portal inbox
subscribeToInbox(portalId: string): boolean

// Subscribe to specific conversation
subscribeToConversation(conversationId: string): boolean

// Emit typing indicator
emitTyping(conversationId: string, isTyping: boolean): boolean

// Mark messages as read
emitReadReceipt(conversationId: string, messageIds: string[]): boolean

// Send custom message
send(type: string, data: any): boolean

// Manual reconnect
reconnect(): void

// Disconnect
disconnect(): void

Provider: InboxProvider

Location: /lib/providers/InboxProvider.tsx

Purpose: High-level inbox state management with WebSocket integration

Integration:

export function InboxProvider({ children }: { children: ReactNode }) {
  const { currentPortal } = usePortals();
  const { token } = useWebSocketToken();
  const { user } = useAuth();

  // Set up WebSocket connection
  const {
    isConnected,
    subscribeToInbox,
    subscribeToConversation,
    emitTyping,
    emitReadReceipt,
  } = useWebSocket({
    businessId: currentPortal.id,
    token,
    userId: user.id,
    onMessage: handleWebSocketMessage,
  });

  // Subscribe to inbox on connect
  useEffect(() => {
    if (isConnected) {
      subscribeToInbox(currentPortal.id);
    }
  }, [isConnected, currentPortal.id]);

  // Handle incoming messages
  function handleWebSocketMessage(message: WebSocketMessage) {
    if (message.type === 'inbox:message:new') {
      dispatch({ type: 'NEW_MESSAGE', payload: message.data });
    }
    // ... handle other message types
  }

  return (
    <InboxContext.Provider value={{ /* ... */ }}>
      {children}
    </InboxContext.Provider>
  );
}

Features:

  • Automatic subscription to portal inbox
  • Real-time message updates
  • Typing indicator management
  • Read receipt handling
  • Conversation state synchronization

Context: WebSocketContext

Location: /lib/providers/WebSocketContext.tsx

Purpose: Provide WebSocket connection to entire app

Usage:

export function WebSocketProvider({ children }: { children: ReactNode }) {
  const { currentPortal } = usePortals();
  const { token } = useWebSocketToken();
  const { user } = useAuth();

  const wsConfig = useWebSocket({
    businessId: currentPortal?.id || '',
    token: token || '',
    userId: user?.id || '',
    autoConnect: !!token && !!currentPortal && !!user,
  });

  return (
    <WebSocketContext.Provider value={wsConfig}>
      {children}
    </WebSocketContext.Provider>
  );
}

// In any component:
const { isConnected, connectionState } = useWebSocketContext();

Deployment

Development Setup

  1. Start PostgreSQL:

    docker-compose up -d postgres
    
  2. Run Prisma Migrations:

    pnpm prisma migrate dev
    
  3. Start Realtime Server:

    cd realtime-server
    pnpm install
    pnpm dev
    
  4. Start Next.js App:

    pnpm dev
    
  5. Verify Connection:

Docker Deployment

Build Image:

cd realtime-server
docker build -t petunia-realtime:latest .

Run Container:

docker run -d \
  --name petunia-realtime \
  -p 3001:3001 \
  -e JWT_SECRET=$NEXTAUTH_SECRET \
  -e REDIS_URL=$REDIS_URL \
  -e ALLOWED_ORIGINS=$ALLOWED_ORIGINS \
  petunia-realtime:latest

Health Check:

curl http://localhost:3001
# Expected: {"status":"ok","service":"petunia-realtime-server",...}

CI/CD Integration

GitHub Actions (.github/workflows/deploy.yml):

- name: Build realtime-server
  run: |
    cd realtime-server
    docker build -t ${{ secrets.REGISTRY }}/petunia-realtime:${{ github.sha }} .
    docker push ${{ secrets.REGISTRY }}/petunia-realtime:${{ github.sha }}

- name: Deploy realtime-server
  run: |
    kubectl set image deployment/realtime-server \
      realtime=${{ secrets.REGISTRY }}/petunia-realtime:${{ github.sha }}

Kubernetes Deployment

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: realtime-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: realtime-server
  template:
    metadata:
      labels:
        app: realtime-server
    spec:
      containers:
      - name: realtime
        image: petunia-realtime:latest
        ports:
        - containerPort: 3001
        env:
        - name: JWT_SECRET
          valueFrom:
            secretKeyRef:
              name: petunia-secrets
              key: jwt-secret
        - name: REDIS_URL
          value: redis://redis-service:6379
        - name: ALLOWED_ORIGINS
          value: https://app.petunia.gardenpatch.xyz
        livenessProbe:
          httpGet:
            path: /
            port: 3001
          initialDelaySeconds: 10
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /
            port: 3001
          initialDelaySeconds: 5
          periodSeconds: 10

Service:

apiVersion: v1
kind: Service
metadata:
  name: realtime-service
spec:
  type: LoadBalancer
  ports:
  - port: 3001
    targetPort: 3001
  selector:
    app: realtime-server

Horizontal Scaling

Requirements:

  • Redis instance for Socket.IO adapter
  • Sticky sessions (or disable connectionStateRecovery)

Enable Redis Adapter:

// realtime-server/index.ts
if (REDIS_URL) {
  const pubClient = new Redis(REDIS_URL);
  const subClient = pubClient.duplicate();
  io.adapter(createAdapter(pubClient, subClient));
}

Verify Scaling:

kubectl scale deployment/realtime-server --replicas=5
kubectl get pods -l app=realtime-server
# Should show 5 pods running

Troubleshooting

Connection Issues

Problem: WebSocket fails to connect

Check:

  1. Realtime server is running:
    curl http://localhost:3001
    
  2. JWT token is valid:
    curl http://localhost:3000/api/auth/ws-token \
      -H "Cookie: next-auth.session-token=..."
    
  3. CORS origins configured:
    echo $ALLOWED_ORIGINS
    # Should include client URL
    
  4. Browser console errors:
    // Look for Socket.IO errors
    socket.on('connect_error', (error) => {
      console.error('Connection failed:', error);
    });
    

Solution:

  • Verify NEXTAUTH_SECRET matches between apps
  • Check firewall rules for port 3001
  • Ensure WebSocket protocol not blocked by proxy

Authentication Errors

Problem: "Authentication token required" or "Invalid token"

Check:

  1. Token generation:
    const { token, error } = useWebSocketToken();
    console.log({ token, error });
    
  2. Token payload:
    // Decode JWT (use jwt.io)
    const payload = JSON.parse(atob(token.split('.')[1]));
    console.log(payload);
    
  3. Portal access:
    SELECT * FROM PortalUser WHERE userId = '...' AND portalId = '...';
    

Solution:

  • Refresh token: const { refresh } = useWebSocketToken(); refresh();
  • Check user has PortalUser record for portal
  • Verify JWT_SECRET environment variable set

Typing Indicators Not Working

Problem: Typing indicators don't appear or get stuck

Check:

  1. Subscription to typing room:
    socket.emit('typing:subscribe', { conversationId });
    
  2. Cleanup service running:
    # Check realtime-server logs
    docker logs petunia-realtime | grep cleanup
    
  3. Database records:
    SELECT * FROM TypingIndicator WHERE expiresAt > NOW();
    

Solution:

  • Re-subscribe to conversation: socket.emit('typing:subscribe', ...)
  • Manually stop typing: socket.emit('typing:stop', { conversationId })
  • Check cleanup service didn't crash (restart realtime-server)

Presence Not Updating

Problem: Users stuck in "online" or "offline" state

Check:

  1. Presence subscription:
    socket.emit('presence:subscribe');
    
  2. Database state:
    SELECT * FROM UserPresence
    WHERE userId = '...' AND portalId = '...';
    
  3. Disconnect handling:
    socket.on('disconnect', (reason) => {
      console.log('Disconnected:', reason);
    });
    

Solution:

  • Manual presence update: socket.emit('presence:update', { status: 'online' })
  • Clear stale records:
    DELETE FROM UserPresence
    WHERE lastSeen < NOW() - INTERVAL '24 hours';
    
  • Restart realtime-server to reset all presence

High Memory Usage

Problem: Realtime server memory grows over time

Check:

  1. Number of connections:
    curl http://localhost:3001 | jq .connections
    
  2. Cleanup service metrics:
    docker logs petunia-realtime | grep "Cleaned up"
    
  3. Orphaned timers:
    // In typing.ts
    cleanupTimers.size; // Should be small
    

Solution:

  • Restart server to clear in-memory state
  • Reduce CLEANUP_INTERVAL_MS for more frequent cleanup
  • Add memory limit in Docker: --memory=512m
  • Enable Redis adapter to distribute load

Debug Logging

Enable Debug Mode:

# Realtime server
LOG_LEVEL=debug pnpm dev

# Client-side
localStorage.debug = 'socket.io-client:*';

Useful Log Filters:

# Connection events
docker logs petunia-realtime | grep "connected\|disconnected"

# Authentication
docker logs petunia-realtime | grep "auth"

# Typing indicators
docker logs petunia-realtime | grep "typing"

# Cleanup service
docker logs petunia-realtime | grep "cleanup"

Performance Monitoring

Metrics Service:

// realtime-server/services/metrics.ts
metricsService.increment('connections.total');
metricsService.gauge('connections.active', io.engine.clientsCount);
metricsService.histogram('auth.latency_ms', latency);

Export to Monitoring:

  • Add Prometheus exporter
  • Send to DataDog/New Relic
  • Stream to CloudWatch

Load Testing & Performance Baselines

Overview

WebSocket reconnect storm load tests verify server stability under various stress conditions. Tests are located at realtime-server/tests/load/websocket-load.spec.ts.

Note: These are CI-safe sanity check thresholds, not production benchmarks. Production load testing should use dedicated infrastructure with higher connection counts.

Load Test Scenarios

ScenarioDescriptionKey Metrics
Thundering Herd100 clients reconnect simultaneouslyConnection P95 < 5s, P99 < 10s
Gradual Storm100 clients reconnect over 10 secondsConnection P50 < 3s, P95 < 5s
Sustained Load50 connections for 30 seconds95%+ retention, latency P95 < 2s
Message Flood20 clients × 10 msg/secSend P95 < 200ms, 10+ msg/sec throughput
Connection Churn30 stable + 50 churn cycles100% stable retention, P95 cycle < 500ms
Partial Outage50 connections, 50% disconnect/reconnect90%+ reconnect, P95 < 5s

Performance Baselines (CI-Safe)

Thundering Herd (100 simultaneous):
  - Connection success rate: ≥90%
  - Connection time P95: <5000ms
  - Connection time P99: <10000ms

Gradual Storm (100 over 10s):
  - Connection success rate: ≥95%
  - Connection time P50: <3000ms
  - Connection time P95: <5000ms

Sustained Load (50 for 30s):
  - Connection retention: ≥95%
  - Message latency P95: <2000ms
  - Message delivery rate: ≥90%

Message Flood (20×10 msg/sec):
  - Send latency P95: <200ms
  - Throughput: ≥10 msg/sec delivered

Connection Churn (30+50 cycles):
  - Stable connection retention: 100%
  - Churn cycle success rate: ≥95%
  - Churn cycle time P95: <500ms

Partial Outage (50% reconnect):
  - Reconnection success rate: ≥90%
  - Reconnection time P95: <5000ms

Running Load Tests

# Navigate to realtime-server
cd realtime-server

# Install dependencies
pnpm install

# Run load tests (isolated)
pnpm test -- --testPathPattern="load"

# Run with verbose output
pnpm test -- --testPathPattern="load" --reporter=verbose

Note: Load tests may take 5-10 minutes to complete due to sustained load scenarios.


Version History

VersionDateChanges
2.2.02025-12-28Added WebSocket reconnect storm load tests (MVP 7.18 gap closed)
2.1.12025-12-05Added verification snapshot, CI/test links, and owner mapping
2.1.02025-12-04Added Redis pub/sub bridge with reliability guarantees, auto-reconnect, retry queue
2.0.02025-12-04Replaced in-memory storage with Prisma persistence
1.0.02025-12-03Initial implementation with Socket.IO


Maintained by: Petunia Engineering Team Last Review: 2025-12-28