Voice Calls Runbook

This runbook covers operational procedures for the Voice Call system, including outbound AI-powered calls, inbound call handling, transcription, and call analytics.

Owner: Platform Team Version: 1.0.0 Last Updated: 2025-12-27 Related Docs: PLATFORM_INTEGRATIONS.md

---

Table of Contents

1. Prerequisites
2. Health Check Commands
3. Common Failure Modes
4. Verification Commands
5. Rollback Steps
6. Escalation

---

Prerequisites

Required Services

Service	Purpose	Health Check
PostgreSQL (Supabase)	Call logs, transcriptions	pnpm db:health
ElevenLabs	Text-to-speech, voice AI	voiceHealthCheck.checkProvider('elevenlabs')
Twilio Voice	Phone infrastructure	Twilio Dashboard
Retell AI	Alternative voice provider	Retell Dashboard
Cartesia	Alternative voice provider	Cartesia Dashboard
OpenAI Whisper	Transcription	OpenAI Dashboard

Required Environment Variables

# Database
DATABASE_URL="postgresql://..."

# ElevenLabs (Primary Voice Provider)
ELEVENLABS_API_KEY="..."
ELEVENLABS_VOICE_ID="..."

# Twilio Voice
TWILIO_ACCOUNT_SID="..."
TWILIO_AUTH_TOKEN="..."
TWILIO_PHONE_NUMBER="+1..."
# Public base URL used to generate TwiML <Play>/<Gather> URLs
NEXT_PUBLIC_SITE_URL="https://your-domain.com"

# (Optional) If you validate Twilio signatures in production, set the canonical webhook URL used for signature calc
TWILIO_VOICE_WEBHOOK_URL="https://your-domain.com/api/webhooks/twilio/voice"

# Phone TTS provider for Twilio <Play> (default: elevenlabs). Options: elevenlabs | cartesia
TWILIO_VOICE_TTS_PROVIDER="elevenlabs"

# Retell AI (Alternative)
RETELL_API_KEY="..."

# Cartesia (Alternative)
CARTESIA_API_KEY="..."

# Transcription
OPENAI_API_KEY="..."

Key Files

Component	File Path
Unified Voice Service	lib/services/voice/unifiedVoiceService.ts
Voice Health Check	lib/services/voice/voiceHealthCheck.ts
ElevenLabs Provider	lib/services/voice/elevenLabsProvider.ts
Retell Provider	lib/services/voice/retellProvider.ts
Cartesia Provider	lib/services/voice/cartesiaProvider.ts
Transcription Service	lib/services/voice/voiceTranscriptionService.ts
Whisper Transcriber	lib/services/voice/transcription/whisperTranscriber.ts
Failover Logic	lib/services/voice/failover/failoverStrategy.ts
Circuit Breaker	lib/services/voice/failover/circuitBreaker.ts
Call Log Ingestion	lib/services/voice/callLogIngestionService.ts
Inbound Call Handler	lib/services/voice/inboundCallHandler.ts
Voice Analytics	lib/services/voice/voiceAnalyticsService.ts
Twilio Voice Webhook (turn-based loop)	app/api/webhooks/twilio/voice/route.ts
Twilio Voice TTS (signed stream)	app/api/webhooks/twilio/voice/tts/route.ts

---

Architecture Snapshot (What We Actually Run Today)

Twilio Inbound Voice (Turn-Based “Phone Chat”)

• Twilio number Voice webhook → POST /api/webhooks/twilio/voice
• TwiML returned:
  • <Play> signed URL to GET /api/webhooks/twilio/voice/tts (streaming TTS)
  • <Gather input="speech"> posts back to /api/webhooks/twilio/voice

Important: This is a half‑duplex turn loop (speak → respond → speak). It is production-shaped and measurable, but it is not full duplex interruption.

Twilio Media Streams (Not Implemented Yet)

True interruptible, sub‑300ms “human receptionist” turn‑taking requires Twilio Media Streams (bi‑directional audio WebSocket) plus streaming STT/TTS. Track as a separate project; do not assume it exists.

Health Check Commands

Quick Provider Health Check

# Check all voice provider health
curl https://your-domain.com/api/admin/voice/health

# Expected response:
# {
#   "providers": [
#     {
#       "provider": "elevenlabs",
#       "status": "healthy",
#       "details": {
#         "apiKeyValid": true,
#         "apiReachable": true,
#         "quotaAvailable": true,
#         "voicesAvailable": true,
#         "latency": 245
#       }
#     }
#   ]
# }

ElevenLabs Health Check

# Check ElevenLabs API status
curl https://api.elevenlabs.io/v1/user \
  -H "xi-api-key: $ELEVENLABS_API_KEY"

# Check quota usage
curl https://api.elevenlabs.io/v1/user/subscription \
  -H "xi-api-key: $ELEVENLABS_API_KEY"

# Verify voices are available
curl https://api.elevenlabs.io/v1/voices \
  -H "xi-api-key: $ELEVENLABS_API_KEY"

Twilio Voice Health Check

# Check Twilio account status
curl -u "$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN" \
  "https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID.json"

# Check recent calls
curl -u "$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN" \
  "https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Calls.json?PageSize=5"

Transcription Health Check

# Test Whisper API availability
curl https://api.openai.com/v1/audio/transcriptions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -F file=@test-audio.wav \
  -F model="whisper-1"

Database Health

# Check call log stats
psql $DATABASE_URL -c "
SELECT
  status,
  COUNT(*) as call_count,
  AVG(duration) as avg_duration,
  MAX(created_at) as last_call
FROM \"CallLog\"
WHERE created_at > NOW() - INTERVAL '24 hours'
GROUP BY status;
"

# Check transcription backlog
psql $DATABASE_URL -c "
SELECT COUNT(*) as pending_transcriptions
FROM \"CallLog\"
WHERE transcription IS NULL
AND status = 'completed'
AND created_at > NOW() - INTERVAL '24 hours';
"

---

Common Failure Modes

1. Calls Not Connecting

Symptom	Likely Cause	Resolution
All calls fail to connect	Twilio credentials invalid	Verify TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN
Calls ring but no audio	Voice provider down	Check ElevenLabs/Retell status, failover may activate
Specific numbers fail	Number blocked or invalid	Check Twilio carrier lookup for number validity
Calls drop after connecting	TwiML webhook failing	Check /api/voice/twiml endpoint

Diagnostic Commands:

# Check recent failed calls
psql $DATABASE_URL -c "
SELECT id, phone_number, status, error_message, created_at
FROM \"CallLog\"
WHERE status = 'failed'
AND created_at > NOW() - INTERVAL '1 hour'
ORDER BY created_at DESC
LIMIT 20;
"

# Check Twilio error logs
curl -u "$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN" \
  "https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Calls.json?Status=failed&PageSize=10"

2. Recording/Transcription Issues

Symptom	Likely Cause	Resolution
No transcriptions appearing	Whisper API key invalid	Verify OPENAI_API_KEY
Transcription delayed	Queue backlog	Check retry queue status
Partial transcriptions	Audio file corrupted	Check recording storage
Transcription errors	Rate limit exceeded	Reduce concurrent transcription jobs

Diagnostic Commands:

# Check transcription queue
psql $DATABASE_URL -c "
SELECT
  status,
  COUNT(*) as count,
  MIN(created_at) as oldest
FROM \"TranscriptionJob\"
GROUP BY status;
"

# Check for failed transcriptions
psql $DATABASE_URL -c "
SELECT id, call_id, error_message, retry_count, created_at
FROM \"TranscriptionJob\"
WHERE status = 'failed'
AND created_at > NOW() - INTERVAL '24 hours'
ORDER BY created_at DESC
LIMIT 10;
"

3. Call Quality Problems

Symptom	Likely Cause	Resolution
Audio choppy/robotic	High latency to voice provider	Check provider latency, consider failover
Voice sounds wrong	Wrong voice ID configured	Verify ELEVENLABS_VOICE_ID
Echo/feedback	Audio routing issue	Check TwiML configuration
Silence during calls	TTS generation failing	Check voice provider logs

Diagnostic Commands:

# Check voice provider latency
curl -w "Time: %{time_total}s\n" -o /dev/null -s \
  "https://api.elevenlabs.io/v1/voices" \
  -H "xi-api-key: $ELEVENLABS_API_KEY"

# Check circuit breaker status
curl https://your-domain.com/api/admin/voice/circuit-breaker

4. Provider Failover Issues

Symptom	Likely Cause	Resolution
Failover not triggering	Circuit breaker thresholds not met	Adjust circuit breaker configuration
Failover too aggressive	Threshold too low	Increase failure threshold
No fallback provider	Secondary not configured	Configure Retell or Cartesia as fallback

Diagnostic Commands:

# Check failover configuration
curl https://your-domain.com/api/admin/voice/failover/config

# Check provider failure counts
psql $DATABASE_URL -c "
SELECT provider, COUNT(*) as failure_count
FROM \"VoiceProviderEvent\"
WHERE event_type = 'failure'
AND created_at > NOW() - INTERVAL '1 hour'
GROUP BY provider;
"

---

Verification Commands

Test Voice Health Check Programmatically

import { voiceHealthCheck } from '@/lib/services/voice/voiceHealthCheck';

// Check all providers
const results = await voiceHealthCheck.checkAllProviders();
console.log(results);

// Check specific provider
const elevenLabsStatus = await voiceHealthCheck.checkProvider('elevenlabs');
console.log(elevenLabsStatus);

Test Outbound Call

# Initiate test call via API
curl -X POST https://your-domain.com/api/voice/call \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "phoneNumber": "+1234567890",
    "script": "Hello, this is a test call.",
    "portalId": "your-portal-id"
  }'

Test TwiML Generation

# Test TwiML webhook locally
curl -X POST http://localhost:3000/api/voice/twiml \
  -d "CallSid=CA123&From=+1234567890&To=+0987654321"

# Expected: Valid TwiML response

Run Voice Tests

# Run voice service tests
pnpm test -- --testPathPattern="voice"

# Run specific provider tests
pnpm test -- --testPathPattern="elevenLabsProvider"
pnpm test -- --testPathPattern="retellProvider"
pnpm test -- --testPathPattern="voiceTranscriptionService"

---

Rollback Steps

Disable Voice Feature

# Set feature flag to disable voice calls
curl -X POST https://your-domain.com/api/admin/features \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"feature": "voice.enabled", "enabled": false}'

# Or set environment variable
# VOICE_ENABLED=false

Force Provider Failover

# Force switch to secondary provider
curl -X POST https://your-domain.com/api/admin/voice/failover/force \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"targetProvider": "retell"}'

Reset Circuit Breaker

# Reset circuit breaker state
curl -X POST https://your-domain.com/api/admin/voice/circuit-breaker/reset \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Clear Voice Health Cache

import { voiceHealthCheck } from '@/lib/services/voice/voiceHealthCheck';

// Clear cache to force re-check
voiceHealthCheck.clearCache();

Rollback Voice Provider Configuration

# If ElevenLabs is having issues, temporarily switch to Retell
# Update environment variable:
# VOICE_PRIMARY_PROVIDER=retell

# Restart application to pick up changes
vercel redeploy

---

Escalation

When to Escalate

Severity	Criteria	Action
P1 - Critical	All voice calls failing	Immediate on-call page
P2 - High	Primary provider down, failover active	Escalate within 30 minutes
P3 - Medium	Call quality issues, partial failures	Escalate within 2 hours
P4 - Low	Transcription delays, minor issues	Normal ticket queue

Escalation Contacts

Role	Contact	Availability
On-Call Engineer	PagerDuty	24/7
Voice Team Lead	Slack #voice	Business hours
ElevenLabs Support	support@elevenlabs.io	Business hours
Twilio Support	support.twilio.com	24/7
Retell Support	support@retellai.com	Business hours

Information to Include

When escalating, include:

1. Provider Status: Which voice providers are affected
2. Error Details: Specific error messages from logs
3. Call Volume: Number of failed vs successful calls
4. Failover Status: Whether failover is active
5. Recent Changes: Any recent deployments or config changes

---

Monitoring & Alerts

Key Metrics to Watch

Metric	Healthy Range	Alert Threshold
Call success rate	> 95%	< 90%
Voice provider latency (p95)	< 500ms	> 1s
Transcription backlog	< 50	> 200
Active calls	Varies	Sudden drop to 0
Circuit breaker open	0	> 0
Quota remaining (ElevenLabs)	> 10%	< 5%

Alert Configuration

Alerts are configured via:

• Sentry for error tracking
• Slack #voice-alerts channel for operational issues
• PagerDuty for P1/P2 incidents

Voice-Specific Dashboards

• ElevenLabs Usage: https://elevenlabs.io/dashboard
• Twilio Console: https://console.twilio.com/
• Retell Dashboard: https://dashboard.retellai.com/
• OpenAI Usage: https://platform.openai.com/usage

---

Provider-Specific Notes

ElevenLabs

• Quota Management: Monitor character usage, alerts at 5% remaining
• Voice Cloning: Custom voices require separate verification
• Rate Limits: 100 concurrent requests default, contact for increase

Twilio

• Number Verification: New numbers may require verification period
• Recording Storage: Recordings auto-delete after 30 days by default
• Geographic Restrictions: Some regions require special setup

Retell AI

• Conversation Mode: Supports multi-turn conversations
• Knowledge Base: Can inject business knowledge for context
• Latency: May be slower than ElevenLabs for simple TTS

---

Last verified: 2025-12-27