Documentation

VOICE_CALLS

docs/runbooks/VOICE_CALLS.md

Voice Calls Runbook

This runbook covers operational procedures for the Voice Call system, including outbound AI-powered calls, inbound call handling, transcription, and call analytics.

Owner: Platform Team Version: 1.0.0 Last Updated: 2025-12-27 Related Docs: PLATFORM_INTEGRATIONS.md


Table of Contents

  1. Prerequisites
  2. Health Check Commands
  3. Common Failure Modes
  4. Verification Commands
  5. Rollback Steps
  6. Escalation

Prerequisites

Required Services

ServicePurposeHealth Check
PostgreSQL (Supabase)Call logs, transcriptionspnpm db:health
ElevenLabsText-to-speech, voice AIvoiceHealthCheck.checkProvider('elevenlabs')
Twilio VoicePhone infrastructureTwilio Dashboard
Retell AIAlternative voice providerRetell Dashboard
CartesiaAlternative voice providerCartesia Dashboard
OpenAI WhisperTranscriptionOpenAI Dashboard

Required Environment Variables

# Database
DATABASE_URL="postgresql://..."

# ElevenLabs (Primary Voice Provider)
ELEVENLABS_API_KEY="..."
ELEVENLABS_VOICE_ID="..."

# Twilio Voice
TWILIO_ACCOUNT_SID="..."
TWILIO_AUTH_TOKEN="..."
TWILIO_PHONE_NUMBER="+1..."
# Public base URL used to generate TwiML <Play>/<Gather> URLs
NEXT_PUBLIC_SITE_URL="https://your-domain.com"

# (Optional) If you validate Twilio signatures in production, set the canonical webhook URL used for signature calc
TWILIO_VOICE_WEBHOOK_URL="https://your-domain.com/api/webhooks/twilio/voice"

# Phone TTS provider for Twilio <Play> (default: elevenlabs). Options: elevenlabs | cartesia
TWILIO_VOICE_TTS_PROVIDER="elevenlabs"

# Retell AI (Alternative)
RETELL_API_KEY="..."

# Cartesia (Alternative)
CARTESIA_API_KEY="..."

# Transcription
OPENAI_API_KEY="..."

Key Files

ComponentFile Path
Unified Voice Servicelib/services/voice/unifiedVoiceService.ts
Voice Health Checklib/services/voice/voiceHealthCheck.ts
ElevenLabs Providerlib/services/voice/elevenLabsProvider.ts
Retell Providerlib/services/voice/retellProvider.ts
Cartesia Providerlib/services/voice/cartesiaProvider.ts
Transcription Servicelib/services/voice/voiceTranscriptionService.ts
Whisper Transcriberlib/services/voice/transcription/whisperTranscriber.ts
Failover Logiclib/services/voice/failover/failoverStrategy.ts
Circuit Breakerlib/services/voice/failover/circuitBreaker.ts
Call Log Ingestionlib/services/voice/callLogIngestionService.ts
Inbound Call Handlerlib/services/voice/inboundCallHandler.ts
Voice Analyticslib/services/voice/voiceAnalyticsService.ts
Twilio Voice Webhook (turn-based loop)app/api/webhooks/twilio/voice/route.ts
Twilio Voice TTS (signed stream)app/api/webhooks/twilio/voice/tts/route.ts

Architecture Snapshot (What We Actually Run Today)

Twilio Inbound Voice (Turn-Based “Phone Chat”)

  • Twilio number Voice webhook → POST /api/webhooks/twilio/voice
  • TwiML returned:
    • <Play> signed URL to GET /api/webhooks/twilio/voice/tts (streaming TTS)
    • <Gather input="speech"> posts back to /api/webhooks/twilio/voice

Important: This is a half‑duplex turn loop (speak → respond → speak). It is production-shaped and measurable, but it is not full duplex interruption.

Twilio Media Streams (Not Implemented Yet)

True interruptible, sub‑300ms “human receptionist” turn‑taking requires Twilio Media Streams (bi‑directional audio WebSocket) plus streaming STT/TTS. Track as a separate project; do not assume it exists.

Health Check Commands

Quick Provider Health Check

# Check all voice provider health
curl https://your-domain.com/api/admin/voice/health

# Expected response:
# {
#   "providers": [
#     {
#       "provider": "elevenlabs",
#       "status": "healthy",
#       "details": {
#         "apiKeyValid": true,
#         "apiReachable": true,
#         "quotaAvailable": true,
#         "voicesAvailable": true,
#         "latency": 245
#       }
#     }
#   ]
# }

ElevenLabs Health Check

# Check ElevenLabs API status
curl https://api.elevenlabs.io/v1/user \
  -H "xi-api-key: $ELEVENLABS_API_KEY"

# Check quota usage
curl https://api.elevenlabs.io/v1/user/subscription \
  -H "xi-api-key: $ELEVENLABS_API_KEY"

# Verify voices are available
curl https://api.elevenlabs.io/v1/voices \
  -H "xi-api-key: $ELEVENLABS_API_KEY"

Twilio Voice Health Check

# Check Twilio account status
curl -u "$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN" \
  "https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID.json"

# Check recent calls
curl -u "$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN" \
  "https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Calls.json?PageSize=5"

Transcription Health Check

# Test Whisper API availability
curl https://api.openai.com/v1/audio/transcriptions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -F file=@test-audio.wav \
  -F model="whisper-1"

Database Health

# Check call log stats
psql $DATABASE_URL -c "
SELECT
  status,
  COUNT(*) as call_count,
  AVG(duration) as avg_duration,
  MAX(created_at) as last_call
FROM \"CallLog\"
WHERE created_at > NOW() - INTERVAL '24 hours'
GROUP BY status;
"

# Check transcription backlog
psql $DATABASE_URL -c "
SELECT COUNT(*) as pending_transcriptions
FROM \"CallLog\"
WHERE transcription IS NULL
AND status = 'completed'
AND created_at > NOW() - INTERVAL '24 hours';
"

Common Failure Modes

1. Calls Not Connecting

SymptomLikely CauseResolution
All calls fail to connectTwilio credentials invalidVerify TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN
Calls ring but no audioVoice provider downCheck ElevenLabs/Retell status, failover may activate
Specific numbers failNumber blocked or invalidCheck Twilio carrier lookup for number validity
Calls drop after connectingTwiML webhook failingCheck /api/voice/twiml endpoint

Diagnostic Commands:

# Check recent failed calls
psql $DATABASE_URL -c "
SELECT id, phone_number, status, error_message, created_at
FROM \"CallLog\"
WHERE status = 'failed'
AND created_at > NOW() - INTERVAL '1 hour'
ORDER BY created_at DESC
LIMIT 20;
"

# Check Twilio error logs
curl -u "$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN" \
  "https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Calls.json?Status=failed&PageSize=10"

2. Recording/Transcription Issues

SymptomLikely CauseResolution
No transcriptions appearingWhisper API key invalidVerify OPENAI_API_KEY
Transcription delayedQueue backlogCheck retry queue status
Partial transcriptionsAudio file corruptedCheck recording storage
Transcription errorsRate limit exceededReduce concurrent transcription jobs

Diagnostic Commands:

# Check transcription queue
psql $DATABASE_URL -c "
SELECT
  status,
  COUNT(*) as count,
  MIN(created_at) as oldest
FROM \"TranscriptionJob\"
GROUP BY status;
"

# Check for failed transcriptions
psql $DATABASE_URL -c "
SELECT id, call_id, error_message, retry_count, created_at
FROM \"TranscriptionJob\"
WHERE status = 'failed'
AND created_at > NOW() - INTERVAL '24 hours'
ORDER BY created_at DESC
LIMIT 10;
"

3. Call Quality Problems

SymptomLikely CauseResolution
Audio choppy/roboticHigh latency to voice providerCheck provider latency, consider failover
Voice sounds wrongWrong voice ID configuredVerify ELEVENLABS_VOICE_ID
Echo/feedbackAudio routing issueCheck TwiML configuration
Silence during callsTTS generation failingCheck voice provider logs

Diagnostic Commands:

# Check voice provider latency
curl -w "Time: %{time_total}s\n" -o /dev/null -s \
  "https://api.elevenlabs.io/v1/voices" \
  -H "xi-api-key: $ELEVENLABS_API_KEY"

# Check circuit breaker status
curl https://your-domain.com/api/admin/voice/circuit-breaker

4. Provider Failover Issues

SymptomLikely CauseResolution
Failover not triggeringCircuit breaker thresholds not metAdjust circuit breaker configuration
Failover too aggressiveThreshold too lowIncrease failure threshold
No fallback providerSecondary not configuredConfigure Retell or Cartesia as fallback

Diagnostic Commands:

# Check failover configuration
curl https://your-domain.com/api/admin/voice/failover/config

# Check provider failure counts
psql $DATABASE_URL -c "
SELECT provider, COUNT(*) as failure_count
FROM \"VoiceProviderEvent\"
WHERE event_type = 'failure'
AND created_at > NOW() - INTERVAL '1 hour'
GROUP BY provider;
"

Verification Commands

Test Voice Health Check Programmatically

import { voiceHealthCheck } from '@/lib/services/voice/voiceHealthCheck';

// Check all providers
const results = await voiceHealthCheck.checkAllProviders();
console.log(results);

// Check specific provider
const elevenLabsStatus = await voiceHealthCheck.checkProvider('elevenlabs');
console.log(elevenLabsStatus);

Test Outbound Call

# Initiate test call via API
curl -X POST https://your-domain.com/api/voice/call \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "phoneNumber": "+1234567890",
    "script": "Hello, this is a test call.",
    "portalId": "your-portal-id"
  }'

Test TwiML Generation

# Test TwiML webhook locally
curl -X POST http://localhost:3000/api/voice/twiml \
  -d "CallSid=CA123&From=+1234567890&To=+0987654321"

# Expected: Valid TwiML response

Run Voice Tests

# Run voice service tests
pnpm test -- --testPathPattern="voice"

# Run specific provider tests
pnpm test -- --testPathPattern="elevenLabsProvider"
pnpm test -- --testPathPattern="retellProvider"
pnpm test -- --testPathPattern="voiceTranscriptionService"

Rollback Steps

Disable Voice Feature

# Set feature flag to disable voice calls
curl -X POST https://your-domain.com/api/admin/features \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"feature": "voice.enabled", "enabled": false}'

# Or set environment variable
# VOICE_ENABLED=false

Force Provider Failover

# Force switch to secondary provider
curl -X POST https://your-domain.com/api/admin/voice/failover/force \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"targetProvider": "retell"}'

Reset Circuit Breaker

# Reset circuit breaker state
curl -X POST https://your-domain.com/api/admin/voice/circuit-breaker/reset \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Clear Voice Health Cache

import { voiceHealthCheck } from '@/lib/services/voice/voiceHealthCheck';

// Clear cache to force re-check
voiceHealthCheck.clearCache();

Rollback Voice Provider Configuration

# If ElevenLabs is having issues, temporarily switch to Retell
# Update environment variable:
# VOICE_PRIMARY_PROVIDER=retell

# Restart application to pick up changes
vercel redeploy

Escalation

When to Escalate

SeverityCriteriaAction
P1 - CriticalAll voice calls failingImmediate on-call page
P2 - HighPrimary provider down, failover activeEscalate within 30 minutes
P3 - MediumCall quality issues, partial failuresEscalate within 2 hours
P4 - LowTranscription delays, minor issuesNormal ticket queue

Escalation Contacts

RoleContactAvailability
On-Call EngineerPagerDuty24/7
Voice Team LeadSlack #voiceBusiness hours
ElevenLabs Supportsupport@elevenlabs.ioBusiness hours
Twilio Supportsupport.twilio.com24/7
Retell Supportsupport@retellai.comBusiness hours

Information to Include

When escalating, include:

  1. Provider Status: Which voice providers are affected
  2. Error Details: Specific error messages from logs
  3. Call Volume: Number of failed vs successful calls
  4. Failover Status: Whether failover is active
  5. Recent Changes: Any recent deployments or config changes

Monitoring & Alerts

Key Metrics to Watch

MetricHealthy RangeAlert Threshold
Call success rate> 95%< 90%
Voice provider latency (p95)< 500ms> 1s
Transcription backlog< 50> 200
Active callsVariesSudden drop to 0
Circuit breaker open0> 0
Quota remaining (ElevenLabs)> 10%< 5%

Alert Configuration

Alerts are configured via:

  • Sentry for error tracking
  • Slack #voice-alerts channel for operational issues
  • PagerDuty for P1/P2 incidents

Voice-Specific Dashboards


Provider-Specific Notes

ElevenLabs

  • Quota Management: Monitor character usage, alerts at 5% remaining
  • Voice Cloning: Custom voices require separate verification
  • Rate Limits: 100 concurrent requests default, contact for increase

Twilio

  • Number Verification: New numbers may require verification period
  • Recording Storage: Recordings auto-delete after 30 days by default
  • Geographic Restrictions: Some regions require special setup

Retell AI

  • Conversation Mode: Supports multi-turn conversations
  • Knowledge Base: Can inject business knowledge for context
  • Latency: May be slower than ElevenLabs for simple TTS

Last verified: 2025-12-27