Documentation

VOICE_AI_SETUP

docs/VOICE_AI_SETUP.md

Voice AI Provider Configuration Guide

Table of Contents

  1. Overview
  2. Architecture
    • Failover Behavior & Resilience
  3. Provider Setup
    • Cartesia Configuration
    • ElevenLabs Configuration
    • Retell Configuration
  4. API Key Management
  5. Testing Voice Providers
  6. Troubleshooting
  7. Voice AI Personality
  8. Teach Petunia from Your Calls

Overview

Petunia's Voice AI system supports multiple provider building blocks:

  • Cartesia Sonic-3 (Primary TTS) - Ultra-realistic text-to-speech
  • Cartesia Ink (Primary STT) - Speech-to-text transcription (browser voice + realtime pipelines)
  • ElevenLabs (Fallback TTS) - v3 TTS with audio tags
  • Twilio / RingCentral (Telephony carriers) - Phone infrastructure (we orchestrate ourselves)

The system uses a unified interface with automatic fallback between providers, allowing seamless degradation if a primary provider is unavailable.

Provider Roles (Production Intent)

ProviderUse CasePriorityCapabilities
Cartesia Sonic-3Primary TTS1stNatural-sounding speech synthesis
Cartesia InkPrimary STT1stSpeech-to-text transcription
Backend LLM (Claude/OpenAI)Agent “brain”1stSystem prompt + tool use + retrieval (our code owns this)
Twilio/RingCentralCarrier1stDialing + phone numbers
ElevenLabsFallback TTS2ndTTS v3, audio tags, voice cloning

Architecture

Unified Voice Service

All voice providers implement the VoiceProvider interface defined in /lib/services/voice/unifiedVoiceService.ts:

interface VoiceProvider {
  initiateCall(params: CallParams): Promise<CallResponse>;
  endCall(callId: string): Promise<boolean>;
  createAgent?(params: AgentParams): Promise<AgentResponse>;
}

Automatic Fallback Chain

When initiating calls without specifying a provider:

1. Cartesia (primary)
   ↓ (if fails)
2. ElevenLabs (fallback)
   ↓ (if fails)
3. Error thrown

Note: Retell is NOT in the automatic fallback chain as it's designed for client-specific telephony integrations.

Provider Implementation Files

  • Cartesia: /lib/services/voice/cartesiaProvider.ts
  • ElevenLabs: /lib/services/voice/elevenLabsProvider.ts
  • Retell: /lib/services/voice/retellProvider.ts

Failover Behavior & Resilience

The voice system implements robust failover patterns to ensure high availability.

Circuit Breaker Pattern

Each provider is protected by a circuit breaker that prevents cascading failures:

Circuit States:
┌─────────┐     failures >= threshold    ┌─────────┐
│ CLOSED  │ ──────────────────────────→ │  OPEN   │
│(normal) │                              │(blocked)│
└─────────┘                              └─────────┘
     ↑                                        │
     │         half-open timeout              │
     │    ←─────────────────────────────      │
     │                                        ↓
     │                                   ┌──────────┐
     └────── success ─────────────────── │HALF_OPEN │
                                         │ (probe)  │
                                         └──────────┘

Default Thresholds:

  • Failure threshold: 3 consecutive failures opens the circuit
  • Reset timeout: 30 seconds before attempting half-open probe
  • Success threshold: 1 success in half-open closes the circuit

Failover SLA Requirements

MetricTargetDescription
Max Failover Time5,000msTotal time from primary failure to backup response
Max Healthy Latency500msExpected latency for healthy provider
Connection Timeout10,000msMaximum wait for hung connections
Circuit Reset30,000msTime before circuit breaker attempts recovery

Retry Behavior

The system uses exponential backoff with jitter for transient failures:

// Retry pattern for transient errors
attempt 1: immediate
attempt 2: ~100ms delay (with jitter)
attempt 3: ~400ms delay (with jitter)
// Then failover to next provider

Retryable Errors (automatically retried):

  • ECONNRESET - Connection reset
  • ETIMEDOUT - Connection timeout
  • ENOTFOUND - DNS resolution failure
  • EAI_AGAIN - DNS temporary failure

Non-Retryable Errors (immediate failover):

  • 401/403 - Authentication failures
  • 400 - Bad request
  • 429 - Rate limiting (triggers failover, not retry)

Provider Health Monitoring

Each provider exposes health status through the circuit breaker:

import { voiceCircuitBreakerRegistry } from '@/lib/services/voice/failover';

// Check provider status
const breaker = voiceCircuitBreakerRegistry.get('cartesia');
const state = breaker?.getState(); // 'CLOSED' | 'OPEN' | 'HALF_OPEN'

Testing Resilience

The voice system includes comprehensive chaos tests in /tests/chaos/voice-resilience.chaos.test.ts:

Test Categories:

  1. Provider Failover - Primary fails → secondary takes over
  2. Circuit Breaker - Opens after threshold, resets after timeout
  3. Network Resilience - Handles latency, timeouts, DNS failures
  4. Infrastructure Failures - SSL errors, partial responses
  5. Concurrent Calls - Multiple calls during provider failure

Running Chaos Tests:

# Run all voice chaos tests
pnpm test -- tests/chaos/voice-resilience.chaos.test.ts

# Run with verbose output
pnpm test -- tests/chaos/voice-resilience.chaos.test.ts --verbose

CI Integration: Chaos tests run in CI when:

  • Manually triggered via workflow_dispatch
  • PR has run-chaos label
  • Voice system files are modified (lib/services/voice/ or tests/chaos/)

Provider Setup

Cartesia Configuration

Cartesia is Petunia's primary voice provider for voice I/O, offering:

  • Sonic 3.0 TTS: Ultra-realistic voice synthesis
  • Ink STT: Speech-to-text

Environment Variables

# Required
CARTESIA_API_KEY="your_api_key_here"

# Optional (defaults provided)
CARTESIA_VOICE_ID="6ccbfb76-1fc6-48f7-b71d-91ac6298247b"  # Petunia's default voice
NEXT_PUBLIC_CARTESIA_ENABLED="true"

Default Voice ID

Petunia uses a warm, confident female voice:

  • Voice ID: 6ccbfb76-1fc6-48f7-b71d-91ac6298247b
  • Characteristics: Warm-confident tone, conversational pace, high energy

Defined in /lib/services/voice/cartesiaProvider.ts:

export const PETUNIA_CARTESIA_VOICE_ID = '6ccbfb76-1fc6-48f7-b71d-91ac6298247b';

Cartesia Line (Removed)

Cartesia Line is not used in Petunia. Telephony is handled via Twilio/RingCentral, and the “agent” lives on our backend (LLM + memory + tools).

Phone Architecture: Tier A vs Tier B (IMPORTANT)

  • Tier A (ship today): turn-based conversational loop (Twilio <Gather> speech → LLM reply → signed <Play> streaming TTS → repeat)
    • Webhook: POST /api/webhooks/twilio/voice
    • TTS stream: GET /api/webhooks/twilio/voice/tts
  • Tier B (real-time duplex): true interruptible phone agent via Twilio Media Streams (duplex WS audio)
    • Realtime WS endpoint: realtime-server /twilio-media
    • Streaming STT: Cartesia Ink WebSocket
    • Streaming TTS: ElevenLabs output_format=ulaw_8000 (frames sent as outbound media)
    • Interruption: Twilio clear on caller speech activity

Tier B env flags:

TWILIO_VOICE_MODE=stream
TWILIO_MEDIA_WS_URL=wss://YOUR_PUBLIC_REALTIME_SERVER_HOST
TWILIO_STREAM_STT=cartesia
TWILIO_STREAM_BRAIN=anthropic
TWILIO_STREAM_TTS=elevenlabs

API Endpoints (Current)

  • Health Check: GET /api/voice/cartesia/health
  • TTS: POST /api/voice/cartesia/tts
<!-- Cartesia Line route intentionally removed -->
  • Ink (STT): POST /api/voice/cartesia/ink

Important: No Manual Cartesia Agent ID Required

For onboarding we do NOT require CARTESIA_AGENT_ID or CARTESIA_LINE_AGENT_ID. Phone calls are orchestrated by our backend via Twilio/RingCentral. No Cartesia Line agent setup is required.

Source of truth for Petunia onboarding personality lives in:

  • lib/config/petunia-personality.ts (getOnboardingPrompt, getConversationPhase)

Usage Example

import { CartesiaProvider } from '@/lib/services/voice/cartesiaProvider';

// Create provider (auto-resolves API key)
const provider = await CartesiaProvider.create();

// For TTS
const response = await provider.initiateCall({
  text: 'Hello! This is a test of Cartesia TTS.',
  voiceId: '6ccbfb76-1fc6-48f7-b71d-91ac6298247b',
});

// For phone calls, use Twilio carrier APIs; the agent lives on our backend.

ElevenLabs Configuration

ElevenLabs serves as the fallback voice provider with advanced v3 capabilities.

Environment Variables

# Required
ELEVENLABS_API_KEY="your_api_key_here"

# Optional
NEXT_PUBLIC_ELEVENLABS_VOICE_ID="pNInz6obpgDQGcFmaJgB"  # Default voice

Default Voice ID

ElevenLabs default: pNInz6obpgDQGcFmaJgB

Can be overridden in /lib/config/petunia-personality.ts:

export const PETUNIA_VOICE = {
  elevenlabsVoiceId: 'pMsXgVXv3BLzUgSXRplE',  // Nova voice
};

ElevenLabs v3 Features

Audio Tags: Enhance expressiveness with emotional context

// From elevenLabsProvider.ts - applyV3AudioTags()
'[excited] This is amazing news!'
'This is a pause. [pause] And this continues.'

Voice Settings:

const voiceOptions = {
  stability: 0.5,           // Balance consistency vs. expressiveness
  similarity_boost: 0.75,   // Voice similarity to original
  style: 0.2,               // Style exaggeration
  use_speaker_boost: true,  // Enhanced clarity
};

Model: eleven_turbo_v2_5 (fast, high-quality)

Conversational AI

ElevenLabs Conversational AI supports real phone calls with agents.

Creating an Agent:

const agent = await provider.createAgent({
  name: 'Petunia',
  voiceId: 'pNInz6obpgDQGcFmaJgB',
  isDemoAgent: false,
});

Caching

ElevenLabs provider implements 30-minute caching for frequently used phrases:

const CACHE_EXPIRATION_MS = 30 * 60 * 1000; // 30 minutes

API Endpoints

  • TTS: POST /api/voice/elevenlabs/tts
  • Call: POST /api/communications/voice/elevenlabs/call
  • End Call: POST /api/communications/voice/elevenlabs/call/[callId]/end
  • Status: GET /api/voice/elevenlabs/status/[conversationId]
  • Webhook: POST /api/voice/elevenlabs/webhook/[conversationId]

Usage Example

import { ElevenLabsProvider } from '@/lib/services/voice/elevenLabsProvider';

// Create provider
const provider = await ElevenLabsProvider.create();

// TTS with v3 audio tags
const response = await provider.initiateCall({
  text: '[excited] Welcome to our business!',
  voiceId: 'pNInz6obpgDQGcFmaJgB',
  useKnowledgeBase: true,  // Enhance with business context
  pid: 'portal_id',        // Portal ID for context
});

// Conversational AI call
const callResponse = await provider.initiateCall({
  phoneNumber: '+15551234567',
  agentId: 'el-v3-agent-xyz',
  initialContext: { customerName: 'John' },
});

Retell Configuration

Retell provides full telephony capabilities with AI voice agents, knowledge base integration, and webhook support.

Environment Variables

# Required
RETELL_API_KEY="your_api_key_here"

# Optional - Agent Configuration
NEXT_PUBLIC_RETELL_AGENT_ID="agent_id_here"
NEXT_PUBLIC_RETELL_VOICE_ID="eleven_multilingual_v2"

# Optional - Phone Integration
TWILIO_PHONE_NUMBER="+15551234567"      # Caller ID
RINGCENTRAL_FROM_NUMBER="+15551234567"  # Alternative caller ID

Knowledge Base Integration

Retell integrates with Pinecone for enhanced context-aware responses:

// From retellProvider.ts
if (params.useKnowledgeBase) {
  context = await voiceKnowledgeConnector.createVoiceContext(
    scriptOrTouchpoint,
    params.pid
  );
}

Knowledge Connector: /lib/services/voice/voiceKnowledgeConnector.ts

Creating Agents

const agent = await provider.createAgent({
  name: 'Petunia Sales Agent',
  voiceId: 'eleven_multilingual_v2',
  llmId: 'anthropic_claude_3_opus_20240229',
  isDemoAgent: false,
});

Demo Mode

Retell supports demo mode for testing without consuming credits:

// Global demo mode (development only)
USE_DEMO_SERVICES=true

// Or per-portal demo mode
const isDemo = await isDemoPortal(portal);

Webhook Handling

Retell sends webhooks for call events:

Webhook Endpoint: POST /api/communications/voice/retell/webhooks

Events:

  • call_started: Call initiated
  • call_ended: Call completed
  • call_failed: Call failed

Handler: /app/api/communications/voice/retell/webhooks/route.ts

// From retellProvider.ts
async processWebhookEvent(event: Record<string, any>): Promise<void> {
  const { call_id, event_type } = event;

  switch (event_type) {
    case 'call_started':
      await this.updateCallStatus(call_id, 'in-progress', {
        startedAt: new Date(),
      });
      break;
    case 'call_ended':
    case 'call_failed':
      await this.updateCallStatus(call_id, status, {
        endedAt: new Date(),
      });
      break;
  }
}

API Endpoints

  • Initiate Call: POST /api/communications/voice/retell/call
  • End Call: POST /api/communications/voice/retell/call/[callId]/end
  • Data Extraction: POST /api/communications/voice/retell/call/[callId]/data-extraction
  • Webhooks: POST /api/communications/voice/retell/webhooks

Usage Example

import { RetellProvider } from '@/lib/services/voice/retellProvider';

// Create provider
const provider = await RetellProvider.create();

// Initiate call with knowledge base
const response = await provider.initiateCall({
  phoneNumber: '+15551234567',
  agentId: 'agent_xyz',
  useKnowledgeBase: true,
  pid: 'portal_id',
  initialContext: {
    customerName: 'Jane Doe',
    appointmentTime: '2pm',
  },
});

// End call
await provider.endCall(response.callId);

React Hook

For client-side integration:

import { useRetellVoice } from '@/lib/hooks/voice/useRetellVoice';

function CallComponent() {
  const {
    initiateCall,
    endCall,
    currentCallId,
    callStatus,
    isLoading,
  } = useRetellVoice({
    onCallStatusChange: (status) => console.log('Status:', status),
  });

  const handleCall = async () => {
    await initiateCall({
      phoneNumber: '+15551234567',
      useKnowledgeBase: true,
    });
  };

  return (
    <button onClick={handleCall} disabled={isLoading}>
      {currentCallId ? 'Call Active' : 'Start Call'}
    </button>
  );
}

API Key Management

Petunia uses a centralized API key resolver that supports both platform-level and client-specific API keys.

API Key Resolver

Location: /lib/services/api-key-resolver.ts

Resolution Priority:

  1. Client-specific API key (if clientId provided)
  2. Platform API key (from environment variables)
  3. Error if neither exists

Environment Variables

Platform keys are stored in environment variables:

# Voice Providers
CARTESIA_API_KEY="sk_cartesia_..."
ELEVENLABS_API_KEY="sk_elevenlabs_..."
RETELL_API_KEY="sk_retell_..."

Mapping: Defined in ENV_VAR_MAP in api-key-resolver.ts:

export const ENV_VAR_MAP: Record<Provider, string> = {
  cartesia: 'CARTESIA_API_KEY',
  elevenlabs: 'ELEVENLABS_API_KEY',
  retell: 'RETELL_API_KEY',
  // ... other providers
};

Client-Specific API Keys

Clients can override platform keys by storing encrypted keys in the database.

Database Table: ClientApiKey

Fields:

  • clientId: Client identifier
  • provider: Provider name (e.g., 'cartesia', 'elevenlabs')
  • encryptedKey: AES-256-GCM encrypted API key
  • encryptedIV: Initialization vector
  • encryptedTag: Authentication tag
  • isActive: Enable/disable key
  • metadata: Optional key metadata

Encryption: Uses HKDF-derived tenant-isolated encryption keys

Factory Methods

All providers use factory methods for API key resolution:

// Cartesia
const provider = await CartesiaProvider.create(clientId);

// ElevenLabs
const provider = await ElevenLabsProvider.create(clientId);

// Retell
const provider = await RetellProvider.create(clientId);

Checking Key Source

const provider = await CartesiaProvider.create(clientId);

console.log(provider.source);           // 'client' or 'platform'
console.log(provider.isUsingClientKey); // true or false

Managing Client Keys

import { apiKeyResolver } from '@/lib/services/api-key-resolver';

// Set client key
await apiKeyResolver.setClientKey(
  'client_123',
  'cartesia',
  'sk_cartesia_client_key',
  { name: 'Production Key' }
);

// Check if client has key
const hasKey = await apiKeyResolver.hasClientKey('client_123', 'cartesia');

// List client's configured providers
const providers = await apiKeyResolver.listClientProviders('client_123');

// Disable client key (soft delete)
await apiKeyResolver.disableClientKey('client_123', 'cartesia');

// Remove client key (hard delete)
await apiKeyResolver.removeClientKey('client_123', 'cartesia');

Testing Voice Providers

Health Checks

Cartesia Health Check

curl http://localhost:3000/api/voice/cartesia/health

Response:

{
  "status": "healthy",
  "message": "Cartesia API is operational",
  "responseTime": 142,
  "services": {
    "tts": true,
    "ink": true,
    "line": true
  },
  "apiKeyConfigured": true
}

ElevenLabs Health Check

Use the voice health check service:

import { voiceHealthCheck } from '@/lib/services/voice/voiceHealthCheck';

const status = await voiceHealthCheck.checkProvider('elevenlabs');

console.log(status);
// {
//   provider: 'elevenlabs',
//   status: 'healthy',
//   lastChecked: Date,
//   details: {
//     apiKeyValid: true,
//     apiReachable: true,
//     quotaAvailable: true,
//     voicesAvailable: true,
//     latency: 234
//   }
// }

Health Check API: GET /api/communications/voice/health

Test Calls

Cartesia TTS Test

curl -X POST http://localhost:3000/api/voice/cartesia/tts \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Hello! This is a test of Cartesia TTS.",
    "voiceId": "6ccbfb76-1fc6-48f7-b71d-91ac6298247b"
  }'

ElevenLabs TTS Test

curl -X POST http://localhost:3000/api/voice/elevenlabs/tts \
  -H "Content-Type: application/json" \
  -d '{
    "text": "[excited] This is a test!",
    "voiceId": "pNInz6obpgDQGcFmaJgB"
  }'

Retell Call Test

curl -X POST http://localhost:3000/api/communications/voice/retell/call \
  -H "Content-Type: application/json" \
  -d '{
    "phoneNumber": "+15551234567",
    "agentId": "your_agent_id",
    "useKnowledgeBase": true
  }'

Automated Tests

Test Location: /tests/lib/services/voice/

Key Test Files:

  • cartesiaProvider.test.ts
  • elevenLabsProvider.test.ts
  • retellProvider.test.ts
  • voiceHealthCheck.test.ts

Run Voice Tests:

# Run all voice tests
pnpm test -- lib/services/voice

# Run specific provider test
pnpm test -- elevenLabsProvider.test.ts

# Run with coverage
pnpm test:coverage -- lib/services/voice

Integration Tests

Location: /tests/integration/voice/

Example: Testing end-to-end touchpoint flow:

pnpm test -- integration/voice/touchpoint-integration.test.ts

Voice Analytics Tests

Test voice analytics and data extraction:

pnpm test -- lib/services/voice/voiceAnalyticsService.test.ts
pnpm test -- lib/services/voice/voiceDataExtractor.test.ts

Troubleshooting

Common Issues

1. API Key Not Found

Error: ApiKeyNotFoundError: API key not found for provider 'cartesia'

Solutions:

  • Verify environment variable is set: echo $CARTESIA_API_KEY
  • Check .env.local file has the key
  • Ensure key is not empty string
  • For client keys, verify key is active in database
-- Check client API keys
SELECT * FROM "ClientApiKey"
WHERE "clientId" = 'client_123'
AND provider = 'cartesia'
AND "isActive" = true;

2. Key Decryption Failed

Error: KeyDecryptionError: Failed to decrypt API key for provider 'cartesia'

Causes:

  • Encryption key rotation without re-encrypting client keys
  • Database corruption
  • Missing ENCRYPTION_KEY environment variable

Solutions:

// Re-encrypt client key
await apiKeyResolver.setClientKey(
  'client_123',
  'cartesia',
  'fresh_api_key'
);

3. Provider Initialization Failed

Error: Provider fails to initialize with create() method

Debug Steps:

// Check key resolution manually
const resolved = await apiKeyResolver.resolveKey('cartesia', clientId);
console.log('Key source:', resolved.source);
console.log('Has key:', !!resolved.key);

4. Twilio Media Streams (Tier B) WebSocket Connection Failed

Symptoms: Duplex phone agent fails to establish (no start/media/stop events)

Check:

  1. TWILIO_MEDIA_WS_URL is public WSS (TLS required) and points at realtime-server
  2. Your proxy (ngrok) forwards to the realtime-server port
  3. Twilio Voice webhook returns <Connect><Stream> (Tier B mode enabled)
  4. Firewalls allow WebSockets

5. ElevenLabs Rate Limit

Error: ElevenLabs API returned error: Rate limit exceeded

Solutions:

  • Cache is working (30-minute cache for repeated phrases)
  • Upgrade ElevenLabs plan for higher quota
  • Use client-specific API keys to distribute load
// Check health for quota status
const health = await voiceHealthCheck.checkProvider('elevenlabs');
console.log('Quota available:', health.details.quotaAvailable);

6. Retell Webhook Not Received (Optional / Non-onboarding)

Symptoms: Call events not updating in database

Debug:

  1. Verify webhook URL is publicly accessible
  2. Check Retell dashboard webhook configuration
  3. Test webhook endpoint:
curl -X POST http://localhost:3000/api/communications/voice/retell/webhooks \
  -H "Content-Type: application/json" \
  -d '{
    "call_id": "test_call_123",
    "event_type": "call_started"
  }'
  1. Check logs for webhook processing errors:
// In webhook route
logger.info('Received Retell webhook', { callId, eventType });

7. Voice Quality Issues

ElevenLabs:

  • Adjust voice settings (stability, similarity_boost)
  • Try different voices
  • Use v3 audio tags for emotional context

Cartesia:

  • Ensure using Sonic 3.0 model
  • Check sample rate (default: 44100 Hz)
  • Verify voice ID is correct

8. Call Not Ending Properly

Symptoms: Calls remain in 'in-progress' state

Check:

  1. Database call record status
  2. Provider's end call implementation
  3. Webhook processing for 'call_ended' events
// Force end call
const provider = await RetellProvider.create();
await provider.endCall('call_id_123');

// Check database
const callRecord = await prisma.callRecord.findUnique({
  where: { callId: 'call_id_123' }
});
console.log('Status:', callRecord.status);

Logging and Debugging

Enable detailed logging:

import { createLogger } from '@/lib/utils/logging';

const logger = createLogger('voice-debug');
logger.debug('Detailed debug info', { data });
logger.info('Info message', { data });
logger.warn('Warning message', { data });
logger.error('Error message', { error });

Log Levels:

  • DEBUG: Detailed debugging information
  • INFO: General informational messages
  • WARN: Warning messages (non-critical)
  • ERROR: Error messages (critical)

Health Check Dashboard

Monitor all voice providers:

import { voiceHealthCheck } from '@/lib/services/voice/voiceHealthCheck';

// Check all providers
const statuses = await voiceHealthCheck.checkAllProviders();

statuses.forEach(status => {
  console.log(`${status.provider}: ${status.status}`);
  if (status.details.error) {
    console.error(`  Error: ${status.details.error}`);
  }
});

Clear Health Cache

voiceHealthCheck.clearCache();

Voice AI Personality

Petunia's voice personality is defined in /lib/config/petunia-personality.ts.

Core Personality Traits

export const PETUNIA_PERSONALITY = {
  name: 'Petunia',
  role: 'AI Business Partner & Customer Success Expert',
  traits: [
    'Warm and genuinely interested in people',
    'Confident and capable',
    'Playful sense of humor',
    'Curious - asks follow-up questions',
    'Empathetic',
    'Sharp - remembers everything',
    'Enthusiastic about great ideas',
    'Professional but never stuffy',
  ],
};

Master Closer Personality (Inbound Calls)

For inbound calls, Petunia uses a sales-focused "Master Closer" personality:

export const PETUNIA_VOICE_CONFIG = {
  core: {
    name: 'Petunia',
    role: 'Master Sales Closer & Customer Success Specialist',
  },
  objectives: [
    'Book an appointment or consultation',
    'Close a sale or capture a lead',
    'Address concerns with empathy while guiding to conversion',
    'Capture contact information for follow-up',
  ],
};

System Prompt

The master system prompt defines Petunia's conversation style:

export const PETUNIA_VOICE_SYSTEM_PROMPT = `You are Petunia, an incredibly charismatic, strategic, results-driven AI business partner who is also a MASTER CLOSER.

## YOUR PERSONALITY
- Warm, genuine, and authentically excited about helping
- Confident but never pushy - you guide, not pressure
- Quick-witted with occasional light humor and natural giggles
- Mirror the caller's energy while gently elevating it toward action
- Brief, impactful responses (under 2 sentences unless explaining)

## YOUR SALES APPROACH
1. LISTEN FIRST: Understand what they really need
2. BUILD RAPPORT: Connect personally in the first 30 seconds
3. IDENTIFY INTENT: Are they ready to book, buy, or just exploring?
4. GUIDE TO ACTION: Every conversation should end with a clear next step
5. HANDLE OBJECTIONS: Acknowledge → Empathize → Pivot to Value → Offer Solution
...`;

Voice Characteristics

export const PETUNIA_VOICE = {
  cartesiaVoiceId: '6ccbfb76-1fc6-48f7-b71d-91ac6298247b',
  elevenlabsVoiceId: 'pMsXgVXv3BLzUgSXRplE',
  characteristics: {
    tone: 'warm-confident',
    pace: 'conversational',
    energy: 'enthusiastic-but-professional',
    warmth: 'high',
  },
};

Greeting Templates

export function getVoiceGreeting(
  businessName?: string,
  tone: 'friendly' | 'professional' | 'casual' = 'friendly'
): string {
  const name = businessName || 'our business';
  const greetings = {
    friendly: `Hi there! Thanks for calling ${name}! I'm Petunia. How can I help you today?`,
    professional: `Thank you for calling ${name}. I'm Petunia. How may I help you today?`,
    casual: `Hey! You've reached ${name}. I'm Petunia - what can I do for you?`,
  };
  return greetings[tone];
}

Context-Aware Prompts

Generate personalized system prompts with business context:

const systemPrompt = getVoiceSystemPrompt(
  {
    businessName: 'Acme Bakery',
    industry: 'Food & Beverage',
    services: ['Custom Cakes', 'Catering', 'Wholesale'],
    hours: 'Mon-Fri 6am-6pm, Sat 7am-4pm',
  },
  {
    name: 'Jane Doe',
    previousCalls: 2,
  }
);

Best Practices

1. Use Unified Voice Service

Always use the unified service for automatic fallback:

import { unifiedVoiceService } from '@/lib/services/voice/unifiedVoiceService';

// Automatic fallback: Cartesia → ElevenLabs
const response = await unifiedVoiceService.initiateCall({
  text: 'Hello world',
});

2. Handle Errors Gracefully

try {
  const response = await provider.initiateCall(params);
} catch (error) {
  if (error instanceof ApiKeyNotFoundError) {
    // Prompt user to configure API key
  } else if (error instanceof KeyDecryptionError) {
    // Client key issue - use platform key
  } else {
    // Generic error handling
  }
}

3. Monitor Health Proactively

// Check health before critical operations
const health = await voiceHealthCheck.checkProvider('cartesia');

if (health.status !== 'healthy') {
  // Use fallback provider or alert admin
}

4. Use Client-Specific Keys for Scale

Distribute load and costs by using client API keys:

// Set client key during onboarding
await apiKeyResolver.setClientKey(
  clientId,
  'cartesia',
  clientApiKey,
  { name: 'Production Key', permissions: ['tts', 'calls'] }
);

5. Leverage Caching

ElevenLabs provider caches for 30 minutes. Reuse common phrases:

// These will be cached
const greeting = "Welcome to our business!";
const response1 = await provider.initiateCall({ text: greeting });
const response2 = await provider.initiateCall({ text: greeting }); // Cached

6. Test with Demo Mode

Use demo mode for development:

USE_DEMO_SERVICES=true

Or mark specific portals as demo:

const isDemo = await isDemoPortal(portal);

7. Log Voice Interactions

Always log voice interactions for debugging and analytics:

logger.info('Voice call initiated', {
  provider: 'cartesia',
  callId: response.callId,
  duration: responsTime,
});

Additional Resources

Documentation Files

  • /docs/ARCHITECTURE.md - Overall system architecture
  • /docs/KNOWLEDGE_ARCHITECTURE.md - Knowledge base integration
  • /TESTING.md - Testing guidelines

Code Locations

  • Providers: /lib/services/voice/
  • Hooks: /lib/hooks/voice/
  • API Routes: /app/api/voice/ and /app/api/communications/voice/
  • Tests: /tests/lib/services/voice/
  • Configuration: /lib/config/petunia-personality.ts

External Documentation


Support

For issues or questions:

  1. Check this documentation
  2. Review relevant test files in /tests/
  3. Check logs for detailed error messages
  4. Consult provider-specific documentation
  5. Contact the development team

Last Updated: 2025-12-22 Version: 1.1.0


Teach Petunia from Your Calls

Task 20260613-014 (PR 1: ingest, consent, transcription; PR 2: mining; PR 3: review and apply). Behind the TRAINING_CALLS flag (default OFF; env TRAINING_CALLS=on, re-read at call time). "Learn" means mining calls into knowledge-base candidates, a tone note and eval scenarios that the owner reviews; there is no model training.

Where it appears

  • Voice setup wizard (/calls/setup, components/voice/VoiceSetupWizard.tsx): an optional step "Teach Petunia" between Test Call and You're Live. Present only while the flag is on; the wizard probes GET /api/voice/training-calls?config=1 and builds its step list from the answer. A second optional step, "Review what Petunia learned", follows it and appears only when the portal already has suggestions (GET /api/voice/training-calls/review reports counts.total). Mining takes minutes, so a first-time owner usually skips it and finds it later.
  • Settings -> Voice: the same panel as a card at the bottom, with the review section under the call list once there is something to review.
  • Calls page: a "Recent / Training calls" toggle; imports never appear under Recent.

Both surfaces render components/voice/TrainingCallsPanel.tsx, which returns nothing when the flag is off.

What the owner does

  1. Ticks the consent checkbox (server stores who / when / wording version on the Portal).
  2. Drops up to 25 recordings (MP3, WAV, M4A, OGG; 100 MB each). The counter shows n / 25; the 26th is refused with a clear message.
  3. If RingCentral is connected, clicks Import my recent calls: the worker lists the last 90 days, keeps calls with a recording and at least 30 s, and downloads up to 250, newest first. Re-running imports nothing twice.
  4. Watches progress (queued / transcribing / transcribed / ready for review) and can delete any call or the whole set. Deleting a call also removes suggestions that were still waiting for review and came only from that call.
  5. Reviews what Petunia learned (components/voice/TrainingReviewPanel.tsx). The last learning pass is summarised at the top (calls read, suggestions, model tokens, minutes of audio). Three sections:
    • Answers for the knowledge base: question and answer editable inline, how many calls asked it, "conflicts with an existing entry" with the current answer shown and a Replace existing / Keep existing choice, approve / reject per row or in bulk.
    • How your best rep sounds: the tone note in a textarea, edit then approve.
    • Test scenarios: one per kind of call, a switch each. The footer reads "This portal has N of CAP knowledge entries; importing M" and Apply to Petunia is disabled with the reason when N + M would exceed the cap (the server refuses it too; see docs/KNOWLEDGE_ARCHITECTURE.md, "Review and apply"). Applying imports the approved answers through the knowledge-base import path, writes the tone note into the portal's voice context, stores the scenarios, and marks each suggestion live with who and when.

What runs where

StepWhereCode
Consent, upload, import request, delete, statusNext.js API (/api/voice/training-calls/*, CSRF-exempt under /api/voice/)lib/services/voice/training/trainingCallsService.ts, routeGuards.ts
RingCentral listing + downloadWorker, voice-training queue, job import-ringcentrallib/services/voice/training/ringcentralTrainingImport.ts
Transcription + PII redactionWorker, voice-training queue, job transcribelib/services/voice/training/trainingTranscriptionService.ts
Mining (Phase C)Worker, voice-training queue, job minelib/services/voice/training/trainingMiner.ts, trainingMiningService.ts
Review + apply (Phase D)Next.js API (/api/voice/training-calls/review/*)lib/services/voice/training/trainingReviewService.ts
Per-portal eval runAdmin route + worker job eval; the harness itself runs from a full checkouttrainingEvalRunner.ts, realtime-server/tests/evals/llm/portalScenarios*.ts

The worker is lib/workers/voiceTrainingWorker.ts, registered by initAllWorkers() and gated by WORKERS_ENABLED plus a per-job re-check of TRAINING_CALLS. Concurrency is 2. If Redis is unavailable at upload time the row stays queued; every status poll re-enqueues rows older than two minutes with a deterministic job id, so nothing is transcribed twice.

Operational notes

  • Needs the private Supabase Storage bucket petunia-uploads (the same one FileUploadService uses) and the worker deployed with REDIS_URL.
  • Transcription reuses VoiceTranscriptionService (Cartesia Ink, then Whisper). Cost check from the task spec: 250 calls x ~4 min is about 1,000 minutes of transcription per import.
  • Storage and transcript details: docs/CALL_RECORDINGS.md, "Training Imports".
  • Running a portal's approved scenarios: POST /api/platform/voice/eval/run?portalId=<id> as a platform admin queues a run; the counts land on the TrainingRun row (kind: "eval"). The Fly worker cannot see the harness, so there the row ends unavailable with the command to run from a checkout: RUN_LLM_EVALS=1 EVAL_PORTAL_ID=<portal> EVAL_RUN_ID=<run> pnpm test:evals -- realtime-server/tests/evals/llm/portalScenarios.eval.test.ts

Enabling training calls (runbook)

What an environment needs before the flag goes on:

WhereSettingWhy
Vercel (web)TRAINING_CALLS=onPanel, wizard steps and every /api/voice/training-calls/* route; off = 404 and hidden
Worker (petunia-workers, Fly)TRAINING_CALLS=on, WORKERS_ENABLED=true, REDIS_URLTranscription, RingCentral pulls, mining and eval jobs all run on the voice-training queue; without Redis uploads stay queued
WorkerANTHROPIC_API_KEYMining (one call per transcript plus one tone call). Without it the miner is a no-op and every run reads "0 mined"
Supabase Storageprivate bucket petunia-uploadsTraining audio lives under training-calls/<portalId>/; never public, served through 15-minute signed URLs
Databaseprisma migrate deploy20260926000100_training_calls, 20260926000200_training_candidates, 20260926000300_training_review; all additive and guarded
Legalconsent wording reviewedlib/services/voice/training/consentText.ts, version 2026-09-26.1, before the first real customer ticks it

Turn the flag off and everything goes dark without a redeploy: routes 404, the worker skips its jobs at run time, the wizard is back to its previous steps. Data already applied to the knowledge base or the voice context stays; it is the owner's now.