Documentation

OPERATIONS_RUNBOOK

docs/OPERATIONS_RUNBOOK.md

Petunia Operations Runbook

Version: 1.0.0 Last Updated: 2025-12-22 Classification: Operations / On-Call Reference

This runbook provides operations engineers with everything needed to diagnose, triage, and resolve production incidents. Keep this document open during on-call shifts.


Table of Contents

  1. Emergency Contacts & Escalation
  2. Health Check Reference
  3. Kill Switches & Feature Flags
  4. Third-Party Status Pages
  5. Common Incident Runbooks
  6. Cron Job Monitoring
  7. Queue System Operations
  8. Circuit Breaker Operations
  9. Graceful Degradation
  10. Rate Limit Bypass (Emergency Only)
  11. Rollback Procedures
  12. Communication Templates
  13. Recovery Procedures

Emergency Contacts & Escalation

Escalation Matrix

SeverityResponse TimeWho to ContactAction
SEV-1 (Production Down)< 15 minOn-call → Engineering Lead → CTOPage immediately, all hands
SEV-2 (Major Feature Broken)< 1 hourOn-call → Engineering LeadPage on-call, notify team
SEV-3 (Degraded Performance)< 4 hoursOn-callSlack notification
SEV-4 (Minor Issue)Next business dayCreate ticketDocument only

Severity Definitions

SEV-1 - Critical (Production Down)

  • Complete service outage
  • Authentication system failure
  • Data corruption or loss
  • Payment processing failure affecting all users
  • Voice system completely unavailable

SEV-2 - Major (Feature Broken)

  • Single major feature unavailable (voice, inbox, dashboard)
  • Significant performance degradation (>5x latency)
  • Partial payment failures (>10% transactions)
  • Elevated error rates (>5% of requests)

SEV-3 - Degraded

  • Non-critical feature issues
  • Minor performance degradation
  • Isolated user reports
  • Third-party service degradation (non-critical)

SEV-4 - Minor

  • Cosmetic issues
  • Feature requests misrouted as bugs
  • Single-user edge cases
  • Documentation errors

Contact Channels

ChannelPurposeResponse Time
PagerDutySEV-1/SEV-2 incidentsImmediate
#alerts-critical (Slack)Automated alerts< 5 min
#engineering (Slack)Team coordinationBusiness hours
Email: ops@petunia.aiNon-urgent issues24 hours

Health Check Reference

Health Endpoints

EndpointPurposeExpected Response
GET /api/healthBasic liveness200 OK + { status: "healthy" }
GET /api/health/statusComprehensive status200 OK + detailed component status
GET /api/readyKubernetes readiness200 OK when ready to serve traffic

Health Check Components

The /api/health/status endpoint returns status for each component:

{
  "status": "healthy | degraded | unhealthy",
  "timestamp": "ISO-8601",
  "components": {
    "database": { "status": "up", "latency_ms": 5 },
    "redis": { "status": "up", "latency_ms": 2 },
    "voice_primary": { "status": "up", "provider": "elevenlabs" },
    "voice_fallback": { "status": "up", "provider": "twilio" },
    "email": { "status": "up" },
    "webhooks": { "status": "up" },
    "websocket": { "status": "up" }
  }
}

Interpreting Health Status

StatusMeaningAction Required
healthyAll systems operationalNone
degradedSome components impairedMonitor, prepare for escalation
unhealthyCritical component failureImmediate investigation

Component-Specific Health Checks

Database Health (lib/services/core/database-health.ts)

# Manual check via psql
PGPASSWORD="$DB_PASSWORD" psql -h $DB_HOST -U postgres -d postgres -c "SELECT 1"

# Check connection pool
curl -s http://localhost:3000/api/health/status | jq '.components.database'

Redis Health

# Manual check
redis-cli -h $REDIS_HOST ping

# Check from application
curl -s http://localhost:3000/api/health/status | jq '.components.redis'

Voice Provider Health (lib/services/voice/voiceHealthCheck.ts)

# Check all voice providers
curl -s http://localhost:3000/api/health/status | jq '.components | with_entries(select(.key | startswith("voice")))'

Kill Switches & Feature Flags

Emergency Kill Switches

These environment variables can disable problematic features without deployment:

VariableEffectRecovery
VOICE_ENABLED=falseDisable all voice featuresSet to true
AI_ENABLED=falseDisable AI response generationSet to true
AUTORESPONDER_ENABLED=falseDisable autorespondersSet to true
WEBHOOKS_ENABLED=falseDisable outbound webhooksSet to true
BILLING_ENABLED=falseDisable billing operationsSet to true

Feature Flags (Runtime)

Feature flags are stored in the database and can be toggled via admin API:

# Disable a feature
curl -X POST /api/admin/features/voice-transcription \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"enabled": false}'

# Check feature status
curl /api/admin/features | jq '.features'

Circuit Breaker Manual Controls

Voice circuit breakers can be manually controlled:

// Force open (block all requests to provider)
voiceCircuitBreakerRegistry.getBreaker('elevenlabs').forceOpen('Manual intervention: API degraded');

// Reset to closed (allow requests)
voiceCircuitBreakerRegistry.getBreaker('elevenlabs').reset();

// Check current state
voiceCircuitBreakerRegistry.getAllMetrics();

Third-Party Status Pages

Bookmark these for incident correlation:

ServiceStatus PageCritical For
ElevenLabshttps://status.elevenlabs.ioVoice synthesis
OpenAIhttps://status.openai.comAI responses
Anthropichttps://status.anthropic.comAI responses
Twiliohttps://status.twilio.comVoice/SMS fallback
Stripehttps://status.stripe.comPayments
Supabasehttps://status.supabase.comDatabase/Auth
Vercelhttps://www.vercel-status.comHosting
Sentryhttps://status.sentry.ioError tracking
Redis (Upstash)https://status.upstash.comCaching/Queues

Checking Third-Party Health Programmatically

# Quick status check script
for service in "status.elevenlabs.io" "status.openai.com" "status.stripe.com"; do
  echo -n "$service: "
  curl -s -o /dev/null -w "%{http_code}" "https://$service" && echo " OK" || echo " FAIL"
done

Common Incident Runbooks

1. Database Connection Failures

Symptoms:

  • Error: ECONNREFUSED, CONNECTION_TIMEOUT, POOL_EXHAUSTED
  • Health check shows database: down
  • Elevated 500 errors across all endpoints

Investigation:

# Check database connectivity
PGPASSWORD="$DB_PASSWORD" psql -h $DB_HOST -U postgres -c "SELECT 1"

# Check connection pool status
curl -s localhost:3000/api/health/status | jq '.components.database'

# Check for connection leaks
PGPASSWORD="$DB_PASSWORD" psql -h $DB_HOST -U postgres -c \
  "SELECT count(*), state FROM pg_stat_activity WHERE datname='postgres' GROUP BY state"

Resolution:

  1. If Supabase is down → Check https://status.supabase.com
  2. If connection pool exhausted → Restart application pods
  3. If credentials invalid → Verify DATABASE_URL in Vercel dashboard
  4. If network issue → Check Vercel/Supabase region connectivity

Recovery Verification:

curl -s localhost:3000/api/health | jq '.status'
# Expected: "healthy"

2. Voice Provider Failures

Symptoms:

  • Error: VOICE_SYNTHESIS_FAILED, VOICE_PROVIDER_UNAVAILABLE
  • Circuit breaker in OPEN state
  • Calls dropping or silent audio

Investigation:

# Check circuit breaker states
curl -s localhost:3000/api/health/status | jq '.components | with_entries(select(.key | startswith("voice")))'

# Check ElevenLabs status
curl -s https://api.elevenlabs.io/v1/user -H "xi-api-key: $ELEVENLABS_API_KEY"

# Check Twilio fallback
curl -s "https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID.json" \
  -u "$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN"

Resolution:

  1. If ElevenLabs down:

    • Failover should be automatic to Twilio
    • Verify: TWILIO_VOICE_ENABLED=true
    • Manual override: Set VOICE_PRIMARY_PROVIDER=twilio
  2. If both providers down:

    • Enable voice disable kill switch: VOICE_ENABLED=false
    • Notify users via status page
    • Monitor third-party status pages
  3. If circuit breaker stuck open:

    // Reset circuit breaker
    voiceCircuitBreakerRegistry.getBreaker('elevenlabs').reset();
    

Recovery Verification:

# Test voice synthesis
curl -X POST localhost:3000/api/voice/test-synthesis \
  -H "Authorization: Bearer $TEST_TOKEN" \
  -d '{"text": "Hello world"}'

3. Payment Processing Failures

Symptoms:

  • Error: BILLING_CHARGE_FAILED, WEBHOOK_SIGNATURE_INVALID
  • Stripe webhooks failing
  • Users unable to subscribe/pay

Investigation:

# Check Stripe API connectivity
curl -s https://api.stripe.com/v1/balance \
  -u "$STRIPE_SECRET_KEY:"

# Check webhook endpoint status
curl -s https://api.stripe.com/v1/webhook_endpoints \
  -u "$STRIPE_SECRET_KEY:" | jq '.data[].status'

# Check failed webhook queue
curl -s localhost:3000/api/admin/webhooks/failed | jq '.pending'

Resolution:

  1. If Stripe API down:

    • Check https://status.stripe.com
    • Failed charges will retry automatically
    • Consider enabling BILLING_ENABLED=false for severe outages
  2. If webhook signature failures:

    • Verify STRIPE_WEBHOOK_SECRET matches Stripe dashboard
    • Check for clock drift between servers
  3. If webhook processing failures:

    • Failed webhooks are queued for retry
    • Check: GET /api/admin/webhooks/failed
    • Manual retry: POST /api/admin/webhooks/retry/{eventId}

Recovery Verification:

# Verify Stripe connectivity
curl -s localhost:3000/api/payments/health

# Check webhook queue is draining
curl -s localhost:3000/api/admin/webhooks/stats

4. Authentication Failures

Symptoms:

  • Error: AUTH_SESSION_EXPIRED, AUTH_TOKEN_INVALID
  • Users unable to log in
  • Session-related 401 errors spike

Investigation:

# Check Supabase Auth service
curl -s "$NEXT_PUBLIC_SUPABASE_URL/auth/v1/health"

# Check session endpoint
curl -s localhost:3000/api/auth/session \
  -H "Cookie: sb-access-token=..."

# Check for JWT secret mismatch
echo $SUPABASE_JWT_SECRET | cut -c1-10

Resolution:

  1. If Supabase Auth down:

  2. If JWT validation failing:

    • Verify SUPABASE_JWT_SECRET matches project settings
    • Check for environment variable propagation issues
  3. If session storage issues:

    • Clear Redis session cache if corrupted
    • Force re-authentication via cookie invalidation

Recovery Verification:

# Test authentication flow
curl -X POST localhost:3000/api/auth/signin \
  -d '{"email":"test@example.com","password":"test"}'

5. AI Response Failures

Symptoms:

  • Error: AI_GENERATION_FAILED, AI_RATE_LIMITED
  • Autoresponders not generating responses
  • Knowledge base queries timing out

Investigation:

# Check OpenAI status
curl -s https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY" | head -1

# Check Anthropic status
curl -s https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01"

# Check rate limit headers from last request
curl -I https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY"

Resolution:

  1. If rate limited:

    • AI requests use exponential backoff automatically
    • Check quota in provider dashboards
    • Consider enabling AI_ENABLED=false temporarily
  2. If provider down:

    • Failover to secondary provider (OpenAI ↔ Anthropic)
    • Set AI_PRIMARY_PROVIDER=anthropic (or openai)
  3. If responses slow:

    • Check for prompt size issues
    • Monitor token usage
    • Consider reducing AI_MAX_TOKENS

Recovery Verification:

# Test AI generation
curl -X POST localhost:3000/api/ai/test \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"prompt": "Say hello"}'

6. WebSocket Connection Issues

Symptoms:

  • Real-time updates not working
  • Inbox messages delayed
  • Error: WEBSOCKET_CONNECTION_FAILED

Investigation:

# Check WebSocket server status
curl -s localhost:3000/api/health/status | jq '.components.websocket'

# Check active connections
curl -s localhost:3000/api/admin/websocket/stats

# Test WebSocket connectivity
wscat -c "wss://app.petunia.ai/api/websocket?token=..."

Resolution:

  1. If WebSocket server crashed:

    • Check pod/container logs
    • Restart WebSocket service
    • Clients will auto-reconnect
  2. If connection limits reached:

    • Scale WebSocket pods horizontally
    • Check for connection leaks (zombie connections)
  3. If authentication issues:

    • Verify token validation logic
    • Check CORS configuration

Recovery Verification:

# Monitor reconnections
curl -s localhost:3000/api/admin/websocket/stats | jq '.reconnections_per_minute'

Cron Job Monitoring

Active Cron Jobs

JobSchedulePurposeFailure Impact
dunningEvery 4 hoursPayment retryRevenue loss
review-aggregationDaily 2 AMAggregate reviewsStale data
webhook-retryEvery 5 minRetry failed webhooksEvent loss
overage-settlementHourlySettle billing overagesBilling delays
data-retentionDaily 3 AMClean old dataStorage growth
lead-scoringEvery 15 minUpdate lead scoresStale scores
campaign-schedulerEvery minuteSchedule campaignsMissed sends
analytics-rollupHourlyAggregate metricsDashboard gaps
health-checkEvery minuteSystem monitoringBlind spots
session-cleanupEvery 6 hoursClean expired sessionsMemory growth
cache-warmupOn deployPre-warm cachesCold start latency
subscription-syncEvery 30 minSync Stripe dataBilling mismatch

Cron Job Health Monitoring

# Check cron job execution history
curl -s localhost:3000/api/admin/cron/history | jq '.[0:5]'

# Check for failed jobs
curl -s localhost:3000/api/admin/cron/failed

# Manually trigger a job
curl -X POST localhost:3000/api/cron/dunning \
  -H "Authorization: Bearer $CRON_SECRET"

Cron Failure Recovery

If a cron job is failing repeatedly:

  1. Check logs for error details
  2. Verify environment variables are set
  3. Check for database lock contention
  4. Verify external service connectivity
  5. Manual execution to test:
    curl -X POST localhost:3000/api/cron/{job-name} \
      -H "Authorization: Bearer $CRON_SECRET"
    

Critical Cron Dependencies:

JobRequiresRecovery if Missing
dunningStripe APIPayment retries delayed
webhook-retryDatabaseEvents may be lost
analytics-rollupDatabase, RedisManual re-aggregation
lead-scoringAI APIScores become stale

Queue System Operations

Queue Architecture

Petunia uses two queue systems:

  1. BullMQ (Redis-backed) - High-throughput, ephemeral jobs

    • Voice processing
    • Email sending
    • Real-time notifications
  2. Database-backed Queues - Durable, audit-required jobs

    • Webhook delivery
    • Billing operations
    • Critical notifications

Queue Monitoring

# BullMQ queue stats
curl -s localhost:3000/api/admin/queues/bullmq | jq '.'

# Database queue stats
curl -s localhost:3000/api/admin/queues/database | jq '.'

# Check for stuck jobs
curl -s localhost:3000/api/admin/queues/stuck

Queue Recovery Procedures

If BullMQ queue is backed up:

# Check queue depth
redis-cli -h $REDIS_HOST LLEN bull:voice-processing:wait

# If queue is stuck, clear stale jobs
curl -X POST localhost:3000/api/admin/queues/bullmq/voice-processing/clean \
  -H "Authorization: Bearer $ADMIN_TOKEN"

If database queue has failed jobs:

# View failed jobs
curl -s localhost:3000/api/admin/queues/database/failed

# Retry specific job
curl -X POST localhost:3000/api/admin/queues/database/retry/{jobId} \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Retry all failed jobs
curl -X POST localhost:3000/api/admin/queues/database/retry-all \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Queue Job States

StateMeaningAction
waitingQueued, awaiting processingNormal
activeCurrently being processedNormal
completedSuccessfully processedAuto-cleaned
failedProcessing failed, may retryCheck errors
delayedScheduled for futureNormal
stalledWorker died mid-processingAuto-retry

Circuit Breaker Operations

Understanding Circuit States

StateBehaviorTrigger
CLOSEDNormal operationSuccess threshold met
OPENAll requests rejectedFailure threshold exceeded
HALF_OPENLimited test requestsRecovery time elapsed

Circuit Breaker Configuration

// Default configuration (circuitBreaker.ts:53-59)
{
  failureThreshold: 5,      // Failures before opening
  successThreshold: 2,      // Successes to close from half-open
  recoveryTimeMs: 30000,    // Time before testing recovery
  failureWindowMs: 60000,   // Window for counting failures
  operationTimeoutMs: 30000 // Individual operation timeout
}

Monitoring Circuit Breakers

# Get all circuit breaker states
curl -s localhost:3000/api/admin/circuit-breakers | jq '.'

# Response format:
# {
#   "elevenlabs": { "state": "CLOSED", "failures": 0, "lastFailure": null },
#   "twilio": { "state": "HALF_OPEN", "failures": 3, "lastFailure": "2025-12-22T..." }
# }

Manual Circuit Breaker Control

Force open (emergency block):

curl -X POST localhost:3000/api/admin/circuit-breakers/elevenlabs/force-open \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"reason": "Provider experiencing degradation"}'

Reset to closed:

curl -X POST localhost:3000/api/admin/circuit-breakers/elevenlabs/reset \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Graceful Degradation

Degradation Hierarchy

When systems fail, Petunia degrades gracefully in this order:

  1. Voice Synthesis

    • Primary: ElevenLabs
    • Fallback: Twilio
    • Graceful: Pre-recorded fallback messages
  2. AI Responses

    • Primary: OpenAI GPT-4
    • Fallback: Anthropic Claude
    • Graceful: Template-based responses
  3. Real-time Updates

    • Primary: WebSocket
    • Fallback: Polling (30s interval)
    • Graceful: Manual refresh prompt
  4. Payments

    • Primary: Stripe direct
    • Fallback: Queued for retry
    • Graceful: Grace period extension

Enabling Degraded Modes

# Enable polling fallback for WebSocket issues
curl -X POST localhost:3000/api/admin/features/websocket-fallback \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"enabled": true}'

# Enable template responses for AI failures
curl -X POST localhost:3000/api/admin/features/ai-fallback-templates \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"enabled": true}'

Rate Limit Bypass (Emergency Only)

When to Use Rate Limit Bypass

ONLY use in genuine emergencies:

  • Critical business operation blocked by rate limits
  • Incident recovery requiring rapid API calls
  • Data migration or batch processing

DO NOT use for:

  • Regular operations
  • Avoiding rate limits during normal usage
  • Client requests (unless approved by engineering lead)

Bypass Procedures

1. Temporary User Bypass (5 minutes)

# Grant temporary bypass to specific user
curl -X POST localhost:3000/api/admin/rate-limit/bypass \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{
    "userId": "user-uuid",
    "durationMinutes": 5,
    "reason": "Incident recovery - INC-2025-12-22-001"
  }'

2. Temporary IP Bypass

# Whitelist IP for emergency operations
curl -X POST localhost:3000/api/admin/rate-limit/whitelist-ip \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{
    "ip": "192.168.1.100",
    "durationMinutes": 30,
    "reason": "Data migration batch job"
  }'

3. Global Rate Limit Increase (DANGER)

# Temporarily increase global limits (requires CTO approval)
curl -X POST localhost:3000/api/admin/rate-limit/global \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{
    "multiplier": 2.0,
    "durationMinutes": 60,
    "approvedBy": "cto@petunia.ai",
    "reason": "SEV-1 incident recovery"
  }'

Audit Requirements

All rate limit bypasses are logged:

  • User who granted bypass
  • Target user/IP
  • Duration and reason
  • Timestamp

Review bypasses weekly in security audit.


Rollback Procedures

Vercel Deployment Rollback

Symptoms requiring rollback:

  • New deployment causing errors
  • Feature regression identified
  • Performance degradation after deploy

Steps:

  1. Identify the last good deployment:

    # List recent deployments
    vercel list --prod
    
    # Or via Vercel dashboard → Deployments
    
  2. Rollback to previous deployment:

    # Instant rollback via CLI
    vercel rollback <deployment-url>
    
    # Example
    vercel rollback petunia-abc123.vercel.app
    
  3. Verify rollback:

    # Check health
    curl -s https://app.petunia.gardenpatch.xyz/api/health
    
    # Verify version
    curl -s https://app.petunia.gardenpatch.xyz/api/version
    

Database Migration Rollback

WARNING: Database rollbacks are complex. Only attempt with DBA guidance.

  1. Check migration status:

    npx prisma migrate status
    
  2. Create rollback script:

    -- Document the rollback SQL before applying any migration
    -- Store in: /prisma/migrations/{migration}/rollback.sql
    
  3. Apply rollback:

    # Execute rollback SQL
    psql $DATABASE_URL -f prisma/migrations/{migration}/rollback.sql
    
    # Mark migration as rolled back
    npx prisma migrate resolve --rolled-back {migration-name}
    

Feature Flag Rollback

For feature-specific issues:

# Disable problematic feature instantly
curl -X POST localhost:3000/api/admin/features/{feature-name} \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"enabled": false}'

Environment Variable Rollback

# Via Vercel CLI
vercel env rm PROBLEMATIC_VAR production
vercel env add SAFE_VAR production

# Force redeploy to pick up changes
vercel --prod

Communication Templates

Status Page Templates

Investigating (Initial Alert)

Title: Investigating [Component] Issues
Status: Investigating

We are currently investigating reports of [brief description].

Our team is actively looking into this issue. We will provide updates as we learn more.

Posted: [TIME] UTC

Identified (Root Cause Found)

Title: [Component] Issues - Root Cause Identified
Status: Identified

We have identified the root cause of [issue description].

Root Cause: [Brief technical explanation]
Impact: [What users experienced]
ETA for fix: [Time estimate]

We are actively working on a resolution.

Posted: [TIME] UTC

Monitoring (Fix Applied)

Title: [Component] Issues - Fix Applied
Status: Monitoring

We have applied a fix for [issue description].

We are monitoring the situation to ensure stability. Some users may need to refresh or re-login to see improvements.

Posted: [TIME] UTC

Resolved

Title: [Component] Issues - Resolved
Status: Resolved

The issue affecting [component] has been resolved.

Duration: [START TIME] to [END TIME] UTC ([X] minutes)
Impact: [Summary of impact]
Root Cause: [Brief explanation]

We apologize for any inconvenience. A detailed post-mortem will follow.

Posted: [TIME] UTC

Internal Communication Templates

Slack Alert (SEV-1)

🚨 **SEV-1 INCIDENT DECLARED** 🚨

**Issue:** [Brief description]
**Impact:** [User impact]
**Incident Commander:** @[name]
**War Room:** [Zoom/Slack channel link]

All hands requested. Please acknowledge if you're joining the response.

Email to Affected Customers

Subject: [Petunia] Service Disruption - [Date]

Dear [Customer Name],

We experienced a service disruption today that may have affected your account.

What happened:
[Brief, non-technical explanation]

Duration:
[Start time] to [End time] (approximately [X] minutes)

What we're doing:
[Brief explanation of fix and prevention]

We sincerely apologize for any inconvenience. If you experienced any issues during this time, please contact support@petunia.ai.

Best regards,
The Petunia Team

Recovery Procedures

Full System Recovery

After a major outage, follow this recovery checklist:

  1. Verify Infrastructure

    curl -s localhost:3000/api/health/status | jq '.'
    
  2. Check Database Integrity

    # Run integrity check
    curl -X POST localhost:3000/api/admin/database/integrity-check \
      -H "Authorization: Bearer $ADMIN_TOKEN"
    
  3. Process Backlogged Queues

    # Check queue depths
    curl -s localhost:3000/api/admin/queues/stats
    
    # If backlogged, scale workers
    # (Vercel function concurrency handles this automatically)
    
  4. Verify Third-Party Connectivity

    # Run connectivity tests
    curl -X POST localhost:3000/api/admin/connectivity-test \
      -H "Authorization: Bearer $ADMIN_TOKEN"
    
  5. Clear Stale Caches

    # Invalidate problematic caches
    curl -X POST localhost:3000/api/admin/cache/invalidate \
      -H "Authorization: Bearer $ADMIN_TOKEN" \
      -d '{"pattern": "user:*"}'
    
  6. Resume Cron Jobs

    # Trigger missed jobs
    for job in dunning webhook-retry analytics-rollup; do
      curl -X POST localhost:3000/api/cron/$job \
        -H "Authorization: Bearer $CRON_SECRET"
    done
    
  7. Monitor Error Rates

    • Check Sentry for new errors
    • Monitor /api/health/status every 5 minutes
    • Watch for circuit breaker state changes

Post-Incident Checklist

  • Incident documented in /docs/security/incident-log.md
  • Root cause identified
  • Timeline established
  • Customer impact assessed
  • Remediation actions identified
  • Follow-up tasks created
  • Post-mortem scheduled (for SEV-1/SEV-2)

Appendix: Quick Commands Reference

Health & Status

curl localhost:3000/api/health                    # Basic health
curl localhost:3000/api/health/status             # Detailed status
curl localhost:3000/api/ready                     # Readiness probe

Feature Control

curl localhost:3000/api/admin/features            # List all features
curl -X POST localhost:3000/api/admin/features/{name} -d '{"enabled": false}'

Queue Management

curl localhost:3000/api/admin/queues/stats        # Queue statistics
curl localhost:3000/api/admin/queues/failed       # Failed jobs
curl -X POST localhost:3000/api/admin/queues/retry-all

Circuit Breakers

curl localhost:3000/api/admin/circuit-breakers    # All states
curl -X POST localhost:3000/api/admin/circuit-breakers/{name}/reset
curl -X POST localhost:3000/api/admin/circuit-breakers/{name}/force-open

Cron Jobs

curl localhost:3000/api/admin/cron/history        # Execution history
curl -X POST localhost:3000/api/cron/{name}       # Manual trigger

Database

curl localhost:3000/api/admin/database/stats      # Connection stats
curl -X POST localhost:3000/api/admin/database/integrity-check

Rate Limits (Emergency)

curl -X POST localhost:3000/api/admin/rate-limit/bypass -d '{"userId":"...","durationMinutes":5}'
curl -X POST localhost:3000/api/admin/rate-limit/whitelist-ip -d '{"ip":"...","durationMinutes":30}'

Rollback

vercel list --prod                                # List deployments
vercel rollback <deployment-url>                  # Instant rollback

Webhooks

curl localhost:3000/api/admin/webhooks/failed     # Failed webhooks
curl -X POST localhost:3000/api/admin/webhooks/retry/{eventId}  # Retry one
curl localhost:3000/api/admin/webhooks/stats      # Webhook statistics

Document History

VersionDateAuthorChanges
1.0.02025-12-22Operations TeamInitial comprehensive runbook