Documentation

INTEGRATION_MONITORING

docs/INTEGRATION_MONITORING.md

Integration Connection Health Monitoring

This document provides comprehensive guidance on monitoring the health and status of platform integrations in Petunia. It covers health check patterns, sync status tracking, monitoring dashboards, failure modes, and troubleshooting procedures.

Owner: Platform Team Version: 1.0.0 Last Updated: 2025-12-13

Table of Contents

  1. Overview
  2. Connection Models
  3. Health Check System
  4. Sync Status Tracking
  5. Monitoring Dashboard
  6. API Endpoints
  7. Failure Modes and Recovery
  8. Alerting Recommendations
  9. Troubleshooting Guide
  10. Best Practices

Overview

Petunia integrates with multiple external platforms (Yelp, Google Business, Facebook, etc.) to provide unified communication and review management. The health monitoring system ensures these integrations remain operational and provides visibility into connection status, sync performance, and error tracking.

Key Components

  • Connection Models: Platform-level connection definitions
  • Connection Integrations: Client-specific integration instances
  • Health Checks: Automated health verification with historical tracking
  • Sync Jobs: Durable queues for data synchronization
  • Monitoring Dashboard: Admin UI for health visualization

Health Status States

StatusDescriptionTypical Cause
healthyConnection operating normallyAll checks passing
warningMinor issues detectedRate limit approaching, slow responses
errorConnection failingAPI authentication failure, service outage
unknownHealth status not yet determinedNew connection, check pending

Connection Models

Base Connection Model

The Connection model represents platform-level integration definitions:

model Connection {
  id                    String                  @id
  name                  String                  @unique
  description           String
  icon                  String?
  category              ConnectionCategory      // ai, messaging, social, email, voice, chat, review, business
  isEnabled             Boolean                 @default(false)
  isConfigured          Boolean                 @default(false)
  type                  ConnectionType          // platform, client
  managedBy             ConnectionOwner
  health                Json                    // Current health status
  config                Json?
  ConnectionHealthCheck ConnectionHealthCheck[]
  ConnectionIntegration ConnectionIntegration[]
}

Health JSON Structure:

{
  "status": "healthy",
  "uptime": 99.5,
  "responseTime": 120,
  "errorRate": 0.005,
  "lastChecked": "2025-12-13T10:00:00Z",
  "issues": [],
  "metrics": {
    "checkDuration": 150
  }
}

Connection Health Check Model

Historical health check records:

model ConnectionHealthCheck {
  id           String                 @id
  connectionId String
  status       ConnectionHealthStatus
  uptime       Float?
  responseTime Int?                   // milliseconds
  errorRate    Float?
  issues       Json?                  // Array of issue objects
  checkedAt    DateTime               @default(now())
  Connection   Connection             @relation(fields: [connectionId], references: [id])
}

Platform-Specific Connection Models

Each integration has its own connection model with platform-specific fields:

YelpConnection:

model YelpConnection {
  id                      String                @id
  businessId              String                @unique
  businessName            String
  accessToken             String?               // Encrypted
  refreshToken            String?               // Encrypted
  tokenExpiresAt          DateTime?
  syncEnabled             Boolean               @default(true)
  syncInterval            Int                   @default(15)  // minutes
  lastSyncAt              DateTime?
  webhookEnabled          Boolean               @default(false)
  webhookSecret           String?
  autoResponderEnabled    Boolean               @default(false)
  connectionIntegrationId String                @unique
}

GoogleConnection:

model GoogleConnection {
  id                      String                @id
  businessId              String                @unique
  businessName            String
  accessToken             String?               // Encrypted
  refreshToken            String?               // Encrypted
  tokenExpiresAt          DateTime?
  syncEnabled             Boolean               @default(true)
  syncInterval            Int                   @default(15)  // minutes
  lastSyncAt              DateTime?
  webhookEnabled          Boolean               @default(false)
  autoResponderEnabled    Boolean               @default(false)
  connectionIntegrationId String                @unique
}

FacebookConnection:

model FacebookConnection {
  id                      String                  @id
  pageId                  String                  @unique
  pageName                String
  accessToken             String?                 // Encrypted
  tokenExpiresAt          DateTime?
  syncEnabled             Boolean                 @default(true)
  syncInterval            Int                     @default(15)  // minutes
  lastSyncAt              DateTime?
  webhookEnabled          Boolean                 @default(false)
  webhookSecret           String?
  autoResponderEnabled    Boolean                 @default(false)
  connectionIntegrationId String                  @unique
}

Health Check System

ConnectionHealthService

The ConnectionHealthService (/lib/services/connectionHealthService.ts) provides centralized health monitoring:

Check Single Connection

import { ConnectionHealthService } from '@/lib/services/connectionHealthService';

const result = await ConnectionHealthService.checkConnectionHealth(
  connectionId,
  {
    timeout: 30000,        // Max wait time
    metrics: true,         // Collect performance metrics
    detailed: true,        // Include diagnostics
    saveResult: true       // Persist to database
  }
);

// Result structure:
{
  connectionId: string;
  status: 'healthy' | 'warning' | 'error' | 'unknown';
  uptime?: number;
  responseTime?: number;
  errorRate?: number;
  issues: ConnectionIssue[];
  checkedAt: Date;
  metrics?: Record<string, any>;
}

Check All Connections

const results = await ConnectionHealthService.checkAllConnections(
  {
    category: 'social',    // Optional filter
    isEnabled: true
  },
  {
    saveResult: true
  }
);

// Returns array of HealthCheckResult objects

Get Health History

const history = await ConnectionHealthService.getHealthHistory(
  connectionId,
  50  // limit
);

// Returns ConnectionHealthCheck[] ordered by checkedAt DESC

System Health Overview

const overview = await ConnectionHealthService.getSystemHealthOverview();

// Returns:
{
  totalConnections: number;
  healthyCount: number;
  warningCount: number;
  errorCount: number;
  unknownCount: number;
  healthyPercentage: number;
  averageResponseTime?: number;
  averageUptime?: number;
  issuesByCategory: Record<string, number>;
}

Health Check Patterns by Category

Messaging Services (Twilio, RingCentral)

  • Check: Message delivery success rate
  • Metrics: Queue size, delivery success rate
  • Issues: Delivery failures, delayed messages

Social Platforms (Yelp, Facebook, Google)

  • Check: API rate limits, token validity
  • Metrics: API quota remaining, reset time
  • Issues: Rate limit exceeded, API degraded performance

Email Services (SendGrid, Mailgun)

  • Check: Email delivery rate, bounce rate
  • Metrics: Delivery rate, bounce rate, open rate
  • Issues: High bounce rate, delivery failures

Voice Services (Retell, ElevenLabs, Cartesia)

  • Check: Call success rate, audio quality
  • Metrics: Call success, audio quality, transcription accuracy
  • Issues: Service outage, quality degradation

Review Platforms (Yelp, Google, Facebook)

  • Check: Review sync status, API authentication
  • Metrics: Sync delay, API quota, reviews per hour
  • Issues: API authentication failure, sync delays

Sync Status Tracking

Sync Job Models

Platform-specific sync jobs track data synchronization:

YelpReviewSyncJob:

model YelpReviewSyncJob {
  id               String   @id
  yelpConnectionId String
  status           String   // pending, running, completed, failed, dead_letter
  priority         Int      @default(0)
  attempts         Int      @default(0)
  scheduledAt      DateTime
  startedAt        DateTime?
  completedAt      DateTime?
  lastError        String?
  lastErrorAt      DateTime?
  payload          Json?
  result           Json?
  createdAt        DateTime @default(now())
  updatedAt        DateTime @updatedAt
}

GoogleReviewSyncJob:

model GoogleReviewSyncJob {
  id                 String   @id
  googleConnectionId String
  status             String   // pending, running, completed, failed, dead_letter
  priority           Int      @default(0)
  attempts           Int      @default(0)
  scheduledAt        DateTime
  startedAt          DateTime?
  completedAt        DateTime?
  lastError          String?
  lastErrorAt        DateTime?
  payload            Json?
  result             Json?
  createdAt          DateTime @default(now())
  updatedAt          DateTime @updatedAt
}

Sync Job Lifecycle

1. PENDING → Job created and scheduled
   ↓
2. RUNNING → Worker claimed job and executing
   ↓
3a. COMPLETED → Sync successful
3b. FAILED → Error occurred, retry scheduled
3c. DEAD_LETTER → Max retries exceeded

Sync Job Service Pattern

// Enqueue sync job
const job = await yelpReviewSyncJobService.enqueueJob(
  yelpConnectionId,
  {
    scheduledAt: new Date(),
    priority: 0,
    payload: { fullSync: false }
  }
);

// Claim next job (worker pattern)
const job = await yelpReviewSyncJobService.claimNextJob();

// Mark job as completed
await yelpReviewSyncJobService.markCompleted(
  job.id,
  {
    total: 25,
    added: 5,
    updated: 2
  }
);

// Mark job as failed
await yelpReviewSyncJobService.markFailed(
  job.id,
  error,
  3  // attempt number
);

Sync Configuration

Each connection tracks sync settings:

FieldTypeDescription
syncEnabledBooleanWhether automatic sync is enabled
syncIntervalIntMinutes between syncs (default: 15)
lastSyncAtDateTimeTimestamp of last successful sync

CallLogSyncState Pattern

For voice integrations (RingCentral):

model CallLogSyncState {
  id                      String                 @id
  connectionIntegrationId String
  phoneNumber             String                 // E.164 format
  status                  CallLogSyncStateStatus // active, paused, error
  cursor                  Json?                  // { lastStartTime, lastCallId, pageToken }
  lastSyncedAt            DateTime?
  lastErrorCode           String?
  lastErrorMessage        String?
  lastErrorAt             DateTime?
}

Monitoring Dashboard

ConnectionHealthMonitor Component

Location: /components/connections/admin/ConnectionHealthMonitor.tsx

Features:

  • Real-time health status visualization
  • Search and filter by status
  • Sort by multiple metrics
  • Refresh on demand
  • Detailed issue tracking

Usage:

import { ConnectionHealthMonitor } from '@/components/connections/admin/ConnectionHealthMonitor';

<ConnectionHealthMonitor connections={connections} />

Metrics Displayed:

  • Total connections count
  • Healthy percentage
  • Warning count
  • Error count
  • Per-connection uptime
  • Response time
  • Error rate
  • Last checked timestamp
  • Active issues

Admin Dashboard Integration

The health monitor integrates with the admin dashboard at:

  • /app/admin/connections - Connection overview
  • /app/admin/connections/[connectionId] - Detailed connection view

API Endpoints

Health Check Endpoints

GET /api/connections/[connectionId]/health

Retrieve health history for a specific connection.

Query Parameters:

  • limit - Number of records (default: 10, max: 100)

Response:

{
  "success": true,
  "data": {
    "currentHealth": {
      "status": "healthy",
      "uptime": 99.5,
      "responseTime": 120,
      "errorRate": 0.005,
      "lastChecked": "2025-12-13T10:00:00Z"
    },
    "healthHistory": [
      {
        "id": "health_xxx",
        "connectionId": "conn_xxx",
        "status": "healthy",
        "uptime": 99.5,
        "responseTime": 120,
        "errorRate": 0.005,
        "issues": [],
        "checkedAt": "2025-12-13T10:00:00Z"
      }
    ]
  }
}

POST /api/connections/[connectionId]/health

Trigger an on-demand health check.

Response:

{
  "success": true,
  "message": "Health check triggered successfully",
  "data": {
    "connectionId": "conn_xxx",
    "status": "healthy",
    "uptime": 99.5,
    "responseTime": 120,
    "errorRate": 0.005,
    "issues": [],
    "checkedAt": "2025-12-13T10:00:00Z"
  }
}

Sync Job Endpoints

GET /api/connections/yelp/jobs

Get Yelp sync job metrics.

Response:

{
  "success": true,
  "data": {
    "pending": 5,
    "running": 2,
    "completed": 150,
    "failed": 3,
    "dead_letter": 0,
    "recentJobs": [...]
  }
}

GET /api/connections/google/jobs

Get Google sync job metrics (similar structure).


Failure Modes and Recovery

Common Failure Modes

1. Token Expiration

Symptoms:

  • 401 Unauthorized responses
  • Health status: error
  • Issue: "API authentication failure"

Recovery:

// Automatic token refresh (OAuth connections)
if (connection.refreshToken && isTokenExpired(connection.tokenExpiresAt)) {
  await refreshAccessToken(connection);
}

// Manual reconnection required
// Notify client to reconnect via OAuth flow

Prevention:

  • Monitor tokenExpiresAt field
  • Trigger refresh 24 hours before expiry
  • Alert clients when manual reconnection needed

2. Rate Limiting

Symptoms:

  • 429 Too Many Requests
  • Health status: warning or error
  • Issue: "API rate limit exceeded"

Recovery:

// Exponential backoff with jitter
const backoffMs = Math.min(
  baseBackoffMs * Math.pow(2, attemptNumber),
  maxBackoffMs
);
await sleep(backoffMs + Math.random() * 1000);

Prevention:

  • Track API quota usage
  • Implement request throttling
  • Monitor metrics.apiRemainingQuota
  • Alert at 80% quota consumption

3. Webhook Failures

Symptoms:

  • Missing webhook events
  • Health status: warning
  • Issue: "Webhook delivery delayed"

Recovery:

// Fall back to polling
if (webhookEnabled && !recentWebhookActivity) {
  logger.warn('Webhook inactive, triggering poll sync');
  await triggerPollSync(connection);
}

Prevention:

  • Monitor webhook delivery timestamps
  • Implement webhook health checks
  • Set up webhook retry mechanisms
  • Alert on 3+ consecutive failures

4. Sync Job Deadlocks

Symptoms:

  • Jobs stuck in running state
  • Increasing pending backlog
  • No completedAt timestamp

Recovery:

// Timeout stuck jobs (>30 minutes)
const stuckJobs = await prisma.yelpReviewSyncJob.updateMany({
  where: {
    status: 'running',
    startedAt: { lt: new Date(Date.now() - 30 * 60 * 1000) }
  },
  data: {
    status: 'failed',
    lastError: 'Job timeout - worker may have crashed'
  }
});

Prevention:

  • Implement job heartbeats
  • Set maximum job duration
  • Monitor worker health
  • Alert on jobs >15 minutes in running

5. Data Sync Conflicts

Symptoms:

  • Duplicate records
  • Missing data
  • Sync result errors

Recovery:

// Idempotency via external ID
await prisma.yelpReview.upsert({
  where: { externalId: review.id },
  create: reviewData,
  update: reviewData
});

Prevention:

  • Always use externalId for deduplication
  • Implement optimistic locking
  • Track sync cursors properly
  • Alert on high duplicate rates

Alerting Recommendations

Critical Alerts (PagerDuty/On-Call)

ConditionThresholdAction
Multiple connections down>3 connections in error stateInvestigate platform issue
Service-wide outageAll connections for one category failingCheck external platform status
Dead letter queue growth>5 jobs in dead_letterManual intervention required
Auth failure spike>10% connections with auth errorsVerify API credentials

Alert Query:

const criticalIssues = await prisma.connection.count({
  where: {
    health: {
      path: ['status'],
      equals: 'error'
    },
    isEnabled: true
  }
});

if (criticalIssues > 3) {
  await sendPagerDutyAlert('Multiple connections down');
}

Warning Alerts (Slack/Email)

ConditionThresholdAction
High error rate>5% error rate for >1 hourReview logs, check for patterns
Slow response timesAvg response time >2sCheck network/API performance
Sync backlog growingPending jobs increasing over timeScale workers or investigate
Token expiration soon<24 hours until expiryNotify client to reconnect

Alert Query:

const healthyPercentage = (healthyCount / totalConnections) * 100;

if (healthyPercentage < 90) {
  await sendSlackAlert(`Connection health at ${healthyPercentage.toFixed(1)}%`);
}

Monitoring Metrics

Set up CloudWatch/Datadog dashboards for:

  1. Health Status Distribution

    • Pie chart: healthy vs warning vs error
    • Trend over 24 hours
  2. Response Time Trends

    • Line graph: avg response time per category
    • Alert on sustained increases
  3. Sync Job Metrics

    • Bar chart: pending/running/completed/failed
    • Dead letter queue size
  4. Error Rate by Category

    • Heatmap: error rate per connection category
    • Identify problematic integrations
  5. Uptime SLO

    • Gauge: Overall platform uptime
    • Target: 99.9% uptime

Troubleshooting Guide

Diagnosis Workflow

1. Check Connection Health Status
   └─> GET /api/connections/[id]/health

2. Review Health Check Issues
   └─> Check `issues` array in response

3. Examine Sync Job Status
   └─> GET /api/connections/[platform]/jobs

4. Check Recent Error Logs
   └─> Filter logs by connectionId

5. Verify Platform Credentials
   └─> Check tokenExpiresAt, webhookEnabled

6. Test Platform API Directly
   └─> Use platform-specific test endpoints

Common Issues and Solutions

Issue: "Connection not receiving messages"

Checklist:

  • Verify isEnabled: true on ConnectionIntegration
  • Check webhook is configured in platform dashboard
  • Verify webhook secret matches
  • Test webhook endpoint with curl
  • Check for webhook signature validation errors in logs
  • Confirm portal mapping in database

Resolution:

# Test webhook endpoint
curl -X POST https://your-domain.com/api/webhooks/yelp/messages \
  -H "Content-Type: application/json" \
  -H "X-Yelp-Webhook-Signature: test" \
  -d '{"event_type":"message.created","data":{}}'

# Check webhook configuration
SELECT * FROM "YelpConnection" WHERE "webhookEnabled" = true;

Issue: "Sync jobs stuck in pending"

Checklist:

  • Verify cron job is running (check logs)
  • Check for worker process errors
  • Look for database connection issues
  • Verify no jobs stuck in running state
  • Check system resources (CPU/memory)

Resolution:

// Force process pending jobs
import { yelpReviewSyncJobService } from '@/lib/jobs/yelpReviewSyncJobService';

const job = await yelpReviewSyncJobService.claimNextJob();
if (job) {
  await processYelpReviewSync(job);
}

Issue: "High error rate on connection"

Checklist:

  • Check external platform status page
  • Review error messages in issues array
  • Verify API credentials are valid
  • Check for rate limiting (429 responses)
  • Look for network connectivity issues

Resolution:

// Trigger manual health check
const result = await ConnectionHealthService.checkConnectionHealth(
  connectionId,
  { detailed: true, metrics: true }
);

console.log('Issues detected:', result.issues);

Issue: "Token expired but no notification sent"

Checklist:

  • Verify token expiry monitoring is enabled
  • Check notification service logs
  • Confirm client email is correct
  • Test notification endpoint manually

Resolution:

// Manual token refresh for OAuth connections
import { googleConnectionService } from '@/lib/connections/google/googleConnectionService';

await googleConnectionService.refreshToken(connectionId);

Debug Mode

Enable detailed logging for troubleshooting:

// Set environment variable
DEBUG=petunia:connections:*

// Or in code
logger.setLevel('debug');

// Logs will include:
// - API request/response details
// - Health check execution steps
// - Sync job processing details
// - Error stack traces

Best Practices

1. Regular Health Checks

  • Run automated health checks every 15 minutes
  • Store results for trend analysis
  • Alert on consecutive failures (3+)

2. Proactive Token Management

  • Refresh tokens 24 hours before expiry
  • Implement graceful token refresh with fallback
  • Notify clients when manual reconnection needed

3. Sync Job Optimization

  • Use exponential backoff for retries
  • Implement job prioritization
  • Monitor dead letter queue closely
  • Archive completed jobs after 30 days

4. Error Handling

  • Always include detailed error context
  • Categorize errors (transient vs permanent)
  • Implement circuit breakers for external APIs
  • Log all API failures with request/response

5. Monitoring and Alerting

  • Set realistic SLOs (99.5% uptime per connection)
  • Create tiered alerting (critical vs warning)
  • Use runbooks for common failure scenarios
  • Review metrics weekly for trends

6. Client Communication

  • Send proactive notifications for token expiry
  • Provide clear reconnection instructions
  • Display connection health in client dashboard
  • Offer self-service troubleshooting tools

7. Testing

  • Write integration tests for health checks
  • Mock external API failures
  • Test token refresh flows
  • Simulate rate limiting scenarios

8. Documentation

  • Keep platform API documentation up to date
  • Document known issues and workarounds
  • Maintain changelog of integration changes
  • Share postmortems for major outages


Changelog

Version 1.0.0 (2025-12-13)

  • Initial documentation
  • Comprehensive health monitoring patterns
  • Sync job tracking guidance
  • Troubleshooting procedures
  • Alerting recommendations