Documentation

BILLING_PAYMENTS

docs/runbooks/BILLING_PAYMENTS.md

Billing & Payments Runbook

[!WARNING] The prices, allowances and rates in this document are NOT current and must not be quoted to a customer. The single source of truth for tier pricing is lib/types/pricing.ts; the public shelf is app/(public)/pricing/page.tsx.

A 2026-09-05 sweep found eight mutually incompatible tier tables across this repo, seven of them in documents marked "Production Ready", "Complete", or carrying no status line at all. Numbers here were correct for some earlier draft of the product and were never revised. They are left in place because the surrounding architecture and flow descriptions are still accurate and useful — only the figures are stale.

If you need a real number, read lib/types/pricing.ts. Do not copy one from here.

This runbook covers operational procedures for billing and payment processing including Stripe integration, subscription management, credit wallet system, and webhook handling.

Owner: Platform Team Version: 1.0.0 Last Updated: 2025-12-27 Related Docs: PLATFORM_INTEGRATIONS.md


Table of Contents

  1. Prerequisites
  2. Health Check Commands
  3. Common Failure Modes
  4. Verification Commands
  5. Rollback Steps
  6. Escalation

Prerequisites

Required Services

ServicePurposeHealth Check
PostgreSQL (Supabase)Subscription datapnpm db:health
Stripe APIPayment processingStripe Dashboard
Redis (Upstash)Webhook idempotencyUpstash Dashboard
Stripe ConnectMarketplace paymentsStripe Connect Dashboard

Required Environment Variables

# Database
DATABASE_URL="postgresql://..."

# Stripe Platform
STRIPE_SECRET_KEY="sk_live_..."
STRIPE_PUBLISHABLE_KEY="pk_live_..."
STRIPE_WEBHOOK_SECRET="whsec_..."

# Stripe Connect (for marketplace)
STRIPE_CONNECT_WEBHOOK_SECRET="whsec_..."

# Pricing Configuration
STRIPE_STARTER_PRICE_ID="price_..."
STRIPE_GROWTH_PRICE_ID="price_..."
STRIPE_SCALE_PRICE_ID="price_..."
STRIPE_ENTERPRISE_PRICE_ID="price_..."

Key Files

ComponentFile Path
Credit Wallet Servicelib/services/billing/credit-wallet-service.ts
Stripe Webhook Handlerslib/services/billing/stripe-webhook-handlers.ts
Stripe Connect Handlerslib/services/billing/stripe-connect-webhook-handlers.ts
Processing Fee Servicelib/services/billing/processing-fee-service.ts
Processing Fee Reportslib/services/billing/processing-fee-reports.ts
Webhook Retry Servicelib/services/billing/webhook-retry-service.ts
Billing Middlewarelib/middleware/billing/
Platform Webhook Routeapp/api/webhooks/stripe/route.ts
Connect Webhook Routeapp/api/webhooks/stripe/connect/route.ts

Pricing Tiers

TierFeaturesMonthly Price
StarterBasic features$29/mo
Growth+ Advanced features$99/mo
Scale+ Priority support$299/mo
EnterpriseCustomContact sales

Health Check Commands

Quick Health Check

# Check Stripe webhook endpoint
curl https://your-domain.com/api/webhooks/stripe
# Expected: {"status":"ok","webhook":"stripe"}

# Check Stripe Connect webhook endpoint
curl https://your-domain.com/api/webhooks/stripe/connect
# Expected: {"status":"ok","webhook":"stripe-connect"}

Stripe API Health Check

# Check Stripe API connectivity
curl https://api.stripe.com/v1/balance \
  -u "$STRIPE_SECRET_KEY:"

# List recent events
curl https://api.stripe.com/v1/events?limit=5 \
  -u "$STRIPE_SECRET_KEY:"

# Check webhook endpoint status
curl https://api.stripe.com/v1/webhook_endpoints \
  -u "$STRIPE_SECRET_KEY:"

Database Health

# Check subscription status distribution
psql $DATABASE_URL -c "
SELECT
  subscription_status,
  COUNT(*) as count,
  ROUND(COUNT(*)::numeric / SUM(COUNT(*)) OVER() * 100, 2) as percentage
FROM \"Portal\"
WHERE subscription_status IS NOT NULL
GROUP BY subscription_status
ORDER BY count DESC;
"

# Check MRR by tier
psql $DATABASE_URL -c "
SELECT
  subscription_tier,
  COUNT(*) as subscriptions,
  SUM(
    CASE subscription_tier
      WHEN 'starter' THEN 29
      WHEN 'growth' THEN 99
      WHEN 'scale' THEN 299
      ELSE 0
    END
  ) as mrr
FROM \"Portal\"
WHERE subscription_status = 'active'
GROUP BY subscription_tier
ORDER BY mrr DESC;
"

# Check credit wallet balances
psql $DATABASE_URL -c "
SELECT
  COUNT(*) as total_wallets,
  SUM(balance) as total_balance,
  AVG(balance) as avg_balance,
  MIN(balance) as min_balance,
  MAX(balance) as max_balance
FROM \"CreditWallet\";
"

# Check recent payment failures
psql $DATABASE_URL -c "
SELECT
  portal_id,
  subscription_tier,
  last_payment_error,
  payment_failed_at
FROM \"Portal\"
WHERE payment_failed_at > NOW() - INTERVAL '7 days'
ORDER BY payment_failed_at DESC
LIMIT 20;
"

Webhook Processing Health

# Check webhook processing stats
psql $DATABASE_URL -c "
SELECT
  event_type,
  COUNT(*) as total,
  COUNT(CASE WHEN status = 'processed' THEN 1 END) as processed,
  COUNT(CASE WHEN status = 'failed' THEN 1 END) as failed,
  MAX(processed_at) as last_processed
FROM \"WebhookEvent\"
WHERE created_at > NOW() - INTERVAL '24 hours'
GROUP BY event_type
ORDER BY total DESC;
"

# Check pending webhook retries
psql $DATABASE_URL -c "
SELECT
  event_type,
  retry_count,
  next_retry_at,
  last_error
FROM \"WebhookEvent\"
WHERE status = 'pending_retry'
ORDER BY next_retry_at ASC
LIMIT 20;
"

Common Failure Modes

1. Payment Failures

SymptomLikely CauseResolution
All payments failingStripe API key invalidVerify API keys
Card declinedInsufficient fundsContact customer
3DS failuresAuthentication requiredEnable 3DS flow
Currency mismatchWrong currency codeCheck currency settings

Diagnostic Commands:

# Check recent payment intents
curl "https://api.stripe.com/v1/payment_intents?limit=10" \
  -u "$STRIPE_SECRET_KEY:"

# Check failed charges
curl "https://api.stripe.com/v1/charges?limit=10&status=failed" \
  -u "$STRIPE_SECRET_KEY:"

# Check payment failure logs
grep "payment.*fail\|PaymentError" /var/log/petunia/app.log | tail -20

2. Subscription Issues

SymptomLikely CauseResolution
Subscription not createdCheckout incompleteCheck session status
Wrong tier assignedPrice ID mismatchVerify price mapping
Subscription stuckWebhook not processedCheck webhook logs
Cancel not workingSubscription ID wrongVerify subscription ID

Diagnostic Commands:

# Check subscription in Stripe
curl "https://api.stripe.com/v1/subscriptions/$SUBSCRIPTION_ID" \
  -u "$STRIPE_SECRET_KEY:"

# Check customer subscriptions
curl "https://api.stripe.com/v1/subscriptions?customer=$CUSTOMER_ID" \
  -u "$STRIPE_SECRET_KEY:"

# Check subscription sync
psql $DATABASE_URL -c "
SELECT
  id,
  stripe_customer_id,
  stripe_subscription_id,
  subscription_status,
  subscription_tier,
  updated_at
FROM \"Portal\"
WHERE stripe_subscription_id = 'sub_xxx';
"

3. Webhook Failures

SymptomLikely CauseResolution
Webhooks not receivedEndpoint not configuredConfigure in Stripe
Signature invalidWrong webhook secretCheck STRIPE_WEBHOOK_SECRET
Events missingEndpoint disabledRe-enable in Stripe
Duplicate processingIdempotency failedCheck idempotency keys

Diagnostic Commands:

# Check recent webhook events in Stripe
curl "https://api.stripe.com/v1/events?limit=20" \
  -u "$STRIPE_SECRET_KEY:"

# Check webhook logs
grep "stripe.*webhook\|WebhookError" /var/log/petunia/app.log | tail -50

# Check failed webhook deliveries in Stripe Dashboard
# https://dashboard.stripe.com/webhooks

4. Credit Wallet Issues

SymptomLikely CauseResolution
Balance not updatingTransaction failedCheck transaction logs
Negative balanceOverdraw allowedVerify balance check
Credits expiredExpiration not processedRun expiration job
Wrong credit amountCalculation errorVerify credit rules

Diagnostic Commands:

# Check wallet transactions
psql $DATABASE_URL -c "
SELECT
  id,
  wallet_id,
  amount,
  type,
  description,
  created_at
FROM \"CreditTransaction\"
WHERE wallet_id = 'wallet-id'
ORDER BY created_at DESC
LIMIT 20;
"

# Check wallet balance history
psql $DATABASE_URL -c "
SELECT
  DATE(created_at) as date,
  SUM(CASE WHEN amount > 0 THEN amount ELSE 0 END) as credits_added,
  SUM(CASE WHEN amount < 0 THEN ABS(amount) ELSE 0 END) as credits_used,
  SUM(amount) as net_change
FROM \"CreditTransaction\"
WHERE wallet_id = 'wallet-id'
AND created_at > NOW() - INTERVAL '30 days'
GROUP BY DATE(created_at)
ORDER BY date DESC;
"

Verification Commands

Test Stripe Integration

import Stripe from 'stripe';

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!);

// Check balance
const balance = await stripe.balance.retrieve();
console.log('Balance:', balance);

// Create test customer
const customer = await stripe.customers.create({
  email: 'test@example.com',
  name: 'Test Customer',
});
console.log('Customer:', customer.id);

// Create checkout session
const session = await stripe.checkout.sessions.create({
  customer: customer.id,
  mode: 'subscription',
  line_items: [{
    price: process.env.STRIPE_GROWTH_PRICE_ID,
    quantity: 1,
  }],
  success_url: 'https://your-domain.com/success',
  cancel_url: 'https://your-domain.com/cancel',
});
console.log('Checkout URL:', session.url);

Test Credit Wallet Service

import { CreditWalletService } from '@/lib/services/billing/credit-wallet-service';

const walletService = new CreditWalletService();

// Get wallet balance
const balance = await walletService.getBalance('portal-id');
console.log('Balance:', balance);

// Add credits
await walletService.addCredits('portal-id', 100, 'manual_adjustment');

// Use credits
const success = await walletService.useCredits('portal-id', 10, 'voice_call');
console.log('Credits used:', success);

Test Webhook Processing

# Trigger test webhook event
curl -X POST https://your-domain.com/api/webhooks/stripe \
  -H "Content-Type: application/json" \
  -H "Stripe-Signature: test" \
  -d '{
    "type": "customer.subscription.created",
    "data": {"object": {"id": "sub_test"}}
  }'

# Note: In production, use Stripe CLI for testing
stripe trigger checkout.session.completed

Run Billing Tests

# Run billing tests
pnpm test -- --testPathPattern="billing"

# Run webhook tests
pnpm test -- --testPathPattern="webhooks/stripe"

# Run credit wallet tests
pnpm test -- --testPathPattern="credit-wallet"

Rollback Steps

Disable Billing Features

# Disable new subscriptions
curl -X POST https://your-domain.com/api/admin/features \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"feature": "billing.subscriptions.enabled", "enabled": false}'

# Disable credit usage
curl -X POST https://your-domain.com/api/admin/features \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"feature": "billing.credits.enabled", "enabled": false}'

Pause Webhook Processing

# Disable webhook processing (emergency)
# In Stripe Dashboard: Webhooks > Endpoint > Disable

# Or via API
curl -X POST "https://api.stripe.com/v1/webhook_endpoints/$ENDPOINT_ID" \
  -u "$STRIPE_SECRET_KEY:" \
  -d "disabled=true"

Manually Sync Subscription

# Force sync subscription from Stripe
psql $DATABASE_URL -c "
UPDATE \"Portal\"
SET
  subscription_status = 'active',
  subscription_tier = 'growth',
  updated_at = NOW()
WHERE stripe_subscription_id = 'sub_xxx';
"

Refund Payment

# Create refund via Stripe API
curl -X POST "https://api.stripe.com/v1/refunds" \
  -u "$STRIPE_SECRET_KEY:" \
  -d "charge=ch_xxx" \
  -d "reason=requested_by_customer"

Rollback Deployment

# If billing changes caused issues
vercel list --app petunia

# Rollback
vercel rollback <previous-deployment-url>

# Verify webhook is functional
curl https://your-domain.com/api/webhooks/stripe

Escalation

When to Escalate

SeverityCriteriaAction
P1 - CriticalAll payments failingImmediate on-call page
P2 - HighSubscription sync brokenEscalate within 30 minutes
P3 - MediumCredit balance issuesEscalate within 2 hours
P4 - LowReporting discrepanciesNormal ticket queue

Escalation Contacts

RoleContactAvailability
On-Call EngineerPagerDuty24/7
Finance TeamSlack #financeBusiness hours
Stripe Supportsupport.stripe.com24/7

Information to Include

When escalating, include:

  1. Customer ID: Stripe customer ID
  2. Subscription ID: Stripe subscription ID
  3. Error Messages: Specific Stripe error codes
  4. Payment Amount: Transaction amount
  5. Recent Changes: Deployments, config changes

Monitoring & Alerts

Key Metrics to Watch

MetricHealthy RangeAlert Threshold
Payment success rate> 95%< 90%
Webhook processing rate> 99%< 95%
MRR growthPositiveSudden drop > 10%
Churn rate< 5%/month> 8%/month
Credit usage rateStableSudden spike > 200%

Alert Configuration

  • Sentry: Payment and webhook errors
  • Slack #billing-alerts: Payment failures, subscription changes
  • PagerDuty: P1/P2 incidents

Billing-Specific Dashboards


Webhook Event Handling

Platform Webhook Events

EventHandlerAction
checkout.session.completedhandleCheckoutCompleteCreate subscription
customer.subscription.createdhandleSubscriptionChangeSync subscription
customer.subscription.updatedhandleSubscriptionChangeUpdate tier/status
customer.subscription.deletedhandleSubscriptionCanceledMark as canceled
invoice.payment_succeededhandlePaymentSucceededRecord payment
invoice.payment_failedhandlePaymentFailedSend failure email
invoice.createdhandleInvoiceCreatedTrack invoice
customer.createdhandleCustomerCreatedLink customer

Connect Webhook Events

EventHandlerAction
account.updatedhandleAccountUpdatedUpdate Connect status
payment_intent.succeededhandlePaymentSucceededRecord marketplace payment
transfer.createdhandleTransferCreatedTrack payout

Webhook Retry Strategy

// Exponential backoff for retries
const RETRY_DELAYS = [
  5 * 60 * 1000,    // 5 minutes
  30 * 60 * 1000,   // 30 minutes
  2 * 60 * 60 * 1000, // 2 hours
  24 * 60 * 60 * 1000, // 24 hours
];

// Maximum retries before marking as failed
const MAX_RETRIES = 4;

Credit System

Credit Types

TypeDescriptionExpiration
subscriptionMonthly allocationEnd of billing period
bonusPromotional credits90 days
purchasedBought creditsNever
rolloverUnused from previous30 days

Credit Usage

ActionCredits
Voice call (per minute)1 credit
SMS message0.5 credits
AI response0.25 credits
WhatsApp message0.5 credits

Last verified: 2025-12-27