Documentation

KNOWLEDGE_ARCHITECTURE

docs/KNOWLEDGE_ARCHITECTURE.md

Knowledge Management Architecture

Overview

Petunia's knowledge management system is a multi-layered architecture that combines vector embeddings, semantic search, and retrieval-augmented generation (RAG) to deliver intelligent, context-aware responses. The system learns from every interaction and continuously improves accuracy through feedback loops.

Version: 1.0.0 Last Modified: 2025-12-13 Status: Production

Architecture Layers

1. Data Storage Layer

KnowledgeBaseEntry (PostgreSQL)

Primary relational storage for structured knowledge entries.

Schema:

model KnowledgeBaseEntry {
  id           String   @id
  portalId     String
  category     String
  title        String
  question     String
  answer       String
  keywords     String[]
  metadata     Json?
  isPublic     Boolean  @default(false)
  requiresAuth Boolean  @default(false)
  channels     String[]
  embedding    Float[]       // Vector embedding for semantic search
  searchText   String        // Full-text search optimization
  createdAt    DateTime @default(now())
  updatedAt    DateTime
  Portal       Portal   @relation(fields: [portalId], references: [id], onDelete: Cascade)

  @@index([category])
  @@index([portalId])
  @@index([searchText])
}

Key Features:

  • Multi-tenant isolation via portalId
  • Category-based organization
  • Channel-specific knowledge (Yelp, SMS, email, etc.)
  • Access control (public/private, auth requirements)
  • Vector embeddings stored inline for quick access
  • Optimized search text for keyword fallback

Location: /prisma/schema.prisma (line 3637-3658)

Vector Storage (Upstash Vector)

High-performance vector database for semantic similarity search.

Storage Types:

enum KnowledgeType {
  BUSINESS_INFO = 'business_info',           // Core business details
  CUSTOMER_PROFILE = 'customer_profile',     // Customer preferences
  CONVERSATION_PATTERN = 'conversation_pattern', // Past interactions
  COMPETITOR_INSIGHT = 'competitor_insight', // Competitive intelligence
  PRODUCT_SERVICE = 'product_service',       // Products/services
  FAQ = 'faq',                              // Common questions
  REVIEW_SENTIMENT = 'review_sentiment',    // Review analysis
  CAMPAIGN_PERFORMANCE = 'campaign_performance', // Marketing data
  YELP_REVIEW = 'yelp_review',             // Yelp reviews
  YELP_MESSAGE = 'yelp_message',           // Yelp messages
  SOCIAL_MENTION = 'social_mention',       // Social media
}

Metadata Structure:

interface KnowledgeMetadata {
  portalId: string;          // Multi-tenant isolation
  type: KnowledgeType;       // Knowledge category
  category?: string;         // Subcategory
  source: string;            // Origin (onboarding, conversation, review)
  timestamp: string;         // When stored
  relevanceScore?: number;   // Quality score
  tags?: string[];          // Searchable tags
  content: string;          // Original text
  [key: string]: any;       // Flexible metadata
}

Location: /lib/services/knowledge/upstash-knowledge.ts

Search Index (Upstash Redis)

Fast full-text search for messages, customers, and documents.

Document Types:

enum SearchDocumentType {
  MESSAGE = 'message',
  CUSTOMER = 'customer',
  LEAD = 'lead',
  REVIEW = 'review',
  CAMPAIGN = 'campaign',
  FAQ = 'faq',
  KNOWLEDGE = 'knowledge',
}

Indexing Strategy:

  • Word tokenization with 3-character minimum
  • Timeline-based sorting (zset)
  • Customer-specific indexes
  • Channel-based filtering
  • Real-time indexing on write

Location: /lib/services/knowledge/upstash-search.ts

Pinecone Vector Database

Long-term vector storage for voice knowledge and specialized contexts.

Features:

  • Namespace isolation
  • Metadata filtering
  • Voice-optimized retrieval
  • Batch operations (100 vectors per batch)
  • Automatic fallback mode for testing

Location: /lib/services/ai/pinecone.ts


2. Embedding Layer

OpenAI Embeddings

Model: text-embedding-3-large Dimensions: 3072 Use Cases:

  • Knowledge base entries
  • Customer queries
  • Business context
  • Reviews and feedback

Caching Strategy:

// 1-hour cache for embeddings
const cacheKey = `embedding:${text.substring(0, 100)}`;
const cached = await caches.api.get<number[]>(cacheKey);
if (cached) return cached;

// Generate and cache
const embedding = await openai.embeddings.create({
  model: 'text-embedding-3-large',
  input: text,
});
await caches.api.set(cacheKey, embedding, 3600);

Deduplication:

// Prevent duplicate embeddings
const dedupeKey = `dedupe:${portalId}:${customerId}:${message.substring(0, 50)}`;
const isDuplicate = await caches.temp.get(dedupeKey);
if (isDuplicate) return;
await caches.temp.set(dedupeKey, true, 3600);

Location: /lib/services/knowledge/upstash-knowledge.ts (generateEmbedding)


3. RAG (Retrieval Augmented Generation)

Knowledge-Enhanced AI Service

Purpose: Combines multiple knowledge sources to generate contextually rich responses.

Context Gathering Pipeline:

async function gatherContext(query: string, context: AIContext) {
  // 1. Business Information
  const businessContext = await businessKnowledge.getRelevantContext(
    portalId, query,
    { types: [KnowledgeType.BUSINESS_INFO, KnowledgeType.PRODUCT_SERVICE] }
  );

  // 2. Customer History
  const customerInsights = await customerKnowledge.getCustomerInsights(
    portalId, customerId
  );

  // 3. Similar Conversations
  const similarInteractions = await messageSearch.search(
    portalId, query,
    { types: [SearchDocumentType.MESSAGE], limit: 3 }
  );

  // 4. Relevant FAQs
  const relevantFAQs = await customerSearch.search(
    portalId, query,
    { types: [SearchDocumentType.FAQ], limit: 2 }
  );

  // 5. Active Campaigns/Offers
  const relevantOffers = await customerSearch.search(
    portalId, query,
    { types: [SearchDocumentType.CAMPAIGN], limit: 1 }
  );

  // 6. Competitor Intelligence
  const competitorContext = await businessKnowledge.getRelevantContext(
    portalId, query,
    { types: [KnowledgeType.COMPETITOR_INSIGHT], limit: 2 }
  );

  return { businessContext, customerInsights, similarInteractions,
           relevantFAQs, relevantOffers, competitorContext };
}

Enhanced Prompt Construction:

const prompt = `
You are a helpful assistant for ${businessName}.

About the business:
${businessContext.content}

Customer history:
${customerInsights.map(i => i.content).join('\n')}

Similar past responses:
${similarInteractions.map(i => i.content).join('\n')}

Relevant FAQs:
${faqs.map(f => `${f.title}: ${f.content}`).join('\n')}

Current offers:
${offers.map(o => o.content).join('\n')}

Use a ${tone} tone.
Customer question: ${query}

Provide a helpful, accurate response based on the context above.
`;

AI Models:

  • Primary: Claude 3 Opus (Anthropic)
  • Fallback: GPT-4 Turbo (OpenAI)
  • Deterministic Fallback: Context-based template responses

Confidence Scoring:

function calculateConfidence(context: KnowledgeContextData): number {
  let score = 0.5; // Base confidence
  if (context.business?.content) score += 0.2;
  if (context.customer?.insights?.length > 0) score += 0.1;
  if (context.similar?.length > 0) score += 0.1;
  if (context.faqs?.length > 0) score += 0.05;
  if (context.offers?.length > 0) score += 0.05;
  return Math.min(1.0, score);
}

Location: /lib/services/ai/knowledge-ai.ts


Similarity Algorithm: Cosine similarity

function cosineSimilarity(a: number[], b: number[]): number {
  if (!a || !b || a.length !== b.length) return 0;

  let dotProduct = 0;
  let normA = 0;
  let normB = 0;

  for (let i = 0; i < a.length; i++) {
    dotProduct += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }

  return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}

Search Process:

  1. Generate query embedding (3072 dimensions)
  2. Apply metadata filters (portalId, type, categories)
  3. Compute similarity scores
  4. Filter by minimum score (default: 0.7)
  5. Sort by relevance
  6. Return top-K results (default: 10)

Upstash Vector Query:

const results = await vectorIndex.query({
  vector: queryEmbedding,
  topK: options?.limit || 10,
  includeMetadata: true,
  filter: { portalId, type: { $in: types } }
});

Location: /lib/services/knowledge/upstash-knowledge.ts (getRelevantContext)

Keyword Search Fallback

Tokenization:

function tokenize(text: string): string[] {
  return text
    .toLowerCase()
    .replace(/[^\w\s]/g, ' ')
    .split(/\s+/)
    .filter(word => word.length > 2)
    .slice(0, 100); // Limit to prevent excessive indexing
}

Relevance Scoring:

function calculateRelevance(query: string, content: string): number {
  const contentLower = content.toLowerCase();
  let score = 0;

  // Exact match bonus
  if (contentLower.includes(query.toLowerCase())) {
    score += 10;
  }

  // Word match scoring
  const queryWords = tokenize(query);
  for (const word of queryWords) {
    const matches = contentLower.match(new RegExp(`\\b${word}\\b`, 'gi'));
    if (matches) {
      score += matches.length * 2;
    }
  }

  return Math.min(100, score);
}

Location: /lib/services/knowledge/upstash-search.ts


5. Multi-Tenant Knowledge Isolation

Portal-Level Isolation

All knowledge operations require portalId:

// Storage
await businessKnowledge.storeBusinessKnowledge(portalId, data);

// Retrieval
const context = await businessKnowledge.getRelevantContext(portalId, query);

// Search
const results = await messageSearch.search(portalId, query);

// Vector query
await vectorIndex.query({
  vector: embedding,
  filter: { portalId }, // REQUIRED
});

Database Constraints:

model KnowledgeBaseEntry {
  Portal Portal @relation(fields: [portalId], references: [id], onDelete: Cascade)
  @@index([portalId])
}

Benefits:

  • Hard isolation at data layer
  • Cascade deletes when portal removed
  • Optimized queries via indexes
  • Prevents cross-tenant data leakage

Namespace Isolation (Upstash)

Namespaces:

  • business - Business context and services
  • customer - Customer profiles and history
  • messages - Conversation search
  • customers - Customer search
  • Portal-specific namespaces for specialized data

Client Override Support:

// Use client-specific API keys when available
const service = await UpstashKnowledgeService.getInstance('business', clientId);

Location: /lib/services/knowledge/upstash-knowledge.ts


6. File Upload & Processing Pipeline

Upload Service

Supported File Types:

  • Images: JPEG, PNG, GIF
  • Documents: PDF, TXT
  • Maximum size: 10MB (configurable)

Processing Pipeline:

async function upload(file: Buffer, filename: string, userId: string) {
  // 1. Rate limiting (10 uploads/minute)
  await uploadRateLimiter.check(userId);

  // 2. File validation
  const fileType = await detectFileType(file);
  validateFileType(fileType, allowedTypes);
  validateFileSize(file.length, maxSize);

  // 3. Security scanning
  await scanFile(file);

  // 4. Image processing (if applicable)
  if (fileType.startsWith('image/')) {
    file = await compressImage(file);
    thumbnailKey = await createThumbnail(file, key, userId);
  }

  // 5. Upload to S3
  await s3Client.send(new PutObjectCommand({
    Bucket: bucket,
    Key: key,
    Body: file,
    ContentType: fileType,
    ServerSideEncryption: 'AES256',
  }));

  return { url, key, size, type, thumbnailUrl };
}

Image Optimization:

// Compression
const compressed = await sharp(file)
  .jpeg({ quality: 85, progressive: true })
  .toBuffer();

// Thumbnail generation
const thumbnail = await sharp(file)
  .resize(200, 200, { fit: 'cover' })
  .jpeg({ quality: 80 })
  .toBuffer();

Location: /lib/services/file-upload.ts

Document Processing

OCR Integration:

  • Endpoint: /app/api/uploads/ocr/route.ts
  • Use case: Extract text from images and PDFs
  • Service: AWS Textract or similar

Knowledge Extraction:

  1. Upload document
  2. Extract text via OCR
  3. Split into chunks (8000 chars max)
  4. Generate embeddings for each chunk
  5. Store in vector database with metadata
  6. Index for search

7. Knowledge Integration Layer

Real-Time Learning Hooks

AutoResponder Integration:

async function onAutoResponderMessage(data: {
  portalId: string;
  customerId: string;
  incomingMessage: string;
  generatedResponse: string;
  sentiment?: string;
}) {
  // Queue for batch processing
  queueInteraction(portalId, {
    customerId, message: incomingMessage,
    response: generatedResponse, sentiment
  });

  // Index immediately for search
  await messageSearch.indexDocument({
    id: `auto_${portalId}_${Date.now()}`,
    portalId, customerId, type: SearchDocumentType.MESSAGE,
    content: incomingMessage
  });
}

Yelp Integration:

// Reviews
async function onYelpReview(data: {
  portalId: string;
  reviewId: string;
  rating: number;
  text: string;
}) {
  // Store review with sentiment
  await businessKnowledge.storeYelpReview(portalId, data);

  // Extract competitor mentions
  const competitors = await getCompetitorKeywords(portalId);
  for (const competitor of competitors) {
    if (text.toLowerCase().includes(competitor)) {
      await businessKnowledge.storeCompetitorInsight(portalId, {
        competitorName: competitor,
        observation: text
      });
    }
  }

  // Analyze negative reviews
  if (rating <= 2) {
    await analyzeNegativeFeedback(portalId, text);
  }
}

// Messages
async function onYelpMessage(data: {
  portalId: string;
  conversationId: string;
  message: string;
  isFromBusiness: boolean;
}) {
  if (isFromBusiness) return; // Only learn from customers

  const conversation = await prisma.yelpConversation.findUnique({
    where: { id: conversationId }
  });

  queueInteraction(portalId, {
    customerId: conversation.userId,
    message: data.message,
    channel: 'yelp'
  });
}

Social Media Integration:

async function onSocialMention(data: {
  portalId: string;
  platform: 'facebook' | 'instagram' | 'twitter';
  content: string;
  authorId: string;
  sentiment?: string;
}) {
  queueInteraction(portalId, {
    customerId: authorId,
    message: content,
    channel: platform,
    sentiment
  });
}

Campaign Learning:

async function onCampaignComplete(data: {
  portalId: string;
  campaignName: string;
  performance: { sent, opened, clicked, converted };
  bestPerformingMessage?: string;
}) {
  const context = `
    Campaign "${campaignName}" Results:
    Open Rate: ${(opened/sent * 100).toFixed(1)}%
    Click Rate: ${(clicked/opened * 100).toFixed(1)}%
    Conversion: ${(converted/clicked * 100).toFixed(1)}%
    Best Message: ${bestPerformingMessage}
  `;

  await businessKnowledge.storeBusinessKnowledge(portalId, {
    description: context,
    source: 'campaign_performance'
  });
}

Location: /lib/services/knowledge/knowledge-integration.ts

Batch Processing

Queue-Based Processing:

class KnowledgeIntegration {
  private processingQueue: Map<string, Interaction[]> = new Map();
  private batchTimer: NodeJS.Timeout;

  startBatchProcessor() {
    // Process every 30 seconds
    this.batchTimer = setInterval(async () => {
      for (const [portalId, interactions] of this.processingQueue) {
        // Filter trivial interactions
        const meaningful = interactions.filter(i =>
          i.message && i.message.length > 10
        );

        if (meaningful.length > 0) {
          await businessKnowledge.batchStoreInteractions(
            portalId, meaningful
          );
        }
      }
      this.processingQueue.clear();
    }, 30000);
  }
}

Benefits:

  • Reduces API calls
  • Prevents rate limiting
  • Improves throughput
  • Batches embeddings

8. Feedback Learning System

Response Tracking

Track Every AI Response:

async function trackResponse(data: {
  portalId: string;
  customerId: string;
  query: string;
  response: string;
  confidence: number;
}): Promise<string> {
  const responseId = `resp_${Date.now()}_${random()}`;

  await AIResponseTracking.create({
    id: responseId,
    portalId, customerId, query, response,
    confidence, status: 'pending_feedback'
  });

  // Check implicit feedback after 5 minutes
  setTimeout(() => {
    checkImplicitFeedback(responseId, portalId, customerId);
  }, 5 * 60 * 1000);

  return responseId;
}

Explicit Feedback

User Corrections:

interface ResponseFeedback {
  responseId: string;
  portalId: string;
  originalQuery: string;
  generatedResponse: string;
  wasHelpful: boolean;
  humanCorrection?: string;
  customerSatisfaction?: 1 | 2 | 3 | 4 | 5;
  resolutionTime?: number;
  requiredHumanIntervention: boolean;
}

async function recordFeedback(feedback: ResponseFeedback) {
  // Queue for batch processing
  learningQueue.get(portalId).push(feedback);

  // Update tracking record
  await AIResponseTracking.update({
    where: { id: feedback.responseId },
    data: { wasHelpful, humanCorrection, satisfaction }
  });

  // Learn from correction immediately
  if (humanCorrection && !wasHelpful) {
    await learnFromCorrection(feedback);
  }
}

Implicit Feedback

Detect Satisfaction from Behavior:

async function checkImplicitFeedback(
  responseId: string,
  portalId: string,
  customerId: string
) {
  // Check if customer sent more messages
  const followUpMessages = await prisma.message.findMany({
    where: { portalId, contactId: customerId,
      createdAt: { gte: new Date(Date.now() - 5 * 60 * 1000) }
    }
  });

  // No follow-up = satisfactory
  const wasHelpful = followUpMessages.length === 0;

  // Check for clarification requests
  const needsClarification = detectClarificationPhrases(
    followUpMessages.map(m => m.content).join(' ')
  );

  await recordFeedback({
    responseId, portalId,
    wasHelpful: wasHelpful && !needsClarification,
    requiredHumanIntervention: needsClarification
  });
}

const clarificationPhrases = [
  'what do you mean', 'i don\'t understand',
  'can you explain', 'not what i asked',
  'that doesn\'t help', 'confused'
];

Learning from Mistakes

Pattern Identification:

function identifyMistakePattern(
  query: string,
  wrongResponse: string,
  correctResponse: string
): { type: string; severity: 'low' | 'medium' | 'high' } {
  // Factual errors
  if (wrongResponse.includes('open') && correctResponse.includes('closed')) {
    return { type: 'factual_error', severity: 'high' };
  }

  // Missing information
  if (correctResponse.length > wrongResponse.length * 1.5) {
    return { type: 'incomplete_response', severity: 'medium' };
  }

  // Missed intent
  if (query.includes('price') && !wrongResponse.includes('price')) {
    return { type: 'missed_intent', severity: 'high' };
  }

  // Tone issues
  if (wrongResponse.includes('sorry') && !correctResponse.includes('sorry')) {
    return { type: 'over_apologetic', severity: 'low' };
  }

  return { type: 'general_mistake', severity: 'medium' };
}

Store Corrections:

async function learnFromCorrection(feedback: ResponseFeedback) {
  // Store as high-priority learning example
  await businessKnowledge.storeBusinessKnowledge(portalId, {
    businessName: 'Learning Example',
    industry: 'correction',
    description: `
      When asked: "${feedback.originalQuery}",
      DON'T say: "${feedback.generatedResponse}",
      INSTEAD say: "${feedback.humanCorrection}"
    `
  });

  // Track mistake patterns
  const pattern = identifyMistakePattern(
    feedback.originalQuery,
    feedback.generatedResponse,
    feedback.humanCorrection
  );

  const patterns = await caches.config.get<MistakePattern[]>(
    `mistakes:${portalId}`
  ) || [];

  const existing = patterns.find(p => p.pattern === pattern.type);
  if (existing) {
    existing.frequency++;
    existing.examples.push({
      query: feedback.originalQuery,
      wrong: feedback.generatedResponse,
      correct: feedback.humanCorrection
    });
  } else {
    patterns.push({ pattern: pattern.type, frequency: 1, examples: [...] });
  }

  await caches.config.set(`mistakes:${portalId}`, patterns, 86400 * 7);
}

Automatic Retraining

Trigger When Accuracy Drops:

async function updateMetrics(portalId: string) {
  const feedback = await AIResponseTracking.findMany({
    where: { portalId, status: 'feedback_received',
      createdAt: { gte: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000) }
    }
  });

  const accuracy = feedback.filter(f => f.wasHelpful).length / feedback.length;

  // Alert if accuracy < 70% with sufficient data
  if (accuracy < 0.7 && feedback.length > 20) {
    logger.warn('AI accuracy below threshold', { portalId, accuracy });
    await triggerRetraining(portalId);
  }
}

async function triggerRetraining(portalId: string) {
  // Gather all corrections
  const corrections = await AIResponseTracking.findMany({
    where: { portalId, humanCorrection: { not: null }, wasHelpful: false },
    take: 50,
    orderBy: { createdAt: 'desc' }
  });

  // Create training examples
  for (const correction of corrections) {
    await businessKnowledge.storeBusinessKnowledge(portalId, {
      businessName: 'Training Data',
      industry: 'retraining',
      description: `Q: ${correction.query}\nA: ${correction.humanCorrection}`
    });
  }

  // Clear caches to use new knowledge
  await caches.api.invalidatePattern(`context:${portalId}:*`);
}

Location: /lib/services/knowledge/feedback-learning.ts


API Routes

Knowledge Base Management

GET /api/autoresponder/knowledge

  • Get all knowledge entries or specific entry by ID
  • Query params: businessId, id (optional)
  • Returns: { data: { knowledge | knowledgeBase }, success: true }

POST /api/autoresponder/knowledge

  • Create or update knowledge entry
  • Body: { businessId, entry: { title, content, category, tags } }
  • Tier limits enforced (prevents bypass via fake IDs)
  • Returns: { data: { entry }, success: true }

DELETE /api/autoresponder/knowledge

  • Delete knowledge entry
  • Query params: businessId, id
  • Returns: { data: { success: true }, success: true }

Location: /app/api/autoresponder/knowledge/route.ts


Services & Components

Core Services

KnowledgeBaseService

  • /lib/services/knowledgeBaseService.ts
  • PostgreSQL-backed knowledge management
  • Semantic search with OpenAI embeddings
  • Import/export (CSV, JSON)
  • Category statistics

UpstashKnowledgeService

  • /lib/services/knowledge/upstash-knowledge.ts
  • Vector-based knowledge storage
  • Real-time embedding generation
  • Business/customer namespace isolation
  • Sentiment analysis

UpstashSearchService

  • /lib/services/knowledge/upstash-search.ts
  • Full-text search with Redis
  • Real-time indexing
  • Multi-field queries
  • Autocomplete suggestions

KnowledgeEnhancedAI

  • /lib/services/ai/knowledge-ai.ts
  • RAG implementation
  • Multi-source context aggregation
  • Claude/GPT-4 integration
  • Confidence scoring

KnowledgeIntegration

  • /lib/services/knowledge/knowledge-integration.ts
  • Real-time learning hooks
  • Batch processing queue
  • Cross-platform integration (Yelp, social, campaigns)

FeedbackLearningSystem

  • /lib/services/knowledge/feedback-learning.ts
  • Response tracking
  • Explicit/implicit feedback
  • Mistake pattern detection
  • Automatic retraining

UI Components

KnowledgeBaseManager

  • /components/knowledge-base/KnowledgeBaseManager.tsx
  • CRUD interface for knowledge entries
  • Category management
  • Bulk import/export

Performance Optimizations

Caching Strategy

Embedding Cache:

// 1-hour cache for embeddings
caches.api.set(`embedding:${text.substring(0, 100)}`, embedding, 3600);

Context Cache:

// 5-minute cache for context queries
caches.api.set(`context:${portalId}:${query}`, results, 300);

Search Results Cache:

// 2-minute cache for search results
caches.api.set(`search:${portalId}:${query}`, results, 120);

Category Stats Cache:

// 5-minute in-memory cache
this.categoryCache.set(organizationId, stats);
setTimeout(() => this.categoryCache.delete(organizationId), 5 * 60 * 1000);

Batch Operations

Embedding Generation:

  • Process 10 embeddings in parallel
  • Prevents API rate limiting
  • Reduces latency

Vector Storage:

  • Batch upsert: 100 vectors per request
  • Reduces network overhead
  • Improves throughput

Search Indexing:

  • Queue interactions (30-second batches)
  • Filters trivial messages
  • Reduces write load

Security & Access Control

Multi-Tenant Isolation

Database Level:

  • All queries filtered by portalId
  • Cascade deletes on portal removal
  • Indexed for performance

Vector Storage:

  • Metadata-based filtering
  • Namespace isolation
  • Client override support

Access Control

Knowledge Entry Permissions:

isPublic: boolean;      // Public vs private entries
requiresAuth: boolean;  // Authentication required
channels: string[];     // Channel-specific access

API Authentication:

export const GET = withAuth(async (request, session) => {
  // session.user.id validated
  // portalId ownership verified
});

Rate Limiting

File Uploads:

  • 10 uploads per minute per user
  • Configurable limits

API Requests:

  • Standard rate limiting via middleware
  • Tier-based quotas

Monitoring & Metrics

Learning Metrics

interface LearningMetrics {
  portalId: string;
  totalResponses: number;
  helpfulResponses: number;
  accuracy: number;                    // % helpful responses
  averageSatisfaction: number;        // 1-5 scale
  averageResolutionTime: number;      // seconds
  commonMistakes: Array<{
    pattern: string;
    frequency: number;
    correction: string;
  }>;
}

Alerts:

  • Accuracy < 70% with 20+ responses
  • Triggers automatic retraining
  • Logs to analytics dashboard

Performance Metrics

  • Embedding generation time
  • Vector search latency
  • Cache hit rates
  • Queue processing time
  • Batch operation throughput

Training candidates (mining)

Task 20260613-014, PR 2/3. Owners hand Petunia real calls (uploads or a connected phone system, see CALL_RECORDINGS.md → Training Imports). Once a batch is transcribed and redacted, a worker mines the transcripts into candidates the owner reviews before anything goes live. "Learn" means mining calls into KB, persona and tests; there is no model training. Behind TRAINING_CALLS (default OFF).

Pipeline

TrainingCall (transcribed)             ExtractedData.training_transcript (REDACTED)
        │                                          │
        │  worker "mine" job                       ▼
        │  (auto when the batch is done,   1. per-transcript extraction, one LLM call each
        │   or POST /api/voice/               → strict JSON, zod-validated, retry once,
        │   training-calls/mine)               cached in ExtractedData.training_extraction
        ▼                                          │
   status "mining"                                 ▼
        │                                  2. cross-call clustering (pure, no model)
        │                                     → ≤30 KB candidates ranked by frequency
        │                                     → one tone note (≤200 words, one small LLM call)
        │                                     → one eval scenario per intent cluster
        │                                     → conflictsWith against KnowledgeBaseEntry
        ▼                                          │
   status "mined"  ◄───────────────────────────────┤
   (or "failed" + lastError for a call whose        ▼
    extraction was invalid twice)          3. TrainingCandidate rows, all `pending`
                                              + TrainingRun (counts, tokens, minutes)

Nothing in this phase writes KnowledgeBaseEntry, the voice persona or the eval set. The owner review step (next section) imports approved KB candidates through the KB import path, writes the approved tone note into the voice context field, and stores approved scenarios per portal.

1. Per-transcript extraction

lib/services/voice/training/mining/transcriptExtraction.ts. Input is the redacted transcript only ([phone number], [email], [street number], [card number] placeholders stay as-is). One claude-sonnet-5 call per transcript (MINING_MODEL in training/constants.ts) returns one JSON object validated by transcriptExtractionSchema:

FieldWhat
callerIntent, intentLabelOne sentence + a snake_case label used to cluster intents across calls
callerOpeningLineThe caller's first substantive line, PII-free; becomes the eval scenario's opening turn
questions[]{ question, repAnswer | null, category }, category from a fixed list (pricing, hours, service_area, …)
businessFacts[]{ kind, statement, asQuestion }: hours, service area, prices, policies, brands, as the rep stated them
objections[]{ objection, rebuttal | null }
outcomebooked | quoted | callback | transferred | resolved | no_action | voicemail | unknown
toneMarkers[] (1–5), phrasesToReuse[], phrasesToAvoid[]Feed the tone note; phrases are verbatim
transcriptConfidencehigh | medium | low; low (or the transcriber's own flag) marks the call low-confidence in every candidate's evidence

An answer that is not valid JSON for the schema, or was cut off at max_tokens, is sent back once with the validation error. A second failure throws TranscriptExtractionError; the run marks that TrainingCall failed with the reason in lastError and carries on with the rest. Transport errors (429/5xx) propagate so BullMQ retries the job.

The validated extraction is cached as ExtractedData.dataType = "training_extraction" (replace, never append). A later run over the same portal reuses it, so re-clustering after a few more uploads pays only for the new calls.

2. Cross-call clustering

lib/services/voice/training/mining/candidateClustering.ts, pure functions:

  • KB candidates. Every answered question and every business fact becomes a Q/A item. Items are clustered greedily on question similarity within one category (a pricing question never merges with a scheduling one). Similarity is Jaccard overlap on normalized, stopword-free, lightly stemmed tokens with a floor of two shared tokens (textSimilarity, threshold QUESTION_MATCH_THRESHOLD = 0.5). Text overlap rather than embeddings: candidates are not KnowledgeBaseEntry rows yet, so embedding them would be an OpenAI call per candidate for a decision the owner reviews anyway. The shortest question in a cluster is the label, the longest answer is kept. Ranked by evidence.frequency (distinct calls), capped at MAX_KB_CANDIDATES = 30, deterministic for the same input.
  • Conflicts. detectConflicts() compares each candidate to the portal's live KnowledgeBaseEntry rows: a matching question with a disagreeing answer gets conflictsWith = <entry id>; a matching question with the same answer gets matchesExisting. Neither is dropped; the owner decides.
  • Eval scenarios. One per intent cluster (same intentLabel, or a similar intent sentence). { intent, callerOpeningLine, expectedBehavior, mustNotSay[], evidence }, where expectedBehavior is composed from the cluster's top ranked answer and its usual outcome and mustNotSay is the reps' own phrases-to-avoid from those calls. toHarnessScenario() maps this onto the voice eval harness's EvalScenario (realtime-server/tests/evals/llm/harness.ts): callerTurns = [callerOpeningLine], rubric = [expectedBehavior, …"avoid saying X" lines], category: 'intent', engine: 'fullduplex'; businessContext is left for PR 3 to fill from the portal's KB. The harness has no per-portal scenario source today; this is the smallest shape it loads.
  • Tone note. mining/toneNote.ts aggregates tone markers (by distinct call) and phrases, then one small model call writes ≤200 words in three parts (how the best rep sounds, phrases to reuse, phrases to avoid). On model failure a deterministic note is built from the same aggregates. Always clamped to TONE_NOTE_MAX_WORDS.

3. Persistence and status

lib/services/voice/training/trainingMiningService.ts runs one pass per portal (runTrainingMining): claims transcribed calls → mining, includes already mined calls (cached extractions) so ranking spans the whole set, calls the TrainingMiner, writes TrainingCandidate rows, moves calls to mined (or failed), and closes the TrainingRun. A thrown error puts claimed calls back to transcribed, marks the run failed, and rethrows.

TableRow
TrainingCandidatekind (kb | tone | eval), status (pending | approved | rejected, everything mining writes is pending), category, payload (kb: {question, answer, category, matchesExisting?}; tone: {text, wordCount}; eval: {intent, callerOpeningLine, expectedBehavior, mustNotSay}), evidence ({callRecordIds, frequency, lowConfidenceCallRecordIds}), rank, conflictsWith, reviewedBy/At (null until PR 3)
TrainingRunstatus, trigger (auto | manual), requestedBy, callCount, transcriptsMined, callsFailed, kbCandidates, evalScenarios, toneNote, model, llmCalls, inputTokens, outputTokens, transcriptionMinutes, lastError

A new run replaces the portal's still-pending rows; approved and rejected rows are kept as the audit of what was decided. Migration 20260926000200_training_candidates (additive, guarded).

Triggers. The worker's transcribe job, once its row is transcribed, calls scheduleMiningIfBatchComplete(portalId): when nothing in the portal is still queued, transcribing or importing it queues one mine job, delayed 20 s and deduped per portal per minute (enqueueMining), so a 25-file upload becomes one run. An auto run that finds nothing new is skipped. POST /api/voice/training-calls/mine (owner/admin) queues a manual run, which re-clusters from cache even when nothing is new. GET on the same route returns recent runs with their cost and candidate counts by kind/status (no payloads).

Admin visibility. GET /api/platform/training-candidates?portalId=…[&kind=][&status=] (platform admin, 404 while the flag is off) returns the candidates with payload and evidence plus the last ten runs. This is what the owner review step builds on.

Cost per portal is on the run row: inputTokens / outputTokens for the model calls (one extraction per new transcript, plus one tone call per run) and transcriptionMinutes summed from TrainingCall.durationSec over the calls covered. A 250-call pull of 4-minute calls is roughly 250 × (~1.5k in + ~0.5k out) tokens ≈ 0.4 M in / 0.13 M out on Sonnet, on top of ~1,000 transcription minutes; log lines Mining run completed carry the same numbers.

Key files

  • lib/services/voice/training/trainingMiner.ts — TrainingMiner interface, noopTrainingMiner, createLlmTrainingMiner(), getTrainingMiner() (flag- and key-gated)
  • lib/services/voice/training/mining/{transcriptLlm,transcriptExtraction,candidateClustering,toneNote,types}.ts
  • lib/services/voice/training/trainingMiningService.ts, trainingQueue.ts (mine job), lib/workers/voiceTrainingWorker.ts
  • app/api/voice/training-calls/mine/route.ts, app/api/platform/training-candidates/route.ts
  • Tests: tests/voice/training/{transcriptExtraction,candidateClustering,trainingMiningService,voiceTrainingWorker}.test.ts, tests/integration/voice/training-mining.test.ts (five fixture transcripts, scripted model)

Review and apply (owner review, PR 3)

Task 20260613-014, PR 3/3. The owner decides what goes live. "Review what Petunia learned" is a step in the voice setup wizard (after "Teach Petunia", shown only when the portal has candidates) and the same view inside the Training Calls card under Settings -> Voice (components/voice/TrainingReviewPanel.tsx). Behind TRAINING_CALLS like the rest.

   TrainingCandidate (pending)
        │  approve / edit / reject       PATCH /api/voice/training-calls/review/:id
        │  bulk approve / reject          PATCH /api/voice/training-calls/review
        ▼
   TrainingCandidate (approved, appliedAt null)
        │  "Apply to Petunia"             POST  /api/voice/training-calls/review/apply
        │
        ├─ kb   ──► budget check ──► KnowledgeBaseService.importEntries()  (the import path)
        │            existing + creates ≤ VOICE_KB_ENTRY_CAP, else 409 and nothing written
        ├─ tone ──► AutoresponderSettings.businessContext, between markers
        └─ eval ──► TrainingEvalScenario (harness EvalScenario shape), one row per candidate
        ▼
   TrainingCandidate (approved, appliedAt / appliedBy set)   status never changes after apply

The review state machine

lib/services/voice/training/trainingReviewService.ts, patchCandidate():

ActionEffect
editpayload validated per kind (kb: question/answer/category; tone: text, ≤ TONE_NOTE_MAX_WORDS; eval: intent/opening line/expected behavior/must-not-say); status goes back to pending whatever it was, because changed text needs a fresh approve. matchesExisting is dropped, conflictsWith is kept.
approveapproved. A kb row with conflictsWith cannot be approved without resolveConflict: "replace" → approved and the live entry is updated at apply time; "keep-existing" → the candidate is rejected. Neither → 409 conflict_unresolved. A stored choice carries over to a later bare approve.
rejectrejected.
any, on an applied row409 already_applied. Live rows are frozen; change the KB entry itself.

Every transition stamps reviewedBy / reviewedAt. Bulk approve reports conflicting rows as skipped with the code instead of failing the batch, so the UI can ask for the choice.

The voice KB budget (the rule that matters)

The voice line reads only the newest KNOWLEDGE_ENTRY_CAP KnowledgeBaseEntry rows per portal (realtime-server/services/knowledgeService.ts) and silently never sees the rest. So the importer enforces the same number server-side, before anything is written:

  • N = the portal's entries now, M = approved, unapplied kb candidates that would create a row (a "replace" updates an existing row and does not count; a replace whose target entry has since been deleted becomes a create and does count).
  • N + M > VOICE_KB_ENTRY_CAP → 409 { code: "kb_cap_exceeded", existing: N, selected: M, cap } and nothing is imported. The owner rejects some candidates or removes entries first. The UI footer shows "This portal has N of CAP knowledge entries; importing M" and disables Apply.
  • VOICE_KB_ENTRY_CAP lives in lib/services/voice/training/constants.ts and must equal the realtime-server constant; tests/voice/training/voiceKbCapGuard.test.ts reads that file and fails when they drift. It was 50 until 2026-09-25 and is 150 now. Do not bypass the check in the UI; the server is the gate.

What apply writes, and how

  • Knowledge base. Never a raw KnowledgeBaseEntry insert. Creates go in one call to KnowledgeBaseService.importEntries(entries, portalId, { format: "json" }), the function behind POST /api/knowledge-base/import, which sets id, searchText and the embedding. Each entry is the import schema (category, title, question, answer, keywords, metadata, isPublic: false, requiresAuth: false, channels: []), with metadata = { source: "training-calls", trainingCandidateId, trainingRunId, callRecordIds, frequency } as the reverse audit link. A "replace" goes through the same function with { id: <existing entry>, ...entry } and overwrite: true, i.e. updateEntry(), which re-embeds; the candidate records appliedEntryId. Per-entry import errors leave that candidate approved-but-unapplied and are returned in kb.failed.
  • Tone note. AutoresponderSettings.businessContext is the free-text field the voice prompt already renders as Context: ... (buildKnowledgePromptSection), so the note is written there, between [Learned from your calls: how to sound] and [/Learned from your calls] (toneNoteBlock.ts). A later apply replaces the block; the owner's own text above it is not touched; the field's 10,000-character ceiling is respected (an over-long result is reported, not truncated). No new prompt path. The settings row is created if the portal has none.
  • Eval scenarios. Each approved eval candidate becomes a TrainingEvalScenario row whose scenario column is the harness's EvalScenario from toHarnessScenario(), with businessContext built from the portal's KB after the import and fullDuplex { businessName, businessIndustry, businessCity } from the Business. scenarioId is unique per portal (a second candidate with the same intent gets a -2 suffix). Rows can be enabled or disabled from the review UI; the loader skips disabled ones. Re-applying an edited scenario updates its row.

The budget check is the only thing that aborts the whole apply. After it, the three sections are applied in turn and each reports its own failures (kb.failed, tone.error, eval.failed): an owner who approved twenty answers and a tone note does not lose the answers to a persona write that failed. Marks (appliedAt, appliedBy) are written only for rows whose own write succeeded.

Delete completeness

Deleting a training call (DELETE /api/voice/training-calls/:id) also removes every still-pending candidate whose evidence cites only deleted calls; a pending candidate that also cites a surviving call keeps it. "Delete all" removes every pending candidate. Approved, applied and rejected rows are the owner's decisions and stay, with their evidence.callRecordIds as the audit (candidateCleanup.ts).

Running a portal's scenarios

Stored scenarios are data for the existing harness; the harness's schema is not forked.

  • realtime-server/tests/evals/llm/portalScenarios.ts loads a portal's enabled rows and validates each against the harness shape (zod, strict; a row with an unknown field is reported as invalid, never silently dropped).

  • realtime-server/tests/evals/llm/portalScenarios.eval.test.ts runs them through evaluateScenario() and reports counts (scenarios passed / failed, failing rubric lines per scenario). It needs RUN_LLM_EVALS=1, EVAL_PORTAL_ID and ANTHROPIC_API_KEY; the promotion gate sets only the first, so it stays skipped there. Per the spec these are a report to compare run over run, not a gate: a failing scenario is recorded, and the it fails only with EVAL_PORTAL_STRICT=1.

  • POST /api/platform/voice/eval/run?portalId= (platform admin) creates a TrainingRun with kind: "eval" and queues an eval job on the voice-training queue. The worker (trainingEvalRunner.ts) spawns the harness when it can see it (a full checkout with vitest and the key) and writes evalPassed / evalFailed / evalResults on the run row; the eval test also writes the row itself when EVAL_RUN_ID is set, so a run started by hand lands in the same place. The worker image ships only lib/, so on Fly the row ends unavailable with the exact command in lastError (also echoed as fallbackCommand by the route):

    RUN_LLM_EVALS=1 EVAL_PORTAL_ID=<portal> EVAL_RUN_ID=<run> pnpm test:evals -- realtime-server/tests/evals/llm/portalScenarios.eval.test.ts
    

    GET on the same route lists the portal's eval runs and scenarios. The owner panel shows the last completed run's counts.

Key files

  • lib/services/voice/training/trainingReviewService.ts (state machine, budget, apply), toneNoteBlock.ts, candidateCleanup.ts, trainingEvalRunner.ts, reviewRouteHelpers.ts
  • app/api/voice/training-calls/review/{route,[candidateId]/route,apply/route}.ts, app/api/platform/voice/eval/run/route.ts
  • components/voice/TrainingReviewPanel.tsx; wizard step review in VoiceSetupWizard.tsx; embedded in TrainingCallsPanel.tsx
  • realtime-server/tests/evals/llm/{portalScenarios,portalScenarios.eval.test}.ts
  • Migration 20260926000300_training_review (additive, guarded)
  • Tests: tests/voice/training/{trainingReviewService,candidateCleanup,trainingReviewRoutes,trainingEvalRunner,voiceKbCapGuard}.test.ts, tests/integration/voice/training-review.test.ts

Future Enhancements

Planned Features

  1. Advanced Document Processing:

    • PDF parsing with layout preservation
    • Table extraction
    • Multi-language OCR
  2. Knowledge Graph:

    • Entity relationships
    • Automatic link discovery
    • Visual knowledge explorer
  3. Active Learning:

    • Uncertainty sampling
    • Human-in-the-loop labeling
    • Semi-supervised learning
  4. Fine-Tuned Models:

    • Domain-specific embeddings
    • Portal-specific language models
    • Custom entity recognition
  5. Knowledge Versioning:

    • Track knowledge evolution
    • Rollback capabilities
    • A/B testing for responses

Technical Debt

  1. Vector Storage Optimization:

    • Migrate to dedicated vector DB (Pinecone/Weaviate)
    • Implement approximate nearest neighbor
    • Add vector compression
  2. Search Improvements:

    • Implement BM25 ranking
    • Add faceted search
    • Support fuzzy matching
  3. Scalability:

    • Partition large knowledge bases
    • Implement sharding strategy
    • Add read replicas

References

  • /docs/ARCHITECTURE.md - Overall system architecture
  • /TESTING.md - Testing knowledge services
  • /docs/CLIENT_INTEGRATIONS.md - Third-party integrations

Key Files

Schema:

  • /prisma/schema.prisma - KnowledgeBaseEntry model

Services:

  • /lib/services/knowledgeBaseService.ts
  • /lib/services/knowledge/upstash-knowledge.ts
  • /lib/services/knowledge/upstash-search.ts
  • /lib/services/knowledge/knowledge-integration.ts
  • /lib/services/knowledge/feedback-learning.ts
  • /lib/services/ai/knowledge-ai.ts
  • /lib/services/ai/pinecone.ts
  • /lib/services/file-upload.ts

API Routes:

  • /app/api/autoresponder/knowledge/route.ts
  • /app/api/uploads/ocr/route.ts

Components:

  • /components/knowledge-base/KnowledgeBaseManager.tsx

External Services

  • Upstash Vector: Vector similarity search
  • Upstash Redis: Full-text search and caching
  • Pinecone: Long-term vector storage
  • OpenAI: Embedding generation (text-embedding-3-large)
  • Anthropic: Claude Opus for RAG responses
  • AWS S3: File storage and serving

Changelog

1.1.0 (2026-09-26):

  • Added "Training candidates (mining)": how training-call transcripts become reviewable KB, tone and eval candidates (task 20260613-014, PR 2/3)

1.0.0 (2025-12-13):

  • Initial comprehensive documentation
  • Documented all architecture layers
  • Added code examples and diagrams
  • Included performance optimizations
  • Security and access control details