Knowledge Management Architecture
Overview
Petunia's knowledge management system is a multi-layered architecture that combines vector embeddings, semantic search, and retrieval-augmented generation (RAG) to deliver intelligent, context-aware responses. The system learns from every interaction and continuously improves accuracy through feedback loops.
Version: 1.0.0 Last Modified: 2025-12-13 Status: Production
Architecture Layers
1. Data Storage Layer
KnowledgeBaseEntry (PostgreSQL)
Primary relational storage for structured knowledge entries.
Schema:
model KnowledgeBaseEntry {
id String @id
portalId String
category String
title String
question String
answer String
keywords String[]
metadata Json?
isPublic Boolean @default(false)
requiresAuth Boolean @default(false)
channels String[]
embedding Float[] // Vector embedding for semantic search
searchText String // Full-text search optimization
createdAt DateTime @default(now())
updatedAt DateTime
Portal Portal @relation(fields: [portalId], references: [id], onDelete: Cascade)
@@index([category])
@@index([portalId])
@@index([searchText])
}
Key Features:
- Multi-tenant isolation via
portalId - Category-based organization
- Channel-specific knowledge (Yelp, SMS, email, etc.)
- Access control (public/private, auth requirements)
- Vector embeddings stored inline for quick access
- Optimized search text for keyword fallback
Location: /prisma/schema.prisma (line 3637-3658)
Vector Storage (Upstash Vector)
High-performance vector database for semantic similarity search.
Storage Types:
enum KnowledgeType {
BUSINESS_INFO = 'business_info', // Core business details
CUSTOMER_PROFILE = 'customer_profile', // Customer preferences
CONVERSATION_PATTERN = 'conversation_pattern', // Past interactions
COMPETITOR_INSIGHT = 'competitor_insight', // Competitive intelligence
PRODUCT_SERVICE = 'product_service', // Products/services
FAQ = 'faq', // Common questions
REVIEW_SENTIMENT = 'review_sentiment', // Review analysis
CAMPAIGN_PERFORMANCE = 'campaign_performance', // Marketing data
YELP_REVIEW = 'yelp_review', // Yelp reviews
YELP_MESSAGE = 'yelp_message', // Yelp messages
SOCIAL_MENTION = 'social_mention', // Social media
}
Metadata Structure:
interface KnowledgeMetadata {
portalId: string; // Multi-tenant isolation
type: KnowledgeType; // Knowledge category
category?: string; // Subcategory
source: string; // Origin (onboarding, conversation, review)
timestamp: string; // When stored
relevanceScore?: number; // Quality score
tags?: string[]; // Searchable tags
content: string; // Original text
[key: string]: any; // Flexible metadata
}
Location: /lib/services/knowledge/upstash-knowledge.ts
Search Index (Upstash Redis)
Fast full-text search for messages, customers, and documents.
Document Types:
enum SearchDocumentType {
MESSAGE = 'message',
CUSTOMER = 'customer',
LEAD = 'lead',
REVIEW = 'review',
CAMPAIGN = 'campaign',
FAQ = 'faq',
KNOWLEDGE = 'knowledge',
}
Indexing Strategy:
- Word tokenization with 3-character minimum
- Timeline-based sorting (zset)
- Customer-specific indexes
- Channel-based filtering
- Real-time indexing on write
Location: /lib/services/knowledge/upstash-search.ts
Pinecone Vector Database
Long-term vector storage for voice knowledge and specialized contexts.
Features:
- Namespace isolation
- Metadata filtering
- Voice-optimized retrieval
- Batch operations (100 vectors per batch)
- Automatic fallback mode for testing
Location: /lib/services/ai/pinecone.ts
2. Embedding Layer
OpenAI Embeddings
Model: text-embedding-3-large
Dimensions: 3072
Use Cases:
- Knowledge base entries
- Customer queries
- Business context
- Reviews and feedback
Caching Strategy:
// 1-hour cache for embeddings
const cacheKey = `embedding:${text.substring(0, 100)}`;
const cached = await caches.api.get<number[]>(cacheKey);
if (cached) return cached;
// Generate and cache
const embedding = await openai.embeddings.create({
model: 'text-embedding-3-large',
input: text,
});
await caches.api.set(cacheKey, embedding, 3600);
Deduplication:
// Prevent duplicate embeddings
const dedupeKey = `dedupe:${portalId}:${customerId}:${message.substring(0, 50)}`;
const isDuplicate = await caches.temp.get(dedupeKey);
if (isDuplicate) return;
await caches.temp.set(dedupeKey, true, 3600);
Location: /lib/services/knowledge/upstash-knowledge.ts (generateEmbedding)
3. RAG (Retrieval Augmented Generation)
Knowledge-Enhanced AI Service
Purpose: Combines multiple knowledge sources to generate contextually rich responses.
Context Gathering Pipeline:
async function gatherContext(query: string, context: AIContext) {
// 1. Business Information
const businessContext = await businessKnowledge.getRelevantContext(
portalId, query,
{ types: [KnowledgeType.BUSINESS_INFO, KnowledgeType.PRODUCT_SERVICE] }
);
// 2. Customer History
const customerInsights = await customerKnowledge.getCustomerInsights(
portalId, customerId
);
// 3. Similar Conversations
const similarInteractions = await messageSearch.search(
portalId, query,
{ types: [SearchDocumentType.MESSAGE], limit: 3 }
);
// 4. Relevant FAQs
const relevantFAQs = await customerSearch.search(
portalId, query,
{ types: [SearchDocumentType.FAQ], limit: 2 }
);
// 5. Active Campaigns/Offers
const relevantOffers = await customerSearch.search(
portalId, query,
{ types: [SearchDocumentType.CAMPAIGN], limit: 1 }
);
// 6. Competitor Intelligence
const competitorContext = await businessKnowledge.getRelevantContext(
portalId, query,
{ types: [KnowledgeType.COMPETITOR_INSIGHT], limit: 2 }
);
return { businessContext, customerInsights, similarInteractions,
relevantFAQs, relevantOffers, competitorContext };
}
Enhanced Prompt Construction:
const prompt = `
You are a helpful assistant for ${businessName}.
About the business:
${businessContext.content}
Customer history:
${customerInsights.map(i => i.content).join('\n')}
Similar past responses:
${similarInteractions.map(i => i.content).join('\n')}
Relevant FAQs:
${faqs.map(f => `${f.title}: ${f.content}`).join('\n')}
Current offers:
${offers.map(o => o.content).join('\n')}
Use a ${tone} tone.
Customer question: ${query}
Provide a helpful, accurate response based on the context above.
`;
AI Models:
- Primary: Claude 3 Opus (Anthropic)
- Fallback: GPT-4 Turbo (OpenAI)
- Deterministic Fallback: Context-based template responses
Confidence Scoring:
function calculateConfidence(context: KnowledgeContextData): number {
let score = 0.5; // Base confidence
if (context.business?.content) score += 0.2;
if (context.customer?.insights?.length > 0) score += 0.1;
if (context.similar?.length > 0) score += 0.1;
if (context.faqs?.length > 0) score += 0.05;
if (context.offers?.length > 0) score += 0.05;
return Math.min(1.0, score);
}
Location: /lib/services/ai/knowledge-ai.ts
4. Knowledge Indexing & Search
Semantic Search
Similarity Algorithm: Cosine similarity
function cosineSimilarity(a: number[], b: number[]): number {
if (!a || !b || a.length !== b.length) return 0;
let dotProduct = 0;
let normA = 0;
let normB = 0;
for (let i = 0; i < a.length; i++) {
dotProduct += a[i] * b[i];
normA += a[i] * a[i];
normB += b[i] * b[i];
}
return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}
Search Process:
- Generate query embedding (3072 dimensions)
- Apply metadata filters (portalId, type, categories)
- Compute similarity scores
- Filter by minimum score (default: 0.7)
- Sort by relevance
- Return top-K results (default: 10)
Upstash Vector Query:
const results = await vectorIndex.query({
vector: queryEmbedding,
topK: options?.limit || 10,
includeMetadata: true,
filter: { portalId, type: { $in: types } }
});
Location: /lib/services/knowledge/upstash-knowledge.ts (getRelevantContext)
Keyword Search Fallback
Tokenization:
function tokenize(text: string): string[] {
return text
.toLowerCase()
.replace(/[^\w\s]/g, ' ')
.split(/\s+/)
.filter(word => word.length > 2)
.slice(0, 100); // Limit to prevent excessive indexing
}
Relevance Scoring:
function calculateRelevance(query: string, content: string): number {
const contentLower = content.toLowerCase();
let score = 0;
// Exact match bonus
if (contentLower.includes(query.toLowerCase())) {
score += 10;
}
// Word match scoring
const queryWords = tokenize(query);
for (const word of queryWords) {
const matches = contentLower.match(new RegExp(`\\b${word}\\b`, 'gi'));
if (matches) {
score += matches.length * 2;
}
}
return Math.min(100, score);
}
Location: /lib/services/knowledge/upstash-search.ts
5. Multi-Tenant Knowledge Isolation
Portal-Level Isolation
All knowledge operations require portalId:
// Storage
await businessKnowledge.storeBusinessKnowledge(portalId, data);
// Retrieval
const context = await businessKnowledge.getRelevantContext(portalId, query);
// Search
const results = await messageSearch.search(portalId, query);
// Vector query
await vectorIndex.query({
vector: embedding,
filter: { portalId }, // REQUIRED
});
Database Constraints:
model KnowledgeBaseEntry {
Portal Portal @relation(fields: [portalId], references: [id], onDelete: Cascade)
@@index([portalId])
}
Benefits:
- Hard isolation at data layer
- Cascade deletes when portal removed
- Optimized queries via indexes
- Prevents cross-tenant data leakage
Namespace Isolation (Upstash)
Namespaces:
business- Business context and servicescustomer- Customer profiles and historymessages- Conversation searchcustomers- Customer search- Portal-specific namespaces for specialized data
Client Override Support:
// Use client-specific API keys when available
const service = await UpstashKnowledgeService.getInstance('business', clientId);
Location: /lib/services/knowledge/upstash-knowledge.ts
6. File Upload & Processing Pipeline
Upload Service
Supported File Types:
- Images: JPEG, PNG, GIF
- Documents: PDF, TXT
- Maximum size: 10MB (configurable)
Processing Pipeline:
async function upload(file: Buffer, filename: string, userId: string) {
// 1. Rate limiting (10 uploads/minute)
await uploadRateLimiter.check(userId);
// 2. File validation
const fileType = await detectFileType(file);
validateFileType(fileType, allowedTypes);
validateFileSize(file.length, maxSize);
// 3. Security scanning
await scanFile(file);
// 4. Image processing (if applicable)
if (fileType.startsWith('image/')) {
file = await compressImage(file);
thumbnailKey = await createThumbnail(file, key, userId);
}
// 5. Upload to S3
await s3Client.send(new PutObjectCommand({
Bucket: bucket,
Key: key,
Body: file,
ContentType: fileType,
ServerSideEncryption: 'AES256',
}));
return { url, key, size, type, thumbnailUrl };
}
Image Optimization:
// Compression
const compressed = await sharp(file)
.jpeg({ quality: 85, progressive: true })
.toBuffer();
// Thumbnail generation
const thumbnail = await sharp(file)
.resize(200, 200, { fit: 'cover' })
.jpeg({ quality: 80 })
.toBuffer();
Location: /lib/services/file-upload.ts
Document Processing
OCR Integration:
- Endpoint:
/app/api/uploads/ocr/route.ts - Use case: Extract text from images and PDFs
- Service: AWS Textract or similar
Knowledge Extraction:
- Upload document
- Extract text via OCR
- Split into chunks (8000 chars max)
- Generate embeddings for each chunk
- Store in vector database with metadata
- Index for search
7. Knowledge Integration Layer
Real-Time Learning Hooks
AutoResponder Integration:
async function onAutoResponderMessage(data: {
portalId: string;
customerId: string;
incomingMessage: string;
generatedResponse: string;
sentiment?: string;
}) {
// Queue for batch processing
queueInteraction(portalId, {
customerId, message: incomingMessage,
response: generatedResponse, sentiment
});
// Index immediately for search
await messageSearch.indexDocument({
id: `auto_${portalId}_${Date.now()}`,
portalId, customerId, type: SearchDocumentType.MESSAGE,
content: incomingMessage
});
}
Yelp Integration:
// Reviews
async function onYelpReview(data: {
portalId: string;
reviewId: string;
rating: number;
text: string;
}) {
// Store review with sentiment
await businessKnowledge.storeYelpReview(portalId, data);
// Extract competitor mentions
const competitors = await getCompetitorKeywords(portalId);
for (const competitor of competitors) {
if (text.toLowerCase().includes(competitor)) {
await businessKnowledge.storeCompetitorInsight(portalId, {
competitorName: competitor,
observation: text
});
}
}
// Analyze negative reviews
if (rating <= 2) {
await analyzeNegativeFeedback(portalId, text);
}
}
// Messages
async function onYelpMessage(data: {
portalId: string;
conversationId: string;
message: string;
isFromBusiness: boolean;
}) {
if (isFromBusiness) return; // Only learn from customers
const conversation = await prisma.yelpConversation.findUnique({
where: { id: conversationId }
});
queueInteraction(portalId, {
customerId: conversation.userId,
message: data.message,
channel: 'yelp'
});
}
Social Media Integration:
async function onSocialMention(data: {
portalId: string;
platform: 'facebook' | 'instagram' | 'twitter';
content: string;
authorId: string;
sentiment?: string;
}) {
queueInteraction(portalId, {
customerId: authorId,
message: content,
channel: platform,
sentiment
});
}
Campaign Learning:
async function onCampaignComplete(data: {
portalId: string;
campaignName: string;
performance: { sent, opened, clicked, converted };
bestPerformingMessage?: string;
}) {
const context = `
Campaign "${campaignName}" Results:
Open Rate: ${(opened/sent * 100).toFixed(1)}%
Click Rate: ${(clicked/opened * 100).toFixed(1)}%
Conversion: ${(converted/clicked * 100).toFixed(1)}%
Best Message: ${bestPerformingMessage}
`;
await businessKnowledge.storeBusinessKnowledge(portalId, {
description: context,
source: 'campaign_performance'
});
}
Location: /lib/services/knowledge/knowledge-integration.ts
Batch Processing
Queue-Based Processing:
class KnowledgeIntegration {
private processingQueue: Map<string, Interaction[]> = new Map();
private batchTimer: NodeJS.Timeout;
startBatchProcessor() {
// Process every 30 seconds
this.batchTimer = setInterval(async () => {
for (const [portalId, interactions] of this.processingQueue) {
// Filter trivial interactions
const meaningful = interactions.filter(i =>
i.message && i.message.length > 10
);
if (meaningful.length > 0) {
await businessKnowledge.batchStoreInteractions(
portalId, meaningful
);
}
}
this.processingQueue.clear();
}, 30000);
}
}
Benefits:
- Reduces API calls
- Prevents rate limiting
- Improves throughput
- Batches embeddings
8. Feedback Learning System
Response Tracking
Track Every AI Response:
async function trackResponse(data: {
portalId: string;
customerId: string;
query: string;
response: string;
confidence: number;
}): Promise<string> {
const responseId = `resp_${Date.now()}_${random()}`;
await AIResponseTracking.create({
id: responseId,
portalId, customerId, query, response,
confidence, status: 'pending_feedback'
});
// Check implicit feedback after 5 minutes
setTimeout(() => {
checkImplicitFeedback(responseId, portalId, customerId);
}, 5 * 60 * 1000);
return responseId;
}
Explicit Feedback
User Corrections:
interface ResponseFeedback {
responseId: string;
portalId: string;
originalQuery: string;
generatedResponse: string;
wasHelpful: boolean;
humanCorrection?: string;
customerSatisfaction?: 1 | 2 | 3 | 4 | 5;
resolutionTime?: number;
requiredHumanIntervention: boolean;
}
async function recordFeedback(feedback: ResponseFeedback) {
// Queue for batch processing
learningQueue.get(portalId).push(feedback);
// Update tracking record
await AIResponseTracking.update({
where: { id: feedback.responseId },
data: { wasHelpful, humanCorrection, satisfaction }
});
// Learn from correction immediately
if (humanCorrection && !wasHelpful) {
await learnFromCorrection(feedback);
}
}
Implicit Feedback
Detect Satisfaction from Behavior:
async function checkImplicitFeedback(
responseId: string,
portalId: string,
customerId: string
) {
// Check if customer sent more messages
const followUpMessages = await prisma.message.findMany({
where: { portalId, contactId: customerId,
createdAt: { gte: new Date(Date.now() - 5 * 60 * 1000) }
}
});
// No follow-up = satisfactory
const wasHelpful = followUpMessages.length === 0;
// Check for clarification requests
const needsClarification = detectClarificationPhrases(
followUpMessages.map(m => m.content).join(' ')
);
await recordFeedback({
responseId, portalId,
wasHelpful: wasHelpful && !needsClarification,
requiredHumanIntervention: needsClarification
});
}
const clarificationPhrases = [
'what do you mean', 'i don\'t understand',
'can you explain', 'not what i asked',
'that doesn\'t help', 'confused'
];
Learning from Mistakes
Pattern Identification:
function identifyMistakePattern(
query: string,
wrongResponse: string,
correctResponse: string
): { type: string; severity: 'low' | 'medium' | 'high' } {
// Factual errors
if (wrongResponse.includes('open') && correctResponse.includes('closed')) {
return { type: 'factual_error', severity: 'high' };
}
// Missing information
if (correctResponse.length > wrongResponse.length * 1.5) {
return { type: 'incomplete_response', severity: 'medium' };
}
// Missed intent
if (query.includes('price') && !wrongResponse.includes('price')) {
return { type: 'missed_intent', severity: 'high' };
}
// Tone issues
if (wrongResponse.includes('sorry') && !correctResponse.includes('sorry')) {
return { type: 'over_apologetic', severity: 'low' };
}
return { type: 'general_mistake', severity: 'medium' };
}
Store Corrections:
async function learnFromCorrection(feedback: ResponseFeedback) {
// Store as high-priority learning example
await businessKnowledge.storeBusinessKnowledge(portalId, {
businessName: 'Learning Example',
industry: 'correction',
description: `
When asked: "${feedback.originalQuery}",
DON'T say: "${feedback.generatedResponse}",
INSTEAD say: "${feedback.humanCorrection}"
`
});
// Track mistake patterns
const pattern = identifyMistakePattern(
feedback.originalQuery,
feedback.generatedResponse,
feedback.humanCorrection
);
const patterns = await caches.config.get<MistakePattern[]>(
`mistakes:${portalId}`
) || [];
const existing = patterns.find(p => p.pattern === pattern.type);
if (existing) {
existing.frequency++;
existing.examples.push({
query: feedback.originalQuery,
wrong: feedback.generatedResponse,
correct: feedback.humanCorrection
});
} else {
patterns.push({ pattern: pattern.type, frequency: 1, examples: [...] });
}
await caches.config.set(`mistakes:${portalId}`, patterns, 86400 * 7);
}
Automatic Retraining
Trigger When Accuracy Drops:
async function updateMetrics(portalId: string) {
const feedback = await AIResponseTracking.findMany({
where: { portalId, status: 'feedback_received',
createdAt: { gte: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000) }
}
});
const accuracy = feedback.filter(f => f.wasHelpful).length / feedback.length;
// Alert if accuracy < 70% with sufficient data
if (accuracy < 0.7 && feedback.length > 20) {
logger.warn('AI accuracy below threshold', { portalId, accuracy });
await triggerRetraining(portalId);
}
}
async function triggerRetraining(portalId: string) {
// Gather all corrections
const corrections = await AIResponseTracking.findMany({
where: { portalId, humanCorrection: { not: null }, wasHelpful: false },
take: 50,
orderBy: { createdAt: 'desc' }
});
// Create training examples
for (const correction of corrections) {
await businessKnowledge.storeBusinessKnowledge(portalId, {
businessName: 'Training Data',
industry: 'retraining',
description: `Q: ${correction.query}\nA: ${correction.humanCorrection}`
});
}
// Clear caches to use new knowledge
await caches.api.invalidatePattern(`context:${portalId}:*`);
}
Location: /lib/services/knowledge/feedback-learning.ts
API Routes
Knowledge Base Management
GET /api/autoresponder/knowledge
- Get all knowledge entries or specific entry by ID
- Query params:
businessId,id(optional) - Returns:
{ data: { knowledge | knowledgeBase }, success: true }
POST /api/autoresponder/knowledge
- Create or update knowledge entry
- Body:
{ businessId, entry: { title, content, category, tags } } - Tier limits enforced (prevents bypass via fake IDs)
- Returns:
{ data: { entry }, success: true }
DELETE /api/autoresponder/knowledge
- Delete knowledge entry
- Query params:
businessId,id - Returns:
{ data: { success: true }, success: true }
Location: /app/api/autoresponder/knowledge/route.ts
Services & Components
Core Services
KnowledgeBaseService
/lib/services/knowledgeBaseService.ts- PostgreSQL-backed knowledge management
- Semantic search with OpenAI embeddings
- Import/export (CSV, JSON)
- Category statistics
UpstashKnowledgeService
/lib/services/knowledge/upstash-knowledge.ts- Vector-based knowledge storage
- Real-time embedding generation
- Business/customer namespace isolation
- Sentiment analysis
UpstashSearchService
/lib/services/knowledge/upstash-search.ts- Full-text search with Redis
- Real-time indexing
- Multi-field queries
- Autocomplete suggestions
KnowledgeEnhancedAI
/lib/services/ai/knowledge-ai.ts- RAG implementation
- Multi-source context aggregation
- Claude/GPT-4 integration
- Confidence scoring
KnowledgeIntegration
/lib/services/knowledge/knowledge-integration.ts- Real-time learning hooks
- Batch processing queue
- Cross-platform integration (Yelp, social, campaigns)
FeedbackLearningSystem
/lib/services/knowledge/feedback-learning.ts- Response tracking
- Explicit/implicit feedback
- Mistake pattern detection
- Automatic retraining
UI Components
KnowledgeBaseManager
/components/knowledge-base/KnowledgeBaseManager.tsx- CRUD interface for knowledge entries
- Category management
- Bulk import/export
Performance Optimizations
Caching Strategy
Embedding Cache:
// 1-hour cache for embeddings
caches.api.set(`embedding:${text.substring(0, 100)}`, embedding, 3600);
Context Cache:
// 5-minute cache for context queries
caches.api.set(`context:${portalId}:${query}`, results, 300);
Search Results Cache:
// 2-minute cache for search results
caches.api.set(`search:${portalId}:${query}`, results, 120);
Category Stats Cache:
// 5-minute in-memory cache
this.categoryCache.set(organizationId, stats);
setTimeout(() => this.categoryCache.delete(organizationId), 5 * 60 * 1000);
Batch Operations
Embedding Generation:
- Process 10 embeddings in parallel
- Prevents API rate limiting
- Reduces latency
Vector Storage:
- Batch upsert: 100 vectors per request
- Reduces network overhead
- Improves throughput
Search Indexing:
- Queue interactions (30-second batches)
- Filters trivial messages
- Reduces write load
Security & Access Control
Multi-Tenant Isolation
Database Level:
- All queries filtered by
portalId - Cascade deletes on portal removal
- Indexed for performance
Vector Storage:
- Metadata-based filtering
- Namespace isolation
- Client override support
Access Control
Knowledge Entry Permissions:
isPublic: boolean; // Public vs private entries
requiresAuth: boolean; // Authentication required
channels: string[]; // Channel-specific access
API Authentication:
export const GET = withAuth(async (request, session) => {
// session.user.id validated
// portalId ownership verified
});
Rate Limiting
File Uploads:
- 10 uploads per minute per user
- Configurable limits
API Requests:
- Standard rate limiting via middleware
- Tier-based quotas
Monitoring & Metrics
Learning Metrics
interface LearningMetrics {
portalId: string;
totalResponses: number;
helpfulResponses: number;
accuracy: number; // % helpful responses
averageSatisfaction: number; // 1-5 scale
averageResolutionTime: number; // seconds
commonMistakes: Array<{
pattern: string;
frequency: number;
correction: string;
}>;
}
Alerts:
- Accuracy < 70% with 20+ responses
- Triggers automatic retraining
- Logs to analytics dashboard
Performance Metrics
- Embedding generation time
- Vector search latency
- Cache hit rates
- Queue processing time
- Batch operation throughput
Training candidates (mining)
Task 20260613-014, PR 2/3. Owners hand Petunia real calls (uploads or a connected phone
system, see CALL_RECORDINGS.md → Training Imports). Once a batch is transcribed and
redacted, a worker mines the transcripts into candidates the owner reviews before anything
goes live. "Learn" means mining calls into KB, persona and tests; there is no model training.
Behind TRAINING_CALLS (default OFF).
Pipeline
TrainingCall (transcribed) ExtractedData.training_transcript (REDACTED)
│ │
│ worker "mine" job ▼
│ (auto when the batch is done, 1. per-transcript extraction, one LLM call each
│ or POST /api/voice/ → strict JSON, zod-validated, retry once,
│ training-calls/mine) cached in ExtractedData.training_extraction
▼ │
status "mining" ▼
│ 2. cross-call clustering (pure, no model)
│ → ≤30 KB candidates ranked by frequency
│ → one tone note (≤200 words, one small LLM call)
│ → one eval scenario per intent cluster
│ → conflictsWith against KnowledgeBaseEntry
▼ │
status "mined" ◄───────────────────────────────┤
(or "failed" + lastError for a call whose ▼
extraction was invalid twice) 3. TrainingCandidate rows, all `pending`
+ TrainingRun (counts, tokens, minutes)
Nothing in this phase writes KnowledgeBaseEntry, the voice persona or the eval set. The
owner review step (next section) imports approved KB candidates through the KB import path,
writes the approved tone note into the voice context field, and stores approved scenarios per
portal.
1. Per-transcript extraction
lib/services/voice/training/mining/transcriptExtraction.ts. Input is the redacted
transcript only ([phone number], [email], [street number], [card number] placeholders
stay as-is). One claude-sonnet-5 call per transcript (MINING_MODEL in
training/constants.ts) returns one JSON object validated by transcriptExtractionSchema:
| Field | What |
|---|---|
callerIntent, intentLabel | One sentence + a snake_case label used to cluster intents across calls |
callerOpeningLine | The caller's first substantive line, PII-free; becomes the eval scenario's opening turn |
questions[] | { question, repAnswer | null, category }, category from a fixed list (pricing, hours, service_area, …) |
businessFacts[] | { kind, statement, asQuestion }: hours, service area, prices, policies, brands, as the rep stated them |
objections[] | { objection, rebuttal | null } |
outcome | booked | quoted | callback | transferred | resolved | no_action | voicemail | unknown |
toneMarkers[] (1–5), phrasesToReuse[], phrasesToAvoid[] | Feed the tone note; phrases are verbatim |
transcriptConfidence | high | medium | low; low (or the transcriber's own flag) marks the call low-confidence in every candidate's evidence |
An answer that is not valid JSON for the schema, or was cut off at max_tokens, is sent back
once with the validation error. A second failure throws TranscriptExtractionError; the
run marks that TrainingCall failed with the reason in lastError and carries on with the
rest. Transport errors (429/5xx) propagate so BullMQ retries the job.
The validated extraction is cached as ExtractedData.dataType = "training_extraction"
(replace, never append). A later run over the same portal reuses it, so re-clustering after a
few more uploads pays only for the new calls.
2. Cross-call clustering
lib/services/voice/training/mining/candidateClustering.ts, pure functions:
- KB candidates. Every answered question and every business fact becomes a Q/A item.
Items are clustered greedily on question similarity within one category (a pricing
question never merges with a scheduling one). Similarity is Jaccard overlap on normalized,
stopword-free, lightly stemmed tokens with a floor of two shared tokens
(
textSimilarity, thresholdQUESTION_MATCH_THRESHOLD = 0.5). Text overlap rather than embeddings: candidates are notKnowledgeBaseEntryrows yet, so embedding them would be an OpenAI call per candidate for a decision the owner reviews anyway. The shortest question in a cluster is the label, the longest answer is kept. Ranked byevidence.frequency(distinct calls), capped atMAX_KB_CANDIDATES = 30, deterministic for the same input. - Conflicts.
detectConflicts()compares each candidate to the portal's liveKnowledgeBaseEntryrows: a matching question with a disagreeing answer getsconflictsWith = <entry id>; a matching question with the same answer getsmatchesExisting. Neither is dropped; the owner decides. - Eval scenarios. One per intent cluster (same
intentLabel, or a similar intent sentence).{ intent, callerOpeningLine, expectedBehavior, mustNotSay[], evidence }, whereexpectedBehavioris composed from the cluster's top ranked answer and its usual outcome andmustNotSayis the reps' own phrases-to-avoid from those calls.toHarnessScenario()maps this onto the voice eval harness'sEvalScenario(realtime-server/tests/evals/llm/harness.ts):callerTurns = [callerOpeningLine],rubric = [expectedBehavior, …"avoid saying X" lines],category: 'intent',engine: 'fullduplex';businessContextis left for PR 3 to fill from the portal's KB. The harness has no per-portal scenario source today; this is the smallest shape it loads. - Tone note.
mining/toneNote.tsaggregates tone markers (by distinct call) and phrases, then one small model call writes ≤200 words in three parts (how the best rep sounds, phrases to reuse, phrases to avoid). On model failure a deterministic note is built from the same aggregates. Always clamped toTONE_NOTE_MAX_WORDS.
3. Persistence and status
lib/services/voice/training/trainingMiningService.ts runs one pass per portal
(runTrainingMining): claims transcribed calls → mining, includes already mined calls
(cached extractions) so ranking spans the whole set, calls the TrainingMiner, writes
TrainingCandidate rows, moves calls to mined (or failed), and closes the TrainingRun.
A thrown error puts claimed calls back to transcribed, marks the run failed, and rethrows.
| Table | Row |
|---|---|
TrainingCandidate | kind (kb | tone | eval), status (pending | approved | rejected, everything mining writes is pending), category, payload (kb: {question, answer, category, matchesExisting?}; tone: {text, wordCount}; eval: {intent, callerOpeningLine, expectedBehavior, mustNotSay}), evidence ({callRecordIds, frequency, lowConfidenceCallRecordIds}), rank, conflictsWith, reviewedBy/At (null until PR 3) |
TrainingRun | status, trigger (auto | manual), requestedBy, callCount, transcriptsMined, callsFailed, kbCandidates, evalScenarios, toneNote, model, llmCalls, inputTokens, outputTokens, transcriptionMinutes, lastError |
A new run replaces the portal's still-pending rows; approved and rejected rows are kept as
the audit of what was decided. Migration 20260926000200_training_candidates (additive,
guarded).
Triggers. The worker's transcribe job, once its row is transcribed, calls
scheduleMiningIfBatchComplete(portalId): when nothing in the portal is still queued,
transcribing or importing it queues one mine job, delayed 20 s and deduped per portal per
minute (enqueueMining), so a 25-file upload becomes one run. An auto run that finds nothing
new is skipped. POST /api/voice/training-calls/mine (owner/admin) queues a manual run, which
re-clusters from cache even when nothing is new. GET on the same route returns recent runs
with their cost and candidate counts by kind/status (no payloads).
Admin visibility. GET /api/platform/training-candidates?portalId=…[&kind=][&status=]
(platform admin, 404 while the flag is off) returns the candidates with payload and evidence
plus the last ten runs. This is what the owner review step builds on.
Cost per portal is on the run row: inputTokens / outputTokens for the model calls
(one extraction per new transcript, plus one tone call per run) and transcriptionMinutes
summed from TrainingCall.durationSec over the calls covered. A 250-call pull of 4-minute
calls is roughly 250 × (~1.5k in + ~0.5k out) tokens ≈ 0.4 M in / 0.13 M out on Sonnet, on
top of ~1,000 transcription minutes; log lines Mining run completed carry the same numbers.
Key files
lib/services/voice/training/trainingMiner.ts—TrainingMinerinterface,noopTrainingMiner,createLlmTrainingMiner(),getTrainingMiner()(flag- and key-gated)lib/services/voice/training/mining/{transcriptLlm,transcriptExtraction,candidateClustering,toneNote,types}.tslib/services/voice/training/trainingMiningService.ts,trainingQueue.ts(minejob),lib/workers/voiceTrainingWorker.tsapp/api/voice/training-calls/mine/route.ts,app/api/platform/training-candidates/route.ts- Tests:
tests/voice/training/{transcriptExtraction,candidateClustering,trainingMiningService,voiceTrainingWorker}.test.ts,tests/integration/voice/training-mining.test.ts(five fixture transcripts, scripted model)
Review and apply (owner review, PR 3)
Task 20260613-014, PR 3/3. The owner decides what goes live. "Review what Petunia learned" is
a step in the voice setup wizard (after "Teach Petunia", shown only when the portal has
candidates) and the same view inside the Training Calls card under Settings -> Voice
(components/voice/TrainingReviewPanel.tsx). Behind TRAINING_CALLS like the rest.
TrainingCandidate (pending)
│ approve / edit / reject PATCH /api/voice/training-calls/review/:id
│ bulk approve / reject PATCH /api/voice/training-calls/review
▼
TrainingCandidate (approved, appliedAt null)
│ "Apply to Petunia" POST /api/voice/training-calls/review/apply
│
├─ kb ──► budget check ──► KnowledgeBaseService.importEntries() (the import path)
│ existing + creates ≤ VOICE_KB_ENTRY_CAP, else 409 and nothing written
├─ tone ──► AutoresponderSettings.businessContext, between markers
└─ eval ──► TrainingEvalScenario (harness EvalScenario shape), one row per candidate
▼
TrainingCandidate (approved, appliedAt / appliedBy set) status never changes after apply
The review state machine
lib/services/voice/training/trainingReviewService.ts, patchCandidate():
| Action | Effect |
|---|---|
edit | payload validated per kind (kb: question/answer/category; tone: text, ≤ TONE_NOTE_MAX_WORDS; eval: intent/opening line/expected behavior/must-not-say); status goes back to pending whatever it was, because changed text needs a fresh approve. matchesExisting is dropped, conflictsWith is kept. |
approve | approved. A kb row with conflictsWith cannot be approved without resolveConflict: "replace" → approved and the live entry is updated at apply time; "keep-existing" → the candidate is rejected. Neither → 409 conflict_unresolved. A stored choice carries over to a later bare approve. |
reject | rejected. |
| any, on an applied row | 409 already_applied. Live rows are frozen; change the KB entry itself. |
Every transition stamps reviewedBy / reviewedAt. Bulk approve reports conflicting rows as
skipped with the code instead of failing the batch, so the UI can ask for the choice.
The voice KB budget (the rule that matters)
The voice line reads only the newest KNOWLEDGE_ENTRY_CAP KnowledgeBaseEntry rows per
portal (realtime-server/services/knowledgeService.ts) and silently never sees the rest. So the
importer enforces the same number server-side, before anything is written:
N= the portal's entries now,M= approved, unapplied kb candidates that would create a row (a"replace"updates an existing row and does not count; a replace whose target entry has since been deleted becomes a create and does count).N + M > VOICE_KB_ENTRY_CAP→409 { code: "kb_cap_exceeded", existing: N, selected: M, cap }and nothing is imported. The owner rejects some candidates or removes entries first. The UI footer shows "This portal has N of CAP knowledge entries; importing M" and disables Apply.VOICE_KB_ENTRY_CAPlives inlib/services/voice/training/constants.tsand must equal the realtime-server constant;tests/voice/training/voiceKbCapGuard.test.tsreads that file and fails when they drift. It was 50 until 2026-09-25 and is 150 now. Do not bypass the check in the UI; the server is the gate.
What apply writes, and how
- Knowledge base. Never a raw
KnowledgeBaseEntryinsert. Creates go in one call toKnowledgeBaseService.importEntries(entries, portalId, { format: "json" }), the function behindPOST /api/knowledge-base/import, which setsid,searchTextand the embedding. Each entry is the import schema (category,title,question,answer,keywords,metadata,isPublic: false,requiresAuth: false,channels: []), withmetadata = { source: "training-calls", trainingCandidateId, trainingRunId, callRecordIds, frequency }as the reverse audit link. A"replace"goes through the same function with{ id: <existing entry>, ...entry }andoverwrite: true, i.e.updateEntry(), which re-embeds; the candidate recordsappliedEntryId. Per-entry import errors leave that candidate approved-but-unapplied and are returned inkb.failed. - Tone note.
AutoresponderSettings.businessContextis the free-text field the voice prompt already renders asContext: ...(buildKnowledgePromptSection), so the note is written there, between[Learned from your calls: how to sound]and[/Learned from your calls](toneNoteBlock.ts). A later apply replaces the block; the owner's own text above it is not touched; the field's 10,000-character ceiling is respected (an over-long result is reported, not truncated). No new prompt path. The settings row is created if the portal has none. - Eval scenarios. Each approved
evalcandidate becomes aTrainingEvalScenariorow whosescenariocolumn is the harness'sEvalScenariofromtoHarnessScenario(), withbusinessContextbuilt from the portal's KB after the import andfullDuplex{ businessName, businessIndustry, businessCity }from the Business.scenarioIdis unique per portal (a second candidate with the same intent gets a-2suffix). Rows can be enabled or disabled from the review UI; the loader skips disabled ones. Re-applying an edited scenario updates its row.
The budget check is the only thing that aborts the whole apply. After it, the three sections
are applied in turn and each reports its own failures (kb.failed, tone.error,
eval.failed): an owner who approved twenty answers and a tone note does not lose the answers
to a persona write that failed. Marks (appliedAt, appliedBy) are written only for rows
whose own write succeeded.
Delete completeness
Deleting a training call (DELETE /api/voice/training-calls/:id) also removes every
still-pending candidate whose evidence cites only deleted calls; a pending candidate that
also cites a surviving call keeps it. "Delete all" removes every pending candidate. Approved,
applied and rejected rows are the owner's decisions and stay, with their evidence.callRecordIds
as the audit (candidateCleanup.ts).
Running a portal's scenarios
Stored scenarios are data for the existing harness; the harness's schema is not forked.
-
realtime-server/tests/evals/llm/portalScenarios.tsloads a portal's enabled rows and validates each against the harness shape (zod, strict; a row with an unknown field is reported as invalid, never silently dropped). -
realtime-server/tests/evals/llm/portalScenarios.eval.test.tsruns them throughevaluateScenario()and reports counts (scenarios passed / failed, failing rubric lines per scenario). It needsRUN_LLM_EVALS=1,EVAL_PORTAL_IDandANTHROPIC_API_KEY; the promotion gate sets only the first, so it stays skipped there. Per the spec these are a report to compare run over run, not a gate: a failing scenario is recorded, and theitfails only withEVAL_PORTAL_STRICT=1. -
POST /api/platform/voice/eval/run?portalId=(platform admin) creates aTrainingRunwithkind: "eval"and queues anevaljob on thevoice-trainingqueue. The worker (trainingEvalRunner.ts) spawns the harness when it can see it (a full checkout with vitest and the key) and writesevalPassed/evalFailed/evalResultson the run row; the eval test also writes the row itself whenEVAL_RUN_IDis set, so a run started by hand lands in the same place. The worker image ships onlylib/, so on Fly the row endsunavailablewith the exact command inlastError(also echoed asfallbackCommandby the route):RUN_LLM_EVALS=1 EVAL_PORTAL_ID=<portal> EVAL_RUN_ID=<run> pnpm test:evals -- realtime-server/tests/evals/llm/portalScenarios.eval.test.tsGETon the same route lists the portal's eval runs and scenarios. The owner panel shows the last completed run's counts.
Key files
lib/services/voice/training/trainingReviewService.ts(state machine, budget, apply),toneNoteBlock.ts,candidateCleanup.ts,trainingEvalRunner.ts,reviewRouteHelpers.tsapp/api/voice/training-calls/review/{route,[candidateId]/route,apply/route}.ts,app/api/platform/voice/eval/run/route.tscomponents/voice/TrainingReviewPanel.tsx; wizard stepreviewinVoiceSetupWizard.tsx; embedded inTrainingCallsPanel.tsxrealtime-server/tests/evals/llm/{portalScenarios,portalScenarios.eval.test}.ts- Migration
20260926000300_training_review(additive, guarded) - Tests:
tests/voice/training/{trainingReviewService,candidateCleanup,trainingReviewRoutes,trainingEvalRunner,voiceKbCapGuard}.test.ts,tests/integration/voice/training-review.test.ts
Future Enhancements
Planned Features
-
Advanced Document Processing:
- PDF parsing with layout preservation
- Table extraction
- Multi-language OCR
-
Knowledge Graph:
- Entity relationships
- Automatic link discovery
- Visual knowledge explorer
-
Active Learning:
- Uncertainty sampling
- Human-in-the-loop labeling
- Semi-supervised learning
-
Fine-Tuned Models:
- Domain-specific embeddings
- Portal-specific language models
- Custom entity recognition
-
Knowledge Versioning:
- Track knowledge evolution
- Rollback capabilities
- A/B testing for responses
Technical Debt
-
Vector Storage Optimization:
- Migrate to dedicated vector DB (Pinecone/Weaviate)
- Implement approximate nearest neighbor
- Add vector compression
-
Search Improvements:
- Implement BM25 ranking
- Add faceted search
- Support fuzzy matching
-
Scalability:
- Partition large knowledge bases
- Implement sharding strategy
- Add read replicas
References
Related Documentation
/docs/ARCHITECTURE.md- Overall system architecture/TESTING.md- Testing knowledge services/docs/CLIENT_INTEGRATIONS.md- Third-party integrations
Key Files
Schema:
/prisma/schema.prisma- KnowledgeBaseEntry model
Services:
/lib/services/knowledgeBaseService.ts/lib/services/knowledge/upstash-knowledge.ts/lib/services/knowledge/upstash-search.ts/lib/services/knowledge/knowledge-integration.ts/lib/services/knowledge/feedback-learning.ts/lib/services/ai/knowledge-ai.ts/lib/services/ai/pinecone.ts/lib/services/file-upload.ts
API Routes:
/app/api/autoresponder/knowledge/route.ts/app/api/uploads/ocr/route.ts
Components:
/components/knowledge-base/KnowledgeBaseManager.tsx
External Services
- Upstash Vector: Vector similarity search
- Upstash Redis: Full-text search and caching
- Pinecone: Long-term vector storage
- OpenAI: Embedding generation (text-embedding-3-large)
- Anthropic: Claude Opus for RAG responses
- AWS S3: File storage and serving
Changelog
1.1.0 (2026-09-26):
- Added "Training candidates (mining)": how training-call transcripts become reviewable KB, tone and eval candidates (task 20260613-014, PR 2/3)
1.0.0 (2025-12-13):
- Initial comprehensive documentation
- Documented all architecture layers
- Added code examples and diagrams
- Included performance optimizations
- Security and access control details