Outcome Tracking

Outcome Tracking automatically records what happens after your AI takes actions, helping you measure effectiveness and improve performance over time.

What is Outcome Tracking?

Outcome Tracking is a lightweight system that records:

• What your AI did (actions like booking appointments, sending messages, triggering escalations)
• What happened afterward (did the customer reply? was an appointment created? was there a handoff?)
• When it happened (timestamps for both action and outcome)

This data helps you:

• Measure effectiveness: See which AI actions lead to desired outcomes
• Identify patterns: Understand what works and what doesn't
• Improve over time: Use real results to guide AI improvements
• Make data-driven decisions: Base changes on actual performance data

How It Works

Automatic Recording

Outcome Tracking works automatically in the background:

• No configuration needed: Starts working as soon as it's enabled
• Non-blocking: Recording failures don't affect your main workflows
• Privacy-safe: Sensitive information is automatically removed

What Gets Tracked

Currently, the system tracks:

Appointment Booking

When your AI successfully books an appointment:

• Action: tool_call:book_appointment
• Outcome: appointment_created
• Details: Whether it was a Calendly booking or follow-up mode

Escalations & Handoffs

When a conversation is escalated to a human:

• Action: escalation
• Outcome: handoff_triggered
• Details: Risk level (high_risk or serious_mode), category, confidence

Future Tracking

Additional outcomes will be added over time:

• Customer reply tracking (did customer respond after AI message?)
• Deal stage changes (did conversation move to next stage?)
• Other AI action outcomes

Understanding Outcome Data

Action Types

Actions are categorized by type:

• Tool calls: tool_call:book_appointment, tool_call:qualify_lead, etc.
• Escalations: escalation
• AI messages: ai_message (future)

Outcome Types

Outcomes describe what happened:

• Appointment created: Appointment was successfully booked
• Handoff triggered: Conversation was escalated to human
• Customer replied: Customer responded (future)
• Deal stage changed: Conversation progressed (future)

Outcome Values

Outcome values provide additional context:

• For appointments: calendly or followup_mode
• For escalations: high_risk or serious_mode
• For other outcomes: Relevant context (e.g., time window, stage name)

Metadata

Each outcome includes optional metadata:

• Appointments: Invitee URI, event type, scheduling URL
• Escalations: Category, confidence, risk category
• Other: Context-specific information

Using Outcome Data

Viewing Outcomes

Outcome data is currently stored for future analysis. In upcoming releases, you'll be able to:

• View outcome reports in the dashboard
• Filter by action type, outcome type, date range
• See success rates for different AI actions
• Compare outcomes across different playbooks or assistants

Analyzing Effectiveness

Use outcome data to answer questions like:

• Appointment booking: What percentage of booking attempts succeed?
• Escalations: How often do conversations get escalated? What triggers them?
• Response effectiveness: Do faster responses lead to better outcomes?

Improving Performance

Based on outcome data:

• Low success rates: Review and improve the AI action logic
• High escalation rates: Update playbooks to handle issues better
• Pattern identification: Spot trends that indicate needed improvements

Privacy & Security

Data Protection

• PII removal: Personal information is automatically removed from outcome records
• Tenant isolation: Each tenant only sees their own outcome data
• Secure storage: Outcomes are stored with the same security as other data

What's Not Tracked

The system does NOT track:

• Message content: Actual message text is not stored
• Customer PII: Names, emails, phone numbers are removed
• Sensitive details: Only metadata and outcome types are recorded

Best Practices

Regular Review

• Review outcome data monthly to spot trends
• Compare outcomes across different time periods
• Look for patterns that indicate needed improvements

Action on Data

• Low success rates: Investigate why actions aren't succeeding
• High escalation rates: Review escalation triggers and playbook rules
• Pattern changes: Understand what changed when patterns shift

Integration with Other Analytics

Combine outcome data with:

• Conversation Intelligence: See how outcomes relate to objections and response times
• Training Loop: Compare outcomes with AI decision quality
• Dashboard metrics: Understand how outcomes affect overall performance

Technical Details

Version Tracking

Each outcome record includes version metadata:

• Model: AI model used when action was taken
• Provider: Model provider (e.g., "openai")
• Code Version: Code version when outcome was recorded
• Timestamps: When action was created and when outcome was recorded

This helps you:

• Track outcomes across code/model changes
• Debug issues by knowing exact versions used
• Maintain audit trails

Data Retention

• Outcome records are retained according to your data retention policy
• Old records can be archived or deleted based on your settings
• Historical data is available for trend analysis

Troubleshooting

Outcomes Not Recording

• Check system status: Ensure outcome tracking is enabled
• Verify actions: Make sure AI actions are actually being executed
• Review logs: Check system logs for recording errors (non-blocking errors are logged but don't fail the action)

Missing Outcome Types

• Some outcome types are planned for future releases
• Current tracking focuses on high-value outcomes (appointments, escalations)
• Additional outcomes will be added based on user feedback

Data Questions

• Outcome data is stored for analysis but UI may not be available yet
• Contact support if you need access to outcome data before UI is released
• API access may be available for programmatic analysis

Related Documentation

• Conversation Intelligence - Analyze conversation patterns
• Analytics Overview - General analytics features
• Training Loop - Track AI decision quality
• AI Playbooks - Improve playbooks based on outcomes