autoch.at Documentation

Appearance

AI response quality and safety review ​

Overview ​

This system checks assistant responses for common issues (safety, relevance, and sensitive data handling). When something looks risky, it records the result and can apply improvements within the same conversation.

How It Works ​

Automatic quality monitoring ​

Every AI response is automatically checked for:

  • Content Safety: Inappropriate, harmful, or off-topic content (using both heuristic checks and optional OpenAI moderation)
  • PII Handling: Proper handling of sensitive customer information
  • Response Quality: Accuracy, relevance, and helpfulness
  • Risk Assessment: Potential issues that need attention

Escalation ​

When a response is flagged as high-risk, the system:

  1. The system sends the response (if configured to do so).
  2. A background review evaluates the response and context.
  3. The system records issues and suggested improvements.
  4. Follow-up guidance can be applied to future replies in the same conversation.
  5. Admins can review the record in the admin dashboard.

What Gets Escalated? ​

Responses are escalated when they meet certain risk criteria:

High-Risk Indicators ​

  • PII Detection: When sensitive information appears inappropriately in responses
  • Off-Topic Content: Responses that don't address the customer's question
  • Multiple Warnings: Responses with several quality or safety concerns
  • Serious Mode: Conversations flagged for threats, legal issues, or safety concerns

Customizable Thresholds ​

You can configure what triggers escalation in your Admin Dashboard:

  • Enable/disable escalation for your account
  • Adjust sensitivity thresholds
  • Choose which warning types trigger escalation
  • Set minimum warning counts

What you get ​

  • Visibility into responses that may need attention
  • A consistent place to review safety and quality signals
  • A way to tune thresholds so the system matches your risk tolerance

Viewing Escalations ​

Admin Dashboard ​

Access escalation records in your Admin Dashboard:

  1. Navigate to Admin → Escalations
  2. View all escalated responses with filters
  3. See detailed feedback and improvement suggestions
  4. Review conversation context

Escalation Details ​

Each escalation record shows:

  • Risk Level: Assessment of the response risk
  • Issues Identified: Specific problems found
  • Improvement Instructions: How to improve future responses
  • Conversation Context: Full conversation history
  • Warnings & Errors: Detailed safety check results

Configuration ​

Setting Up Escalation ​

  1. Go to Admin → Escalations → Configuration
  2. Select your tenant
  3. Configure risk thresholds
  4. Enable/disable escalation
  5. Set notification preferences

For Most Businesses:

  • Enable escalation
  • Set PII warning threshold: ON
  • Set off-topic warning threshold: ON
  • Minimum warnings: 2
  • Enable serious mode escalation

For High-Volume Businesses:

  • Adjust thresholds to focus on critical issues
  • Review escalation statistics regularly
  • Fine-tune based on your needs

Cost Efficiency ​

The escalation system is designed to be cost-effective:

  • Separate Model: Uses a specialized, cost-efficient review model
  • Selective Review: Only high-risk responses are reviewed
  • Asynchronous Processing: No impact on response speed
  • One-Time Review: Each high-risk response is reviewed once

Privacy & Security ​

Your Data is Protected ​

  • Full PII in Your Data: Your conversations and records contain full customer information
  • Redacted Logs: System logs use hashed PII for privacy
  • Secure Storage: All escalation records are securely stored
  • Access Control: Only authorized admins can view escalations

What We See ​

  • Escalation records help us improve the system
  • PII in escalation records is hashed in system logs
  • Your actual customer data remains fully accessible to you

Best Practices ​

  1. Regular Review: Check escalation records weekly to identify patterns
  2. Adjust Thresholds: Fine-tune based on your business needs
  3. Learn from Feedback: Use improvement suggestions to update your assistant configuration
  4. Monitor Statistics: Track escalation volume and trends
  5. Update Prompts: Incorporate feedback into your assistant's instructions

Troubleshooting ​

Too Many Escalations? ​

  • Review and adjust risk thresholds
  • Check if content safety settings are too strict
  • Review escalation feedback for common issues
  • Consider updating your assistant's instructions

Not Enough Escalations? ​

  • Lower the minimum warning threshold
  • Enable additional warning types
  • Check that escalation is enabled
  • Verify your content safety settings

Questions? ​

  • Review escalation feedback for guidance
  • Check the PII Handling Guide for privacy details
  • Contact support for assistance with configuration

autoch.at Documentation

Copyright © 2024 autoch.at