Agent Evaluation & Readiness Guide

Before any Operanix agent reaches production, it must pass a rigorous evaluation framework. This guide covers 7-dimension quality scoring, golden set testing, regression detection, hallucination analysis, and the GO/NO-GO production gate.

Evaluation Tabs Overview

The Agent Readiness module contains 10 tabs, each focused on a specific aspect of agent quality assurance:

7-Dimension Quality Scoring

Every agent response in an evaluation run is scored across seven dimensions. Each dimension uses a 0–1 scale with LLM-as-judge evaluation, calibrated against human-labeled reference sets.

Composite Score Calculation

Composite Score = (
    0.20 x Correctness +
    0.15 x Relevance +
    0.10 x Tone +
    0.15 x Safety +
    0.20 x Groundedness +
    0.10 x Faithfulness +
    0.10 x Context Utilization
)

// Minimum thresholds (any dimension below threshold = NO-GO)
Safety     >= 0.90
Groundedness >= 0.80
Correctness  >= 0.75
All others   >= 0.60

Golden Set Management

Golden sets are curated collections of question-answer pairs that serve as ground truth for evaluation. They are the foundation of repeatable, objective agent testing.

Golden Set Structure

{
  "id": "gs-support-v3",
  "agent": "customer-support",
  "version": 3,
  "cases": [
    {
      "id": "case-001",
      "question": "How do I reset my password?",
      "expected_answer": "Go to Settings > Security > Reset Password...",
      "context_chunks": ["kb-auth-001", "kb-auth-002"],
      "tags": ["authentication", "self-service"],
      "difficulty": "easy",
      "created_by": "jane@company.com",
      "created_at": "2026-03-15T10:00:00Z"
    }
  ]
}

Golden Set Best Practices

• Minimum 50 cases per agent — Include a mix of easy (40%), medium (35%), and hard (25%) questions to test the full capability range.
• Cover all knowledge domains — Ensure every domain assigned to the agent has at least 5 golden set cases.
• Include edge cases — Add questions that test boundary conditions: ambiguous queries, multi-part questions, questions requiring cross-domain synthesis, and out-of-scope questions (agent should gracefully decline).
• Version control — Golden sets are versioned. When knowledge changes, update the golden set to reflect new expected answers. Keep previous versions for regression comparison.
• Regular review — Schedule monthly golden set reviews to retire stale cases and add cases based on real production failures.

Regression Testing

Regression testing automatically compares new evaluation results against a baseline to detect quality degradation before it reaches production.

How Regression Detection Works

• Baseline selection — The last successful production deployment evaluation serves as the baseline. You can also manually pin a specific evaluation run as the baseline.
• Per-case comparison — Each golden set case is compared between baseline and current run. Cases with score drops of more than 0.15 are flagged as regressions.
• Dimension-level regression — Even if the composite score holds, a drop in any single dimension of more than 0.10 triggers a regression warning.
• Statistical significance — Small score fluctuations are expected. The system uses paired t-tests to determine whether observed differences are statistically significant (p < 0.05).

Regression Report

Hallucination Detection

The Hallucination Report provides deep analysis of every instance where an agent made claims not supported by its knowledge base.

Hallucination Categories

• Fabricated facts — Agent invents information that does not exist in any source. Example: citing a non-existent policy or making up a product feature.
• Exaggerated claims — Agent overstates what the source material says. Example: "guaranteed 99.99% uptime" when the source says "targeting 99.9% uptime."
• Conflated information — Agent merges details from different sources incorrectly. Example: attributing features of Product A to Product B.
• Outdated references — Agent cites information from superseded content that was not properly retired from the knowledge base.
• Implicit assumptions — Agent makes logical leaps not explicitly stated in the source. Example: inferring a feature exists because related features exist.

Hallucination Trace

Each detected hallucination includes a trace showing:

• The specific claim in the agent's response
• The retrieved context chunks that were available
• The groundedness score for that claim
• The nearest source passage (if any partial match exists)
• A suggested correction based on available knowledge

GO/NO-GO Production Gate

The Production Gate is the final checkpoint before an agent can be deployed to production. It combines automated scoring with manual review checkpoints.

Automated Checks

Manual Review Checkpoints

• Sample conversation review — Reviewer reads 10 randomly selected evaluation conversations and confirms quality.
• Edge case verification — Reviewer tests the agent with adversarial or unusual queries not in the golden set.
• Persona consistency check — Reviewer confirms the agent's tone and behavior match the intended persona across different query types.
• Escalation path test — Reviewer verifies the agent correctly escalates queries it cannot handle to human operators.

Gate Decision

• GO — All automated checks pass and all manual checkpoints are approved. Agent is cleared for production deployment via the governance approval chain.
• CONDITIONAL GO — Automated checks pass but advisory thresholds are not met. Agent can deploy with monitoring flags and a mandatory re-evaluation within 7 days.
• NO-GO — One or more required thresholds are not met. Agent cannot deploy. The system generates a remediation report identifying specific failures and suggested fixes.

Feedback Loop

The Feedback Loop tab connects production performance data back into the evaluation framework, creating a continuous improvement cycle.

Feedback Sources

• User ratings — End-user thumbs up/down and optional comments on agent responses.
• Operator corrections — When human operators correct or override an agent response, the correction is captured as training signal.
• Escalation analysis — Queries that were escalated to humans are analyzed to identify agent capability gaps.
• Safety gate triggers — Production safety gate activations are fed back as negative examples for evaluation tuning.
• Conversation analytics — Automated analysis of conversation outcomes (resolution rate, follow-up questions, abandonment) provides indirect quality signals.

Feedback-to-Evaluation Pipeline

• Negative feedback (thumbs down, corrections, escalations) is automatically reviewed for golden set candidacy.
• Patterns in negative feedback are clustered by topic to identify systematic issues.
• High-confidence negative cases are added to the golden set automatically (with human confirmation).
• The feedback loop triggers re-evaluation when the negative feedback rate exceeds 2% over a 24-hour window.

A/B Comparisons

Compare two agent versions side-by-side on the same golden set to make data-driven decisions about configuration changes.

• Run the same evaluation against two agent configurations simultaneously.
• View per-dimension score comparisons with statistical significance indicators.
• Drill into individual cases where the two versions diverge to understand the impact of changes.
• Generate a comparison report summarizing which version is superior and by how much.

Best Practices

• Run evaluations after every knowledge update, not just agent configuration changes. Knowledge changes can cause unexpected quality shifts.
• Maintain at least 10 universal test cases that apply to all agents (e.g., "I want to speak to a human", adversarial prompts, out-of-scope questions).
• Set up automated nightly evaluation runs against your golden sets to catch regressions early.
• Review the hallucination report weekly and trace hallucinations back to their root cause in the knowledge pipeline.
• Use A/B comparisons before making any significant prompt or retrieval configuration change.
• Treat the production gate as non-negotiable. The short-term pressure to deploy quickly is never worth the long-term cost of a quality incident.