13 terms
Definitions in this topic
- ProductionAgent simulationAgent simulation explained: controlled environments for testing behavior before production.
- ProductionAgent trajectoryAgent trajectories explained: the sequence of decisions and tool calls used to evaluate how an AI agent reached its result.
- Agentic AIAI agent evalsAI agent evals explained: end to end, trajectory, and component level testing, and why single-turn accuracy is not enough.
- ProductionAI evaluation harnessAI evaluation harness explained: the dataset, scorers, and pipeline that turn one off tests into a repeatable gate.
- BusinessAI ROI measurementAI ROI measurement done honestly: per workflow baselines, fully loaded costs, and why hours saved do not equal money saved.
- ProductionArize PhoenixArize Phoenix explained: OpenTelemetry-native LLM tracing and evals, self-hosting, and where it sits against Langfuse and LangSmith.
- ProductionLLM as a judgeWhat LLM as a judge means, the biases judges carry, and how teams validate automated evaluation against human labels before trusting it.
- ProductionOnline vs offline evaluationOnline vs offline AI evaluation: when to test fixed datasets before release and when to monitor live production runs.
- ProductionPrompt Regression TestingPrompt regression testing explained: compare a changed LLM system against representative cases before it reaches production.
- ProductionRAG evaluationRAG evaluation explained: measuring retrieval relevance, answer correctness, faithfulness, and end-to-end usefulness.
- Agentic AIReflection Agent PatternReflection agent pattern explained: the draft, critique, revise loop, and when self-review improves an AI workflow.
- ProductionTask Completion RateTask completion rate explained: an outcome metric for AI agents, and why success percentage alone is not enough.
- ProductionTrace-Based EvaluationTrace-based evaluation explained: evaluate an AI system's full execution path, not only the final answer.