The boundary matters whenever model text becomes code, data, or a tool request. A request for JSON can still omit a required field, invent an identifier, choose an unauthorized action, or produce a valid structure with a wrong claim.
What should validate an LLM response?
Start with a schema and deterministic domain rules. Check types, required fields, value ranges, record ownership, permissions, and any source evidence the action requires. Validation belongs in application code that can reject the output, not only in prompt wording. Structured output makes the shape more reliable, but it does not establish truth.
How do we validate model output in our own pipelines?
With deterministic gates, never the model’s own confidence. We stopped gating on self-reported LLM confidence because the number clusters near certainty on right and wrong answers alike; instead we compute a signal the system can measure, apply a plain threshold rule, and escalate the exceptions to a person. The same shape guards this website: every page an agent drafts must pass audit scripts that reject hard failures, from malformed structure and broken internal links to a missing meta description, before anything deploys. A page that fails a rule does not ship, whatever the model thought of it. The gates are ordinary code, so a rejection is reproducible and the rule that fired is named in the log.
What should happen when validation fails?
Retry a bounded structural error with precise feedback when a corrected response is safe to request. Stop and escalate policy, authorization, or factual failures. A validation retry must never turn a rejection into permission. Retries may repair structure, but they must not negotiate a policy or permission failure. Guardrails enforce policies around the agent loop, tool approval binds a person to a consequential action, and AI agent evals prove the boundary handles realistic failures before release.