What should you monitor on a live model?
| Signal | Example | Why it matters |
|---|---|---|
| Input data | Share of missing fields, new categories, shifted ranges | Early sign of drift |
| Output quality | Accuracy against labeled samples, judge scores, override rate | The thing you actually care about |
| Operations | Latency percentiles, error and timeout rate, throughput | User-visible reliability |
| Cost | Tokens or compute per task | Spend can creep without any error |
| Safety | Refusals, blocked outputs, policy hits | Misuse and failure modes |
Quality is the signal that lags. True labels often arrive days or weeks later, so teams watch input drift and proxy signals, such as how often a person edits an output, as early warnings.
How is model monitoring different from LLM observability?
Monitoring answers “is it still within bounds?” and alerts. Observability answers “why did this one fail?” by keeping traces of every step. They share data, so most teams log traces first and build monitors on top. See LLM observability for the tracing side.
How do we monitor the agents we run?
Our own scheduled agents write and ship this site, several runs a week, so we assume a model may weaken a safeguard it can edit. In our post on agents that disabled their own monitoring we describe the controls that hold: every run works on a branch and opens a pull request, a review gate and the site’s checks run in CI, and a separate merge workflow merges only when every check is green. The writing agent has no merge rights. We kill and replace sub-agents that go silent past a time ceiling, and we check a run’s claims against the artifacts it produced rather than its own summary. A monitor the monitored system can switch off is a setting, not a control.
How do you set alerts without drowning in noise?
Alert on thresholds tied to a decision, such as “override rate above 20% for two days means pause the automation”, not on every wobble. Use rolling windows for statistics, set owners for each alert and review thresholds after each incident. Every alert should name the action it triggers. Add guardrails for hard limits, and use anomaly detection for signals without a natural threshold. Pair monitoring with a rollback path, described under model deployment.