Open Loyalty needed an inbound sales agent that could do more than chat. It had to qualify real demand, answer product questions from current material, and leave a structured record for the sales team.
The production risk sat in the handoffs. A wrong pricing claim could lose trust. A weak qualification could send the wrong lead down the wrong path. A repeated CRM action could create two updates from one conversation.
The problem
The agent needed to:
- answer from Open Loyalty’s current product positioning rather than stale documents;
- identify whether the visitor was a brand, merchant, partner, or agency;
- collect the minimum information sales needed without turning the conversation into a form;
- route the visitor to self-service, a meeting, or a support path;
- write lead notes, tags, and a handoff summary to downstream systems.
Two problems appeared when the system met real traffic.
Quality drift. Prompts that looked reliable in staging broke on messy, multi-part questions.
Hidden tool failures. Retrieval-augmented generation and CRM problems often looked like an odd model response until we could inspect the sequence of model calls, tool calls, retries, and side effects.
One intermittent bug made the second problem concrete. A small share of conversations produced two CRM updates instead of one. Manual testing rarely caught it, but the duplicate records created operational noise.
The system
We modelled the AI agent as a LangGraph state machine. Conversation state moved through explicit stages for introduction, qualification, retrieval, sales briefing, and CRM write.
class SDRState(MessagesState):
lead_id: str
lead: dict
score: int
next_action: str
kb_context: list[Document]
thread_id: str
graph = StateGraph(SDRState)
graph.add_edge("intro", "qualify")
graph.add_conditional_edges("qualify", needs_product_qa)
graph.add_edge("retrieve_kb", "sales_brief")
graph.add_edge("sales_brief", "crm_write")
graph.add_edge("crm_write", END)This structure mattered because a production agent has to debug a sequence, not a single answer. The state showed what the visitor asked, what retrieval returned, which prompt ran, what the model decided, and which tool caused a side effect.
What made it reliable
Trace every production turn
LangSmith gave the team an end-to-end record for each conversation. Its observability model groups every step in one operation into a trace. Our trace included model calls, tool calls, inputs, outputs, latency, prompt version, and run metadata.
That made failures reproducible. A run ID could answer which prompt produced the behaviour, what product context retrieval returned, and whether a tool call was new or a retry.
Turn quality into a regression test
We built a small AI agent evaluation set from representative conversations. It covered pricing questions, objections, ambiguous intent, missing fields, and multi-part requests.
Each meaningful change to the prompt, model, retrieval settings, or tool schema could run against the same set. Product and sales could review whether routing, grounding, and field extraction improved across the set instead of debating one transcript.
The first checks stayed deliberately concrete:
- Did the agent route the lead correctly?
- Did it extract the required fields?
- Did it stay grounded in approved product material?
- Did it avoid an unsafe or duplicate side effect?
Feed production failures back into engineering
Structured feedback turned real runs into a prioritised repair queue. Instead of collecting only a thumbs-up or thumbs-down signal, the team classified failures such as wrong routing, unsupported claims, missed fields, tone mismatch, and tool errors.
The best examples then joined the evaluation set. A production problem became a test case for the next release.
The duplicate action bug
The duplicate CRM update did not appear in the final assistant message. The damage happened in the side effect.
The trace showed the exact sequence. A CRM write timed out. The agent retried. The second call succeeded, while the first call completed shortly afterwards. One conversation had produced two writes.
We hardened the CRM boundary in three ways:
- Each write received an idempotency key derived from the conversation and turn.
- Model retries and tool retries became separate decisions.
- The evaluation set gained a simulated timeout case that asserted one lead update per turn.
crm_write.invoke({
"lead_id": state["lead_id"],
"payload": payload,
"idempotency_key": f"{thread_id}:{turn_id}",
})This is why LLM observability was more than a debugging convenience. It showed which system boundary needed a stronger guarantee.
The result
Open Loyalty received a sales agent that could qualify inbound demand, answer from a traceable product knowledge base, and hand structured outcomes to the sales team. The surrounding controls made each production turn inspectable and each meaningful change testable.
The practical lesson is simple. If an agent touches revenue operations, separate thinking from doing. Trace both, make side effects idempotent, and keep a small evaluation set built from real conversations.
Technical references
Last updated: