Voice agents in production: what to automate first

8 min readSimon BudziakBy Simon Budziak

On this page8 sections
Title card reading Voice agents in production, start narrow. A navy background shows a spoken waveform entering a production workflow that branches into a completed action and a human handoff.

A voice agent should first automate high-volume calls with a predictable request, a verifiable answer, and a recoverable outcome. Appointment booking, order status, opening hours, delivery updates, and basic intake are strong first candidates. Complaints, emergencies, unusual account changes, and decisions with legal, financial, or health consequences should stay with people.

That boundary matters more than how natural the voice sounds. A fluent demo can hide weak permissions, slow tools, missing records, or a handoff that makes the caller repeat everything. Production quality starts with a narrow job, a reliable system of record, and a clear path to a person.

What should a voice agent automate first?

Pick a call type that satisfies five conditions:

  1. The intent is easy to recognize. Callers may phrase it differently, but the underlying job is stable.
  2. The answer comes from an authoritative system. The agent reads the appointment book, order system, CRM, or approved knowledge base rather than guessing.
  3. The action is reversible. A booking can be moved. A delivery preference can be changed. A serious medical or financial decision often cannot be undone so easily.
  4. Success is observable. You can confirm that the booking exists, the record changed, or the caller reached the correct team.
  5. A person can take over. The caller can request a human, and the system can escalate when confidence, permissions, or conversation quality fall.

This is why appointment scheduling often beats open-ended customer support as a first release. The state is structured, the available actions are limited, and the result can be checked immediately. The same logic applies to order status and simple eligibility intake.

The fastest way to find a candidate is to review call reasons, not job titles. Look for a repeated question that already has a standard answer and a defined owner. If staff need judgment, negotiation, empathy, or authority to resolve it, keep that route human until the surrounding process is better defined.

How is a voice agent different from IVR and a chatbot?

An interactive voice response system, or IVR, asks callers to follow a fixed menu. Press one for sales, press two for support. It is predictable because its branches are written in advance.

A chatbot accepts free text but usually operates in a channel where a pause of several seconds is tolerable. Users can reread the answer, edit their message, and ignore the conversation while the system works.

A voice agent accepts natural speech and can use tools while the call is live. It may retrieve an order, check availability, update a record, or hand the conversation to another worker. Modern speech-to-speech systems can process audio directly instead of assembling a separate speech-to-text, reasoning, and text-to-speech chain. OpenAI’s voice-agent documentation notes that this reduces latency and makes interruptions and tool calls feel more natural.

The trade-off is that speech removes the user’s patience buffer. A chat reply can arrive late and still be useful. A silent phone line feels broken.

Latency, interruptions, and repair determine whether callers stay

Measure the time until the caller hears a useful response, not only the time until the whole operation finishes. Google Cloud’s service guidance warns that LLM processing can add significant latency and that streamed time to first response is different from full response time.

That leads to a practical design rule: acknowledge the caller quickly, then do the longer work. The agent can confirm what it understood before checking a slow back-office system. It should never fill the gap with a promise that the tool has not yet confirmed.

Interruptions are normal. People correct themselves, add a date halfway through, or start speaking before the agent finishes. The system needs voice activity detection, cancellation of outdated responses, and a policy for which tool calls may continue after an interruption. A spoken correction should stop the sentence. It should not silently cancel a booking that already committed.

Conversation repair matters just as much. After the first misunderstanding, ask for the missing detail in a shorter way. After repeated failures, stop looping. Google’s voice-agent design guide recommends escalation to a human on the 3rd no-match or no-input event on a page.

Put authentication, permissions, and handoffs before tool use

A voice is not an identity. Caller ID can be spoofed, shared phones exist, and a natural conversation can make a weak authentication flow feel stronger than it is. Verify identity before revealing account data or allowing changes. Match the verification strength to the consequence of the action.

Then apply agent authorization at the tool boundary. The agent should receive only the permissions required for its narrow job. Reading an order status does not require the ability to refund it. Booking an appointment does not require access to every patient note. A tool approval can pause a consequential action until a person confirms it.

A vertical production voice-agent flow. The agent understands the request, retrieves the record, proposes an action, and applies a permission gate. Routine reversible work completes automatically, while sensitive work hands identity, intent, history, and the proposed next action to a person.
Put one explicit permission decision between a plausible answer and a real-world action.

The human-in-the-loop route is part of the product, not an error screen. A useful handoff includes the verified identity, caller intent, facts already collected, tools already called, and the proposed next action. The person should start from context, not from hello.

For callers in the European Union, transparency belongs in the opening too. Article 50 of the EU AI Act consolidated text requires people interacting directly with an AI system to be informed unless it is obvious in the circumstances. Treat disclosure as a design requirement and confirm the exact obligations for your use case with counsel.

Measure a pilot by resolved work, not minutes deflected

A pilot should compare one narrow route with its current baseline. Google’s design guidance identifies 6 core experience metrics. Add action accuracy and handoff quality so the operational outcome is measured too:

  • First-call resolution: did the caller’s job finish during the first contact?
  • Misroute rate: how often did the system send someone to the wrong workflow or team?
  • Average handling time: how long did a successful resolution take, including human time after transfer?
  • Number of turns: how many exchanges were needed before completion or escalation?
  • Customer satisfaction: did callers consider the result clear and acceptable?
  • Handoff quality: did the person receive enough context to continue without repetition?
  • Action accuracy: did the final state in the source system match what the caller requested?

These align with the core measures in Google’s voice-agent quality guidance. Add business-specific failure measures too. A healthcare intake flow may track unsafe routing. A field-service scheduler may track duplicate bookings. A payment reminder may track disputes raised during the call.

Do not optimize average handling time alone. A fast wrong answer is not efficiency. A short call that immediately transfers without useful context may only move work downstream.

A practical launch checklist for a mid-sized company

Start with one call reason and one owner. Then require every item below before expanding:

  • The call reason and excluded cases are written in plain language.
  • Answers come from named systems of record or approved sources.
  • Identity verification is strong enough for the data and action involved.
  • Tool permissions are narrower than the employee role the agent supports.
  • The caller is told they are interacting with AI where required.
  • The agent can be interrupted without leaving tool actions in an unknown state.
  • Repeated misunderstanding triggers a human handoff.
  • The handoff includes identity, intent, collected facts, and tool history.
  • The team has a representative test set covering accents, noise, corrections, silence, and adversarial requests.
  • Production traces connect each spoken promise to the record read or action taken.
  • The pilot has baseline metrics and a named stop condition.

Only add another call reason when the first one is reliable. Each new route expands the language, data, permissions, tools, and failure modes the system must handle. A narrow agent that completes one valuable job is a production system. A broad agent that sounds impressive but cannot be trusted with an action is still a demo.

The decision to make before choosing a platform

Do not begin with a model comparison. Begin with the operational boundary: what may the agent know, what may it change, and when must a person take over? Once those answers are explicit, the platform decision becomes easier. You can evaluate latency, telephony, tool integration, observability, regional deployment, and cost against a real workflow.

The right first voice agent is not the one that can discuss everything. It is the one that finishes a frequent, bounded task and makes the difficult calls easier for the humans who receive them.

Sources

See this working in a system we built