Beyond the Model:
Building a Harness for a Voice Agent

The result gives context for the engineering work below. We did not train or fine-tune the model; we worked on the prompts, state, and control code around it. Without a full harness-off comparison, the score alone cannot tell us how much each change contributed.

When building voice agents, we keep running into a particular kind of failure: the model knows what to do, but the information it sees does not match what has actually happened on the call.

A customer is still reading out an identifier when a partial transcript reaches the model. An order has already been updated when the same request executes again. A reply has not finished playing, but both sentences are already in the conversation history. More prompting may not solve these problems: the model cannot account for state that has never been recorded or passed to it.

Our approach is to have the program keep track of what was received, what was executed, and what was played—then let the model decide what to do next. We use harness to mean the workflow prompts, control code, and state management around the model. This article focuses on the latter two.

The method is straightforward: find where the information diverges, record the state needed to close that gap, control the relevant action, and measure both the benefit and the cost. Waiting for input, suppressing duplicate tool calls, and handling interruptions are three examples of that method.

1. Find where the information diverges

In one retail evaluation call, the agent had completed two returns and was asking the customer to confirm a third. The customer said mm-hmm. That response appeared in the playback record, but ASR produced no text to submit. The confirmation never reached the model before the timeout. The customer then asked about the total refund and ended the call. The third return was never executed.

Looking only at the outcome, it would be easy to label this “the model missed a return” and add “do not overlook customer requests” to the prompt. But align the audio, transcript, and model input, and the break becomes clear: the customer answered, but the model never received the answer. The place to investigate is how the confirmation step handles an empty transcript.

That gives us a change we can test: if the system detects speech during a confirmation step but receives no text, should it ask again? This is still a proposal to validate. Finding the break does not mean we have fixed it.

We usually start with two questions. If the model received all the information and still failed to follow the workflow, the prompt or model may need to change. If information was missing, the status of an action was unclear, or the history differed from what had played, start with the code around the model call.

The distinction is who makes the decision. A prompt asks the model to follow a rule. Control code can delay invoking the model, withhold a tool request, or directly change the history used on the next turn. Even if a blocked call produces a text message for the model, the program has already made the decision to block it.

2. Track state at the input, tools, and history

Place these controls in the full system first. A spoken exchange follows the main ASR → language model (LLM) → TTS → playback pipeline. The Bot Prompt is text supplied when invoking the model. Input waiting, tool checks, and history updates are code running around that pipeline.

The purple Bot Prompt tells the model what it should do. The orange blocks ①②③ are executed by the program. They maintain unsubmitted text, a record of requests within the turn, and reply history. They are not three more system prompts. The next three examples follow those locations.

① Input handling: a pause does not always mean the identifier is complete

The state to retain is the unsubmitted text and the waiting state. If every pause immediately triggers a model call, the model may start a lookup with half an identifier.

Before submitting the text, we inspect its ending. We consider waiting longer only when it looks like spelling, a sequence of individual digits, or a connector such as dash or underscore. If the customer continues, we merge the continuation before passing it to the model. This changes the timing and input of the model call; it does not require the model to first obey a “please wait” instruction.

Detecting silence, finalizing a transcript segment, and deciding to submit the entire utterance are three separate events. An integration needs to define when the timer starts, what happens when speech resumes, and how long the full utterance can be held. Our additional-wait settings were 2.5 seconds for retail and airline, and 1.5 seconds for telecom. These are configuration values, not measured average delays or recommended defaults.

The cost is unnecessary waiting. Complete identifiers and ordinary sentences can also match the rules. Our implementation exempts digit strings of exactly five digits or at least ten digits to reduce cases where a completed identifier is held back. Length alone does not establish validity. The measurements that matter are how often we catch an incomplete input and how often we needlessly delay a complete one. We do not yet have comprehensive counts for either.

ASR spelling cleanup belongs in this same input layer. Splitting YAsInYellow into Y as in Yellow restores separators. Filling in a missing letter from an order database crosses a different boundary. We retain the text before and after processing to inspect unintended changes; we removed one overly aggressive rule after it broke terminology and airport codes.

② Tool checks: preventing duplicate writes

The evaluated system includes a duplicate-write check. It records the tool name and arguments of write requests that have returned without an error in the current turn. If the model asks for the same update again, the program can prevent the request from being sent twice.

We added a check inside the tool loop. Once the first call returns without an error, its request is added to a set. If the same request arrives again within the turn, the program does not forward it to the business tool. Instead, it returns a “not executed” result to the model.

The main branch can be simplified to:

The model requests an update
    ↓
Has the same request returned without an error in this turn?
    ├─ Yes: do not forward it; return “not executed”
    └─ No: call the tool; record the request if no error is returned
    ↓
Return the result to the model as a tool message and continue

The model may request the same action twice, but the program forwards only the first. The feedback is text; the execution decision is code. The check works without the model first understanding “do not repeat this update.”

The boundary lies in how we define “the same request” and “returned without an error.” JSON arguments need a canonical representation for comparison. Errors must be recognized according to the tool protocol so that legitimate retries can pass. The set lasts only until the current turn’s reply is finalized. It does not handle duplicates across turns, across processes, or concurrently in flight.

Timeouts deserve particular care: not receiving a result does not mean the tool did not execute. For operations that could charge a customer twice, the business service still needs an operation ID for idempotency, or a way to query the actual state. This small set is no substitute for a complete retry design.

Business-policy checks can also run before a call, but they need a clear basis for their decisions. Comparing a tool name and arguments is straightforward. Deciding whether a health-related reason permits cancellation requires natural-language understanding, and we have incorrectly blocked a cancellation this way. Whether to allow, warn, or ask for clarification when information is incomplete must be designed for each rule.

③ History correction: generated does not mean heard

The state to retain is estimated playback progress and the corresponding reply text. The model may have written two sentences while the player is still on the first. If the customer interrupts and both sentences remain in history, the next model call will treat words that never played as words already spoken.

Our history-management code directly edits the reply message. It estimates the portion the customer heard, retains that prefix, and removes the rest. Existing tool results remain. If a refund has executed but the notification was interrupted, the next turn should finish telling the customer, not issue another refund.

The cost is estimation error. Progress here is estimated from the duration of audio successfully written to the player, not a receipt confirming playback. The code combines this with audio duration and text length, with a bias toward keeping less text. If text or TTS is still streaming at interruption and no reliable prefix is available, the history code removes the whole reply. This can remove words the customer did hear.

There is an exception to prevent repetition: after the same reply has been removed in full twice, it is retained to avoid regenerating it indefinitely. Existing tool results are unaffected by either truncation or deletion.

This TTS path does not pass character timestamps to the history code, and we have not quantified truncation error. A phone or browser integration should use progress from its player and text–audio alignment where available. Check both excessive deletion and repetition; the estimate is not an exact record of what the customer heard.

All three examples record state the model cannot verify on its own, then use that state to control a specific action. What the model should ask, or the order in which it should troubleshoot, can still live in the prompt.

3. How we decide which checks to keep

The duplicate-write check above uses a recorded fact: an identical request has already returned without an error in this turn. It is enabled in the evaluated system. Other checks prescribe how the model should approach a task. Code can enforce a rule; whether that rule helps is a separate question.

We tested a different check: when user details listed orders the model had not yet inspected, it blocked the first update request and listed the orders still to retrieve. It used only tool results already seen during the call and could block at most once per call. The hypothesis was that inspecting every listed order before making a change would help the model choose correctly. But a task may concern only one order, making the extra lookups unnecessary. We left this check disabled. Here is how we reached that decision.

First, we replayed the context just before an error in three retail tasks, with three samples per task for each treatment:

The expected steps included retrieving order details or asking separately why two orders should be cancelled. 8/9 measures a change in the next action—not successful tool execution or task completion. The sample covered only three tasks; one baseline already tended to ask a follow-up, and the replay feedback differed from the final check. Reference actions were used only for offline evaluation, never supplied to the running check.

Next, we ran five tasks four times with the order-inspection check disabled and four times with it enabled: 20 complete calls per setting, 40 in total. Only this check changed, not the entire harness.

The check blocked seven requests. On two tasks selected to watch for regressions, passes fell from 6/8 to 4/8, reaching the pre-set stopping condition. Mean model interaction requests rose from 10.4 to 11.8 per call; these are requests, not customer turns or latency. The customer’s actual answers also differed between runs: on one task, a color preference was volunteered in the two successful off-runs but in none of the four on-runs. Some calls retrieved more details and still made the wrong update. This small comparison did not establish that the check caused the lower score, but it did not justify enabling it either.

The model more often took the intended next step; we did not establish that this made the full task more likely to succeed. We therefore kept this experimental check disabled. The duplicate-write check in Section 2 remains enabled; this experiment did not test its effect.

For any new rule, we ask two questions: did it change the behavior we targeted, and was that change worth keeping over a complete call? We track both the benefit and the cost:

These are proposed diagnostic measures, not measured gains. Counting triggers is not enough: review recordings and traces to check unnecessary waiting, false blocks, and excessive deletion, alongside task outcomes. A check firing proves that the control ran; it does not prove the rule was worth adding.

4. What the evaluation results tell us

τ-voice is a benchmark for voice agents handling customer-service calls. Agents speak with simulated customers, follow business policies, and use tools to complete tasks such as returns, flight changes, and telecom troubleshooting. Calls include accents, background noise, and interruptions, with both sides able to speak and listen at the same time. The main score, pass@1, measures whether the task was completed; latency and turn-taking are measured separately. Benchmark paper.

We evaluated the system on τ-voice’s retail, airline, and telecom tasks. The agent uses Deepgram nova-3 → gemini-3.7-flash → ElevenLabs flash v2.5, without training or fine-tuning by us. The control code runs inside the agent and leaves the evaluator’s business tools and scoring rules unchanged, within the boundary allowed for standard voice submissions. Submission rules

Batch: va20260926a · September 26, 2026 · Self-evaluated; not yet officially verified. Both groups covered all 278 tasks across the three domains, excluding Banking. Standard used GPT-4.1 (temperature 0) to simulate customers; Custom used GPT-5.5 (xhigh). Both used voice simulator v1.0, ElevenLabs v3, and voices generated by us, not the official leaderboard voices.

Same agent, different customer simulators: complete-system results, not a harness-on versus harness-off comparison.

Standard passed 237/278 tasks and Custom passed 250/278. The overall scores in the table weight the three domain pass rates equally, rather than pooling all tasks into one pass rate.

Comparison with published results

As of October 26, 2026, the tables below cover all 20 published submissions using the GPT-4.1 and GPT-5.5 customer simulators in the original τ-voice track, alongside our two self-evaluations. Retail, airline, and telecom scores are shown separately; their equal-weight mean is used for sorting within each matched-simulator group. Newer τ³-Voice results use a different benchmark and are excluded. Official leaderboard · Source snapshot

Our self-evaluated scores are 2.18 and 2.58 percentage points above the highest published scores in the corresponding groups. Scores can vary between runs, so a lead in one self-evaluation does not establish a stable ranking. Our results have not yet been officially verified. Voices, infrastructure, and error handling can differ across systems, so these margins cannot be attributed directly to the harness.

Task completion also needs to be considered alongside response speed, stopping when interrupted, and talking over the customer. Interaction metrics were computed for this batch, but we do not report latency or cost figures here, or treat a higher pass rate as evidence of a better conversation. Interaction metrics

5. Turn a fix into a method you can reuse

Start with one recurring failure you can reconstruct in your own agent. Find out why it happens and where the system has enough information to prevent it. These four steps turn that investigation into a change you can evaluate.

Step 1: Find the first mismatch

Work backward from the wrong action, aligning four records from the same call on one timeline:

Leave this step with a specific finding: “The confirmation never reached the model,” or “The tool returned, but the program did not record it.” If the model had all the necessary information and still chose incorrectly, revisit the prompt, business rules, or model itself. Runtime changes will not address every failure.

Step 2: Give the rule a precise scope

Write down which state justifies the decision, when the rule applies, and which action it changes. For example: “Within a turn, do not forward an identical write request that has already returned without an error.” Then define when that state is created and cleared, and what happens when the outcome is unknown. A normal return, an explicit failure, and an unconfirmed timeout need different treatment.

Put the control where that information is available: the input buffer controls submission, the tool loop controls forwarding, and history management controls what the next model call receives. Workflow reminders can remain in the prompt. A hard block needs a sufficiently clear basis; if it still requires guessing the customer’s intent, consider clarification or a warning first.

Step 3: Test when the rule should act—and when it should leave things alone

Replay the target failures, then test nearby normal cases for false blocks. A duplicate-write check should suppress an identical request that has already returned without an error, while allowing retries after an error and new requests with different arguments. Write those expectations before looking at the results. A rule firing is not enough to call the problem fixed.

Replay tells you whether the control behaves as intended. It cannot tell you whether added waiting frustrates the customer or whether more tool lookups make a task easier to complete.

Step 4: Decide using complete calls

Keep other settings fixed and toggle only the rule under test. Compare the target error, task completion, and added waiting, follow-up questions, or false blocks. Decide in advance what would justify keeping the rule and what regression would make you stop. If the sample is too small or customer responses differ too much between groups, collecting more evidence is a valid outcome.

For a rule you retain, log the evidence that triggered it, the action taken, and what happened afterward. Keep it independently switchable. When another failure appears, you can then distinguish a model error from stale state or a poorly chosen rule.

A reusable change should answer three questions: which failure does it address, when should it stay out of the way, and what evidence supports keeping it? Establish that for one change before adding the next.