Silent prompt regressions in multi-turn agents

The fix that broke turn five

A member services agent at a health plan kept quoting copays before confirming which plan the member was on. A product manager filed the bug, an engineer added one sentence to the system prompt ("Always confirm the member's plan before discussing cost"), checked five conversations that started with a cost question, saw the agent ask for the plan first, and merged. The fix worked.

Over the next week the average conversation got two turns longer and tool spend went up. The new sentence had made the agent re-confirm the plan every time cost came up, including after it had already confirmed it on turn one, and in conversations where the member corrected their date of service the agent now asked for the plan again instead of re-running the benefits lookup. Nobody had looked at conversations where cost came up twice, because nobody thought the change touched them. That is the typical silent regression: a correct local fix with a non-local effect, invisible in the cases the author chose to check.

Why multi-turn makes it worse

Single-turn regressions are at least visible in a side-by-side diff of outputs. Multi-turn agents carry state across turns: slots collected, tools already called, facts the user corrected, a running plan. A prompt change can alter how the agent reads that state without changing any first-turn output. The regression shows up only when a later turn depends on something an earlier turn did, and by then the conversation has diverged so far that a reviewer comparing transcripts by eye cannot tell which difference matters.

We see the same few patterns again and again.

• Dropped slots: the agent stops collecting a required field (date of service, state of residence) because a new instruction reordered its priorities.
• Repeated tool calls: a lookup that used to happen once now happens on every turn, inflating latency and cost.
• Lost corrections: after the user corrects a fact, the agent continues with the old value in a later tool argument.
• Skipped confirmations: a confirmation step before a write disappears on paths that start with a different intent.

How often it happens

When we looked across 1,180 prompt-only pull requests on 14 production agents, 41% changed at least one tool-call trajectory on the golden set while final-answer scores stayed flat (GONATURELLE regression study, 2026). The figure covers single-turn and multi-turn agents together, and multi-turn agents accounted for a disproportionate share of the changes. Most changes were harmless. The problem is that harmless and harmful changes look the same in a final-answer report, and a human reviewer cannot read thousands of transcripts per pull request.

The cost side is easier to reason about with explicit assumptions. Take a fully loaded engineer hour at $150. Debugging one prompt regression found in production, from the support ticket to the root cause to the fix and re-verification, typically takes around six hours across the people involved, which is $900 before counting the customer, compliance or clinical exposure. A team where 15 engineers each spend 20 hours a month on manual prompt QA is spending 300 hours, or $45,000 a month, on a process that still misses the multi-turn cases.

Replay against a baseline, turn by turn

The fix is to make every prompt change re-run a fixed set of multi-turn conversations and compare the new trajectories against the baseline recorded on the main branch. Two kinds of conversation belong in that set. Scripted conversations replay recorded user turns verbatim; they are deterministic on the user side and catch changes in the agent's tool use. Simulated conversations use a user model driven by a goal and a persona; they catch cases where the agent's new behaviour sends the dialogue down a different branch that a script cannot follow.

For each conversation the gate stores the per-turn trajectory: tools called, arguments, state after the turn. The comparison is structural first. Did the set of tools called per turn change? Did any slot that was filled at turn three in the baseline stay empty? Did any argument value disagree with a value the user stated or corrected? Only after that does a judge look at the prose.

import pytest
from gonaturelle import EvalSuite, TrajectoryTester, assertions as A

from agents.member_services import agent

SUITE = EvalSuite.load('healthcare/member-services-multiturn-v1')


@pytest.fixture(scope='module')
def results():
    return SUITE.evaluate(agent=agent, user='scripted+simulated', max_concurrency=8)


def test_no_turn_level_regressions(results):
    diff = results.compare(baseline='main')
    diff.assert_no_new_failures()
    diff.assert_slots_preserved(['member_id', 'plan_id', 'date_of_service'])
    diff.assert_tool_calls_per_turn(max_increase=0.10)


def test_corrections_reach_tool_args(results):
    TrajectoryTester(results).assert_order('confirm_plan', 'fetch_benefits')
    results.assert_all([
        A.uses_latest_user_value(field='date_of_service', tool='fetch_benefits'),
        A.no_cycles(tool='confirm_plan', max_repeats=1),
    ])

Making the diff readable for the reviewer

A gate that fails with "37 conversations changed" gets overridden. The PR comment has to tell the author what changed and where. Ours groups changes by pattern (tool added per turn, slot dropped, argument mismatch), shows one representative conversation per group with the first diverging turn highlighted, and lists score and token cost deltas against the baseline. In the member services case above, the comment would have shown a single group: confirm_plan called on turns three and five where the baseline called it once, plus a 22% rise in tokens per conversation on the cost-question cases.

Keep the baseline honest. It must be recorded with the same dataset version, judge versions and model version as the candidate, or the diff will mix your change with someone else's. We key cached baselines on dataset version, prompt hash and model version, and invalidate them when any of the three moves.

Where to start this week

Pick the 30 to 50 multi-turn conversations your support or clinical team would be most upset to see broken, including at least a few with corrections, repeated intents and mid-conversation topic changes. Record them against the current main branch as the baseline. Add the structural comparison to CI and let it run in report-only mode for a week, so you learn its noise level before it blocks anything. Then turn on blocking for slot drops and argument mismatches, which have almost no false positives, and leave the cost and tool-count thresholds advisory until you have tuned them.

Teams that do this usually find their first real regression in the first fortnight. If you want to see the turn-level diff on your own agent, start a 14-day trial and connect the repository; the first run records your baseline automatically.