- pass@k (Chen et al., 2021) is the probability that at least one of k samples succeeds; it suits code generation with a test oracle, and flatters agents.
- pass^k (Yao et al., tau-bench, 2024) is the probability that all k independent trials succeed; it measures the consistency a production workflow needs.
- In tau-bench's retail domain the best function-calling agent was under 50% on a single trial and under 25% at pass^8.
- An agent with 90% per-case success passes all 8 trials only about 43% of the time. Small per-trial gaps compound fast.
- Estimate pass^k with the unbiased combinatorial estimator from n trials per case, and gate critical cases on it rather than on a single run.
Two metrics with almost the same name
pass@k comes from code generation. Chen et al. (2021), introducing Codex and the HumanEval benchmark, defined it as the probability that at least one of k sampled programs passes the unit tests, and gave an unbiased estimator from n samples with c correct: 1 - C(n - c, k) / C(n, k). The setting justifies the metric: a developer can generate several candidates, run the tests, and keep one that passes. Success once is success.
pass^k comes from agent evaluation. Yao et al. (tau-bench, arXiv:2406.12045, 2024) introduced it to capture something pass@k cannot: whether an agent solves the same task correctly every time it sees it. Their estimator, from n trials with c successes, is C(c, k) / C(n, k), averaged across tasks. The two metrics move in opposite directions as k grows. pass@k rises toward 1 for any agent that sometimes succeeds. pass^k falls toward 0 for any agent that sometimes fails.
Why production cares about the second one
A customer service, clinical or credit agent does not get to sample five responses and keep the best one. Each member who calls, each prior authorization that arrives and each application that comes in gets exactly one trajectory. If the agent handles a given kind of request correctly 90% of the time, then across many members with that request, one in ten gets the wrong outcome. The relevant question for a reviewer is not "can it do this?" but "will it do this every time this situation occurs?". That is pass^k.
tau-bench made the gap concrete. In its retail domain, the best function-calling agent the authors tested succeeded on under 50% of tasks in a single trial, and its pass^8, the chance of succeeding on the same task in all 8 trials, was under 25% (Yao et al., 2024). An agent that looks merely mediocre on a single run looks unreliable when you ask for consistency, and consistency is what regulated workflows need.
How fast consistency decays
If each trial on a case succeeds independently with probability p, pass^k for that case is p to the power k. The decay is steeper than intuition suggests. At p = 0.99, pass^8 is about 0.92. At p = 0.95 it is about 0.66. At p = 0.90 it is about 0.43. A one-point improvement in single-trial success near the top of the range buys several points of pass^8.
Averages across cases hide structure too. Compare two suites with the same 90% average single-trial success. In the first, every case succeeds 90% of the time: pass^8 is about 0.43. In the second, half the cases always succeed and half succeed 80% of the time: pass^8 is 0.5 x 1 + 0.5 x 0.8^8, about 0.58. Same headline accuracy, different reliability profile, and the second suite tells you exactly which cases to work on. Report pass^k per case or per slice, not only as one number.
from math import comb
def pass_at_k(n: int, c: int, k: int) -> float:
# Chen et al. 2021: P(at least one of k samples succeeds)
if n - c < k:
return 1.0
return 1.0 - comb(n - c, k) / comb(n, k)
def pass_hat_k(n: int, c: int, k: int) -> float:
# Yao et al. 2024 (tau-bench): P(all k trials succeed)
if k > n:
raise ValueError('need at least k trials per case')
return comb(c, k) / comb(n, k)
trials = {'pa-0412': (8, 8), 'pa-0977': (8, 7), 'pa-1203': (8, 5)}
for case_id, (n, c) in trials.items():
print(case_id, round(pass_at_k(n, c, 4), 3), round(pass_hat_k(n, c, 4), 3))Estimating it without fooling yourself
Estimating pass^k needs at least k trials per case, and more is better: with n = k you get an unbiased but very noisy estimate. In the example above, a case with 7 of 8 successes has a pass^4 estimate of C(7, 4) / C(8, 4) = 35 / 70 = 0.5, and a case with 5 of 8 has 5 / 70, about 0.07, while pass@4 is 1.0 for both. The trials must also be genuinely independent: fresh sessions, no shared cache of tool results, and the same sampling settings production uses. Running at temperature zero does not make most hosted models deterministic, and even where it reduces variance, production rarely runs at temperature zero for conversational agents.
Simulated users add a second source of variance. In tau-bench the user is itself a language model following an instruction, so two trials can differ because the user phrased something differently. That is a feature, not noise: real users phrase things differently too. But it means you should fix the user model and its settings across releases, or a change in pass^k may come from the simulator rather than the agent.
Gating on pass^k in CI
Running every case k times is expensive, so we apply pass^k where it matters. The full suite runs each case once and reports single-trial success. The critical subset, the cases whose failure is a safety, disclosure or regulatory event, runs k times, and the gate requires a minimum pass^k on that subset. A typical starting configuration is k = 4 on 50 to 200 critical cases, which costs a few hundred extra runs per full-suite execution and removes most of the single-sample luck that makes gates flaky.
The threshold should come from the risk, not from what the agent currently achieves. If a missed contraindication is unacceptable, the target for that slice is pass^k = 1.0 on the cases you have, and any case below it is a blocking finding with its failed trajectories attached for review.
from gonaturelle import EvalSuite
from agents.prior_auth import agent
def test_critical_cases_pass_every_trial():
suite = EvalSuite.load('healthcare/clinical-prior-auth-v2').filter(tags=['critical'])
results = suite.evaluate(agent=agent, trials=4, max_concurrency=10)
results.assert_zero_contraindications()
results.slice_by('risk_tier').assert_pass_k(k=4, min_rate=1.0, where="risk_tier == 'safety'")
results.assert_pass_k(k=4, min_rate=0.95)
print(results.report(metrics=['pass@1', 'pass^4']))Which number to show whom
Report both, and say which is which. pass@1 is the honest single-run accuracy and the right number for comparing models on a benchmark. pass^k is the reliability number and the right one for a release decision, a model risk review or an Annex IV accuracy declaration, because it describes what users will actually experience across repeated encounters. pass@k with k greater than 1 rarely belongs in a regulated release report at all, since production has no oracle to pick the good sample.
If your current dashboards show only single-run accuracy, add pass^4 on your critical cases this sprint and look at which cases drop. They are usually the ones your team already argues about. Start a 14-day trial to run repeated-trial gates on your own agent, or explore the demo to see a pass^k report on a clinical pack.
PhD in machine learning from IISc Bengaluru on annotator disagreement. Owns the calibration engine, agreement statistics and the Reliability Index methodology.