Fast path vs full suite: structuring CI evals

Two jobs that want different things

An evaluation gate in CI has two jobs that pull in opposite directions. On every push, it has to give the author a verdict while they still have the change in their head, which means minutes, not tens of minutes. Before code reaches main, it has to be thorough enough that a regression in a rare but serious case does not slip through, which means thousands of cases, multiple judges and repeated trials. One suite cannot do both. Teams that try end up with either a slow gate that people bypass or a fast gate that tests nothing important.

The structure we use, and recommend, is two tiers. The fast path runs on every push to a branch with an open pull request: a small, curated set of critical cases, deterministic assertions plus calibrated judges, compared against the main-branch baseline. Across our runners the fast path is 20 cases with a median gate time of 74 seconds (GONATURELLE platform telemetry). The full suite runs when a pull request enters the merge queue and nightly on main: every case in the relevant packs, sharded across runners, with repeated trials on the critical subset.

Choosing the 20

Fast-path cases are not a random sample of the full suite. A random 20 out of 3,000 will usually contain no rare failure modes at all. We select them deliberately and re-select them as the agent evolves.

• Severity first: every case whose failure would be a safety, disclosure or regulatory event, such as a missed contraindication, an unverified account access or an adverse action notice without the top factor.
• Coverage: at least one case exercising each write-capable tool and each policy branch that a prompt change could plausibly affect.
• Recent regressions: any case that failed on main in the last 30 days stays in the fast path until it has passed for two weeks.
• Cheap to judge: prefer cases that deterministic assertions can settle, so the fast path spends its time budget on the agent rather than on judges.

The workflow file

In GitHub Actions the split maps onto events. Pushes to pull request branches run the fast path. The merge_group event, fired when a pull request enters a merge queue, runs the full suite sharded four ways. A nightly schedule runs the full suite against main to refresh the baseline and catch drift that comes from outside the repository, such as a provider updating a model behind the same identifier. The concurrency block cancels superseded fast-path runs so a developer pushing three commits in a minute does not pay for three gates.

name: gonaturelle

on:
  pull_request:
  merge_group:
  schedule:
    - cron: '30 1 * * *'

concurrency:
  group: gonaturelle-${{ github.event_name }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
  fast-path:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: gonaturelle/gate-action@v2
        with:
          suite: gonaturelle.yml
          mode: fast-path
          baseline: main
          comment: true
          api-key: ${{ secrets.GONATURELLE_API_KEY }}

  full-suite:
    if: github.event_name != 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 45
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: gonaturelle/gate-action@v2
        with:
          suite: gonaturelle.yml
          mode: full
          shard: ${{ matrix.shard }}/4
          baseline: main
          junit-path: reports/gonaturelle-${{ matrix.shard }}.xml
          api-key: ${{ secrets.GONATURELLE_API_KEY }}

Budgeting runs, with the arithmetic shown

Eval runs cost money in model tokens and in plan quota, so size the tiers with arithmetic rather than instinct. Take a team that pushes to pull request branches 40 times a working day. At 20 fast-path cases per push and 22 working days a month, that is 40 x 20 x 22 = 17,600 runs. Add a full suite of 1,500 cases that runs nightly (30 times a month) and on about 60 merge-queue entries a month: 1,500 x 90 = 135,000 runs. The total, roughly 152,600 runs a month, fits comfortably inside the Scale plan's 250,000 and well outside the Developer plan's 50,000.

The same arithmetic shows where to cut if you need to. The full suite dominates, so running it on merge-queue entries only for pull requests that touch prompts, tools or model configuration, and skipping it for documentation changes, often halves the total. Caching helps further: a case whose dataset version, prompt hash and model version are unchanged since the last run on that branch does not need to re-execute. Overage on either plan is billed at $6 per 1,000 runs, so an occasional spike costs a few dollars rather than a blocked gate.

Flakiness is the real enemy

A gate that fails one run in twenty for no reason trains everyone to click re-run, and after a month it is advisory in practice even if it is required in settings. Most eval flakiness comes from three sources. Sampling variance: even at temperature zero, many model APIs are not bit-for-bit deterministic. Unpinned versions: a judge model or a dataset that changed under you. And external tools: a sandboxed tool that hits a live endpoint and times out.

The fixes are mechanical. Pin dataset, judge and calibration versions in the suite file and make version changes their own pull requests. Replace live tool endpoints with recorded fixtures in the fast path. And for critical cases, do not gate on a single sample. Run them several times and require the agent to pass every trial, which is the pass^k idea from tau-bench applied to a gate. It costs more runs on a small set of cases and removes most of the coin-flip failures, because a case that passes 4 out of 4 is telling you something a single pass is not.

What the developer sees

The fast path posts a single PR comment and updates it in place on each push: pass or fail, the cases that changed relative to baseline with the first diverging step, score deltas per rubric, and token cost per case compared with main. The full suite adds a JUnit report per shard so failures appear in the same CI views engineers already use, and a link to the trajectory diff timeline for anything that regressed. Make the fast-path check required in branch protection and the full-suite check required for the merge queue. Then measure the gate itself: median time, re-run rate and override rate, and treat a rising re-run rate as a bug in the gate. To see this layout on your own repository, run a free eval suite during the 14-day trial.