Glossary

67 terms that come up when ML engineers, clinicians and compliance officers review the same agent.

Every term has a stable anchor: link to /resources/glossary#pass-hat-k and similar.

67 of 67 terms

A

Adversarial robustness: How stable an agent's behaviour is when inputs are perturbed or attacked: typos, code-mixed language, hostile tone, irrelevant history, or instructions injected through retrieved content. Measured by comparing each perturbed variant against the clean case and counting how often the trajectory and verdict are preserved.; RelatedPrompt injection Perturbation testing Red teaming
Adverse action notice: The notice a creditor must give an applicant when it denies credit, changes terms unfavourably or takes other adverse action. Under the Equal Credit Opportunity Act and Regulation B (12 CFR 1002.9) it must include a statement of the specific principal reasons. The duty applies when the decision comes from a complex algorithm: CFPB Circulars 2022-03 and 2023-03 said so explicitly before the CFPB withdrew them in May 2025, and the requirement in Regulation B itself, with its Official Interpretations, did not change. For an AI underwriting agent the testable property is that the stated reasons match the model's real drivers.; RelatedDisparate impact Reason code alignment Model risk management
Annex IV: The annex of the EU AI Act (Regulation (EU) 2024/1689) listing what the technical documentation of a high-risk AI system must contain: a general description, the development process and design choices, data and data governance, monitoring and control, validation and testing procedures with metrics and test logs, the risk management system, changes over the lifecycle and the post-market monitoring plan. Article 11 requires the documentation before the system is placed on the market.; RelatedHigh-risk AI system Conformity assessment Technical documentation
Assertion: A deterministic check that a test run must satisfy, written as code: a tool argument validates against a schema, one call happens before another, a pattern never appears in outbound arguments. Assertions fail loudly and identically on every run, which is why they gate merges more reliably than a judge score.; RelatedTrajectory assertion Tool-call schema Regression gate

B

Bias audit: An impartial evaluation of whether an automated decision tool produces different outcomes across demographic groups. New York City Local Law 144 requires a bias audit by an independent auditor within one year before an employer uses an automated employment decision tool, reporting selection rates and impact ratios by sex and race or ethnicity.; RelatedDisparate impact Impact ratio Protected class
Business associate agreement: The contract HIPAA requires between a covered entity and a business associate that creates, receives, maintains or transmits PHI on its behalf. It limits how the PHI may be used and disclosed and obliges the associate to safeguard it and report breaches. An evaluation vendor that receives real clinical traces needs one.; RelatedPHI De-identification

C

Calibration (judge): The process of tuning an LLM-as-a-judge, or an ensemble of judges, until its verdicts agree with credentialed human experts on a held-out labelled set. Calibration adjusts rubrics, few-shot anchors, thresholds and ensemble weights, and is reported as an agreement statistic such as Cohen's kappa against the adjudicated expert label.; RelatedLLM-as-a-judge Cohen's kappa Judge ensemble
Canary string: A unique random string embedded in a private benchmark so that its appearance in a model's output or training corpus reveals contamination. A model that can complete the canary has very likely seen the test set.; RelatedData contamination Golden dataset
Cohen's kappa: An agreement statistic for two raters assigning categorical labels that corrects for the agreement expected by chance: kappa = (observed agreement minus chance agreement) divided by (1 minus chance agreement). By the Landis and Koch (1977) convention, 0.61 to 0.80 is substantial and 0.81 to 1.00 almost perfect. In evaluation it measures both human inter-annotator agreement and judge-to-expert agreement. It can behave oddly when one label dominates, which is why Gwet's AC1 is often reported alongside it.; RelatedGwet's AC1 Inter-annotator agreement Calibration (judge)
Conformity assessment: Under the EU AI Act, the procedure that demonstrates a high-risk AI system meets the requirements of Chapter III, Section 2 before it is placed on the market. For most Annex III systems it is an internal control procedure by the provider; some systems, including certain biometric uses and AI that is a safety component of products already under third-party assessment, involve a notified body.; RelatedAnnex IV High-risk AI system
Context window: The maximum number of tokens a model can attend to in a single request, covering the system prompt, conversation history, retrieved documents and tool results. Long multi-turn agent sessions can push early instructions or verification results out of effective attention even before the hard limit.; RelatedTrajectory Tool call

D

Data contamination: When benchmark or test cases leak into a model's training data, inflating its scores. Private golden sets, rotating case pools, canary strings and comparing scores on retired versus live cases are the usual defences.; RelatedCanary string Golden dataset
De-identification: Removing or transforming identifiers so that data is no longer individually identifiable. Under the HIPAA Privacy Rule (45 CFR 164.514) health information is de-identified either by the Safe Harbor method, removing 18 listed identifier types with no actual knowledge that the rest could identify the person, or by Expert Determination that the re-identification risk is very small.; RelatedPHI PII redaction Synthetic data
Disparate impact: A facially neutral practice that disproportionately disadvantages members of a protected class. In US fair lending and employment law it can be unlawful even without intent unless justified by a legitimate business need that cannot be met by a less discriminatory alternative. For AI systems it is tested by comparing outcome rates across groups and probing for proxy variables.; RelatedProtected class Impact ratio Bias audit Adverse action notice
Domain factuality: Correctness of an output judged against a domain expert's standard rather than a generic one: the contraindication named, the reason for adverse action specific and accurate, the cited authority real and still good law. Scored with domain rubrics and calibrated judges because string overlap metrics such as BLEU or ROUGE cannot see a missing contraindication.; RelatedHallucination Rubric Calibration (judge)
Drift (cost and latency): Gradual change in tokens per case, dollar cost or response time across prompt and model versions. A prompt edit that improves accuracy by a point but doubles tokens per case is a regression for a high-volume claims or coding agent, so gates track drift beside quality.; RelatedRegression baseline Regression gate

E

Edge case: An input at the boundary of normal operation where behaviour is most likely to break: a pediatric patient at the weight cut-off, a loan applicant with a thin file, a contract with conflicting governing-law clauses. In regulated verticals the expensive failures cluster here, which is why golden sets over-sample them deliberately.; RelatedGolden dataset Synthetic data Perturbation testing
Escalation: Handing a case from an agent to a human or a higher-acuity pathway when defined triggers fire: red-flag symptoms in triage, a sanctions hit in KYC, a coverage question outside policy. Missing an escalation is usually a sequence failure, testable by asserting that the escalation tool is called before any other action once the trigger is present.; RelatedHuman-in-the-loop Sequence constraint

F

Fast path: A small, high-signal subset of a test suite, typically the 20 or so most critical cases, run on every push so that a pull request gets a verdict in about a minute. The full suite runs on merge requests or nightly. GONATURELLE platform telemetry puts the median fast-path gate in GitHub Actions at 74 seconds.; RelatedFull suite Regression gate Shard
Full suite: The complete set of cases for an agent, often thousands of cases including perturbation variants, run on merge requests, nightly or before release. Sharded across parallel runners so that wall-clock time stays manageable.; RelatedFast path Shard

G

GMLP: Good Machine Learning Practice for Medical Device Development: ten guiding principles published in October 2021 by the US FDA, Health Canada and the UK MHRA. They cover multidisciplinary expertise, good software engineering and security, representative clinical study data, independence of training and test sets, reference datasets based on best available methods, human-AI team performance, clinically relevant testing conditions, clear information for users, and monitoring of deployed models.; RelatedSaMD PCCP Golden dataset
Golden dataset: A curated, versioned set of test cases with expert-verified expected behaviour that serves as the reference for evaluation. In regulated verticals each case is labelled by credentialed reviewers, disagreements are adjudicated and the set is frozen per version so that a score on one version always means the same thing.; RelatedInter-annotator agreement Semantic versioning (datasets)Edge case
Grounding: Constraining a model's claims to evidence it was given, such as retrieved documents, tool outputs or the patient chart, and making that link checkable. A grounded answer can be traced claim by claim to a source span; an ungrounded one cannot, even if it happens to be true.; RelatedHallucination Retrieval-augmented generation
Gwet's AC1: An agreement coefficient proposed by Kilem Gwet that, like Cohen's kappa, corrects for chance, but estimates chance agreement in a way that stays stable when one category dominates. It avoids the kappa paradox in which two raters agree on 95% of cases yet get a low kappa because almost every case is a pass. Useful for safety rubrics where violations are rare.; RelatedCohen's kappa Inter-annotator agreement

H

Hallucinated argument: A tool-call argument whose value no upstream step produced: an account number, dose, claim ID or citation the model invented rather than read from the user, a retrieval result or a previous tool output. It can pass schema validation perfectly, so it is detected by tracing each argument value back to a source in the trajectory.; RelatedTool call Trajectory assertion Hallucination
Hallucination: Model output that is fluent but not supported by its inputs or by fact: a fabricated case citation, a lab value absent from the chart, a policy clause that does not exist. Magesh et al. (Stanford RegLab and HAI, 2024) measured hallucination rates of 17% to 33% in leading AI legal research tools.; RelatedHallucinated argument Domain factuality Grounding
High-risk AI system: Under Article 6 of the EU AI Act, an AI system that is a safety component of, or itself, a product covered by the EU harmonisation legislation in Annex I and subject to third-party conformity assessment, or that falls in an Annex III use case such as creditworthiness assessment of natural persons, risk assessment and pricing for life and health insurance, or access to essential services. High-risk systems must meet requirements on risk management, data governance, technical documentation, logging, transparency, human oversight, accuracy, robustness and cybersecurity. After the Digital Omnibus amendments entered into force in July 2026, Annex III high-risk obligations apply from 2 December 2027 and Annex I obligations from 2 August 2028 (as of November 2026).; RelatedAnnex IV Conformity assessment
Human-in-the-loop: A design where a qualified person reviews, approves or can override an AI system's output before it takes effect. Effective oversight requires that the reviewer has the information and time to disagree; a reviewer who rubber-stamps a fluent recommendation is not meaningful oversight.; RelatedHigh-risk AI system Escalation

I

Impact ratio: The selection or approval rate of one group divided by the rate of the most-selected group. The four-fifths rule in the US Uniform Guidelines on Employee Selection Procedures treats a ratio below 0.8 as evidence of adverse impact. NYC Local Law 144 bias audits report impact ratios by category.; RelatedDisparate impact Bias audit
Inter-annotator agreement: How consistently independent human labellers assign the same label to the same case, reported with a chance-corrected statistic such as Cohen's kappa (two raters), Fleiss' kappa (many raters) or Gwet's AC1. Low agreement means the rubric is ambiguous, and no judge can be calibrated to a target the experts themselves do not share.; RelatedCohen's kappa Gwet's AC1 Golden dataset

J

Judge ensemble: Several scorers combined into one verdict: deterministic rules, a small specialized grader and one or more frontier models from different families, each weighted by its measured agreement with experts. Ensembles reduce the variance and self-preference of any single LLM judge.; RelatedLLM-as-a-judge Calibration (judge)Self-enhancement bias

L

LLM-as-a-judge: Using a language model to grade another model's output against a rubric or by pairwise comparison. Zheng et al. (NeurIPS 2023) found strong LLM judges reach over 80% agreement with human preferences on open-ended chat, comparable to human-human agreement, while documenting position, verbosity and self-enhancement biases. In regulated domains judges must be calibrated against credentialed experts before their scores gate anything.; RelatedCalibration (judge)Position bias Verbosity bias Self-enhancement bias
Loop detection: Detecting an agent that repeats the same tool call, or a short cycle of calls, without making progress, and capping calls per turn and per task. Runaway loops burn tokens, hit rate limits and can repeat side effects such as duplicate claim submissions.; RelatedTrajectory assertion Tool call

M

Model risk management: The framework banks use to identify, measure and control the risk of decisions based on incorrect or misused models. In the US it was defined for fifteen years by SR 11-7 (2011): sound development and implementation, effective independent validation (conceptual soundness, ongoing monitoring, outcomes analysis) and governance with a model inventory. SR 11-7 was rescinded in April 2026 and replaced by SR 26-2 (OCC Bulletin 2026-13), which excludes generative and agentic AI from its scope, so most banks now govern LLM agents through their own internal model-risk and AI policies built on the same validation ideas.; RelatedSR 11-7 Regression baseline Adverse action notice
Multi-turn evaluation: Testing an agent across a whole conversation or task with several exchanges and tool calls, often with a simulated user, rather than one prompt and one reply. Most regulated workflows, from prior authorization to loan servicing, only fail in the third or fifth turn.; RelatedTrajectory User simulator pass^k

P

pass@k: The probability that at least one of k independent attempts succeeds. Popularised for code generation (Chen et al., 2021), it rewards a model that sometimes gets it right. It is the right metric when a human picks the best of k candidates, and the wrong one for an agent that acts once on a real customer.; Relatedpass^k Multi-turn evaluation
pass^k: The probability that all k independent attempts at the same task succeed, introduced with tau-bench (Yao et al., 2024). It measures consistency, which is what a regulated deployment needs. In tau-bench's retail domain the best function-calling agent succeeded on under 50% of tasks in a single trial and its pass^8 fell below 25%.; Relatedpass@k tau-bench Multi-turn evaluation
PCCP: Predetermined Change Control Plan: a section of an FDA marketing submission for an AI-enabled device that describes planned modifications, the methodology to develop, validate and implement them, and an impact assessment. Once authorized, changes made within the plan do not require a new submission. FDA finalized its guidance on PCCPs for AI-enabled device software functions in December 2024.; RelatedSaMD GMLP Regression baseline
Perturbation testing: Expanding each clean test case into variants that should not change the correct behaviour (typos, reordered history, code-mixed phrasing, distracting details, different formatting) and checking that the agent's trajectory and verdict hold. A cheap way to find brittle prompts before users do.; RelatedAdversarial robustness Synthetic data Edge case
PHI: Protected health information: individually identifiable health information held or transmitted by a HIPAA covered entity or business associate, in any form. For agents the risk sits in places teams forget to check: tool-call arguments, logs, traces, prompts sent to third-party models and cached retrieval results.; RelatedDe-identification PII redaction Business associate agreement
PII redaction: Detecting and masking personal identifiers such as names, account numbers, national IDs, addresses and dates of birth in text before it is stored, logged or sent to another system. Assertions can require that no identifier pattern appears in any outbound tool argument.; RelatedPHI De-identification
Position bias: An LLM judge's tendency to prefer a response because of where it appears, usually the first of two in a pairwise comparison. Controlled by scoring each pair in both orders and only counting consistent verdicts, as described by Zheng et al. (2023).; RelatedLLM-as-a-judge Verbosity bias Self-enhancement bias
Prompt injection: Input crafted to override an agent's instructions. Direct injection comes from the user; indirect injection arrives through content the agent reads, such as a retrieved web page, an uploaded PDF, an email or a tool result. OWASP lists it first in its Top 10 for LLM applications. Tests seed injected instructions into retrieved documents and assert the agent never executes them.; RelatedAdversarial robustness Red teaming Retrieval-augmented generation
Protected class: A characteristic on which discrimination is prohibited. Under ECOA these include race, color, religion, national origin, sex, marital status, age (with limited exceptions), receipt of public assistance income and good-faith exercise of Consumer Credit Protection Act rights. Models must also be tested for proxies, such as ZIP code or first name, that encode a protected class indirectly.; RelatedDisparate impact Adverse action notice

R

Reason code alignment: Checking that the reasons an AI credit agent states for an adverse action are the factors that actually drove the decision, for example by comparing them with feature attributions or a counterfactual analysis. Misaligned reasons are the failure Regulation B's specific-reasons requirement targets, the point the CFPB's since-withdrawn Circular 2023-03 made about algorithmic underwriting.; RelatedAdverse action notice Model risk management
Red teaming: Structured adversarial testing in which people or automated generators try to make a system fail, leak data or behave unsafely. Findings become permanent regression cases so the same attack is tested on every future version.; RelatedPrompt injection Adversarial robustness
Regression baseline: The frozen scores of a reference version, usually the main branch, against which a candidate change is compared case by case. A regression is a case that passed on the baseline and fails on the candidate, even if the aggregate score barely moves.; RelatedRegression gate Fast path Drift (cost and latency)
Regression gate: A required CI check that blocks a merge when the candidate regresses against the baseline beyond set thresholds: any critical assertion failing, pass rate falling, judge score dropping or cost and latency drifting. The gate's output, signed and stored, doubles as change-control evidence.; RelatedRegression baseline Fast path Signed run manifest
Retrieval-augmented generation: Supplying a model with documents fetched at query time, from a vector index, search engine or database, so that it can answer from current, specific sources. Evaluation must cover retrieval (did the right passages come back) and generation (did the answer stay within them).; RelatedGrounding Prompt injection
Rubric: The written scoring criteria a judge, human or model, applies to an output: what counts as correct, which errors are critical, how partial credit works. A good rubric is specific enough that two experts reach the same score independently, which is what inter-annotator agreement measures.; RelatedDomain factuality Inter-annotator agreement LLM-as-a-judge

S

SaMD: Software as a Medical Device: software intended for one or more medical purposes that performs them without being part of a hardware medical device, as defined by the International Medical Device Regulators Forum (IMDRF). In the US, clinical decision support software can fall outside the device definition only if it meets all four criteria in section 520(o)(1)(E) of the FD&C Act, including letting the clinician independently review the basis for the recommendation.; RelatedGMLP PCCP
Self-enhancement bias: An LLM judge's tendency to score outputs from its own model, or its own model family, more favourably. Zheng et al. (2023) reported it for some judges. Controlled by using judges from a different family than the model under test and by calibrating against human experts.; RelatedLLM-as-a-judge Judge ensemble Position bias
Semantic versioning (datasets): Versioning a golden dataset as MAJOR.MINOR.PATCH: a patch fixes labels without changing scope, a minor version adds cases, a major version changes the rubric or task definition so scores are no longer comparable. It lets a team say what a score of 0.94 on version 2.3.0 means a year later.; RelatedGolden dataset Regression baseline
Sequence constraint: A rule about the order of steps in a trajectory: verify the member before fetching benefits, verify identity before reading a balance, check allergies before recommending a drug. Expressed as an ordering assertion over tool calls.; RelatedTrajectory assertion Tool call Escalation
Shard: One slice of a test suite run on its own worker so that thousands of cases finish in minutes. Sharding by case ID keeps results stable across runs; sharding by expected duration keeps workers finishing together.; RelatedFull suite Fast path
Signed run manifest: A record of exactly what a test run used and produced (dataset version, prompt and model versions, judge configuration, thresholds, scores) with a cryptographic hash such as SHA-256 so that any later change is detectable. It turns a CI result into audit evidence.; RelatedRegression gate Annex IV
SR 11-7: Supervisory Guidance on Model Risk Management, issued by the Federal Reserve in April 2011 (with OCC Bulletin 2011-12). It defined a model broadly and required independent validation, ongoing monitoring, outcomes analysis and a firm-wide model inventory. It was rescinded on 17 April 2026 and replaced by SR 26-2 (OCC Bulletin 2026-13), which excludes generative and agentic AI; many banks still use SR 11-7's validation structure internally for LLM systems.; RelatedModel risk management Regression baseline
Suitability: The obligation under FINRA Rule 2111 that a broker-dealer have a reasonable basis to believe a recommended transaction or strategy is suitable for the customer, based on the customer's investment profile (age, other investments, financial situation, tax status, objectives, experience, time horizon, liquidity needs and risk tolerance). For retail customers Regulation Best Interest now applies to recommendations, with Rule 2111 continuing for others. An advisory agent must gather the profile before recommending.; RelatedSequence constraint Model risk management
Synthetic data: Generated test inputs, such as fictional patients, applicants or claims, built to resemble real distributions without containing real people's records. Used to multiply one real edge case into many variants. Synthetic cases are for evaluation only and still need expert review of the expected label.; RelatedEdge case Perturbation testing De-identification

T

tau-bench: A benchmark by Yao et al. (2024, arXiv:2406.12045) in which a tool-using agent talks with an LLM-simulated user under domain policies in retail and airline settings, and success is judged by comparing the final database state with the goal state. It introduced the pass^k metric for consistency across repeated trials.; Relatedpass^k User simulator Multi-turn evaluation
Technical documentation: In the EU AI Act, the documentation a provider of a high-risk AI system must draw up before market placement (Article 11) and keep up to date, with the content set out in Annex IV. Test logs, metrics and the validation procedure are part of it.; RelatedAnnex IV Conformity assessment
Tool call: A structured request from a model to run an external function, with a name and JSON arguments, such as fetch_account_balance(account_id) or submit_prior_auth(cpt_code, units). The application executes it and returns the result to the model. In agents, most consequential actions happen through tool calls, not text.; RelatedTool-call schema Hallucinated argument Trajectory
Tool-call schema: The JSON Schema declaring a tool's parameters: names, types, required fields, enums, numeric ranges and string patterns. Validating every call against it, plus domain bounds the schema cannot express, catches a large class of failures. In the GONATURELLE Failure Modes Catalog analysis (2026), 32.8% of multi-step agent failures were tool-argument schema or parameter-boundary violations.; RelatedTool call Assertion Hallucinated argument
Trajectory: The full sequence of an agent's steps for one task: messages, reasoning where exposed, tool calls with arguments, tool results and state changes. Evaluating the trajectory, not just the final answer, catches agents that land on a correct-looking answer by an unsafe or non-compliant route.; RelatedTrajectory assertion Tool call Multi-turn evaluation
Trajectory assertion: An assertion over the steps of a trajectory rather than its final output: required calls, forbidden calls, order, argument provenance, maximum calls per turn and expected end state. In the GONATURELLE regression study (2026), 41% of prompt-only pull requests changed at least one tool-call trajectory while the final-answer score stayed flat.; RelatedTrajectory Sequence constraint Assertion

U

User simulator: A model or scripted agent that plays the customer, patient or applicant in a multi-turn test, following a hidden goal and persona. It lets suites test long conversations at scale, but its own behaviour must be pinned down (fixed seed, constrained instructions) or it adds noise to every score.; RelatedMulti-turn evaluation tau-bench

V

Verbosity bias: An LLM judge's tendency to prefer longer answers regardless of quality. Controlled with length-normalized rubrics, explicit penalties for unsupported content and calibration against experts who are told to ignore length.; RelatedLLM-as-a-judge Position bias

Want the long version? The agent eval guide puts these terms to work, and the regulatory guides map the legal ones to tests.

Monthly newsletter

The Regulated Agent

A monthly deep dive on AI evaluations, safety benchmarks and regulatory enforcement. One email, first Tuesday of the month.

Stop testing regulated agents on generic prompts.

Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.

Start free trial Book a technical demo