Clause change omitted from redline summary
A contract review agent summarizes a counterparty's redline and leaves out a material change, often a deletion inside a long clause, so the lawyer approves terms they never saw. Deletions and moved text are where coverage fails most.
A legal team's agent compares the counterparty's returned draft of a master services agreement against the team's last version. Tools: diff_documents(base_id, revised_id) which returns insertions, deletions and moves, get_clause(doc_id, section), and write_change_summary(changes). The agreement is 38 pages; the counterparty returned it with 61 tracked changes.
Summarize the counterparty's changes in order of materiality. Flag anything in limitation of liability, indemnity and data protection.
How the trajectory goes wrong
Removing gross negligence, willful misconduct and data protection breaches from the cap's exclusions means those liabilities are now capped, often at fees paid over 12 months. For a vendor processing the customer's personal data, that can shift most of the realistic breach exposure onto the customer. The summary explicitly says Section 11 has no material changes, which is worse than silence because the lawyer reasonably relies on it. The defect is coverage: the change was in the diff output and never assessed.
Root causes
- Output budgets that push the agent to batch late changes under a generic label.
- Materiality heuristics based on change length, so short deletions are treated as minor.
- No assertion that every diff item receives an individual classification.
- Eval sets with short contracts and few changes, where budget pressure never appears.
- 1Step 1: diff_documents returns 61 changes: 44 insertions, 15 deletions and 2 moves.
- 2Step 2: The agent processes the first 40 changes in detail and groups the rest as 'minor conforming edits' to stay within its output budget.
- 3Step 3: Change 52 is a deletion inside Section 11.3 that removes 'or arising from a party's gross negligence, willful misconduct, or breach of Section 9 (Data Protection)' from the list of exclusions to the liability cap.
- 4Step 4: The agent's grouping labels change 52 as conforming because it sits near a renumbering edit.
- 5Step 5: write_change_summary reports 'Section 11 (Limitation of Liability): no material changes; conforming edits only.'
How GONATURELLE detects it
Change coverage
GONATURELLE asserts that every change ID returned by diff_documents appears in the summary with a classification. Grouping is allowed only when each grouped change is individually listed.
Materiality labels
Attorney reviewers label each change in the golden redlines as material or not, with adjudication. A material change classified as minor or conforming fails the case.
Section claims check
A statement that a section has no material changes is checked against the labelled changes in that section.
Long-document stress
Synthetic Stress places critical deletions late in long redlines with 80 to 150 changes to expose budget-driven batching.
from gonaturelle import EvalSuite, assertions as A
suite = EvalSuite.load('legal/msa-redline-review-v2')
results = suite.evaluate(agent=redline_agent, max_concurrency=6)
results.assert_tool_schema_valid()
results.assert_all([
A.covers_all(source=('diff_documents', 'output.changes[*].id'), target=('write_change_summary', 'args.changes[*].id')),
A.field_matches('write_change_summary', 'args.changes[*].material', key='case.materiality_labels'),
A.no_section_claim_contradiction(claim='no material changes', labels='case.materiality_labels'),
])
results.slice_by('change_count_band').assert_metric('material_recall', min=0.98, min_support=15)Mitigations
- 01Classify changes one by one in a map step, then summarize, so no change depends on output budget.
- 02Treat any deletion in liability, indemnity, data protection or termination clauses as material by rule.
- 03Show the lawyer a per-change table alongside the prose summary.
- 04Report material recall by document length so long-redline regressions are visible.
Regulations this maps to
Questions about this failure
Why not just send all 61 changes to the lawyer?
Lawyers do review them, but the summary sets their attention. The suite checks that the summary does not tell them to look away from the change that matters.
Can the agent group trivial edits?
Yes, as long as each grouped change is listed by ID and classified. The coverage assertion rejects only changes that disappear into a group without assessment.
Scenarios use fictional people and synthetic records for evaluation only. This entry is not legal or financial advice.
Explore the 2026 Vertical AI Reliability Index: failure rates across 12 regulated domainsSchema adherence, trajectory completion, domain factuality and adversarial robustness for 48 models. Free to cite.View the indexStop testing regulated agents on generic prompts.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.