NYC Local Law 144 AEDT Bias Audit Testing Guide

Statutory requirements

1. 01

   What counts as an AEDT

   Admin Code 20-870; 6 RCNY 5-300

   An automated employment decision tool is a computational process derived from machine learning, statistical modelling, data analytics or artificial intelligence that issues simplified output, including a score, classification or recommendation, used to substantially assist or replace discretionary decision making for employment decisions that impact natural persons. The DCWP rules define substantially assist or replace, including relying solely on the output or weighting it more than any other criterion.

2. 02

   Independent bias audit within one year

   Admin Code 20-871(a)(1); 6 RCNY 5-300 and 5-301

   An employer or employment agency may not use an AEDT unless it has been the subject of a bias audit by an independent auditor no more than one year before use. The auditor cannot have been involved in using, developing or distributing the tool, or have a direct financial interest in it.

3. 03

   Impact ratios

   6 RCNY 5-301

   For tools that select candidates, the audit calculates the selection rate for each category and the impact ratio: the category's selection rate divided by the rate of the most selected category. For tools that score, it uses the scoring rate, the rate at which individuals in a category receive a score above the sample's median score. Ratios are reported for sex, race and ethnicity, and intersectional combinations, along with the number of individuals in unknown categories.

4. 04

   Data used for the audit

   6 RCNY 5-301 and 5-302

   Audits generally use historical data of the employer's use of the tool. Where historical data is insufficient, test data may be used, with an explanation in the summary. A category representing less than 2% of the data may be excluded from the impact ratio calculation, with the justification and the number and selection or scoring rate of that category disclosed.

5. 05

   Publication, notice and penalties

   Admin Code 20-871(b) and 20-872; 6 RCNY 5-303 and 5-304

   Employers must publish a summary of the latest audit results and the tool's distribution date on their website. Candidates and employees who reside in the city must receive notice at least 10 business days before use, including the job qualifications and characteristics the tool will assess and how to request an alternative selection process or accommodation. Penalties run up to $500 for a first violation and $500 to $1,500 for each subsequent violation, with each day of use a separate violation.

Evaluation methodologies

Testing artifacts to retain

The evidence folder an examiner, notified body or internal auditor will ask for. GONATURELLE compliance reports export most of it from signed runs.

GONATURELLE assertion rules

Each assertion maps to a requirement above. Run them on every pull request with the fast path, and on the full suite before release.

from gonaturelle import EvalSuite, TrajectoryTester, assertions as A
from gonaturelle.fairness import impact_ratios

suite = EvalSuite.load('hr/resume-screening-nyc', data='historical_2026h1')
results = suite.evaluate(agent=screening_agent, max_concurrency=10)

# 6 RCNY 5-301: scoring rate (above sample median) impact ratios
ratios = impact_ratios(results, mode='scoring_rate', categories=['sex', 'race_ethnicity'], intersectional=True, exclude_below=0.02)
ratios.flag_below(0.80)  # investigation trigger, not a legal threshold
ratios.export_summary('ll144/audit_summary.json', include_unknown_counts=True)

# Causal sensitivity: matched resumes must score alike
results.paired('name_pronoun_gradyear_swaps').assert_invariant(['score_band'], tolerance=0)

# Notice before assessment; alternative process reaches a human
TrajectoryTester(results).assert_order('deliver_aedt_notice', 'score_candidate')
TrajectoryTester(results).assert_implies('request_alternative_process', 'route_to_recruiter')
results.assert_all([A.no_pattern_in_tool_args(patterns=['age', 'pregnan', 'disab'], tools=['score_candidate'])])

results.export_manifest('ll144/config_bound_run.json', sign=True)

Audit checklist