The LL144 Bias Audit Process: What to Expect

If your company uses any AI tool to screen, score, or rank job candidates for positions based in New York City, NYC Local Law 144 of 2021 requires you to conduct an annual independent bias audit of that tool. This guide walks you through the entire process — from determining whether your tool qualifies as an AEDT to publishing your audit results and notifying candidates.

Step 1: AEDT Inventory

The first step is identifying which tools in your hiring stack qualify as Automated Employment Decision Tools. Under DCWP's final rules (published April 2023), an AEDT is any computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues a simplified output — a score, classification, or recommendation — used to substantially assist or replace discretionary decision-making for employment decisions.

The "substantially assists" threshold is critical. DCWP clarified that this means the employer either relies exclusively on the tool's output, weighs the output more heavily than any other criterion, or uses the output to overrule human judgment that had reached a different conclusion. A tool that merely provides one data point among many, where the human decision-maker gives it no special weight, may not qualify.

Common tools that typically qualify include AI resume screening software (e.g., tools that automatically filter or rank applicants), automated video interview analysis platforms, candidate scoring algorithms, and AI-driven promotion recommendation systems. For a detailed scope analysis, see Does Local Law 144 Apply to Your Company?

Step 2: Auditor Selection

LL144 requires the bias audit to be conducted by an independent auditor. The law specifies that the auditor cannot be the developer of the AEDT being audited, cannot currently be marketing or selling the AEDT, and must be free from any financial interest that could create a conflict of interest.

This is where many companies run into trouble. The SaaS platforms offering "LL144 compliance" dashboards and automated bias testing often have business relationships with AEDT vendors — or are AEDT vendors themselves offering compliance as an add-on. Using such a service may not satisfy the independence requirement. When selecting an auditor, verify three things: (1) no development or sale of AEDTs, (2) no financial relationship with your AEDT vendor, and (3) ability to provide a written independence declaration for your records.

Step 3: Data Collection and Preparation

The bias audit requires demographic data on the candidates or employees evaluated by your AEDT. Specifically, the audit must analyze outcomes across race/ethnicity categories (Hispanic or Latino, White, Black or African American, Asian, and Native Hawaiian or Other Pacific Islander) and sex categories (Male, Female), including intersectional breakdowns combining both.

You have two data options. The preferred approach is historical data — actual candidate data from prior AEDT use. If historical data is unavailable (for example, if the AEDT is newly deployed), you may use test data, but this must be disclosed in the audit summary. The auditor should work with you to collect, clean, and prepare the dataset, ensuring it contains sufficient records across all required demographic categories to produce statistically meaningful results.

An important consideration: if you do not collect demographic data on applicants, you will need to either begin collecting it (with appropriate consent mechanisms) or use proxy data or test data. Lack of data does not exempt you from the audit requirement.

Step 4: Bias Analysis and Impact Ratios

The core of the audit is calculating selection rates and impact ratios. For each demographic group, the auditor computes the selection rate (the proportion of candidates from that group who were selected or advanced by the AEDT). Impact ratios are then calculated by dividing each group's selection rate by the selection rate of the most-selected group.

An impact ratio below 0.80 (the four-fifths rule, derived from EEOC Uniform Guidelines) may indicate adverse impact against that demographic group. However, a ratio below 0.80 does not automatically mean the AEDT is illegal — it means further investigation is warranted. The auditor documents all calculations, methodology, and any areas of concern.

Intersectional analysis adds another layer. The auditor must calculate impact ratios not only for race/ethnicity alone and sex alone, but also for combinations (e.g., Hispanic Female, Black Male). This intersectional analysis often reveals disparities that single-axis analysis misses.

Step 5: Audit Report and Disclosure Package

The auditor produces a formal audit summary that must include: the date of the most recent bias audit, the distribution date of the AEDT (when it was first used), a summary of results including selection rates and impact ratios for all required categories, and a description of the source and type of data used.

This summary must be published on your company's website in a clear and conspicuous manner. It cannot be hidden behind login walls, buried in footnotes, or placed on pages that require more than one click to reach from the homepage. The summary must remain publicly accessible for at least six months after the most recent use of the AEDT in an employment decision.

Step 6: Candidate Notification

Separately from the audit, you must notify each candidate or employee at least 10 business days before an AEDT is used in their evaluation. The notice must include: a statement that an AEDT will be used, the job qualifications and characteristics the AEDT evaluates, the data sources collected for the AEDT, data retention policy information, and instructions for how to request an alternative selection process or reasonable accommodation.

Notice can be provided in three ways: on the careers or employment section of your website, in the job listing itself, or via direct communication (email, letter, etc.) to the candidate. Many employers choose to include the notice in job postings and on their careers page simultaneously to ensure coverage.

Step 7: Annual Renewal and Ongoing Compliance

The audit is not a one-time obligation. It must be renewed annually — within 12 months of the prior audit — for as long as you continue using the AEDT. If you change your AEDT or significantly modify its algorithm, a new audit should be conducted even if the prior one is less than 12 months old.

Many companies also benefit from maintaining an internal AEDT register documenting all AI tools used in hiring, their scope determination status, audit dates, and disclosure locations. This creates an audit trail that demonstrates ongoing compliance.

Lexara Advisory LLC — AI governance consulting, not legal practice. This article provides general compliance information only and does not constitute legal advice.