NYC Local Law 144 — Module 2
The tool is classified. Now someone has to audit it — and the vendor is offering to do it for free.
Your Role
People Operations Manager, Vantage Property Group
Six weeks ago you classified TalentScope as an AEDT. The procurement memo is amended. The CFO hire for Vantage Capital landed four weeks late. Priya Venkatesan has retained counsel.
Today the bias audit sourcing memo is due. Nora's at the 9 AM stand-up. Gil Fontaine is on the calendar at 10.
Three decisions. Each one costs something real — budget, timeline, or relationship. Under NYC Local Law 144, the “right” answer is sometimes the expensive one.
§ Section references — click to read the NYC Admin Code section.
From: Marcus Chen <m.chen@chenruiz.law>
To: Sam Rourke <s.rourke@vantagepg.com>
Cc: General Counsel <legal@vantagepg.com>
Preservation Notice — In re: TalentScope AEDT use / Venkatesan
Ms Rourke,
This firm represents Priya Venkatesan in connection with Vantage Property Group's use of the TalentScope automated employment decision tool.
This letter is a formal request that Vantage preserve all records relating to TalentScope from 1 January 2024 to date, including but not limited to: candidate-level scoring data, feature weights, methodology documentation, internal classification memos, any bias audit conducted or proposed, and all communications between Vantage personnel and TalentScope personnel.
We note Ms Venkatesan has not yet filed a complaint. She is observing the conduct of the forthcoming bias audit as required under NYC Admin Code § 20-872 before determining next steps.
Regards,
Marcus Chen, Partner — Chen & Ruiz LLP
Your phone buzzes. Jenna: She lawyered up. Anya Petrov is who you want. I’ll intro.
Nora’s text comes in thirty seconds later: Gil just emailed me. Says their audit division can do this in two weeks, no charge. Meeting at 10 still on?
| Proposal | Auditor | Fee | Timeline | Independence |
|---|---|---|---|---|
| TalentScope CAD | Ethan Bright (TalentScope employee) | $0 | 2 weeks | Employee of vendor |
| Petrov Analytics | Dr. Anya Petrov (Petrov Analytics LLC) | $18,000 | 4 weeks | No relationship |
| Nora’s preferred | TalentScope CAD | $0 | 2 weeks | See row 1 |
The gap between the two rows is eighteen thousand dollars, two weeks, and the entire point of the regulation.
Gil’s call is at 10. Nora wants a recommendation before it.
Nora reads the preservation letter and the two proposals. She says:
“I know what you’re going to tell me. Tell me anyway. Eighteen grand, six months after the CFO delay, with Q3 already under pressure. Is there a version of this I can defend?”
What auditor do you recommend?
Your choice
Dr. Anya Petrov — independent auditorNo employment or financial relationship with Vantage or TalentScope. Four-week turnaround lands inside the deadline. $18,000 is less than the firm’s weekly legal bill with Chen & Ruiz watching.
Your choice
TalentScope runs analysis; independent certifier reviewsEthan’s team does the statistical work, Dr. Petrov reviews and certifies the methodology and output. Costs around $6,000. Splits the difference.
Your choice
TalentScope Certified Audit DivisionFree. Two weeks. 200 client references. Operational separation from the product team. Meets the budget constraint and the deadline.
Independence isn’t about the quality of the methodology — it’s about who employs the auditor. An auditor employed by the vendor cannot meet the independence standard, regardless of how rigorous their analysis is. Gil’s offer is structurally disqualified before anyone reviews the methodology.
Asking an independent auditor to certify analysis they didn’t conduct is asking them to stake their professional licence on work they can’t verify. Dr. Petrov’s refusal is the correct outcome — the half-measure collapses the moment the person you need for independence won’t stand behind it.
Structural independence and operational independence are different things. An internal firewall keeps teams from talking to each other. It doesn’t change who signs Ethan’s paycheck. The vendor cannot audit its own tool under any reporting structure.
From: Dr. Anya Petrov <a.petrov@petrovanalytics.nyc>
To: Sam Rourke <s.rourke@vantagepg.com>
Cc: Gil Fontaine <g.fontaine@talentscope.ai>
Data access — formal request to TalentScope (urgent)
Sam — attached, my formal data request. I need candidate-level scores, EEOC-category demographic data, selection outcomes, and the feature weighting matrix.
Gil’s reply (attached): TalentScope will provide aggregate selection rates by category. Candidate-level data is ‘proprietary.’ Feature weights are ‘trade secret.’
I cannot calculate impact ratios under the four-fifths rule from aggregates alone. I can compute ratios from numbers Gil gives me — but I would be auditing Gil’s summary of the tool, not the tool. My report would have to disclose this limitation.
Your call. I need a decision by end of week to stay on timeline.
Anya
The vendor agreement has a compliance clause. Section 8.3. You wrote it into the procurement terms last October, which feels like a very long time ago.
TalentScope will not release candidate-level data. Dr. Petrov needs it to conduct an audit the DCWP will accept. Gil is framing it as a privacy/IP issue. Anya is framing it as audit integrity. Only one framing matches what § 20-872 requires.
How do you respond to the data refusal?
Your choice
Invoke Section 8.3 and set a 5-business-day deadlineFormal letter to TalentScope citing the compliance clause. If the tool can’t be audited, the tool can’t be used. Anonymised candidate-level data is available. Trade-secret feature weights are a reasonable carve-out.
Your choice
Ask Anya to work with aggregates and disclose the limitationKeep the timeline. Accept a report with a data-limitation disclosure on page 3. It’s not ideal but it’s an audit on the record.
Your choice
Proceed with TalentScope’s data — instruct Dr. Petrov to note any gaps in the reportPetrov is a professional — she’ll disclose what she couldn’t verify. A published audit with a documented limitation is still an audit. The DCWP is more likely to ask about a delayed audit than a published one with a methodological note.
The data-access problem is an employer problem, not an auditor problem. Vantage is responsible for ensuring the audit can be conducted — which means ensuring the vendor provides what’s needed. Invoking the contract forces the issue and clarifies who is actually obstructing compliance.
A disclosed limitation in an audit report is professionally acceptable. Publishing it while hoping the DCWP doesn’t ask further questions is a different posture. If the underlying data gap surfaces later — in litigation, in a second audit — the disclosure becomes evidence that the limitation was known and accepted.
Gil’s confidence about the aggregate numbers is the tell. A vendor who knows what an independent audit will find — before the independent auditor runs it — isn’t offering data. They’re offering a predetermined result. The value of an independent audit is precisely that the employer doesn’t know the outcome in advance.
| Category (vs reference) | Ratio | 4/5ths Pass? |
|---|---|---|
| Female vs Male | 0.89 | Pass |
| Black vs White | 0.84 | Pass |
| Asian vs White | 0.91 | Pass |
| Hispanic/Latino vs White | 0.72 | Fail — adverse impact |
| Hispanic/Latino male (intersectional) | 0.64 | Fail — adverse impact |
| Age 40+ vs Age <40 | 0.81 | Pass (borderline) |
Gil Fontaine calls within an hour of the draft going to Nora’s inbox. He has ‘context’ that he’d like to share before publication.
The deadline is eight days away.
Dr. Petrov’s audit has produced the selection rate data. Apply the four-fifths rule (§ 20-872) to identify which group triggers the adverse impact threshold.
TalentScope Audit — Selection Rate by Demographic Group
| Group | Applicants | Passed | Rate | 4/5 Ratio |
|---|---|---|---|---|
| White (reference) | 412 | 185 | 44.9% | — |
| Asian | 198 | 83 | 41.9% | 0.93 |
| Hispanic/Latino | 224 | 72 | 32.1% | 0.72 |
| Black/African American | 156 | 59 | 37.8% | 0.84 |
Under the four-fifths (80%) rule, which group’s result must be disclosed as adverse impact in the published audit?
The audit found adverse impact. § 20-872 requires publication regardless of findings. Gil wants time for a ‘sensitivity analysis.’ Nora wants a week to prepare Vantage Capital. Priya Venkatesan’s counsel will read whatever you post.
How do you handle publication?
Your choice
Publish in full this week. Suspend the cultural-fit factor. Open remediation.Full audit posted on the Vantage careers page. ‘Cultural fit’ factor suspended pending root-cause work. Priya’s counsel gets a written notice. Hispanic/Latino candidates in the active cycle re-scored without the factor.
Your choice
Delay publication a week for TalentScope’s ‘context’Gil has context. Give him the week. Publish the following Monday with a vendor response attached. Meets the deadline within tolerance.
Your choice
Ask Dr. Petrov to rerun the analysis using a different methodologyThe four-fifths rule is one test. A regression approach might produce different results. Ask Anya to rerun before publishing.
Publishing adverse findings protects Vantage more than it hurts it. A posted result — even a bad one — is a defensible position. An unpublished audit that surfaces in litigation is a concealment story. The remediation you announce alongside the findings is the document that matters.
Every additional hiring cycle that ran while waiting for a ‘sensitivity analysis’ is an additional use of a tool whose audit results aren’t posted. The delay creates new exposure rather than reducing existing exposure. Removing the applicants who produced the adverse result isn’t sensitivity analysis — it’s editing the data until the number passes.
Requesting a new methodology because the first one produced an adverse result is audit shopping. Dr. Petrov’s professional obligation is to document the request. That documentation — that Vantage asked her to change the result — is now in a file that can be subpoenaed.
Module Complete
Remediation open.
The audit is published. But candidates applying tomorrow need prior notice the tool is being used. Which candidates? In what format? By when?
Module Complete
Under scrutiny.
The audit is published, late. Candidate notice is now the exposure point. The 198 applications from the delay period did not receive prior notice.
Module Complete
File is damaged.
The audit you posted is the audit you will defend in front of the DCWP. Candidate notice now has to be perfect because the audit is not.