In 1964, the Civil Rights Act banned people who had made recruitment decisions from discrimination based on sex or race. Today, software often contributes to these hiring decisions, helping managers review résumés or interpret video interviews.
This worries some tech experts and civil rights groups who are citing evidence that algorithms can replicate or magnify the prejudices displayed by humans. In 2018, Reuters reported that Amazon scrapped a tool that filters résumés based on previous hiring patterns because it discriminates against women.
The legislation proposed in New York City Council aims to update the rules for setting discrimination for the age of algorithms. The bill would require companies to notify candidates when they have been evaluated using software. Companies selling such tools would have to conduct annual audits to ensure that their people sorting technology is non-discriminatory.
The proposal is part of a recent movement at all levels of government, algorithms and software shaping life-changing decisions to impose legal restraints – one that may change gear if Democrats take control of the White House and both Houses of Congress.
More than a dozen U.S. cities have banned the use of facial recognition by the government, and New York State recently passed a two-year moratorium on the use of the technology in schools. Some federal lawmakers have proposed laws regulating facial algorithms and automated decision-making tools used by businesses, including for recruitment. In December, 10 senators called on the Equal Employment Opportunity Commission to bias police bias around AI hiring tools. They feared the technology could deepen racial disparities in employment and affect economic recovery from COVID-19 in marginalized communities. Also last year, a new law went into effect in Illinois that requires applicants' consent prior to using video analytics. A similar law in Maryland limits the use of facial analysis technology in recruitment.
Legislators are more likely to talk about regulating new algorithms and AI tools than about implementing such rules. Months after San Francisco banned facial recognition in 2019, it had to change the ordinance because it accidentally made city-owned iPhones illegal.
The New York City proposal put forward by Democratic Council member Laurie Cumbo would oblige companies to use so-called automated employment decision tools to screen candidates or set conditions such as compensation for disclosing the use of the technology. Providers of such software would have to carry out a "bias audit" of their products every year and make the results available to customers.
The proposal has met opposition from some unusual allies, as well as unresolved questions about how it works. Eric Ellman, senior vice president of public policy at the Consumer Data Industry Association, which represents credit and background checking companies, says the bill could make hiring less fair by re-burdening companies that do background checks on behalf of employers. He argues that such reviews can help managers overcome reluctance to hire certain demographics.
Some civil rights groups and AI experts are also opposed to the bill – for a variety of reasons. Albert Fox Cahn, founder of the Surveillance Technology Oversight Project, organized a letter from 12 groups, including the NAACP and New York University's AI Now Institute, who objected to the proposed bill. Cahn wants to regulate tech retirement, but says the New York proposal could allow discrimination-perpetuating software to be stamped a passed fairness audit.
Cahn wants each law to more broadly define the technology, let vendors not decide how to test their own technology, and let individuals sue to enforce the law. "We haven't seen any meaningful form of enforcement against the discrimination we're concerned about," he says.
Others have concerns but still support the New York proposal. "I hope the bill goes forward," says Julia Stoyanovich, director of the Center for Responsible AI at New York University. "I also hope that it will be revised."
Like Cahn, Stoyanovich is concerned that the bill's examination requirements are not well defined. She still thinks it is worth attending, partly because many citizens were surprised to find that automated tools were widespread when organizing public meetings on recruitment technology at the Queens Public Library. "The reason I'm in favor of it is because it forces people to disclose that they have been partially rated by both a machine and a human," says Stoyanovich. "That will help get the public involved in the conversation."
She welcomed two New York-based startups whose hiring tools would be governed by the new rules. The founders of HiredScore, which tries to highlight promising candidates based on résumés and other data sources, and Pymetrics, which offers online assessments based on cognitive psychology using machine learning, endorsed the bill during a virtual hearing of the city council committee about Technology in November.
Frida Polli, CEO and co-founder of Pymetrics, markets the company's technology as a fairer signal to candidates than traditional measures like resumes, which can put people with less privileged backgrounds at a disadvantage. The company recently had its technology tested for fairness by researchers at Northeastern University. She admits that the bill's exam requirements could be stricter, but says it is unclear how this is supposed to be done in a practical way and it would be better to get something on the books. "The bill is moderate, but powerful," she says.
"Like the Wild West out there"
Robert Holden, chairman of the city council's technology committee, has his own concerns about the ability of the financially troubled city government to define how software settings should be checked. He has also heard from envoys from companies whose software would fall under the proposed rules, which has led to more involvement in the industry than is usual for the city council's business. Some have assured him that the industry can be trusted to regulate itself. Holden says what he has learned so far makes it clear that more transparency is needed. "It's almost like being in the Wild West now," says Holden. "We really have to ensure transparency."
Holden says the bill is likely to face some negotiation and rewriting, as well as possible opposition from the mayor's office, before it can be slated for a final vote by the council. If passed, it will take effect in January 2022.
This story originally appeared on wired.com.