I. Introduction

Our online experiences are increasingly personalized. Facebook and Google micro-target advertisements aimed to meet our immediate needs. Amazon, Netflix, and Spotify offer up books, movies, and music tailored to match our tastes. Our news feeds are populated with stories intended to appeal to our particular interests and biases. This drive toward increasing personalization is powered by complex machine learning algorithms built to discern our preferences and anticipate our behavior. Personalization offers benefits because companies can efficiently offer consumers the precise products and services they desire.

Online personalization, however, has come under considerable criticism lately. Shoshana Zuboff assails our current economic system, which is built on companies amassing and exploiting ever more detailed personal information.¹ 1.Shoshana Zuboff, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power 8–11 (2019).Show More Ryan Calo and Tal Zarsky explain that firms are applying the insights of behavioral science to manipulate consumers by exploiting their psychological or emotional vulnerabilities.² 2.SeeRyan Calo, Digital Market Manipulation, 82 Geo. Wash. L. Rev. 995, 996, 999 (2014); Tal Z. Zarsky, Privacy and Manipulation in the Digital Age, 20 Theoretical Inquiries L. 157, 158, 160–61 (2019).Show More Daniel Susser, Beate Roessler, and Helen Nissenbaum describe how information technology is enabling manipulative practices on a massive scale.³ 3.Daniel Susser, Beate Roessler & Helen Nissenbaum, Online Manipulation: Hidden Influences in a Digital World, 4 Geo. L. Tech. Rev. 1, 2, 10 (2019).Show More Julie Cohen similarly argues that “[p]latform-based, massively-intermediated processes of search and social networking are inherently processes of market manipulation.”⁴ 4.Julie E. Cohen, Law for the Platform Economy, 51 U.C. Davis L. Rev. 133, 165 (2017); see also Julie E. Cohen, Between Truth and Power: The Legal Constructions of Informational Capitalism 75–77, 83–89, 96 (2019) (describing how techniques for behavioral surveillance and micro-targeting contribute to social harms such as polarization and extremism).Show More In the political sphere as well, concerns have been raised about manipulation, with warnings that news personalization is creating “filter bubble[s]” and increasing polarization.⁵ 5.See, e.g., Eli Pariser, The Filter Bubble: What the Internet Is Hiding from You 13–14 (2011); Michael J. Abramowitz, Stop the Manipulation of Democracy Online, N.Y. Times (Dec. 11, 2017), https://www.nytimes.com/2017/12/11/opinion/fake-news-russia-kenya.html [https://perma.cc/9YWF-PED7]; James Doubek, How Disinformation and Distortions on Social Media Affected Elections Worldwide, NPR (Nov. 16, 2017, 2:28 PM), https://www.npr.org­/sections/alltechconsidered/2017/11/16/564542100/how-disinformation-and-distortions-on-social-media-affected-elections-worldwide [https://perma.cc/ZJ97-GQ SZ]; Jon Keegan, Blue Feed, Red Feed: See Liberal Facebook and Conservative Facebook, Side by Side, Wall St. J. (Aug. 19, 2019), http://graphics.wsj.com/blue-feed-red-feed/ [https://perma.cc/GJA8-4U9W].Show More These issues were highlighted by revelations that Cambridge Analytica sent personalized ads based on psychological profiles of eighty-seven million Facebook users in an effort to influence the 2016 presidential election.⁶ 6.Carole Cadwalladr & Emma Graham-Harrison, Revealed: 50 Million Facebook Profiles Harvested for Cambridge Analytica in Major Data Breach,Guardian (Mar. 17, 2018, 6:03 PM), https://www.theguardian.com/news/2018/mar/17/cambridge-analytica-facebook-influ­ence-us-election [https://perma.cc/72CR-9Y8K]; Alex Hern, Cambridge Analytica: How Did It Turn Clicks into Votes?, Guardian (May 6, 2018, 3:00 AM), https://www.theguardian.com/­news/2018/may/06/cambridge-analytica-how-turn-clicks-into-votes-christopher-wylie [https://perma.cc/AD8H-PF3M]; Matthew Rosenberg, Nicholas Confessore & Carole Cadwalladr, How Trump Consultants Exploited the Facebook Data of Millions, N.Y. Times (Mar. 17, 2018), https://www.nytimes.com/2018/03/17/us/politics/cambridge-analytica-trump-campaign.html [https://perma.cc/3WYQ-3YKP].Show More The extensive criticism of personalization is driven by concerns that online manipulation undermines personal autonomy and compromises rational decision making.

Largely overlooked in these discussions is the possibility that online manipulation also threatens equality. Online platforms increasingly operate as key intermediaries in the markets for employment, housing, and financial services—what I refer to as opportunity markets. Predictive algorithms are also used in these markets to segment the audience and determine precisely what information will be delivered to which users. The risk is that in doing so, these intermediaries will direct opportunities in ways that reproduce or reinforce historical forms of discrimination. Predictive algorithms are built by observing past patterns of behavior, and one of the enduring patterns in American economic life is the unequal distribution of opportunities along the lines of race, gender, and other personal characteristics. As a result, these systems are likely to distribute information about future opportunities in ways that reflect existing inequalities and may reinforce historical patterns of disadvantage.

The way in which information about opportunities is distributed matters, because these markets provide access to resources that are critical for human flourishing and well-being. In that sense, access to them is foundational. People need jobs and housing before they can act as consumers or voters. They need access to financial services in order to function in the modern economy. Of course, many other factors contribute to inequality, such as unequal educational resources, lack of access to health care, and over-policing in certain communities. Decisions by landlords, employers, or banks can also contribute to inequality. Tech intermediaries are thus just one part of a much larger picture. Nevertheless, they will be an increasingly important part as more and more transactions are mediated online.⁷ 7.See, e.g., Miranda Bogen & Aaron Rieke, Help Wanted: An Examination of Hiring Algorithms, Equity, and Bias 5–6 (2018) (describing the role of platforms in the hiring process); Geoff Boeing, Online Rental Housing Market Representation and the Digital Reproduction of Urban Inequality, 52 Env’t & Plan. A 449, 450 (2019) (documenting the growing impact of Internet platforms in shaping the rental housing market).Show More Because they control access to information about opportunities, they have the potential to significantly impact how these markets operate.

Online intermediaries have unprecedented potential to finely calibrate the distribution of information. In the past, traditional print or broadcast media might aim at a particular audience, but they could not prevent any particular individual from accessing information that they published. And if an advertiser tried to signal its interest in only a particular group—as has happened with real estate ads that used code words or featured only white models—the attempts at exclusion were plainly visible. In contrast, online intermediaries have the ability to precisely target an audience, selecting some users to receive information and others to be excluded in ways that are not at all transparent.

The issue is illustrated by Facebook’s ad-targeting tools. Several lawsuits alleged that employers or landlords could use the company’s tools to exclude users on the basis of race, gender, or age from their audience.⁸ 8.See infra Section II.B.Show More To a large extent, these concerns were resolved by a recent settlement in which Facebook agreed to bar the use of sensitive demographic variables to target employment, housing, and credit advertisements.⁹ 9.See Galen Sherwin & Esha Bhandari, Facebook Settles Civil Rights Cases by Making Sweeping Changes to Its Online Ad Platform, ACLU (Mar. 19, 2019, 2:00 PM), https://www.aclu.org/blog/womens-rights/womens-rights-workplace/facebook-settles-civil-rights-cases-making-sweeping [https://perma.cc/H6D6-UMJ4].Show More However, the settlement failed to address another potential source of bias—Facebook’s ad-delivery algorithm, which determines which users within a targeted audience actually receive an ad. As explained below, even if an advertiser uses neutral targeting criteria and intends to reach a diverse audience, an ad-targeting algorithm may distribute information about opportunities in a biased way.¹⁰ 10.See infra Section II.C.Show More This is an example of a much broader concern—namely, that when predictive algorithms are used to allocate access to opportunities, there is a significant risk that they will do so in a way that reproduces existing patterns of inequality and disadvantage.

Concerns about the distributive effects of predictive algorithms are relevant to all kinds of opportunity markets, including for housing, employment, and basic financial services. Each of these markets operates somewhat differently and is regulated under different laws. They deserve separate attention and more detailed consideration than can be provided here. This Article focuses on the labor market and the relevant laws regulating it; however, the issues it raises likely plague other opportunity markets as well.

Examining employment practices reveals dramatic change. Just a couple of decades ago, employers had a handful of available strategies for recruiting new workers, such as advertising in newspapers or hiring through an employment agency. Today, firms increasingly rely on tech intermediaries to fill job openings.¹¹ 11.See Bogen & Rieke, supra note 7, at 5–6.Show More Recent surveys suggest that somewhere from 84% to 93% of job recruiters use online strategies to find potential employees.¹² 12.Soc’y for Human Res. Mgmt., SHRM Survey Findings: Using Social Media for Talent Acquisition—Recruitment and Screening 3 (Jan. 7, 2016), https://www.shrm.org/hr-today/trends-and-forecasting/research-and-surveys/Documents/SHRM-Social-Media-Recruiting-Screening-2015.pdf [https://perma.cc/L6NT-N4KL]. The Society for Human Resource Management conducts biennial surveys of job recruiters. The surveys demonstrated an increase in the use of online recruiting by employers, rising from fifty-six percent in 2011 to seventy-seven percent in 2013 to eighty-four percent in 2015.Id.; Soc’y for Human Res. Mgmt., SHRM Survey Findings: Social Networking Websites and Recruiting/Selection 2 (Apr. 11, 2013), https://www.shrm.org/hr-today/trends-and-forecasting/research-and-sur­veys/Pages/shrm-social-networking-websites-recruiting-job-candidates.aspx [https://perma.cc/U4HN-E7U7]; see also Jobvite’s New 2015 Recruiter Nation Survey Reveals Talent Crunch, Jobvite (Sept. 22, 2015), https://www.jobvite.com/news_item/­jobvites-new-2015-recruiter-nation-survey-reveals-talent-crunch-95-recruiters-anticipate-similar-increased-competition-skilled-workers-coming-year-86-expect-exp/ [https://perma.cc /H66S-8E5Z] (stating that 92% of recruiters use social media to discover or evaluate candidates).Show More Employers distribute information about positions through social media. They also rely on specialized job platforms like ZipRecruiter, LinkedIn, and Monster to recruit applicants and recommend the strongest candidates.¹³ 13.See Bogen & Rieke, supra note 7, at 5, 19–20, 24.Show More In addition, passive recruiting—using data to identify workers who are not actively looking for another position—is a growing strategy for recruiting new talent.¹⁴ 14.Id. at 22.Show More

The use of algorithms and artificial intelligence in the hiring process has not gone unnoticed. Numerous commenters and scholars have described how employers are using automated decision systems and have raised concerns that these developments may cause discrimination or threaten employee privacy.¹⁵ 15.See, e.g., Ifeoma Ajunwa, Kate Crawford & Jason Schultz, Limitless Worker Surveillance, 105 Calif. L. Rev. 735, 738–39 (2017); Ifeoma Ajunwa, The Paradox of Automation as Anti-Bias Intervention, 41 Cardozo L. Rev. (forthcoming 2020) (manuscript at 14) (on file with author); Richard A. Bales & Katherine V.W. Stone, The Invisible Web of Work: Artificial Intelligence and Electronic Surveillance in the Workplace, 41 Berkeley J. Lab. & Emp. L. (forthcoming 2020) (manuscript at 3) (on file with author); Solon Barocas & Andrew D. Selbst, Big Data’s Disparate Impact, 104 Calif. L. Rev. 671, 673–75 (2016); Matthew T. Bodie, Miriam A. Cherry, Marcia L. McCormick & Jintong Tang, The Law and Policy of People Analytics, 88 U. Colo. L. Rev. 961, 989–92 (2017); James Grimmelmann & Daniel Westreich, Incomprehensible Discrimination, 7 Calif. L. Rev. Online 164, 170–72, 176–77 (2017); Jeffrey M. Hirsch, Future Work, 2020 U. Ill. L. Rev. (forthcoming 2020) (manuscript at 3) (on file with author); Pauline T. Kim, Data-Driven Discrimination at Work, 58 Wm. & Mary L. Rev. 857, 860–61 (2017) [hereinafter Kim, Data-Driven Discrimination at Work]; Pauline T. Kim, Data Mining and the Challenges of Protecting Employee Privacy Under U.S. Law, 40 Comp. Lab. L. & Pol’y J. 405, 406 (2019); Pauline T. Kim & Erika Hanson, People Analytics and the Regulation of Information Under the Fair Credit Reporting Act, 61 St. Louis U. L.J. 17, 18–19 (2016); Charles A. Sullivan, Employing AI, 63 Vill. L. Rev. 395, 396 (2018).Show More However, previous work has focused on whether employers can or should be held liable when they use predictive algorithms or other artificial intelligence tools to make personnel decisions. What is missing from this literature is close scrutiny of how tech intermediaries are shaping labor markets and the implications for equality.

This Article undertakes that analysis, arguing that the use of predictive algorithms by labor market intermediaries risks reinforcing or even worsening existing patterns of inequality and that these intermediaries should be accountable for those effects. A number of studies have documented instances of biased delivery of employment ads.¹⁶ 16.See infra Section II.C.Show More Although the exact mechanism is unclear, it should not be surprising that predictive algorithms distribute information about job opportunities in biased ways. These algorithms are built by analyzing existing data, and one of the most persistent facts of the U.S. labor market is ongoing occupational segregation along the lines of race and gender.¹⁷ 17.See infra Section II.D.Show More If predictions are based solely on observations about past behavior—without regard to what social forces shaped that behavior—then they are likely to reproduce those patterns.

Tech intermediaries may not intend to cause discriminatory effects, but they are nevertheless responsible for them.¹⁸ 18.Building predictive models involves numerous choices, many of them implicating value judgments. See, e.g., Barocas & Selbst, supra note 15, at 674; Margot E. Kaminski, Binary Governance: Lessons from the GDPR’s Approach to Algorithmic Accountability, 92 S. Cal. L. Rev. 1529, 1539 (2019); David Lehr & Paul Ohm, Playing with the Data: What Legal Scholars Should Learn About Machine Learning, 51 U.C. Davis L. Rev. 653, 703–04 (2017); Andrew D. Selbst & Solon Barocas, The Intuitive Appeal of Explainable Machines, 87 Fordham L. Rev. 1085, 1130–31 (2018).Show More They make choices when designing the algorithms that distribute information about job opportunities or suggest the best matches for job seekers and hiring firms. In doing so, they decide what goals to optimize—typically revenue—and those choices influence how information is channeled, making some opportunities visible and obscuring others. Thus, these technologies shape how the market participants—both workers and employers—perceive their available options and thereby also influence their behavior.¹⁹ 19.Karen Levy and Solon Barocas have explored how the design choices made by platforms “can both mitigate and aggravate bias.” Karen Levy & Solon Barocas, Designing Against Discrimination in Online Markets, 32 Berkeley Tech. L.J. 1183, 1185 (2017). The focus of their analysis is on user bias in online markets like ride matching, consumer-to-consumer sales, short-term rentals, and dating. Id. at 1189–90. Because the design choices platforms make will structure users’ interactions with one another, these choices influence behavior, affecting whether or to what extent users can act on explicit or implicit biases. Levy and Barocas review multiple platforms across domains and develop a taxonomy of policy and design elements that have been used to address the risks of bias. Although the focus of this Article is on the impact of predictive algorithms rather than user bias, the issues are obviously interrelated. Past bias by users can cause predictive algorithms to discriminate. Conversely, algorithmic outputs in the form of recommendations or rankings can activate or exacerbate implicit user biases. To that extent, some, but not all, of the strategies they identify may be relevant to addressing bias in online opportunity markets.Show More When these intermediaries structure access to opportunities in ways that reflect historical patterns of discrimination and exclusion, they pose a threat to workplace equality. Even if the discriminatory effects are unintentional, the harm to workers can be real. Employment discrimination law has long targeted discriminatory effects, not just invidious motivation.²⁰ 20.See Griggs v. Duke Power Co., 401 U.S. 424, 431 (1971).Show More

The risk that tech intermediaries will contribute to workplace inequality poses a number of challenges for the law. Discrimination law has largely focused on employers, examining their decisions and practices for discriminatory intent or impact. However, if bias affects how potential applicants are screened out before they even interact with a hiring firm, then focusing on employer behavior will be inadequate to dismantle patterns of occupational segregation. Holding tech intermediaries directly responsible for their effects on labor markets, however, will raise a different set of challenges. Some of these are legal, such as whether existing law reaches these types of intermediaries,²¹ 21.See infra Part III.Show More and whether they can avoid liability by relying on Section 230 of the Communications Decency Act (CDA),²² 22.47 U.S.C. § 230 (2012).Show More which gives websites a defense to some types of liability. Other obstacles are more practical in nature, which suggests that preventing discriminatory effects may require alternative strategies.²³ 23.See infra Section IV.B.Show More

This Article proceeds as follows. Part II first explores the role that tech intermediaries play in the labor market and how targeting tools can be misused for discriminatory purposes. It next explains that even if employers are no longer permitted to use discriminatory targeting criteria, a significant risk remains that platforms’ predictive algorithms will distribute access to opportunities in ways that reproduce existing patterns of inequality. Because tech intermediaries have a great deal of power to influence labor market interactions, and may do so in ways that are not transparent, I argue in Part II that they should bear responsibility when they cause discriminatory effects.

Parts III and IV consider the relevant legal landscape. Part III discusses how the growing importance of tech intermediaries in the labor market poses challenges for existing anti-discrimination law. It first shows how the question “who is an applicant?”—an issue critical for finding employer liability—is complicated as platforms increasingly mediate job seekers’ interactions with firms. It then explores the possibilities for holding these intermediaries directly liable under existing employment discrimination law, either as employment agencies or for interfering with third party employment relationships. Part IV considers some obstacles to holding tech intermediaries liable for their discriminatory labor market effects. Section IV.A examines and rejects the argument that Section 230 of the Communications Decency Act would automatically bar such claims. Section IV.B explains that significant practical obstacles remain, suggesting that a post hoc liability regime may not be the best way to prevent discriminatory harms. Thus, Section IV.B also argues that we should look to regulatory models in order to minimize the risks of discrimination from the use of predictive algorithms.

1. * Daniel Noyes Kirby Professor of Law, Washington University School of Law, St. Louis, Missouri. I am grateful to Victoria Schwarz, Miranda Bogen, Aaron Riecke, Greg Magarian, Neil Richards, Peggie Smith, Dan Epps, John Inazu, Danielle Citron, Ryan Calo, Andrew Selbst, Margot Kaminski, and Felix Wu for helpful comments on earlier drafts of this Article. I also benefited from feedback from participants at the 2019 Privacy Law Scholar’s Conference, Washington University School of Law’s faculty workshop, and Texas A&M School of Law’s Faculty Speaker Series. Many thanks to Adam Hall, Theanne Liu, Joseph Tomchak, and Samuel Levy for outstanding research assistance. ↑
2. Shoshana Zuboff, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power 8–11 (2019). ↑
3. See Ryan Calo, Digital Market Manipulation, 82 Geo. Wash. L. Rev. 995, 996, 999 (2014); Tal Z. Zarsky, Privacy and Manipulation in the Digital Age, 20 Theoretical Inquiries L. 157, 158, 160–61 (2019). ↑
4. Daniel Susser, Beate Roessler & Helen Nissenbaum, Online Manipulation: Hidden Influences in a Digital World, 4 Geo. L. Tech. Rev. 1, 2, 10 (2019). ↑
5. Julie E. Cohen, Law for the Platform Economy, 51 U.C. Davis L. Rev.
   

   133, 165 (2017)

   ; see also Julie E. Cohen, Between Truth and Power: The Legal Constructions of Informational Capitalism 75–77, 83–89, 96 (2019) (describing how techniques for behavioral surveillance and micro-targeting contribute to social harms such as polarization and extremism). ↑

6. See, e.g., Eli Pariser, The Filter Bubble: What the Internet Is Hiding from You 13–14 (2011); Michael J. Abramowitz, Stop the Manipulation of Democracy Online, N.Y. Times (Dec. 11, 2017), https://www.nytimes.com/2017/12/11/opinion/fake-news-russia-kenya.html [https://perma.cc/9YWF-PED7]; James Doubek, How Disinformation and Distortions
   on Social Media Affected Elections Worldwide, NPR (Nov. 16, 2017, 2:28 PM), https://www.npr.org­/sections/alltechconsidered/2017/11/16/564542100/how-disinformation-and-distortions-on-social-media-affected-elections-worldwide [https://perma.cc/ZJ97-GQ SZ]; Jon Keegan, Blue Feed, Red Feed: See Liberal Facebook and Conservative Facebook, Side by Side, Wall St. J. (Aug. 19, 2019), http://graphics.wsj.com/blue-feed-red-feed/ [https://perma.cc/GJA8-4U9W]. ↑
7. Carole Cadwalladr & Emma Graham-Harrison, Revealed: 50 Million Facebook Profiles Harvested for Cambridge Analytica in Major Data Breach, Guardian (Mar. 17, 2018, 6:03 PM), https://www.theguardian.com/news/2018/mar/17/cambridge-analytica-facebook-influ­ence-us-election [https://perma.cc/72CR-9Y8K]; Alex Hern, Cambridge Analytica: How Did It Turn Clicks into Votes?, Guardian (May 6, 2018, 3:00 AM), https://www.theguardian.com/­news/2018/may/06/cambridge-analytica-how-turn-clicks-into-votes-christopher-wylie [https://perma.cc/AD8H-PF3M]; Matthew Rosenberg, Nicholas Confessore & Carole Cadwalladr, How Trump Consultants Exploited the Facebook Data of Millions, N.Y. Times (Mar. 17, 2018), https://www.nytimes.com/2018/03/17/us/politics/cambridge-analytica-trump-campaign.html [https://perma.cc/3WYQ-3YKP]. ↑
8. See, e.g., Miranda Bogen & Aaron Rieke, Help Wanted: An Examination of Hiring Algorithms, Equity, and Bias 5–6 (2018) (describing the role of platforms in the hiring process); Geoff Boeing, Online Rental Housing Market Representation and the Digital Reproduction of Urban Inequality, 52 Env’t & Plan. A 449, 450 (2019) (documenting the growing impact of Internet platforms in shaping the rental housing market). ↑
9. See infra Section II.B. ↑
10.  See Galen Sherwin & Esha Bhandari, Facebook Settles Civil Rights Cases by Making Sweeping Changes to Its Online Ad Platform, ACLU (Mar. 19, 2019, 2:00 PM), https://www.aclu.org/blog/womens-rights/womens-rights-workplace/facebook-settles-civil-rights-cases-making-sweeping [https://perma.cc/H6D6-UMJ4]. ↑
11. See infra Section II.C. ↑
12. See Bogen & Rieke, supra note 7, at 5–6. ↑
13. Soc’y for Human Res. Mgmt., SHRM Survey Findings: Using Social Media for Talent Acquisition—Recruitment and Screening 3 (Jan. 7, 2016), https://www.shrm.org/hr-today/trends-and-forecasting/research-and-surveys/Documents/SHRM-Social-Media-Recruiting-Screening-2015.pdf [https://perma.cc/L6NT-N4KL]. The Society for Human Resource Management conducts biennial surveys of job recruiters. The surveys demonstrated an increase in the use of online recruiting by employers, rising from fifty-six percent in 2011 to seventy-seven percent in 2013 to eighty-four percent in 2015. Id.; Soc’y for Human Res. Mgmt., SHRM Survey Findings: Social Networking Websites and Recruiting/Selection 2 (Apr. 11, 2013), https://www.shrm.org/hr-today/trends-and-forecasting/research-and-sur­veys/Pages/shrm-social-networking-websites-recruiting-job-candidates.aspx [https://perma.cc/U4HN-E7U7]; see also Jobvite’s New 2015 Recruiter Nation Survey Reveals Talent Crunch, Jobvite (Sept. 22, 2015), https://www.jobvite.com/news_item/­jobvites-new-2015-recruiter-nation-survey-reveals-talent-crunch-95-recruiters-anticipate-similar-increased-competition-skilled-workers-coming-year-86-expect-exp/ [https://perma.cc /H66S-8E5Z] (stating that 92% of recruiters use social media to discover or evaluate candidates). ↑
14. See Bogen & Rieke, supra note 7, at 5, 19–20, 24. ↑
15. Id. at 22. ↑
16.  See, e.g., Ifeoma Ajunwa, Kate Crawford & Jason Schultz, Limitless Worker Surveillance, 105 Calif. L. Rev. 735, 738–39 (2017); Ifeoma Ajunwa, The Paradox of Automation as Anti-Bias Intervention, 41 Cardozo L. Rev. (forthcoming 2020) (manuscript at 14) (on file with author); Richard A. Bales & Katherine V.W. Stone, The Invisible Web of Work: Artificial Intelligence and Electronic Surveillance in the Workplace, 41 Berkeley J. Lab. & Emp. L. (forthcoming 2020) (manuscript at 3) (on file with author); Solon Barocas & Andrew D. Selbst, Big Data’s Disparate Impact, 104 Calif. L. Rev. 671, 673–75 (2016); Matthew T. Bodie, Miriam A. Cherry, Marcia L. McCormick & Jintong Tang, The Law and Policy of People Analytics, 88 U. Colo. L. Rev. 961, 989–92 (2017); James Grimmelmann & Daniel Westreich, Incomprehensible Discrimination, 7 Calif. L. Rev. Online 164, 170–72, 176–77 (2017); Jeffrey M. Hirsch, Future Work, 2020 U. Ill. L. Rev. (forthcoming 2020) (manuscript at 3) (on file with author); Pauline T. Kim, Data-Driven Discrimination at Work, 58 Wm. & Mary L. Rev. 857, 860–61 (2017) [hereinafter Kim, Data-Driven Discrimination at Work]; Pauline T. Kim, Data Mining and the Challenges of Protecting Employee Privacy Under U.S. Law, 40 Comp. Lab. L. & Pol’y J. 405, 406 (2019); Pauline T. Kim & Erika Hanson, People Analytics and the Regulation of Information Under the Fair Credit Reporting Act, 61 St. Louis U. L.J. 17, 18–19 (2016); Charles A. Sullivan, Employing AI, 63 Vill. L. Rev. 395, 396 (2018). ↑
17. See infra Section II.C. ↑
18. See infra Section II.D. ↑
19. Building predictive models involves numerous choices, many of them implicating value judgments. See, e.g., Barocas & Selbst, supra note 15, at 674; Margot E. Kaminski, Binary Governance: Lessons from the GDPR’s Approach to Algorithmic Accountability, 92 S. Cal. L. Rev. 1529, 1539 (2019); David Lehr & Paul Ohm, Playing with the Data: What Legal Scholars Should Learn About Machine Learning, 51 U.C. Davis L. Rev. 653, 703–04 (2017); Andrew D. Selbst & Solon Barocas, The Intuitive Appeal of Explainable Machines, 87 Fordham L. Rev. 1085, 1130–31 (2018). ↑
20. Karen Levy and Solon Barocas have explored how the design choices made by platforms “can both mitigate and aggravate bias.” Karen Levy & Solon Barocas, Designing Against Discrimination in Online Markets, 32 Berkeley Tech. L.J. 1183, 1185 (2017). The focus of their analysis is on user bias in online markets like ride matching, consumer-to-consumer sales, short-term rentals, and dating. Id. at 1189–90. Because the design choices platforms make will structure users’ interactions with one another, these choices influence behavior, affecting whether or to what extent users can act on explicit or implicit biases. Levy and Barocas review multiple platforms across domains and develop a taxonomy of policy and design elements that have been used to address the risks of bias. Although the focus of this Article is on the impact of predictive algorithms rather than user bias, the issues are obviously interrelated. Past bias by users can cause predictive algorithms to discriminate. Conversely, algorithmic outputs in the form of recommendations or rankings can activate or exacerbate implicit user biases. To that extent, some, but not all, of the strategies they identify may be relevant to addressing bias in online opportunity markets. ↑
21. See Griggs v. Duke Power Co., 401 U.S. 424, 431 (1971). ↑
22. See infra Part III. ↑
23. 47 U.S.C. § 230 (2012). ↑
24. See infra Section IV.B. ↑

The post Manipulating Opportunity first appeared on Virginia Law Review.

This Article makes three contributions to the debate about how best to measure algorithmic fairness: one conceptual, one normative, and one legal. Equal predictive accuracy ensures that a score means the same thing for each group at issue. As such, it relates to what one ought to believe about a scored individual. Because questions of fairness usually relate to action, not belief, this measure is ill-suited as a measure of fairness. This is the Article’s conceptual contribution. Second, this Article argues that parity in the ratio of false positives to false negatives is a normatively significant measure. While a lack of parity in this dimension is not constitutive of unfairness, this measure provides important reasons to suspect that unfairness exists. This is the Article’s normative contribution. Interestingly, improving the accuracy of algorithms overall will lessen this unfairness. Unfortunately, a common assumption that anti-discrimination law prohibits the use of racial and other protected classifications in all contexts is inhibiting those who design algorithms from making them as fair and accurate as possible. This Article’s third contribution is to show that the law poses less of a barrier than many assume.

Introduction

At an event celebrating Martin Luther King, Jr. Day, Representative Alexandria Ocasio-Cortez (D-NY) expressed the concern, shared by many, that algorithmic decision making is biased. “Algorithms are still made by human beings, and those algorithms are still pegged to basic human assumptions,” she asserted. “They’re just automated. And if you don’t fix the bias, then you are automating the bias.”¹ 1.Blackout for Human Rights, MLK Now 2019, Riverside Church in the City of N.Y. (Jan. 21, 2019), https://www.trcnyc.org/mlknow2019/ [https://perma.cc/L45Q-SN9T] (interview with Rep. Ocasio-Cortez begins at approximately minute 16, and comments regarding algorithms begin at approximately minute 40); see also Danny Li, AOC Is Right: Algorithms Will Always Be Biased as Long as There’s Systemic Racism in This Country, Slate (Feb. 1, 2019, 3:47 PM), https://slate.com/news-and-politics/2019/02/aoc-algorithms-racist-bias.html [https://perma.cc/S97Z-UH2U] (quoting Ocasio-Cortez’s comments at the event in New York); Cat Zakrzewski, The Technology 202: Alexandria Ocasio-Cortez Is Using Her Social Media Clout To Tackle Bias in Algorithms, Wash. Post: PowerPost (Jan. 28, 2019), https://www.washingtonpost.com/news/powerpost/paloma/the-technology-202/2019/01/28 /the-technology-202-alexandria-ocasio-cortez-is-using-her-social-media-clout-to-tackle-bias-in-algorithms/5c4dfa9b1b326b29c37­78cdd/?utm_term=.541cd0827a23 [https://perma.cc/ LL4Y-FWDK] (discussing Ocasio-Cortez’s comments and reactions to them).Show More The audience inside the room applauded. Outside the room, the reaction was more mixed. “Socialist Rep. Alexandria Ocasio-Cortez . . . claims that algorithms, which are driven by math, are racist,” tweeted a writer for the Daily Wire.² 2.Ryan Saavedra (@RealSaavedra), Twitter (Jan. 22, 2019, 12:27 AM), https://twitter.com/RealSaavedra/status/1087627739861897216 [https://perma.cc/32DD-QK5S]. The coverage of Ocasio-Cortez’s comments is mixed. See, e.g., Zakrzewski, supra note 1 (describing conservatives’ criticism of and other media outlets’ and experts’ support of Ocasio-Cortez’s comments).Show More Math is just math, this commentator contends, and the idea that math can be unfair is crazy.

This controversy is just one of many to challenge the fairness of algorithmic decision making.³ 3.See, e.g., Hiawatha Bray, The Software That Runs Our Lives Can Be Biased—But We Can Fix It, Bos. Globe, Dec. 22, 2017, at B9 (describing a New York City Council member’s proposal to audit the city government’s computer decision systems for bias); Drew Harwell, Amazon’s Facial-Recognition Software Has Fraught Accuracy Rate, Study Finds, Wash. Post, Jan. 26, 2019, at A14 (reporting on an M.I.T. Media Lab study that found that Amazon facial-recognition software is less accurate with regard to darker-skinned women than lighter-skinned men, and Amazon’s criticism of the study); Tracy Jan, Mortgage Algorithms Found To Have Racial Bias, Wash. Post, Nov. 15, 2018, at A21 (reporting on a University of California at Berkeley study that found that black and Latino home loan customers pay higher interest rates than white or Asian customers on loans processed online or in person); Tony Romm & Craig Timberg, Under Bipartisan Fire from Congress, CEO Insists Google Does Not Take Sides, Wash. Post, Dec. 12, 2018, at A16 (reporting on Congresspeople’s concerns regarding Google algorithms which were voiced at a House Judiciary Committee hearing with Google’s CEO).Show More The use of algorithms, and in particular their connection with machine learning and artificial intelligence, has attracted significant attention in the legal literature as well. The issues raised are varied, and include concerns about transparency,⁴ 4.See, e.g., Danielle Keats Citron, Technological Due Process, 85 Wash. U. L. Rev. 1249, 1288–97 (2008); Natalie Ram, Innovating Criminal Justice, 112 Nw. U. L. Rev. 659 (2018); Rebecca Wexler, Life, Liberty, and Trade Secrets: Intellectual Property in the Criminal Justice System, 70 Stan. L. Rev. 1343 (2018).Show More accountability,⁵ 5.See, e.g., Margot E. Kaminski, Binary Governance: Lessons from the GDPR’s Approach to Algorithmic Accountability, 92 S. Cal. L. Rev. 1529 (2019); Joshua A. Kroll et al., Accountable Algorithms, 165 U. Pa. L. Rev. 633 (2017); Anne L. Washington, How To Argue with an Algorithm: Lessons from the COMPAS-ProPublica Debate, 17 Colo. Tech. L.J. 131 (2018) (arguing for standards governing the information available about algorithms so that their accuracy and fairness can be properly assessed). But see Jon Kleinberg et al., Discrimination in the Age of Algorithms (Nat’l Bureau of Econ. Research, Working Paper No. 25548, 2019), http://www.nber.org/papers/w25548 [https://perma.cc/JU6H-HG3W] (analyzing the potential benefits of algorithms as tools to prove discrimination).Show More privacy,⁶ 6.See generally Frank Pasquale, The Black Box Society: The Secret Algorithms That Control Money and Information (2015) (discussing and critiquing internet and finance companies’ non-transparent use of data tracking and algorithms to influence and manage people); Anupam Chander, The Racist Algorithm?, 115 Mich. L. Rev. 1023, 1024 (2017) (reviewing Frank Pasquale, The Black Box Society: The Secret Algorithms That Control Money and Information (2015)) (arguing that instead of “transparency in the design of the algorithm” that Pasquale argues for, “[w]hat we need . . . is a transparency of inputs and results”) (emphasis omitted).Show More and fairness.⁷ 7.See, e.g., Aziz Z. Huq, Racial Equity in Algorithmic Criminal Justice, 68 Duke L.J. 1043 (2019) (arguing that current constitutional doctrine is ill-suited to the task of evaluating algorithmic fairness and that current standards offered in the technology literature miss important policy concerns); Sandra G. Mayson, Bias In, Bias Out, 128 Yale L.J. 2218 (2019) (discussing how past and existing racial inequalities in crime and arrests mean that methods to predict criminal risk based on existing information will result in racial inequality).Show More This Article focuses on fairness—the issue raised by Ocasio-Cortez. It focuses on how we should assess what makes algorithmic decision making fair. Fairness is a moral concept, and a contested one at that. As a result, we should expect that different people will offer well-reasoned arguments for different conceptions of fairness. And this is precisely what we find.

The computer science literature is filled with a proliferation of measures, each purporting to capture fairness along some dimension. This Article provides a pathway through that morass. It makes three contributions: one conceptual, one normative, and one legal. This Article argues that one of the dominant measures of fairness offered in the literature tells us what to believe, not what to do, and thus is ill-suited as a measure of fair treatment. This is the conceptual claim. Second, this Article argues that the ratio between false positives and false negatives offers an important indicator of whether members of two groups scored by an algorithm are treated fairly, vis-à-vis each other. This is the normative claim. Third, this Article challenges a common assumption that anti-discrimination law prohibits the use of racial and other protected classifications in all contexts. Because using race within algorithms can increase both their accuracy and fairness, this misunderstanding has important implications. This Article’s third contribution is to show that the law poses less of a barrier than many assume.

We can use the controversy over a common risk assessment tool used by many states for bail, sentencing, and parole to illustrate the controversy about how best to measure fairness.⁸ 8.See Julia Angwin et al., Machine Bias, ProPublica (May 23, 2016), https://www.pro­publica.org/article/machine-bias-risk-assessments-in-criminal-sentencing [https://perma.cc/BA53-JT7V].Show More The tool, called COMPAS, assigns each person a score that indicates the likelihood that the person will commit a crime in the future.⁹ 9.Equivant, Practitioner’s Guide to COMPAS Core 7 (2019), http://www.equivant.com/wp-content/uploads/Practitioners-Guide-to-COMPAS-Core-040419.pdf [https://perma.cc/LRY6-RXAH].Show More In a high-profile exposé, the website ProPublica claimed that COMPAS treated blacks and whites differently because black arrestees and inmates were far more likely to be erroneously classified as risky than were white arrestees and inmates despite the fact that COMPAS did not explicitly use race in its algorithm.¹⁰ 10.See Angwin et al., supra note 8 (“Northpointe’s core product is a set of scores derived from 137 questions that are either answered by defendants or pulled from criminal records. Race is not one of the questions.”).Show More The essence of ProPublica’s claim was this:

In forecasting who would re-offend, the algorithm made mistakes with black and white defendants at roughly the same rate but in very different ways. The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants. White defendants were mislabeled as low risk more often than black defendants.¹¹ 11.Id.Show More

Northpointe¹² 12.Northpointe, along with CourtView Justice Solutions Inc. and Constellation Justice Systems, rebranded to Equivant in January 2017. Equivant, Frequently Asked Questions 1, http://my.courtview.com/rs/322-KWH-233/images/Equivant%20Customer%20FAQ%20-%20FINAL.pdf [https://perma.cc/7HH8-LVQ6].Show More (the company that developed and owned COMPAS) responded to the criticism by arguing that ProPublica was focused on the wrong measure. In essence, Northpointe stressed the point ProPublica conceded—that COMPAS made mistakes with black and white defendants at roughly equal rates.¹³ 13.See William Dieterich et al., COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity, Northpointe 9–10 (July 8, 2016), http://go.volarisgroup.com/rs/430-MBX-989/images/ProPublica_Commentary_Final_070616.pdf [https://perma.cc/N5RL-M9RN].Show More Although Northpointe and others challenged some of the accuracy of ProPublica’s analysis,¹⁴ 14.For a critique of ProPublica’s analysis, see Anthony W. Flores et al., False Positives, False Negatives, and False Analyses: A Rejoinder to “Machine Bias: There’s Software Used Across the Country To Predict Future Criminals. And It’s Biased Against Blacks.”, 80 Fed. Prob. 38 (2016).Show More the main thrust of Northpointe’s defense was that COMPAS does treat blacks and whites the same. The controversy focused on the manner in which such similarity is assessed. Northpointe focused on the fact that if a black person and a white person were each given a particular score, the two people would be equally likely to recidivate.¹⁵ 15.See Dieterich et al., supra note 13, at 9–11.Show More ProPublica looked at the question from a different angle. Rather than asking whether a black person and a white person with the same score were equally likely to recidivate, it focused instead on whether a black and white person who did not go on to recidivate were equally likely to have received a low score from the algorithm.¹⁶ 16.See Angwin et al., supra note 8 (“In forecasting who would re-offend, the algorithm made mistakes with black and white defendants at roughly the same rate but in very different ways.”).Show More In other words, one measure begins with the score and asks about its ability to predict reality. The other measure begins with reality and asks about its likelihood of being captured by the score.

The easiest way to fix the problem would be to treat the two groups equally in both respects. A high score and low score should mean the same thing for both blacks and whites (the measure Northpointe emphasized), and law-abiding blacks and whites should be equally likely to be mischaracterized by the tool (the measure ProPublica emphasized). Unfortunately, this solution has proven impossible to achieve. In a series of influential papers, computer scientists demonstrated that, in most circumstances, it is simply not possible to equalize both measures.¹⁷ 17.See, e.g., Richard Berk et al., Fairness in Criminal Justice Risk Assessments: The State of the Art, Soc. Methods & Res. OnlineFirst 1, 23 (2018), https://journals.sagepub.com/doi/­10.1177/0049124118782533 [https://perma.cc/GG9L-9AEU] (discussing the required trade­off between predictive accuracy and various fairness measures); Alexandra Chouldechova, Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments, 5 Big Data 153, 157 (2017) (demonstrating that recidivism prediction instruments cannot simultaneously meet all fairness criteria where recidivism rates differ across groups because its error rates will be unbalanced across the groups when the instrument achieves predictive parity); Jon Kleinberg et al., Inherent Trade-Offs in the Fair Determination of Risk Scores, 67 LIPIcs 43:1, 43:5–8 (2017), https://drops.dagstuhl.de/opus/volltexte/2017/8156/pdf/LIPIcs-ITCS-2017-43.pdf [https://perma.cc/S9DM-PER2] (demonstrating how difficult it is for algorithms to simultaneously achieve the fairness goals of calibration and balance in predictions involving different groups).Show More The reason it is impossible relates to the fact that the underlying rates of recidivism among blacks and whites differ.¹⁸ 18.See Bureau of Justice Statistics, U.S. Dep’t of Justice, 2018 Update on Prisoner Recidivism: A 9-Year Follow-up Period (2005–2014) 6 tbl.3 (2018), https://www.bjs.gov/­content/pub/pdf/18upr9yfup0514.pdf [https://perma.cc/3UE3-AS5S] (analyzing rearrests of state prisoners released in 2005 in 30 states and finding that 86.9% of black prisoners and 80.9% of white prisoners were arrested in the nine years following their release); see also Dieterich et al., supra note 13, at 6 (“[I]n comparison with blacks, whites have much lower base rates of general recidivism . . . .”). Of course, the data on recidivism itself may be flawed. This consideration is discussed below. See infra text accompanying notes 33–37.Show More When the two groups at issue (whatever they are) have different rates of the trait predicted by the algorithm, it is impossible to achieve parity between the groups in both dimensions.¹⁹ 19.This is true unless the tool makes no mistakes at all. Kleinberg et al., supra note 17, at 43:5–6.Show More The example discussed in Part I illustrates this phenomenon.²⁰ 20.See infra Section I.A.Show More This fact gives rise to the question: in which dimension is such parity more important and why?

These different measures are often described as different conceptions of fairness.²¹ 21.For example, Berk et al. consider six different measures of algorithmic fairness. See Berk et al., supra note 17, at 12–15.Show More This is a mistake. The measure favored by Northpointe is relevant to what we ought to believe about a particular scored individual. If a high-risk score means something different for blacks than for whites, then we do not know whether to believe (or how much confidence to have) in the claim that a particular scored individual is likely to commit a crime in the future. The measure favored by ProPublica relates instead to what we ought to do. If law-abiding blacks and law-abiding whites are not equally likely to be mischaracterized by the score, we will not know whether or how to use the scores in making decisions. If we are comparing a measure that is relevant to what we ought to believe to one that is relevant to what we ought to do, we are truly comparing apples to oranges.

This conclusion does not straightforwardly suggest that we should instead focus on the measure touted by ProPublica, however. A sophisticated understanding of the significance of these measures is fast-moving and evolving. Some computer scientists now argue that the lack of parity in the ProPublica measure is less meaningful than one might think.²² 22.See Sam Corbett-Davies & Sharad Goel, The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning (arXiv, Working Paper No. 1808.00023v2, 2018), http://arxiv.org/abs/1808.00023 [https://perma.cc/ML4Y-EY6S].Show More The better way to understand the measure highlighted by ProPublica would be to say that it suggests that something is likely amiss. Differences in the ratio of false positive rates to false negative rates indicate that the algorithmic tool may rely on data that are themselves infected with bias or that the algorithm may be compounding a prior injustice. Because these possibilities have normative implications for how the algorithm should be used, this measure relates to fairness.

The most promising way to enhance algorithmic fairness is to improve the accuracy of the algorithm overall.²³ 23.See Sumegha Garg et al., Tracking and Improving Information in the Service of Fairness (arXiv, Working Paper No. 1904.09942v2, 2019), http://arxiv.org/abs/1904.09942 [https://perma.cc/D8ZN-CJ83].Show More And we can do that by permitting the use of protected traits (like race and sex) within the algorithm to determine what other traits will be used to predict the target variable (like recidivism). For example, housing instability might be more predictive of recidivism for whites than for blacks.²⁴ 24.See Sam Corbett-Davies et al., Algorithmic Decision Making and the Cost of Fairness, 2017 Proc. 23d ACM SIGKDD Int’l Conf. on Knowledge Discovery and Data Mining 797, 805.Show More If the algorithm includes a racial classification, it can segment its analysis such that this trait is used to predict recidivism for whites but not for blacks. Although this approach would improve risk assessment and thereby lessen the inequity highlighted by ProPublica, many in the field believe this approach is off the table because it is prohibited by law.²⁵ 25.See id. (“[E]xplicitly including race as an input feature raises legal and policy complications, and as such it is common to simply exclude features with differential predictive power.”).Show More This is not the case.

The use of racial classifications only sometimes constitutes disparate treatment on the basis of race and thus only sometimes gives rise to strict scrutiny. The fact that some uses of racial classifications do not constitute disparate treatment reveals that the concept of disparate treatment is more elusive than is often recognized. This observation is important given the central role that the distinction between disparate treatment and disparate impact plays in equal protection doctrine and statutory anti-discrimination law. In addition, it is important because it opens the door to more creative ways to improve algorithmic fairness.

The Article proceeds as follows. Part I develops the conceptual claim. It shows that the two most prominent types of measures used to assess algorithmic fairness are geared to different tasks. One is relevant to belief and the other to decision and action. This Part begins with a detailed explanation of the two measures and then explores the factors that affect belief and action in individual cases. Turning to the comparative context, Part I argues that predictive parity (the measure favored by Northpointe) is relevant to belief but not directly to the fair treatment of different groups.

Part II makes a normative claim. It argues that differences in the ratio of false positives to false negatives between protected groups (a variation on the measure put forward by ProPublica) suggest unfairness, and it explains why this is so. This Part begins by clarifying three distinct ways in which the concept of fairness is used in the literature. It then explains both the normative appeal of focusing on the parity in the ratio of false positives to false negatives and, at the same time, why doing so can be misleading. Despite these drawbacks, Part II argues that the disparity in the ratio of false positive to false negative rates tells us something important about the fairness of the algorithm.

Part III explores what can be done to diminish this unfairness. It argues that using protected classifications like race and sex within algorithms can improve their accuracy and fairness. Because constitutional anti­discrimination law generally disfavors racial classifications, computer scientists and others who work with algorithms are reluctant to deploy this approach. Part III argues that this reluctance rests on an overly simplistic view of the law. Focusing on constitutional law and on racial classification in particular, this Part argues that the doctrine’s resistance to the use of racial classifications is not categorical. Part III explores contexts in which the use of racial classifications does not constitute disparate treatment on the basis of race and extracts two principles from these examples. Using these principles, this Part argues that the use of protected classifications within algorithms may well be permissible. A conclusion follows.

1. * D. Lurton Massee, Jr. Professor of Law and Roy L. and Rosamond Woodruff Morgan Professor of Law at the University of Virginia School of Law. I would like to thank Charles Barzun, Aloni Cohen, Aziz Huq, Kim Ferzan, Niko Kolodny, Sandy Mayson, Tom Nachbar, Richard Schragger, Andrew Selbst, and the participants in the Caltech 10th Workshop in Decisions, Games, and Logic: Ethics, Statistics, and Fair AI, the Dartmouth Law and Philosophy Workshop, and the computer science department at UVA for comments and critique. In addition, I would like to thank Kristin Glover of the University of Virginia Law Library and Judy Baho for their excellent research assistance. Any errors or confusions are my own. ↑
2. Blackout for Human Rights, MLK Now 2019, Riverside Church in the City of N.Y. (Jan. 21, 2019), https://www.trcnyc.org/mlknow2019/ [https://perma.cc/L45Q-SN9T] (interview with Rep. Ocasio-Cortez begins at approximately minute 16, and comments regarding algorithms begin at approximately minute 40); see also Danny Li, AOC Is Right: Algorithms Will Always Be Biased as Long as There’s Systemic Racism in This Country, Slate (Feb. 1, 2019, 3:47 PM), https://slate.com/news-and-politics/2019/02/aoc-algorithms-racist-bias.html [https://perma.cc/S97Z-UH2U] (quoting Ocasio-Cortez’s comments at the event in New York); Cat Zakrzewski, The Technology 202: Alexandria Ocasio-Cortez Is Using Her Social Media Clout To Tackle Bias in Algorithms, Wash. Post: PowerPost (Jan. 28, 2019), https://www.washingtonpost.com/news/powerpost/paloma/the-technology-202/2019/01/28 /the-technology-202-alexandria-ocasio-cortez-is-using-her-social-media-clout-to-tackle-bias-in-algorithms/5c4dfa9b1b326b29c37­78cdd/?utm_term=.541cd0827a23 [https://perma.cc/ LL4Y-FWDK] (discussing Ocasio-Cortez’s comments and reactions to them). ↑
3. Ryan Saavedra (@RealSaavedra), Twitter (Jan. 22, 2019, 12:27 AM), https://twitter.com/RealSaavedra/status/1087627739861897216 [https://perma.cc/32DD-QK5S]. The coverage of Ocasio-Cortez’s comments is mixed. See, e.g., Zakrzewski, supra note 1 (describing conservatives’ criticism of and other media outlets’ and experts’ support of Ocasio-Cortez’s comments). ↑
4. See, e.g., Hiawatha Bray, The Software That Runs Our Lives Can Be Biased—But We Can Fix It, Bos. Globe, Dec. 22, 2017, at B9 (describing a New York City Council member’s proposal to audit the city government’s computer decision systems for bias); Drew Harwell, Amazon’s Facial-Recognition Software Has Fraught Accuracy Rate, Study Finds, Wash. Post, Jan. 26, 2019, at A14 (reporting on an M.I.T. Media Lab study that found that Amazon facial-recognition software is less accurate with regard to darker-skinned women than lighter-skinned men, and Amazon’s criticism of the study); Tracy Jan, Mortgage Algorithms Found To Have Racial Bias, Wash. Post, Nov. 15, 2018, at A21 (reporting on a University of California at Berkeley study that found that black and Latino home loan customers pay higher interest rates than white or Asian customers on loans processed online or in person); Tony Romm & Craig Timberg, Under Bipartisan Fire from Congress, CEO Insists Google Does Not Take Sides, Wash. Post, Dec. 12, 2018, at A16 (reporting on Congresspeople’s concerns regarding Google algorithms which were voiced at a House Judiciary Committee hearing with Google’s CEO). ↑
5. See, e.g., Danielle Keats Citron, Technological Due Process, 85 Wash. U. L. Rev. 1249, 1288–97 (2008); Natalie Ram, Innovating Criminal Justice, 112 Nw. U. L. Rev. 659 (2018); Rebecca Wexler, Life, Liberty, and Trade Secrets: Intellectual Property in the Criminal Justice System, 70 Stan. L. Rev. 1343 (2018). ↑
6. See, e.g., Margot E. Kaminski, Binary Governance: Lessons from the GDPR’s Approach to Algorithmic Accountability, 92 S. Cal. L. Rev. 1529 (2019); Joshua A. Kroll et al., Accountable Algorithms, 165 U. Pa. L. Rev. 633 (2017); Anne L. Washington, How To Argue with an Algorithm: Lessons from the COMPAS-ProPublica Debate, 17 Colo. Tech. L.J. 131 (2018) (arguing for standards governing the information available about algorithms so that their accuracy and fairness can be properly assessed). But see Jon Kleinberg et al., Discrimination in the Age of Algorithms (Nat’l Bureau of Econ. Research, Working Paper No. 25548, 2019), http://www.nber.org/papers/w25548 [https://perma.cc/JU6H-HG3W] (analyzing the potential benefits of algorithms as tools to prove discrimination). ↑
7. See generally Frank Pasquale, The Black Box Society: The Secret Algorithms That Control Money and Information (2015) (discussing and critiquing internet and finance companies’ non-transparent use of data tracking and algorithms to influence and manage people); Anupam Chander, The Racist Algorithm?, 115 Mich. L. Rev. 1023, 1024 (2017) (reviewing Frank Pasquale, The Black Box Society: The Secret Algorithms That Control Money and Information (2015)) (arguing that instead of “transparency in the design of the algorithm” that Pasquale argues for, “[w]hat we need . . . is a transparency of inputs and results”) (emphasis omitted). ↑
8. See, e.g., Aziz Z. Huq, Racial Equity in Algorithmic Criminal Justice, 68 Duke L.J. 1043 (2019) (arguing that current constitutional doctrine is ill-suited to the task of evaluating algorithmic fairness and that current standards offered in the technology literature miss important policy concerns); Sandra G. Mayson, Bias In, Bias Out, 128 Yale L.J. 2218 (2019) (discussing how past and existing racial inequalities in crime and arrests mean that methods to predict criminal risk based on existing information will result in racial inequality). ↑
9. See Julia Angwin et al., Machine Bias, ProPublica (May 23, 2016), https://www.pro­publica.org/article/machine-bias-risk-assessments-in-criminal-sentencing [https://perma.cc/BA53-JT7V]. ↑
10. Equivant, Practitioner’s Guide to COMPAS Core 7 (2019), http://www.equivant.com/wp-content/uploads/Practitioners-Guide-to-COMPAS-Core-040419.pdf [https://perma.cc/LRY6-RXAH]. ↑
11. See Angwin et al., supra note 8 (“Northpointe’s core product is a set of scores derived from 137 questions that are either answered by defendants or pulled from criminal records. Race is not one of the questions.”). ↑
12. Id. ↑
13. Northpointe, along with CourtView Justice Solutions Inc. and Constellation Justice Systems, rebranded to Equivant in January 2017. Equivant, Frequently Asked Questions 1, http://my.courtview.com/rs/322-KWH-233/images/Equivant%20Customer%20FAQ%20-%20FINAL.pdf [https://perma.cc/7HH8-LVQ6]. ↑
14. See William Dieterich et al., COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity, Northpointe 9–10 (July 8, 2016), http://go.volarisgroup.com/rs/430-MBX-989/images/ProPublica_Commentary_Final_070616.pdf [https://perma.cc/N5RL-M9RN]. ↑
15. For a critique of ProPublica’s analysis, see Anthony W. Flores et al., False Positives, False Negatives, and False Analyses: A Rejoinder to “Machine Bias: There’s Software Used Across the Country To Predict Future Criminals. And It’s Biased Against Blacks.”, 80 Fed. Prob. 38 (2016). ↑
16. See Dieterich et al., supra note 13, at 9–11. ↑
17. See Angwin et al., supra note 8 (“In forecasting who would re-offend, the algorithm made mistakes with black and white defendants at roughly the same rate but in very different ways.”). ↑
18. See, e.g., Richard Berk et al., Fairness in Criminal Justice Risk Assessments: The State of the Art, Soc. Methods & Res. OnlineFirst 1, 23 (2018), https://journals.sagepub.com/doi/­10.1177/0049124118782533 [https://perma.cc/GG9L-9AEU] (discussing the required trade­off between predictive accuracy and various fairness measures); Alexandra Chouldechova, Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments, 5 Big Data 153, 157 (2017) (demonstrating that recidivism prediction instruments cannot simultaneously meet all fairness criteria where recidivism rates differ across groups because its error rates will be unbalanced across the groups when the instrument achieves predictive parity); Jon Kleinberg et al., Inherent Trade-Offs in the Fair Determination of Risk Scores, 67 LIPIcs 43:1, 43:5–8 (2017), https://drops.dagstuhl.de/opus/volltexte/2017/8156/pdf/LIPIcs-ITCS-2017-43.pdf [https://perma.cc/S9DM-PER2] (demonstrating how difficult it is for algorithms to simultaneously achieve the fairness goals of calibration and balance in predictions involving different groups). ↑
19. See Bureau of Justice Statistics, U.S. Dep’t of Justice, 2018 Update on Prisoner Recidivism: A 9-Year Follow-up Period (2005–2014) 6 tbl.3 (2018), https://www.bjs.gov/­content/pub/pdf/18upr9yfup0514.pdf [https://perma.cc/3UE3-AS5S] (analyzing rearrests of state prisoners released in 2005 in 30 states and finding that 86.9% of black prisoners and 80.9% of white prisoners were arrested in the nine years following their release); see also Dieterich et al., supra note 13, at 6 (“[I]n comparison with blacks, whites have much lower base rates of general recidivism . . . .”). Of course, the data on recidivism itself may be flawed. This consideration is discussed below. See infra text accompanying notes 33–37. ↑
20. This is true unless the tool makes no mistakes at all. Kleinberg et al., supra note 17, at 43:5–6. ↑
21. See infra Section I.A. ↑
22. For example, Berk et al. consider six different measures of algorithmic fairness. See Berk et al., supra note 17, at 12–15. ↑
23. See Sam Corbett-Davies & Sharad Goel, The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning (arXiv, Working Paper No. 1808.00023v2, 2018), http://arxiv.org/abs/1808.00023 [https://perma.cc/ML4Y-EY6S]. ↑
24. See Sumegha Garg et al., Tracking and Improving Information in the Service of
    Fairness (arXiv, Working Paper No. 1904.09942v2, 2019), http://arxiv.org/abs/1904.09942 [https://perma.cc/D8ZN-CJ83]. ↑
25. See Sam Corbett-Davies et al., Algorithmic Decision Making and the Cost of Fairness, 2017 Proc. 23d ACM SIGKDD Int’l Conf. on Knowledge Discovery and Data Mining 797, 805. ↑
26. See id. (“[E]xplicitly including race as an input feature raises legal and policy complications, and as such it is common to simply exclude features with differential predictive power.”). ↑

The post Measuring Algorithmic Fairness first appeared on Virginia Law Review.

This Article investigates the legal possibilities and normative appeal of a right to a human decision. I begin by sketching its conditions of technological plausibility. This requires the specification of both a feasible domain of machine decisions and the margins along which machine decisions are distinct from human ones. With this technological accounting in hand, I analyze the normative stakes of a right to a human decision. I consider four potential normative justifications: (a) a concern with population-wide accuracy; (b) a grounding in individual subjects’ interests in participation and reason giving; (c) arguments about the insufficiently reasoned or individuated quality of state action; and (d) objections grounded in negative externalities. None of these yields a general justification for a right to a human decision. Instead of being derived from normative first principles, limits to machine decision making are appropriately found in the technical constraints on predictive instruments. Within that domain, concerns about due process, privacy, and discrimination in machine decisions are typically best addressed through a justiciable “right to a well-calibrated machine decision.”

Introduction

Every tectonic technological change—from the first grain domesticated to the first smartphone set abuzz¹ 1.For recent treatments of these technological causes of social transformations, see generally James C. Scott, Against the Grain: A Deep History of the Earliest States (2017), and Ravi Agrawal, India Connected: How the Smartphone is Transforming the World’s Largest Democracy (2018).Show More—begets a new society. Among the ensuing birth pangs are novel anxieties about how power is distributed—how it is to be gained, and how it will be lost. A spate of sudden advances in the computational technology known as machine learning has stimulated the most recent rush of inky public anxiety. These new technologies apply complex algorithms,² 2.An algorithm is simply a “well-defined set of steps for accomplishing a certain goal.” Joshua A. Kroll et al., Accountable Algorithms, 165 U. Pa. L. Rev. 633, 640 n.14 (2017); see also Thomas H. Cormen et al., Introduction to Algorithms 5 (3d ed. 2009) (defining an algorithm as “any well-defined computational procedure that takes some value, or set of values, as input and produces some value, or set of values, as output” (emphasis omitted)). The task of computing, at its atomic level, comprises the execution of serial algorithms. Martin Erwig, Once Upon an Algorithm: How Stories Explain Computing 1–4 (2017).Show More called machine-learning instruments, to vast pools of public and government data so as to execute tasks previously beyond mere human ability.³ 3.Machine learning is a general purpose technology that, in broad terms, encompasses “algorithms and systems that improve their knowledge or performance with experience.” Peter Flach, Machine Learning: The Art and Science of Algorithms that Make Sense of Data 3 (2012); see also Ethem Alpaydin, Introduction to Machine Learning 2–3 (3d ed. 2014) (defining machine learning in similar terms). For the uses of machine learning, see Susan Athey, Beyond Prediction: Using Big Data for Policy Problems, 355 Science 483, 483 (2017) (noting the use of machine learning to solve prediction problems). I discuss the technological scope of the project, and define relevant terms, infra at text accompanying note 111. I will use the terms “algorithmic tools” and “machine learning” interchangeably, even though the class of algorithms is technically much larger.Show More Corporate and state actors increasingly lean on these tools to make “decisions that affect people’s lives and livelihoods—from loan approvals, to recruiting, legal sentencing, and college admissions.”⁴ 4.Kartik Hosanagar & Vivian Jair, We Need Transparency in Algorithms, But Too Much Can Backfire, Harv. Bus. Rev. (July 23, 2018), https://hbr.org/2018/07/we-need-transparency-in-algorithms-but-too-much-can-backfire [https://perma.cc/7KQ9-QMF3]; accord Cary Coglianese & David Lehr, Regulating by Robot: Administrative Decision Making in the Machine-Learning Era, 105 Geo. L.J. 1147, 1149 (2017).Show More

As a result, many people feel a loss of control over key life decisions.⁵ 5.Shoshana Zuboff, Big Other: Surveillance Capitalism and the Prospects of an Information Civilization, 30 J. Info. Tech. 75, 75 (2015) (describing a “new form of information capitalism [that] aims to predict and modify human behavior as a means to produce revenue and market control”).Show More Machines, they fear, resolve questions of critical importance on grounds that are beyond individuals’ ken or control.⁶ 6.See, e.g., Rachel Courtland, The Bias Detectives, 558 Nature 357, 357 (2018) (documenting concerns among the public that algorithmic risk scores for detecting child abuse fail to account for an “effort . . . to turn [a] life around”).Show More Many individuals experience a loss of elementary human agency and a corresponding vulnerability to an inhuman and inhumane machine logic. For some, “the very idea of an algorithmic system making an important decision on the basis of past data seem[s] unfair.”⁷ 7.Reuben Binns et al., ‘It’s Reducing a Human Being to a Percentage’; Perceptions of Justice in Algorithmic Decisions, 2018 CHI Conf. on Hum. Factors Computing Systems 9 (emphasis omitted).Show More Machines, it is said, want fatally for “empathy.”⁸ 8.Virginia Eubanks, Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor 168 (2017).Show More For others, machine decisions seem dangerously inscrutable, non-transparent, and so hazardously unpredictable.⁹ 9.Will Knight, The Dark Secret at the Heart of AI, MIT Tech. Rev. (Apr. 11, 2017), https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/ [https://perma.cc/L94L-LYTJ] (“The computers that run those services have programmed themselves, and they have done it in ways we cannot understand. Even the engineers who build these apps cannot fully explain their behavior.”).Show More Worse, governments and companies wield these tools freely to taxonomize their populations, predict individual behavior, and even manipulate behavior and preferences in ways that give them a new advantage over the human subjects of algorithmic classification.¹⁰ 10.For consideration of these issues, see Mariano-Florentino Cuéllar & Aziz Z. Huq, Economies of Surveillance, 133 Harv. L. Rev. 1280 (2020), and Mariano-Florentino Cuéllar & Aziz Z. Huq, Privacy’s Political Economy and the State of Machine Learning: An Essay in Honor of Stephen J. Schulhofer, N.Y.U. Ann. Surv. Am. L. (forthcoming 2020).Show More Even the basic terms of political choice seem compromised.¹¹ 11.See, e.g., Daniel Kreiss & Shannon C. McGregor, Technology Firms Shape Political Communication: The Work of Microsoft, Facebook, Twitter, and Google with Campaigns During the 2016 U.S. Presidential Cycle, 35 Pol. Comm. 155, 156–57 (2018) (describing the role of technology firms in shaping campaigns).Show More At the same time that machine learning is poised to recalibrate the ordinary forms of interaction between citizen and government (or big tech), advances in robotics as well as machine learning appear to be about to displace huge tranches of both blue-collar and white-collar labor markets.¹² 12.For what has become the standard view, see Larry Elliott, Robots Will Take Our Jobs. We’d Better Plan Now, Before It’s Too Late, Guardian (Feb. 1, 2018, 1:00 AM), https://www.theguardian.com/commentisfree/2018/feb/01/robots-take-our-jobs-amazon-go-seattle [https://perma.cc/2CFP-3JJV]. For a more nuanced account, see Martin Ford, Rise of the Robots: Technology and the Threat of a Jobless Future 282–83 (2015).Show More A fearful future looms, one characterized by massive economic dislocation, wherein people have lost control of many central life choices, and basic consumer and political preferences are no longer really one’s own.

This Article is about one nascent and still inchoate legal response to these fears: the possibility that an individual being assigned a benefit or a coercive intervention has a right to a human decision rather than a decision reached by a purely automated process (a “machine decision”). European law has embraced the idea. American law, especially in the criminal justice domain, is flirting with it.¹³ 13.See infra text accompanying notes 70–73.Show More My aim in this Article is to test this burgeoning proposal, to investigate its relationship with technological possibilities, and to ascertain whether it is a cogent response to growing distributional, political, and epistemic anxieties. My focus is not on the form of such a right—statutory, constitutional, or treaty-based—or how it is implemented—say, in terms of liability or property rule protection—but more simply on what might ab initio justify its creation.

To motivate this inquiry, consider some of the anxieties unfurling already in public debate: A nursing union, for instance, launched a campaign urging patients to demand human medical judgments rather than technological assessment.¹⁴ 14.‘When It Matters Most, Insist on a Registered Nurse,’ Nat’l Nurses United, https://www.­nationalnursesunited.org/insist-registered-nurse [https://perma.cc/MB66-XTXW] (last visited Jan. 19, 2020).Show More And a majority of patients surveyed in a 2018 Accenture survey preferred treatment by a doctor in person to virtual care.¹⁵ 15.Accenture Consulting, 2018 Consumer Survey on Digital Health: US Results 9 (2018), https://www.accenture.com/_acnmedia/PDF-71/Accenture-Health-2018-Consumer-Survey-Digital-Health.pdf#zoom=50 [https://perma.cc/TU5F-9J82].Show More When California proposed replacing money bail with a “risk-based pretrial assessment” tool, a state court judge warned that “[t]echnology cannot replace the depth of judicial knowledge, experience, and expertise in law enforcement that prosecutors and defendants’ attorneys possess.”¹⁶ 16.Quentin L. Kopp, Replacing Judges with Computers Is Risky, Harv. L. Rev. Blog (Feb. 20, 2018), https://blog.harvardlawreview.org/replacing-judges-with-computers-is-risky/ [https://perma.cc/WS5S-ARVF]. On the current state of affairs, see California Set to Greatly Expand Controversial Pretrial Risk Assessments, Filter (Aug. 7, 2019), https://filtermag.org/­california-slated-to-greatly-expand-controversial-pretrial-risk-assessments/ [https://perma.cc­/2FNX-U3C9].Show More In 2018, the City of Flint, Michigan, discontinued the use of a highly effective machine-learning tool designed to identify defective water pipes, reverting under community pressure to human decision making with a far lower hit rate for detecting defective pipes.¹⁷ 17.Alexis C. Madrigal, How a Feel-Good AI Story Went Wrong in Flint, Atlantic (Jan. 3, 2019), https://www.theatlantic.com/technology/archive/2019/01/how-machine-learning-fou­nd-flints-lead-pipes/578692/ [https://perma.cc/V8VA-F22W].Show More Finally, and perhaps most powerfully, consider the worry congealed in an anecdote told by data scientist Cathy O’Neil: An Arkansas woman named Catherine Taylor is denied federal housing assistance because she fails an automated, “webcrawling[,] data-gathering” background check.¹⁸ 18.Cathy O’Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy 152–53 (2016).Show More It is only when “one conscientious human being” takes the trouble to look into the quality of this machine result that it is discovered that Taylor has been red-flagged in error.¹⁹ 19.Id. at 153.Show More O’Neil’s plainly troubling anecdote powerfully captures the fear that machines will be unfair, incomprehensive, or incompatible with the flexing of elementary human agency: it provides a sharp spur to the inquiry that follows.

The most important formulation of a right to a human decision to date is found in European law. In April 2016, the European Parliament enacted a new regime of data protection in the form of a General Data Protection Regulation (GDPR).²⁰ 20.Regulation 2016/679, of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of Such Data, and Repealing Directive 95/46/EC (General Data Protection Regulation), 2016 O.J. (L 119) (EU) [hereinafter GDPR]; see also Christina Tikkinen-Piri, Anna Rohunen & Jouni Markkula, EU General Data Protection Regulation: Changes and Implications for Personal Data Collecting Companies, 34 Computer L. & Security Rev. 134, 134–35 (2018) (documenting the enactment process of the GDPR).Show More Unlike the legal regime it superseded,²¹ 21.See Directive 95/46, of the European Parliament and of the Council of 24 October 1995 on the Protection of Individuals with Regard to the Processing of Personal Data and on the Free Movement of Such Data, art. 1, 1995 O.J. (L 281) (EC) [hereinafter Directive 95/46].Show More the GDPR as implemented in May 2018 is legally mandatory even in the absence of implementing legislation by member states of the European Union (EU).²² 22.Bryce Goodman & Seth Flaxman, European Union Regulations on Algorithmic Decision Making and a “Right to Explanation,” AI Mag., Fall 2017, at 51–52 (explaining the difference between a non-binding directive and a legally binding regulation under European law).Show More Hence, it can be directly enforced in court through hefty financial penalties.²³ 23.Id. at 52.Show More Article 22 of the GDPR endows a natural individual with “the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.”²⁴ 24.GDPR, supra note 20, arts. 4(1), 22(1) (inter alia, defining “data subject”).Show More That right covers private and some (but not all) state entities.²⁵ 25.See id. art. 4(7)–(8) (defining “controller” and “processor” as key scope terms). The Regulation, however, does not apply to criminal and security investigations. Id. art. 2(2)(d).Show More On its face, it fashions an opt-out of quite general scope from automated decision making.²⁶ 26.As I explain below, this is not the only provision of the GDPR that can be interpreted to create a right to a human decision. See infra text accompanying notes 53–58.Show More

The GDPR also has extraterritorial effect.²⁷ 27.GDPR, supra note 20, art. 3.Show More It reaches platforms, such as Google and Facebook, that offer services within the EU.²⁸ 28.There is sharp divergence in the scholarship over the GDPR’s extraterritorial scope, which ranges from the measured, see Griffin Drake, Note, Navigating the Atlantic: Understanding EU Data Privacy Compliance Amidst a Sea of Uncertainty, 91 S. Cal. L. Rev. 163, 166 (2017) (documenting new legal risks to American companies pursuant to the GDPR), to the alarmist, see Mira Burri, The Governance of Data and Data Flows in Trade Agreements: The Pitfalls of Legal Adaptation, 51 U.C. Davis L. Rev. 65, 92 (2017) (“The GDPR is, in many senses, excessively burdensome and with sizeable extraterritorial effects.”).Show More And American law is also making tentative moves toward a similar right to a human decision. In 2016, for example, the Wisconsin Supreme Court held that an algorithmically generated risk score “may not be considered as the determinative factor in deciding whether the offender can be supervised safely and effectively in the community” as a matter of due process.²⁹ 29.State v. Loomis, 881 N.W.2d 749, 760 (Wis. 2016).Show More That decision precludes full automation of bail determinations. There must be a human judge in the loop. The Wisconsin court’s holding is unlikely to prove unique. State deployment of machine learning has, more generally, elicited sharp complaints sounding in procedural justice and fairness terms.³⁰ 30.See, e.g., Julia Angwin, Jeff Larson, Surya Mattu & Lauren Kirchner, Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And It’s Biased Against Blacks, ProPublica 2 (May 23, 2016), https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing [https://perma.cc/Q9ZU-VY6J] (criticizing machine-learning instruments in the criminal justice context).Show More Further, the Sixth Amendment’s right to a jury trial has to date principally been deployed to resist judicial factfinding.³¹ 31.See, e.g., Apprendi v. New Jersey, 530 U.S. 466, 477 (2000) (explaining that the Fifth and Sixth Amendments “indisputably entitle a criminal defendant to a jury determination that [he] is guilty of every element of the crime with which he is charged, beyond a reasonable doubt” (alteration in original) (internal quotation marks omitted) (quoting United States v. Gaudin, 515 U.S. 506, 510 (1995))).Show More But there is no conceptual reason why the Sixth Amendment could not be invoked to preclude at least some forms of algorithmically generated inputs to criminal sentencing. Indeed, it would seem to follow a fortiori that a right precluding a jury’s substitution with a judge would also block its displacement by a mere machine.

In this Article, I start by situating a right to a human decision in its contemporary technological milieu. I can thereby specify the feasible domain of machine decisions. I suggest this comprises decisions taken at high volume in which sufficient historical data exists to generate effective predictions. Importantly, this excludes many matters presently resolved through civil or criminal trials but sweeps in welfare determinations, hiring decisions, and predictive judgments in the criminal justice contexts of bail and sentencing. Second, I examine the margins along which machine decisions are distinct from human ones. My focus is on a group of related technologies known as machine learning. This is the form of artificial intelligence diffusing most rapidly today.³² 32.See infra text accompanying note 88 (defining machine learning). I am not alone in this focus. Legal scholars are paying increasing attention to new algorithmic technologies. For leading examples, see Kate Crawford & Jason Schultz, Big Data and Due Process: Toward a Framework to Redress Predictive Privacy Harms, 55 B.C. L. Rev. 93, 109 (2014) (arguing for “procedural data due process [to] regulate the fairness of Big Data’s analytical processes with regard to how they use personal data (or metadata . . . )”); Andrew Guthrie Ferguson, Big Data and Predictive Reasonable Suspicion, 163 U. Pa. L. Rev. 327, 383–84 (2015) (discussing the possible use of algorithmic prediction in determining “reasonable suspicion” in criminal law); Kroll et al., supra note 2, at 636–37; Michael L. Rich, Machine Learning, Automated Suspicion Algorithms, and the Fourth Amendment, 164 U. Pa. L. Rev. 871, 929 (2016) (developing a “framework” for integrating machine-learning technologies into Fourth Amendment analysis).Show More A right to a human decision cannot be defined or evaluated without some sense of the technical differences between human decision making and decisions reached by these machine-learning technologies. Indeed, careful analysis of how machine learning is designed and implemented reveals that the distinctions between human and machine decisions are less crisp than might first appear. Claims about a right to human decision, I suggest, are better understood to turn on the timing, and not the sheer fact, of such involvement.

With this technical foundation in hand, I evaluate the right to a human decision in relation to four normative ends it might plausibly be understood to further. A first possibility turns on overall accuracy worries. My second line of analysis takes up the interests of an individual exposed to a machine decision. The most pertinent of these interests hinge upon an individual’s participation in decision making and her opportunity to offer reasons. A third analytic salient tracks ways that a machine instrument might be intrinsically objectionable because it uses a deficient decisional protocol. I focus here on worries about the absence of individualized consideration and a machine’s failure to offer reasoned judgments. Finally, I consider dynamic, system-level effects (i.e., negative spillovers), in particular in relation to social power. None of these arguments ultimately provides sure ground for a legal right to a human decision.

Rather, I suggest that the limits of machine decision making be plotted based on its technical constraints. Machines should not be used when there is no tractable parameter amenable to prediction. For example, if there is no good parameter that tracks job performance, then machine evaluation of those employees should be abandoned. Nor should they be used when decision making entails ethical or otherwise morally charged judgments. Most important, I suggest that machine decisions should be subject to a right to a well-calibrated machine decision that folds in due process, privacy, and equality values.³³ 33.A forthcoming companion piece develops a more detailed account of how this right would be vindicated in practice through a mix of litigation and regulation. See Aziz Z. Huq, Constitutional Rights in the Machine Learning State, 105 Cornell L. Rev. (forthcoming 2020).Show More This is a better response than a right to a human decision to the many instruments now implemented by the government that are highly flawed.³⁴ 34.For a catalog, see Meredith Whittaker et al., AI Now Inst., AI Now Report 2018, at 18–22 (2018), https://ainowinstitute.org/AI_Now_2018_Report.pdf [https://perma.cc/2BCG-M4­54].Show More

My analysis here focuses on state action that imposes benefits or coercion on individuals—and not on either private action or a broader array of state action—for three reasons. First, salient U.S. legal frameworks, unlike the GDPR’s coverage, are largely (although not exclusively) trained on state action. Accordingly, a focus on state action makes sense in terms of explaining and evaluating the current U.S. regulatory landscape. Second, the range of private uses of algorithmic tools is vast and heterogenous. Algorithms are now deployed in private activities ranging from Google’s PageRank instrument,³⁵ 35.See, e.g., David Segal, The Dirty Little Secrets of Search: Why One Retailer Kept Popping Up as No. 1, N.Y. Times, Feb. 13, 2011, at BU1.Show More to “fintech” applied to generate new revenue streams,³⁶ 36.See Falguni Desai, The Age of Artificial Intelligence in Fintech, Forbes (June 30, 2016, 10:42 PM), http://www.forbes.com/sites/falgunidesai/2016/06/30/the-age-of-artificial-intelli­gence-in-fintech [https://perma.cc/DG8N-8NVS] (describing how fintech firms use artificial intelligence to improve investment strategies and analyze consumer financial activity).Show More to medical instruments used to calculate stroke risk,³⁷ 37.See, e.g., Benjamin Letham, Cynthia Rudin, Tyler H. McCormick & David Madigan, Interpretable Classifiers Using Rules and Bayesian Analysis: Building a Better Stroke Prediction Model, 9 Annals Applied Stat. 1350, 1350 (2015).Show More to engineers’ identification of new stable inorganic compounds.³⁸ 38.See, e.g., Paul Raccuglia et al., Machine-Learning-Assisted Materials Discovery Using Failed Experiments, 533 Nature 73, 73 (2016) (identifying new vanadium compounds).Show More Algorithmic tools are also embedded within new applications, such as voice recognition software, translation software, and visual recognition systems.³⁹ 39.Yann LeCun et al., Deep Learning, 521 Nature 436, 438–41 (2015).Show More In contrast, the state is to date an unimaginative user of machine learning, with a relatively constrained domain of deployments.⁴⁰ 40.See infra text accompanying notes 117–21 (describing state uses of machine learning).Show More This makes for a more straightforward analysis. Third, where the state does use algorithmic tools, it often results directly or indirectly in deprivations of liberty, freedom of movement, bodily integrity, or basic income. These normatively freighted machine decisions present arguably the most compelling circumstances for adopting a right to a human decision and so are a useful focus of normative inquiry.

The Article proceeds in three steps. Part I catalogs ways in which law has crafted, or could craft, a right to a human decision. This taxonomical enterprise demonstrates that such a right is far from fanciful. Part II defines the class of computational tools to be considered, explores the manner in which such instruments can be used, and teases out how they are (or are not) distinct from human decisions. Doing so helps illuminate the plausible forms of a right to a human decision. Part III then turns to the potential normative foundations of such a right. It provides a careful taxonomy of those grounds. It then shows why they all fall short. Finally, a brief conclusion inverts the Article’s analytic lens to gesture at the possibility that a right to a well-calibrated machine decision can be imagined, and even defended, on more persuasive terms than a right to a human decision.

1. * Frank and Bernice J. Greenberg Professor of Law, University of Chicago Law School. Thanks to Faith Laken for terrific research aid. Thanks to Tony Casey, David Driesen, Lauryn Gouldin, Daniel Hemel, Darryl Li, Anup Malani, Richard McAdams, Eric Posner, Julie Roin, Lior Strahilevitz, Rebecca Wexler, and Annette Zimmermann for thoughtful conversation. Workshop participants at the University of Chicago, Stanford Law School, the University of Houston, William and Mary Law School, and Syracuse University School of Law also provided thoughtful feedback. I am grateful to Christiana Zgourides, Erin Brown, and the other law review editors for their careful work on this Article. All errors are mine, not the machine’s. ↑
2. For recent treatments of these technological causes of social transformations, see generally James C. Scott, Against the Grain: A Deep History of the Earliest States (2017), and Ravi Agrawal, India Connected: How the Smartphone is Transforming the World’s Largest Democracy (2018). ↑
3. An algorithm is simply a “well-defined set of steps for accomplishing a certain goal.” Joshua A. Kroll et al., Accountable Algorithms, 165 U. Pa. L. Rev. 633, 640 n.14 (2017); see also Thomas H. Cormen et al., Introduction to Algorithms 5 (3d ed. 2009) (defining an algorithm as “any well-defined computational procedure that takes some value, or set of values, as input and produces some value, or set of values, as output” (emphasis omitted)). The task of computing, at its atomic level, comprises the execution of serial algorithms. Martin Erwig, Once Upon an Algorithm: How Stories Explain Computing 1–4 (2017). ↑
4. Machine learning is a general purpose technology that, in broad terms, encompasses “algorithms and systems that improve their knowledge or performance with experience.” Peter Flach, Machine Learning: The Art and Science of Algorithms that Make Sense of Data 3 (2012); see also Ethem Alpaydin, Introduction to Machine Learning 2–3 (3d ed. 2014) (defining machine learning in similar terms). For the uses of machine learning, see Susan Athey, Beyond Prediction: Using Big Data for Policy Problems, 355 Science 483, 483 (2017) (noting the use of machine learning to solve prediction problems). I discuss the technological scope of the project, and define relevant terms, infra at text accompanying note 111. I will use the terms “algorithmic tools” and “machine learning” interchangeably, even though the class of algorithms is technically much larger. ↑
5. Kartik Hosanagar & Vivian Jair, We Need Transparency in Algorithms, But Too Much Can Backfire, Harv. Bus. Rev. (July 23, 2018), https://hbr.org/2018/07/we-need-transparency-in-algorithms-but-too-much-can-backfire [https://perma.cc/7KQ9-QMF3]; accord Cary Coglianese & David Lehr, Regulating by Robot: Administrative Decision Making in the Machine-Learning Era, 105 Geo. L.J. 1147, 1149 (2017). ↑
6. Shoshana Zuboff, Big Other: Surveillance Capitalism and the Prospects of an Information Civilization, 30 J. Info. Tech. 75, 75 (2015) (describing a “new form of information capitalism [that] aims to predict and modify human behavior as a means to produce revenue and market control”). ↑
7. See, e.g., Rachel Courtland, The Bias Detectives, 558 Nature 357, 357 (2018) (documenting concerns among the public that algorithmic risk scores for detecting child abuse fail to account for an “effort . . . to turn [a] life around”). ↑
8. Reuben Binns et al., ‘It’s Reducing a Human Being to a Percentage’; Perceptions of Justice in Algorithmic Decisions, 2018 CHI Conf. on Hum. Factors Computing Systems 9 (emphasis omitted). ↑
9. Virginia Eubanks, Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor 168 (2017). ↑
10. Will Knight, The Dark Secret at the Heart of AI, MIT Tech. Rev. (Apr. 11, 2017), https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/ [https://perma.cc/L94L-LYTJ] (“The computers that run those services have programmed themselves, and they have done it in ways we cannot understand. Even the engineers who build these apps cannot fully explain their behavior.”). ↑
11. For consideration of these issues, see Mariano-Florentino Cuéllar & Aziz Z. Huq, Economies of Surveillance, 133 Harv. L. Rev. 1280 (2020), and Mariano-Florentino Cuéllar & Aziz Z. Huq, Privacy’s Political Economy and the State of Machine Learning: An Essay in Honor of Stephen J. Schulhofer, N.Y.U. Ann. Surv. Am. L. (forthcoming 2020). ↑
12. See, e.g., Daniel Kreiss & Shannon C. McGregor, Technology Firms Shape Political Communication: The Work of Microsoft, Facebook, Twitter, and Google with Campaigns During the 2016 U.S. Presidential Cycle, 35 Pol. Comm. 155, 156–57 (2018) (describing the role of technology firms in shaping campaigns). ↑
13. For what has become the standard view, see Larry Elliott, Robots Will Take Our Jobs. We’d Better Plan Now, Before It’s Too Late, Guardian (Feb. 1, 2018, 1:00 AM), https://www.theguardian.com/commentisfree/2018/feb/01/robots-take-our-jobs-amazon-go-seattle [https://perma.cc/2CFP-3JJV]. For a more nuanced account, see Martin Ford, Rise of the Robots: Technology and the Threat of a Jobless Future 282–83 (2015). ↑
14. See infra text accompanying notes 70–73. ↑
15. ‘When It Matters Most, Insist on a Registered Nurse,’ Nat’l Nurses United, https://www.­nationalnursesunited.org/insist-registered-nurse [https://perma.cc/MB66-XTXW] (last visited Jan. 19, 2020). ↑
16. Accenture Consulting, 2018 Consumer Survey on Digital Health: US Results 9 (2018), https://www.accenture.com/_acnmedia/PDF-71/Accenture-Health-2018-Consumer-Survey-Digital-Health.pdf#zoom=50 [https://perma.cc/TU5F-9J82]. ↑
17. Quentin L. Kopp, Replacing Judges with Computers Is Risky, Harv. L. Rev. Blog
    (Feb. 20, 2018), https://blog.harvardlawreview.org/replacing-judges-with-computers-is-risky/ [https://perma.cc/WS5S-ARVF]. On the current state of affairs, see California Set to Greatly Expand Controversial Pretrial Risk Assessments, Filter (Aug. 7, 2019), https://filtermag.org/­california-slated-to-greatly-expand-controversial-pretrial-risk-assessments/ [https://perma.cc­/2FNX-U3C9]. ↑
18. Alexis C. Madrigal, How a Feel-Good AI Story Went Wrong in Flint, Atlantic (Jan. 3, 2019), https://www.theatlantic.com/technology/archive/2019/01/how-machine-learning-fou­nd-flints-lead-pipes/578692/ [https://perma.cc/V8VA-F22W]. ↑
19. Cathy O’Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy 152–53 (2016). ↑
20. Id. at 153. ↑
21. Regulation 2016/679, of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of Such Data, and Repealing Directive 95/46/EC (General Data Protection Regulation), 2016 O.J. (L 119) (EU) [hereinafter GDPR]; see also Christina Tikkinen-Piri, Anna Rohunen & Jouni Markkula, EU General Data Protection Regulation: Changes and Implications for Personal Data Collecting Companies, 34 Computer L. & Security Rev. 134, 134–35 (2018) (documenting the enactment process of the GDPR). ↑
22. See Directive 95/46, of the European Parliament and of the Council of 24 October 1995 on the Protection of Individuals with Regard to the Processing of Personal Data and on the Free Movement of Such Data, art. 1, 1995 O.J. (L 281) (EC) [hereinafter Directive 95/46]. ↑
23. Bryce Goodman & Seth Flaxman, European Union Regulations on Algorithmic Decision Making and a “Right to Explanation,” AI Mag., Fall 2017, at 51–52 (explaining the difference between a non-binding directive and a legally binding regulation under European law). ↑
24. Id. at 52. ↑
25. GDPR, supra note 20, arts. 4(1), 22(1) (inter alia, defining “data subject”). ↑
26. See id. art. 4(7)–(8) (defining “controller” and “processor” as key scope terms). The Regulation, however, does not apply to criminal and security investigations. Id. art. 2(2)(d). ↑
27. As I explain below, this is not the only provision of the GDPR that can be interpreted to create a right to a human decision. See infra text accompanying notes 53–58. ↑
28. GDPR, supra note 20, art. 3. ↑
29. There is sharp divergence in the scholarship over the GDPR’s extraterritorial scope, which ranges from the measured, see Griffin Drake, Note, Navigating the Atlantic: Understanding EU Data Privacy Compliance Amidst a Sea of Uncertainty, 91 S. Cal. L. Rev. 163, 166 (2017) (documenting new legal risks to American companies pursuant to the GDPR), to the alarmist, see Mira Burri, The Governance of Data and Data Flows in Trade Agreements: The Pitfalls of Legal Adaptation, 51 U.C. Davis L. Rev. 65, 92 (2017) (“The GDPR is, in many senses, excessively burdensome and with sizeable extraterritorial effects.”). ↑
30. State v. Loomis, 881 N.W.2d 749, 760 (Wis. 2016). ↑
31. See, e.g., Julia Angwin, Jeff Larson, Surya Mattu & Lauren Kirchner, Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And It’s Biased Against Blacks, ProPublica 2 (May 23, 2016), https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing [https://perma.cc/Q9ZU-VY6J] (criticizing machine-learning instruments in the criminal justice context). ↑
32. See, e.g., Apprendi v. New Jersey, 530 U.S. 466, 477 (2000) (explaining that the Fifth and Sixth Amendments “indisputably entitle a criminal defendant to a jury determination that [he] is guilty of every element of the crime with which he is charged, beyond a reasonable doubt” (alteration in original) (internal quotation marks omitted) (quoting United States v. Gaudin, 515 U.S. 506, 510 (1995))). ↑
33. See infra text accompanying note 88 (defining machine learning). I am not alone in this focus. Legal scholars are paying increasing attention to new algorithmic technologies. For leading examples, see Kate Crawford & Jason Schultz, Big Data and Due Process: Toward a Framework to Redress Predictive Privacy Harms, 55 B.C. L. Rev. 93, 109 (2014) (arguing for “procedural data due process [to] regulate the fairness of Big Data’s analytical processes with regard to how they use personal data (or metadata . . . )”); Andrew Guthrie Ferguson, Big Data and Predictive Reasonable Suspicion, 163 U. Pa. L. Rev. 327, 383–84 (2015) (discussing the possible use of algorithmic prediction in determining “reasonable suspicion” in criminal law); Kroll et al., supra note 2, at 636–37; Michael L. Rich, Machine Learning, Automated Suspicion Algorithms, and the Fourth Amendment, 164 U. Pa. L. Rev. 871, 929 (2016) (developing a “framework” for integrating machine-learning technologies into Fourth Amendment analysis). ↑
34. A forthcoming companion piece develops a more detailed account of how this right would be vindicated in practice through a mix of litigation and regulation. See Aziz Z. Huq, Constitutional Rights in the Machine Learning State, 105 Cornell L. Rev. (forthcoming 2020). ↑
35. For a catalog, see Meredith Whittaker et al., AI Now Inst., AI Now Report 2018, at 18–22 (2018), https://ainowinstitute.org/AI_Now_2018_Report.pdf [https://perma.cc/2BCG-M4­54]. ↑
36. See, e.g., David Segal, The Dirty Little Secrets of Search: Why One Retailer Kept Popping Up as No. 1, N.Y. Times, Feb. 13, 2011, at BU1. ↑
37. See Falguni Desai, The Age of Artificial Intelligence in Fintech, Forbes (June 30, 2016, 10:42 PM), http://www.forbes.com/sites/falgunidesai/2016/06/30/the-age-of-artificial-intelli­gence-in-fintech [https://perma.cc/DG8N-8NVS] (describing how fintech firms use artificial intelligence to improve investment strategies and analyze consumer financial activity). ↑
38. See, e.g., Benjamin Letham, Cynthia Rudin, Tyler H. McCormick & David Madigan, Interpretable Classifiers Using Rules and Bayesian Analysis: Building a Better Stroke Prediction Model, 9 Annals Applied Stat. 1350, 1350 (2015). ↑
39. See, e.g., Paul Raccuglia et al., Machine-Learning-Assisted Materials Discovery Using Failed Experiments, 533 Nature 73, 73 (2016) (identifying new vanadium compounds). ↑
40. Yann LeCun et al., Deep Learning, 521 Nature 436, 438–41 (2015). ↑
41. See infra text accompanying notes 117–21 (describing state uses of machine learning). ↑

The post A Right to a Human Decision first appeared on Virginia Law Review.