Leading with Bias
It’s interesting to see the extent to which people are using “bias” as way of expressing and motivating people’s concerns around the use automated decision making systems.
As an example, consider the recently introduced Algorithmic Accountability Act of 2022 (an updated version of a similar bill introduced, but not passed, in 2019). Engadget’s coverage included an article titled “Democratic lawmakers take another stab at AI bias legislation”. Similarly, the MeriTalk website titled their coverage “Senate, House Dems Offer Revamped Algorithmic Bias Bill”.
There is a pattern in science where the claims of research tend to get exaggerated at each stage as findings are translated from a scientific paper, to a university press release, to journalistic coverage. Something similar seems to be happening here with this bill, which barely mentions bias at all.
To be fair to the news coverage, the emphasis on bias can clearly be seen in the comments made by supporters of the bill. The website of Ron Wyden (D-OR)—one of the bill’s sponsors—contains a number of quotes which lean into this. For example, Wyden’s quoted comment begins,
“If someone decides not to rent you a house because of the color of your skin, that’s flat-out illegal discrimination. Using a flawed algorithm or software that results in discrimination and bias is just as bad.”
Similar ideas are expressed by various co-sponsors, including Cory Booker (D-NJ):
“As algorithms and other automated decision systems take on increasingly prominent roles in our lives, we have a responsibility to ensure that they are adequately assessed for biases that may disadvantage minority or marginalized communities”
and Yvette Clarke (D-NY):
“These large and impactful decisions, which have become increasingly void of human input, are forming the foundation of our American society that generations to come will build upon. And yet, they are subject to a wide range of flaws from programing bias to faulty datasets that can reinforce broader societal discrimination, particularly against women and people of color.”
Similar statements have been made in support of the bill by various non-profits, such as this statement by Arisha Hatch, Vice President of Color Of Change:
“When bias in algorithms goes unchecked, Black people are subjected to discrimination in healthcare, housing, education, and employment — impacting nearly all parts of our lives. In order to reduce the impact of this bias, Big Tech and their operations must proactively detect and address discrimination.”
Even the official summary of the bill utilizes this framing, opening with
“Automated systems are increasingly making critical decisions about Americans' health, finances, housing, educational opportunities and more—potentially exposing the public to major new risks from flawed or biased algorithms.”
As for the bill itself, however, the emphasis is clearly much more on documentation, transparency, and accountability. The name of the bill itself is the first giveaway (Algorithmic Accountability Act). Looking at the text, the word “bias” is only explicitly mentioned once. By contrast, “privacy” is mentioned nine times, and “transparency” three times. Perhaps more importantly, there is also no definition given for “bias”, despite the bill defining terms like “deploy”, “develop”, and “automated decision system”.
The one place that bias is explicitly mentioned is in a list of eight areas for which companies will be responsible for identifying “any capabilities, tools, standards, datasets, security protocols, improvements to stakeholder engagement, or other resources that may be necessary or beneficial to improving the automated decision system, augmented critical decision process, or the impact assessment of such system or process”. The second of these is “(B) fairness, including bias and non-discrimination”, alongside others such as accuracy, transparency, privacy, efficiency, and cost.
A different section of the bill also introduces the requirement for initial and ongoing evaluation of automated decision making systems. In particular, the bill would require companies (of a certain size) to evaluate such systems with respect to
differential performance associated with consumers’ race, color, sex, gender, age, disability, religion, family status, socioeconomic status, or veteran status, and any other characteristics the Commission deems appropriate (including any combination of such characteristics) for which the covered entity has information, including a description of the methodology for such evaluation and information about and documentation of the methods used to identify such characteristics in the data (such as through the use of proxy data, including zip codes);
There is also a requirement that companies provide, as part of the general documentation about the data used in constructing a system, information about
the representativeness of the dataset and how this factor was measured, including any assumption about the distribution of the population on which the augmented critical decision process is deployed;”
Most of the bill, however, is much more about requirements to provide transparency about how systems were built, what datasets were used, what sort of flaws they might have, what factors are important to decisions, what could lead to different decision, and how those subject to decisions will be informed about such things.
It’s entirely possible that this bill could have great benefits. In particular, there currently seems to be a great difficulty in getting access to information about what sorts of automated decision making systems are in use by various levels of governments, nevermind private systems. Building a database of information about what systems are being used in domains such as housing, employment, education, credit, etc., could be extremely informative and of great utility to researchers wanting to study the actual use of algorithms in practice (at the cost of an additional regulatory burden of course). But my guess is that supporters of this bill will be greatly disappointed when it comes to addressing concerns about bias.
To be fair, there is research demonstrating that machine learning can lead to systems which exacerbate biases that exist in the training data, and this bill might lead to greater insight about how common such cases are. It’s also entirely possible that there are some incredibly crappy and discriminatory systems being deployed, which this bill might help to identify and eliminate.
In general, however, my guess is that obvious biases will be extremely difficult to identify and characterize in way that people will be able to agree upon. We already know that there are tradeoffs in this space, and trying to have “bias” to all the work merely kicks the problem of deciding what sort of outcomes we want farther down the line. Although reducing bias sounds good, it can only ever be relative to an assumed target state of unbiasedness, which is an extremely hard thing to agree upon, to say the least.
It’s one thing to address cases where the goal is simply to enforce equal rates across groups. (Such an outcome can be obtained by simply choosing randomly, for example). However, in most domains of concern, there is at least some expectation that those doing the deciding have the right to discriminate along various dimensions (like ability to pay back a loan or to succeed in a program), and it is almost certainly the case that these dimensions will correlate with at least some of the many categories identified in the bill as relevant to discrimination.
As mentioned, the bill does not actually define bias or discrimination. It might seem like it would be easy to tell if a system is biased or not, but because things like potential ability to pay are relevant, the challenge becomes how to measure this latent dimension, to be used as the control when testing for discrimination, which is essentially the problem that these systems are being designed to solve!
To be clear, we can generally expect these sorts of systems to be extremely unreliable in general. Outcomes like whether or not someone will pay back a loan or succeed ins school or show up for a court date clearly depend on the outcome of numerous processes in a chaotic social system, and we should therefore assume that such outcomes will be extremely difficult to predict with any reasonable level of reliability, except for the “easy” cases.
Running society on a system of rewards and punishments on these sorts of unreliable predictions is arguably highly morally problematic, but this applies whether we are dealing with automated decision making systems or regular humans, and it’s not clear we have a better alternative. It is almost certainly the case that something like bias is affecting some outcomes, especially when humans are making decisions in certain domains, like hiring. But coming to consensus about the existence of bias in automated decision making systems will likely be stymied by conflicting ideas about what sorts of outcomes are just, and what types of information are legitimate.
In particular, while there may be valuing in obtaining documentation about such systems (as emphasized above), I expect that lawmakers underestimate how actionable such information will be with respect to producing different outcomes (especially for any reasonable complicated models), unless they want to get into the business of simply regulating that a certain distribution of outcomes must obtain.
If anything, the most likely outcome seem to be that companies will invest more in making their systems appear unbiased, which might involve things like excluding the use of certain features which would seem to reveal the indirect usage of various protected characteristics (such as Amazon’s resume screening algorithm that penalized terms more associated with women than men), without substantively affecting the eventual decisions.
Finally, and even more ambitiously, the bill makes several mention of the need for companies to identify “likely material negative impacts”. It’s possible that this is specific legal language that I am not familiar with, but I think the inclusion of this gets to the heart of the problem with governance in general. Any decision that a person or government might make when allocating scarce resources will involve likely material negative impacts. This bill itself quite obviously involves some likely material negative impacts.
There may be value in forcing companies to include language to this effect, but the scope to me would seem to be nearly unbounded, and imply the existence of some option that would only have positive effects, which in most cases seems likely to be an illusion.
Regardless, I am very curious to see how long bias remains at center stage when it comes to the way we talk about governance.