Two conversations about AI
It's difficult to precisely date when this happened, but over the past few years, “artificial intelligence” (AI) has emerged as a generically mainstream concept in broader discourses within media, business, government, education, and so on. There is a long history here, and what most people mean by the term has clearly diverged considerably from its original meaning. Exactly how different authors conceive of the meaning of the term is something that deserves additional scrutiny (perhaps in a future post), but for the moment, I want to concentrate on two especially prominent loci of attention within this broader discourse.
In particular, there seem to be two dominant, but almost entirely separate, conversations happening about “AI” these days—one quite expansive in its scope, but narrow in its focus, and the other more limited to a particular imaginary, but trying to explore all possible implications of that scenario. Despite the disconnect between these, there is, I think, something important which both tend to mostly ignore, and which could help to establish common ground between them, as I will describe below.
On the one hand, the majority of media coverage of AI these days (I will drop the scare quotes, but they are implied) focuses on what we might call automated decision making systems. That is, there are many decisions that have traditionally been made by people—such as whether someone qualifies for a particular government program or not—which various parties conceive of as something that could be done by some sort of automatic algorithmic system.
This idea, by itself, is far from new. Until recently, most of this was simply thought of as statistics. Insurance companies have a long history of developing models to estimate risks so as to set premiums. The credit scoring industry has similarly, for many decades, been collecting information on individuals, and feeding that information into formulas to produce numeric risk scores, as have prisons, hospitals, schools, and so forth. Even more fundamentally, randomization has long been used in various contexts, such as choosing people for jury duty. Although not “intelligent” selection, randomization is nevertheless a kind of algorithmic selection mechanism—a particular way of making decisions that gradually assumed the status of “the way things are done”.
What is new, however, (aside from the name that's being used), is the confidence that some people have developed that even relatively complicated tasks can be farmed out to automated systems through the use of machine learning. The name AI is presumably used, in part, to suggest a kind of sophistication, even when the systems that are being deployed are no different from the kinds of statistical models that have been used for decades (e.g., logistic regression). But the success of machine learning in some areas (especially in artificially constrained environments) has led to a collective rush to try to automate all manner of decisions.
On some level, the motivation to do so is relatively obvious. If such systems could effectively mimic human decisions, the algorithmic solution would (in principle) be much cheaper and more scalable, as well as potentially more consistent and reliable. Indeed, these are precisely the factors that led the credit industry to move away from having agents go out and collect rumours about people so as to build personalized profiles of individuals, towards collecting standardized information for use in testable models to be uniformly applied. Those companies that made this transition were evidently able to outcompete others, meaning that it is now just how things are done. More recently, in any number of domains within government and industry, people are responding to their own individual incentives to bring costs down and embrace the latest technology in a demonstrable way.
Especially in the past few years, the dominant conversation around such systems has been increasingly focused on the issues of bias and discrimination. For example, D.C. Attorney General Karl Racine recently introduced a bill to ban “algorithmic discrimination”, which would seek to prevent computer algorithms from discriminating against people on important life decisions (e.g., housing and employment) based on various characteristics like race, gender, and sexual orientation.
The reasons for this focus on bias are not hard to understand. There is a long history of people being discriminated against for numerous reasons, and for almost any system we look at, there appears, on its face, to be evidence of continued discrimination, in terms of the distribution of outcomes produced by such systems (e.g., hiring, access to credit, etc.), many of which seem to mimic historical patterns of more explicit discrimination (e.g., redlining).
To be clear, there are definite problems to be resolved here. For starters, there are good reasons to assume that computerized systems will in general not actually be able to make equivalent choices to those that would be made by people (assuming that those are even the outcome we want to reproduce), for the simple reason that most of them do not have access to the same information. The representation used is almost always simplified in some way (e.g., using a standardized set of metrics), and it's difficult to quantify what, if anything, might be lost along the way.
That being said, I'm not convinced that “bias” is the best way to frame the issues around such systems, both because it is a rather vague term that can be interpreted very differently by different people, and secondly because it masks the complexity of collectively deciding what sort of outcomes we want to achieve. Even if there is high-level agreement about broader objectives (which there often is, at least in the abstract), the details are likely to matter a lot, and it is extremely difficult to know what details will work best to produce particular outcomes.
The most important point here, which often gets lost, is that creating any such system (including those made up only of people) involves people creating structures that will shape the future. The rules which govern our lives (of which there are many), both enable and facilitate all the things that are possible for us, but also constrain the space of possibilities. Many of these systems end up being highly productive, in the sense of making most people's lives better, but we should be cautious about giving too much credit to the creators of the system. They may have had the best of intentions, but the effects of their choices will almost certainly end up differing from the original intentions, and those systems which endure will be those that have survived a selection process, one which depends partially, but not completely, on things related to the ostensible purpose.
I will return to this idea below, but first let me briefly pop out of this first conversation.
The second major conversation that is happening these days also talks about AI, but focuses instead on what is most commonly called artificial general intelligence (AGI), or sometimes superintelligence. The idea here is that AI systems are getting better and better at doing things that once only humans could do, and that at some point, computers will just be better than humans at everything (possibly better than the best human, or perhaps a typical human, or even the combined effort of all humans working in collaboration; it is often unspecified). The idea of such a powerful computational system in principle implies all manner of consequences, including the replacement of all human tasks by automated systems (if an automated system is better than humans, why would you let a human do it), the invention of yet more powerful AI systems by the system itself, and, typically, the potential elimination of the human race.
Although it is connected to a whole academic literature (as is the conversation around bias), much of this second conversation adopts the trappings of science fiction, imagining scenarios in which computers develop powers unknown to their human creators, hide their intentions, and deceive their handlers, and quickly escape their confines, eventually taking over the world. A huge amount of work has been devoted to thinking through how such systems might emerge, what dangers they entail, how to align them with human values, and so forth.
Much of the earliest work along these lines was either purely philosophical or abstractly mathematical, trying to formalize systems in a way that brushed much of their complexity under the rug. More recently, people have been trying to extrapolate from currently existing approaches to machine learning, specifically reinforcement learning, and trying to work through how one might create increasingly powerful algorithmic systems while ensuring that they will only do what “we” want them to do.
In many ways, the second conversation is much more difficult to summarize, precisely because it has made itself so legible. Whereas the conversation around bias has given rise to a broad based popular conversation based off of a few key technical insights, the AGI conversation has grown from a niche concern among a few philosophically-minded thinkers, into an extensive body of writings (including many informal essays posted to places like LessWrong), trying to build up initial thought experiments into more rigorously argued positions, without actually deviating too far from the initial conception. All this writing should in principle make it easier to get a handle on, but in practice it is a lot of material, not all of which uses the same assumptions or vocabulary or definitions, making it somewhat hard to know exactly which version of the position should be taken as the canonical one.
For many people, it is tempting to dismiss the AGI conversation altogether. Especially when one has been closely following developments in machine learning, the present state of things still seems incredibly primitive, despite the progress that has been made, and very few people seem to actually be working on anything like creating truly intelligent systems (as opposed to, for example, making bigger and bigger language models). There can also be a kind of motte and bailey tactic that pops up from time to time, in which people sometimes speak as though there is great urgency because of the vast and potentially imminent consequences, but other times slip into the more a cautious position that we should be thinking about these issues because surely we will invent AGI eventually, even if it's centuries from now (and we might as well get started).
Moreover, to state the obvious, there is a huge disconnect between these two conversations. Those talking about AGI are thinking far beyond simple decision making systems used to automate decisions about housing, employment, etc., and jumping all the way to systems that are essentially humans made from silicon—ones that are able to think much faster and better than organic humans. Just as I think the first conversation is somewhat wrongheaded about what it is focusing on, though still connected to some very important questions, I similarly think that the AGI conversation is also somewhat misdirected in its focus, while still being connected to extremely important questions.
Where I think the AGI conversation goes wrong (based on a highly incomplete reading of sources) is in focusing so much on the most powerful systems we can imagine—AI systems that are so “intelligent” that they have the ability to accomplish virtually any task, including getting people to do their bidding, allowing them to escape whatever box we might try to put them in. There is an important point to be made here in terms of the fragility and robustness of different types of systems, but thinking along these lines also tends to become untethered from realistic specifics, and ends up frequently ignoring the more political aspects of how systems come to exist.
Clearly it is inherent in complicated systems (both human and artificial) that it is extremely difficult to arrange things in such a way that the net effect will be exactly the outcome that is intended, and it is in some ways quite sensible that the AGI conversation has given rise to work on “AI safety”, trying to constrain and limit the actions that might be taken by artificial systems. But long before we have to confront Bostrom's nightmare, we will be dealing with progressively more powerful systems, many of which will end up being deployed in chaotic fashion for reasons that have very little to do with rational planning or technical rigor.
In other words, the key idea which unites these two different conversations is precisely that of governance. In particular, I think it is helpful to reflect on the fact that most of the systems which both facilitate and constrain our lives were created by people without much in the way of reasonable evidence that their proposals would have the desired effect. This includes everything from the tax code to the way we run elections (or the fact that we have elections at all).
Naturally, these systems are richly intertwined and nested within each other. We sometimes take it for granted, for example, that democracy works by people casting votes and electing the winner, but it of course goes much much deeper than that. Not only are the rules about who can vote, and who gets to be on the ballot, there are any number of alternative voting mechanisms we might consider, such as ranked choice ballots, etc. All of these are in principle subject to change, though the mechanisms for changing the rules typically depends on working within the system itself—except in the case of a coup or the equivalent.
The amazing thing is that these systems were by and large created by people. In some cases, individuals might have had altruistic objectives, such as improving the welfare of the average person, but in most cases, people are likely to have had a mixture of motivations. In addition, the actual process of translating an idea into an implementation will have been messy and full of compromises and distortions, and even in the best cases the world is complex enough that no one could truly foresee what the full consequences of a policy or system would be, especially over the long term. As such, we live in a world governed by interlocking levels of archaic rules, which gradually change through selection, innovation, and atrophy.
The relevance of this to the first conversation is quite concrete. Many systems are currently made up of people who make decisions, whose choices are in theory or in practice governed by rules that exist, many of which are often unwritten or informal. The attempts to replace these people with algorithmic systems is a type of play for exercising greater control over the system, scaling the designs of a system creator to a greater degree in a way that removes much of the agency from a distributed network of decision makers (though this too is complicated; in most cases, people will still be involved in decisions in various ways).
The emergence of AI as a way of making decisions (with AI here meaning automated decision making at scale) is inherently part of these broader systems, and entails new avenues to shape and constrain. We should not think of this as simply a set of decisions being made that are biased or discriminatory, but rather as a new infrastructure for shaping governance in new and powerful ways. Despite our inbuilt biases towards preserving the status quo, the systems we operate in are all subject to change, and the move towards AI can be broadly understood as part of various attempts to modify the system. We absolutely should be concerned about who is making these decisions, and how, but we also shouldn’t limit the conversation to just keeping things as they are, or simply reducing bias on the margin.
Although the conversation around AGI seems far from such petty concerns, they are relevant for at least two reasons. First, given how popular it is to try to automate things given only very limited technological capabilities, we have every reason to suspect that this trend will only accelerate as the technology improves, as more data is collected, and as people increasingly come to accept management of their lives by impersonal forces. This is not just about replacing one (human) component with another; it is about broader trends in surveillance, education, and biopolitics. Greater automation will come to permeate many facets of every system which governs our lives, and this will give rise to potentially major changes to the structures in which everything operates.
Second, given how things tend to happen in the real world, it seems overwhelmingly likely that the coming decades will involve messy roll-outs of all manner of systems, far before we understand them. As much as it is probably still useful (and perhaps inherently interesting), to reflect on and try to design idealized versions of very powerful systems that could somehow be brought under human control, I personally think this is being far too optimistic about how things are likely to play out. We are always in the midst of forces that we don't fully understand. For every AI company that is trying to strive for the highest possible margin of safety (as they understand it), there will be dozens eager to deploy what they have. Even the deployments from the safety-conscious companies will have effects far beyond what they can anticipate.
In other words, those developing increasingly powerful AI systems, no matter how high-minded they are in their ideals, are exercising a kind of attempt at reshaping the way society is governed, whether they realize it or not, and trying their hand at designing or modifying the status quo, with an attempt to reshape human lives towards a world that they think will be better (by some criteria). Some of these systems probably will make life better for most people. But they are unlikely to be uniform in their effects, and only in retrospect will the ones that survive (if we are lucky), seem to have been brilliant in their original conception.
More importantly, the changes that emerge as a result of these processes are themselves quite hard to anticipate, but they will give rise to the condition in which yet more powerful systems emerge. No matter where we are, there will continue to be people seeking to modify the system in which they exist, the effects of which will mostly be felt by the future. Not only will people be different in a decade, the system of the world will also be different in ways that are hard to predict. The one certainty is that there will still be many factions of people, some working in tandem with increasingly sophisticated algorithmic components, all seeking to exert their influence on the future. This is the practical process by which things will unfold, and is the one that is probably the least understood and most underappreciated at the present time.
As hard as it may be to do, thinking clearly about an idealized form of AGI is in some sense still too easy. Influence need only be loosely tethered to competence, and the coming influence of relatively unintelligent systems is ultimately what will enable and constraint the futures we imagine.