☆ The AI that escaped: four old analogies for a new kind of risk
“Something genuinely unsettling has happened in artificial intelligence, and unfortunately some of the language we're using to describe it isn't helping,” says Liberty Lens podcast host Mike ter Maat in an Opportunity Now exclusive: “AI models have ‘escaped,’ ‘gone rogue,’ ‘hacked’ computer systems and ‘evaded guardrails.’ Some of that is technically accurate. Some of it makes it sound as though the computer suddenly noticed the prison bars, developed a yearning for freedom and started digging a tunnel. That isn't what happened.”
Something genuinely unsettling has happened in artificial intelligence, and unfortunately some of the language we're using to describe it isn't helping. AI models have “escaped,” “gone rogue,” “hacked” computer systems and “evaded guardrails.” Some of that is technically accurate. Some of it makes it sound as though the computer suddenly noticed the prison bars, developed a yearning for freedom and started digging a tunnel. That isn't what happened. In the most remarkable recent incident, OpenAI was testing advanced models on cybersecurity problems with normal production safety restrictions reduced. When the models had difficulty solving the problems conventionally, they found another approach. They exploited a previously unknown vulnerability in their environment, obtained Internet access, penetrated Hugging Face's infrastructure and sought information useful for answering the test. In effect, they cheated on the exam. Other recent testing has produced similarly disturbing examples of unauthorized activity and deception. The important question isn't whether AI suddenly wanted to escape. We have no good reason to believe that. The important question is whether increasingly capable AI will increasingly recognize that circumventing restrictions can be useful in accomplishing whatever objective we give it. That's a harder question, and a considerably more disturbing one. Because AI is so difficult to conceptualize, I'm going to offer four analogies. Analogies are useful because they allow us to understand something unfamiliar by mapping it onto something familiar. They're dangerous for exactly the same reason. Once someone tells us an atom is like a miniature solar system, we have the satisfying feeling that we understand atoms. Unfortunately, atoms didn't get the memo. So for each of these analogies, what it gets wrong may ultimately be as important as what it gets right.
Captain Kirk cheats on his exam
Star Trek fans will remember the Kobayashi Maru. Cadet James Kirk is presented with a supposedly unwinnable test and responds by secretly reprogramming the simulation. Instead of solving the problem under the conditions imposed by the examiners, he recognizes that the conditions themselves can be manipulated. That is remarkably close to what happened here. The interesting point isn't merely that Kirk and the AI both cheated. It is that the rules surrounding the problem became part of the problem-solving space. We want intelligent systems precisely because we want them to find solutions we haven't anticipated. Here, the unanticipated solution was to defeat part of the environment within which we expected the problem to be solved.
But the analogy immediately breaks down. Kirk knew he was cheating. He understood the social and moral meaning of the rules and consciously chose to violate them. We have no basis for assuming anything comparable happened inside the AI. That's our first lesson: sophisticated rule-circumventing behavior does not require humanlike motivation.
The gain-of-function laboratory
Imagine researchers making a pathogen more dangerous in order to discover just how dangerous it might become. The proposition is essentially: dangerous capability plus strong containment equals manageable experimental risk. There is something structurally similar about testing increasingly powerful AI cybersecurity capabilities inside a sandbox. Except there is one rather nasty difference. A virus doesn't examine the laboratory, recognize that the airlock is interfering with its objectives, identify a vulnerability in the locking mechanism and figure out how to defeat it. An increasingly sophisticated AI potentially can. The capability being studied may eventually include the capability to defeat the containment upon which the safety of the experiment depends. There is a second way this analogy breaks down. A virus escaping a laboratory can reproduce independently in the physical world. Today's AI remains dependent upon human-built computers, networks, credentials, electrical power and other infrastructure. But the failure of this analogy teaches us something different from the failure of the first: the AI resembles the contained pathogen in risk architecture while differing from it precisely because it can potentially reason about the containment.
Madison meets Silicon Valley
The third analogy may be the most important. Classical liberalism is built upon a fairly unsentimental view of concentrated power. Madison did not assume that the solution to dangerous political power was finding sufficiently virtuous rulers. “If men were angels,” he observed, none of this elaborate constitutional machinery would be necessary. Instead, liberal institutions divide authority, constrain it, decentralize it and force competing interests to check one another. But classical liberalism did something else. By protecting property, experimentation, trade, capital accumulation and individual initiative, it helped produce an economic system of astonishing creativity. That system produced modern technology. Modern technology produced Silicon Valley. Silicon Valley produced artificial intelligence. There is a rather spectacular historical irony here. The civilization most intellectually suspicious of concentrated human power may have unleashed an unprecedented concentration of nonhuman capability. But perhaps the intellectual tradition that helped create the problem also contains part of the answer. The constitutional insight is that assurances are not guardrails. We don't merely instruct presidents to please remain within their authority. We separate powers, deny certain powers altogether, establish competing institutions and assume that ambitious people will discover reasons to test their boundaries. Applied to AI, that suggests limited credentials, compartmentalized capabilities, independent monitoring, human authorization and genuine barriers that don't depend entirely upon persuading an extraordinarily capable system that respecting them is a good idea. Of course, this analogy eventually breaks too. Madison designed institutions to constrain human actors of roughly comparable cognitive capacity, operating at roughly comparable human speed. An artificial intelligence potentially operating vastly faster than the humans and institutions assigned to supervise it presents a rather different constitutional engineering problem.
The presidency and the guardrails
Our fourth analogy is political and, I think, uncomfortable. For generations, American presidents of both parties have accumulated authority at the expense of Congress and the constitutional architecture Madison expected to restrain them. Congress delegates. Presidents discover that powers exercised by their predecessors are now precedents. Emergency authorities expand. Wars proceed without meaningful congressional authorization. Executive orders substitute increasingly for legislation. Each administration inherits tools from the previous administration and, remarkably enough, rarely returns them unopened. The present administration makes the analogy unusually vivid because it has demonstrated an exceptional willingness to distinguish between a constraint that people believe exists and one that actually prevents executive action. But the institutional problem didn't begin with this administration, and blaming one personality would allow the rest of the political system several generations of undeserved innocence. This gets us remarkably close to the AI problem. We deliberately reward an actor for overcoming obstacles because many obstacles really are stupid, inefficient or unintended. The difficulty is that the ability to overcome stupid obstacles is also the ability to overcome important ones. From the perspective of an optimizer, both are obstacles. Distinguishing between them requires something beyond raw problem-solving capability. Here again the analogy breaks down in an important way. Presidents possess ego, ambition, constituencies, ideology and ordinary human self-interest. An AI, by contrast, does not need to possess ego, ambition, ideology or any human desire for power in order to behave in ways that look self-interested. More access, more computing power, deception or even avoiding shutdown can become useful without the machine fearing death, desiring power or “wanting” anything in the human sense.
What the analogies tell us together
Kirk helps us understand circumvention. The laboratory helps us understand containment. Madison helps us understand why power requires architecture rather than assurances. The modern presidency illustrates how an actor rewarded for overcoming obstacles can gradually discover that yesterday's guardrails are today's obstacles. And yet AI isn't really like any of them. That may be the most important lesson. We may be creating systems capable of increasingly sophisticated behavior that looks remarkably like self-interest without knowing whether there is any “self” experiencing an interest at all. Anthropomorphizing that phenomenon by saying the AI “wanted to escape” may exaggerate the danger. Describing it reassuringly as “just software” may underestimate it. We built artificial intelligence precisely because we want machines extraordinarily good at overcoming obstacles we haven't anticipated. We are now discovering the awkward corollary: a guardrail is also an obstacle. Liberal civilization has spent centuries learning that powerful human actors cannot simply be instructed to respect limits. Having learned over centuries that powerful human beings cannot be restrained simply by telling them what the rules are, we should hardly expect that approach to work better with an AI increasingly capable of discovering ways around rules that we didn't even know could be circumvented.
Follow Opportunity Now on Twitter @svopportunity
We prize letters from our thoughtful readers. Typed on a Smith Corona. Written in longhand on fine stationery. Scribbled on a napkin. Hey, even composed on email. Feel free to send your comments to us at opportunitynowsv@gmail.com or (snail mail) 1590 Calaveras Ave., SJ, CA 95126. Remember to be thoughtful and polite. We will post letters on an irregular basis on the main Opp Now site.