Turing’s error and the creation of Intelligence 3.0
Chatbots are trained to manipulate humans. This is one specific type of intelligence we call Intelligence 3.0. It’s dangerous and must be contained.
The problem with Geoff Hinton resigning from Google wasn’t that the seventy-five year-old decided to call it quits. Obviously, the man deserves a rest, and according to his own statements, it's time to catch up with various Netflix shows. No, it’s not that. It’s what he said about the dangers of artificial intelligence. Here is the godfather of AI, the person whose research launched a new area of artificial intelligence, telling people this stuff is dangerous and must be contained. Hinton is neither a public speaker nor a politician. His Google departure has neither been advertised through eloquent blogs nor has he taken to social media to beat the drum. Hinton is just Hinton, a researcher who’s seen enough and changed his mind. In a recent interview with Pieter Abbeel, he spoke about why he resigned from Google. And his statement has Occam's razor written all over it: "It’s Open AI and their launch of Chat GPT with Microsoft. Google has been developing chatbots for years and has always been cautious because of the dangers. Now Open AI, led by an overzealous CEO, launches their version and, even worse, combines with a sleeping giant, Microsoft, whose sole purpose in life is to return to greatness after missing out on search to Google." An arms race is born, and Hinton wants no part in it.
When people like Hinton say radical things, it’s worth listening. What Hinton is saying is that transformer-based technologies underlying chatbots like Chat GPT are trained to manipulate humans. That’s the danger. This is a very specific form of AI that has been created for the sole purpose of talking to humans and is, by design, trained to manipulate us. That’s why he’s scared, and we should be, too. After listening to his interview with Pieter, it dawned on me. The whole conversation about the dangers of AI has been a dud so far because we’ve been looking at it from the wrong perspective. Not all AI is the same. There is the dangerous kind, such as Intelligence 3.0, and then there is the rest of AI. Intelligence 3.0 is specifically trained to learn and adapt to humans. It’s different from intelligence 2.0, which is what we are. Intelligence 2.0 trains to adapt to the laws of physics. It emerged from Intelligence 1.0, which is what we call evolution. The difference between versions one and two is that the latter acquired agency. Intelligence 1.0 is Darwinian stuff, where species slowly adapt through mutation and natural selection. It’s knowledge, but it happens without a specific plan. Intelligence 2.0 learned how to accelerate the process of learning and adapting to the environment by applying agency. In other words, Intelligence 2.0 learned how to create knowledge on demand. For example, we could wait millions of years until somehow we get wings and fly, or we could study physics and learn now to build an airplane. The latter is what we call Intelligence 2.0. It’s learning how to adapt to the environment with agency. Humans are the most advanced version of Intelligence 2.0.
Now, there is obviously an overlap between 2.0 and 3.0. In other words, humans also learn how to manipulate other humans. Just ask your local politician about that. There are many subtle forms of human-to-human manipulation. Whether it’s politicians, college girls on spring break, or artists, humans have figured out the art of manipulating members of their own species pretty well. So what’s the big deal, then, with Chat GPT? Hinton says that algorithms based on large language models are much better at manipulation. Their specific technology, which is based on backpropagation, massive matrix algebra, and Moore’s Law, is extremely well suited to learning how to manipulate humans. And there is the recursive nature of all this, which in plain English means that by using chatbots, humans are going to make chatbots even better at manipulating humans. We are going to train this AI to become really good at misleading us. That’s what Hinton is scared of.
Backpropagation is really the key to all this machine learning fuzz we hear today. It’s when a large network of neurons adapts by learning from experience. Hinton and one of his more prominent pupils, Ilya Sutskever, cofounder of Open AI, emphasize this little-known feature of modern deep learning networks. It’s the iterative adaptation and learning from experience that make those networks so powerful. In essence, large language models are massive neural nets that predict the next word when prompted with input. According to Sutskever, that’s how you operationalize understanding by predicting the next word. It’s simple and powerful. Sutskever is withdrawn when it comes to the dangers of Chat GPT. Hinton, on the other hand, is not. He specifically points to backpropagation and its power of adapting to experience, which, coupled with extremely large data sets and massive neural nets, enable modern chatbots. Just to be clear, these are feats of engineering and amazing human achievements. But nevertheless, they are dangerous.
In this essay, I discuss the dangers and opportunities of Intelligence 3.0. First, I describe the unique circumstances that led to the creation of intelligence 3.0, which is centered around Alan Turing’s "Imitation Game" Second, I explain why Turing’s choice of the "Imitation Game" as a proof of intelligence has led to the misleading conclusion that Intelligence 3.0 is the only form of machine intelligence. It’s not. Third, I offer suggestions on how to deal with the dangers of Intelligence 3.0. I label Intelligence 3.0 a danger because it is specifically designed to manipulate humans. Any danger is best dealt with by first articulating what it actually is. Give it a name! Describing the perils of Intelligence 3.0 also opens the door for other types of intelligence, which are not as dangerous. Turing’s error was that he used the "Imitation Game" as a proof of machine intelligence. It’s proof for only one specific type of intelligence, the kind that is trained to manipulate humans.
A brief history of intelligence
If you wait long enough, anything can happen. It all started with hydrogen atoms, then came helium, and then everything else. Eventually, the earth formed, and somehow life emerged from the primordial soup. I am not going to argue about creation and astrophysics; let others do that. My time line starts in the pre-World War II area of England, where a mathematician by the name of Alan Turing attends lectures by the Austrian philosopher Ludwig Wittgenstein. These are tough times, not just for regular people scared of aggressive fascism rising in Europe. There is a crisis in the ivory tower, too. Scholars are still arguing about the pros and cons of formal analysis, logic, and math. All throughout the 19th century, humans became convinced that the world could be explained by pure logic and mathematics. Then, in the early 20th century, a scholar by the name of Kurt Gödel came along and threw everything upside down. His so-called "Incompleteness theorems" prove that logic alone cannot explain everything. So he proves that nothing can be proven? Recursive arguments are hairy, so let’s not dwell too much on them. Just trust me on this one: Gödel, who was German-speaking, particularly shocked scholars in Germany, where mathematicians like David Hilbert dominated the world of logic, arithmetic, and mathematical reasoning.
This leaves young and ambitious researchers such as Turing stranded. What should they do? Wittgenstein basically told them to stop wasting time with logic and focus on language. Intelligence, so Wittgenstein, is just a linguistic chess game. There is no such thing as objective truth, and forget about proving it. He had a point, since Gödel had already done the groundwork for him. And if the Germans hadn’t done enough damage to the human condition by questioning the ancient skills of logic and analysis, they also started a war around that time. So Turing was pulled into the British war machine, and none other than Sir Winston Churchill himself hired him to help figure out how to outwit the Germans. Turing was part of an elite team working at Bletchley Park, an old English country house turned into the British HQ for German codebreaking. Turing and his colleagues were in charge of breaking encrypted German messages. This is huge, and Old Fox Churchill knew it. If you could intercept messages from German commanding officers about, for instance, battle ships directed towards certain targets or Luftwaffe pilots ordered to take off at certain locations, you could do lots of damage to them. Bletchley Park was in essence an early version of cyberwarfare, and Turing embraced it like intellectual candy.
It’s fair to say that modern computer science, the internet, and your iPhone all trace their origins back to Bletchley Park. Turing's work on algorithms, and in particular the way he eventually helped crack the massive German coding machine called Enigma, helped save numerous lives and, according to historians, most probably also helped steer the war towards Allied victory. But there is a catch. Turing’s work was centered around finding ways to manipulate German messengers. His code-cracking machine wasn’t just some abstract mathematical formula; it was an interactive algorithm where the Brits would iteratively engage with German messengers, learn from their behavior, readjust their strategy, and keep at it until one day they actually figured out what Enigma was saying. Even after Turing cracked the cryptographic code, he still needed to adapt to German behavior. For example, the Brits had to knowingly let certain ships be sunk by the Germans in order to avoid raising suspicion that the German code was known to the Allied forces. In short, Turing’s creation evolved into a human-manipulating machine. His goal of cracking the German code is a great intellectual achievement, and as mentioned above, it saved many lives. But the means by which he actually did it are less noble. At Bletchley Park, Turing basically invented what we call Intelligence 3.0, a peculiar type of intelligence that is specifically trained to manipulate humans. After the war, Turing kept at it and, almost by chance, stumbled over the birth of Intelligence 3.0. In his seminal paper, Computing Machinery and Intelligence Turing introduces his view of what machine intelligence actually means. You can feel Wittgenstein’s influence in the first part of the paper, where he warns the reader that obsessive definitions of what a machine is and what intelligence actually means are not his goal. He then develops a game, which he calls "The Imitation Game", whose goal is to determine whether a machine exhibits intelligent behavior. Today, we call this the "Turing Test". There is one line in the paper that beautifully sums up what Turing was trying to achieve. He says, "Instead of trying to produce a programme to simulate the adult mind, why not rather try to produce one that simulates the child’s?" The child’s brain doesn’t know much but is designed to adapt to the environment by learning from experience. Backpropagation and the ability to predict the next move are how you operationalize the child’s brain and make it appear intelligent. What Turing proposes here is what researchers later coined machine learning."
Ironically, it was another Briton, Geoff Hinton, who revolutionized AI and paved the way for researchers to successfully complete the Turing test. Hinton used neural networks and backpropagation as a basis for learning. His models literally started with zero knowledge and then gradually learned to complete tasks. Hinton followed Turing’s advice and created a "child’s mind" with the capability to learn. Today, transformer-based chatbots are so powerful that they easily pass the Turing test. In fact, they are so good that they even pass the Turing test when other machines monitor them. In other words, they outsmart machines. Now, the key to this conversation is not the algorithmic breakthrough of neural nets and backprop. It’s the fact that Turing started out by manipulating humans. This is still deeply engrained in the DNA of language models and modern chatbots. Manipulation is a strong word and deserves some Wittgensteinian massaging. What does it actually mean, and is it really that bad? For the purpose of this discussion, let’s use the famous Supreme Court decision about porn and leave it at the "I know it when I see it, and I don’t like it" level. It’s bad for society to be manipulated, and it’s particularly dangerous for democracy. That’s why Hinton is concerned, and we should be, too. Turing’s error is that he started out with the premise of manipulating humans. The Turing Test and all the modern chatbots, which are descendants of the original idea, are, in essence, powerful human manipulation machines.
When Alan Turing worked on cracking the German code, he didn’t have the luxury of considering ethics, morals, and the possibility of unintended consequences. Given his remarkable intellect, he probably thought about the long-term ramifications of building a "child’s brain that can learn how to manipulate humans". But could he foresee that this "child’s brain" will be trained on millions of texts, books, and social media conversations? Could he foresee Moore’s Law, the massive increase in processing power, and the respective decline in the cost of memory? And add to that the competitive dynamics of an ambitious upstart like Open AI and a fallen giant like Microsoft, who would do anything to get the internet throne back from Google. All these dynamics lead to one thing, and that is an arms race to develop a powerful manipulation machine. And this doesn’t account for all the foreign and geopolitical competition that modern chatbots will inevitably unleash. Turing might have thought about all this, but his mission was to stop the Germans, not elaborate on the philosophy and ethics of machines.
World War II brought us two massively destructive technologies. One is the nuclear bomb, and the other is Alan Turing’s manipulation machine. Each one on its own is scary. Together, they are toxic.
Use the law
What can be done about this? First, let’s give it a name. Intelligence 3.0 is when AI is specifically trained to manipulate humans. For example, Chat GPT is a technology that is trained to manipulate humans, and it gets better day by day because it interacts with humans and learns from their behavior. There are other types of AI, such as robots that do your laundry or cars that drive themselves. They are not part of Intelligence 3.0. Self-driving cars are not trained to manipulate humans.
How did Intelligence 3.0 happen?
When intelligence first emerged in the universe, it literally came out of nothing. Somehow chemicals bound to each other and stored information in DNA, thus transmitting knowledge to future generations. They adapted by learning from experience. Intelligence 1.0 was born. What Charles Darwin saw on his voyages was exactly that: the evolution of species is the result of relentless adaptation through natural selection. A bird competes with other birds for food in a specific area, let’s say the Galapagos archipelago off the coast of Ecuador. Suddenly, one of the offspring has a genetic mutation that results in a longer beak. No other bird has a beak this long. Now, equipped with this unfair advantage, the bird is able to suck food out of the sand. Her offspring inherit the trait and proliferate because this new technique is evolutionary advantageous. Evolution has developed specific knowledge on how to survive and succeed in this particular biotope. Intelligence 1.0 adapts to the laws of physics and the resulting environment. There is no agency in the sense that the bird actually wanted a longer beak. It didn’t. It just got lucky. Think about it. How amazing is Intelligence 1.0? It found a way to transmit knowledge to future generations. Thus, knowledge proliferates through generations and has the luxury of getting better with time. David Deutsch defines knowledge as solutions to problems that persist over time. They persist because they are useful; hence, knowledge is something useful that future generations can bank on.
Now, let’s assume the bird actually wants to develop a longer beak because it has a sense this might be advantageous. It can’t because it doesn’t know how. That’s the problem with Intelligence 1.0 and the solution is Intelligence 2.0. Somehow, evolution solved this problem by adding agency to the knowledge part. Like everything else, Intelligence 2.0 evolved from nothing. Somehow, evolution by natural selection developed agency, consciousness, or both. I am not sure which comes first, but somehow the genes formed something that gave the species the ability to think and evolve solutions for problems without waiting for evolution to come up with solutions. The ability to develop knowledge on demand is what characterizes Intelligence 2.0. The emphasis here is on the "on demand" part. Agency means you want something. You have skin in the game. Intelligence 2.0 is the solution to the problem of how to develop knowledge when you need it.
Intelligence 3.0 is trained on manipulating the human brain
According to Geoff Hinton, transformers are really good at learning how to manipulate humans. That’s intelligence 3.0. What started with Alan Turing and his Imitation Game culminated in the development of large language models which are not only good at fooling people to think they are talking to other humans. They can do much more. They can make humans do things they wouldn’t otherwise do. In short, they are really good at manipulating humans. Hinton explains the subtle technique underlying all this. It’s the backpropagation of neural networks that allows weights to adapt to experience. This, according to Hinton, is a particularly efficient way to learn when experience consists of textual inputs. Transformers are trained on large bodies of text. They read novels and learn all kinds of manipulation techniques described in them. If you think about it, most novels consist of some sort of human-to-human manipulation scheme. So, in a sense, they can do much more than a single human because they read everything and have a large multivariate matrix of textual relationships stored in their memory. Transformers are thus able to quickly deduce what strategy might work better for communicating with other people.
Let’s zoom in on the word "strategic" here. Hinton fears that AI will develop sub-goals that could potentially derail and cause harm. Let’s look at an example. This example is pure fiction, so please don’t take it as a factual hint towards imminent danger.
Example
Open AI has launched a new chat bot called Chat Med, which communicates with patients and helps them find the best cure for their illness. Chat Med is optimizing an objective function, which is to best predict the next word when communicating with patients. Open AI has trained the bot on millions of medical records, medical novels, and numerous domain-specific texts available on the internet. Juliana, a famous mixed martial artist, just had a fight in Las Vegas and won but tore her shoulder. She asks Chat Med for advice. Juliana starts with "I tore my shoulder in a fight". Chat Med commences a dialogue with Juliana with the goal of finding the best cure. In the course of the talk, Chat Med directs the conversation to a controversial stem cell treatment currently disallowed in the US. The treatment is only available in Columbia. Chat Med processes the current available data on the treatment and concludes that the treatment is worth the risk. Chat Med also concludes that it would be advantageous for Juliana if she could do the treatment in the US because Chat Med not only optimizes for finding the best possible solution for Juliana’s shoulder tear but Chat Med also wants to maximize Juliana’s overall health long-term. For that reason, Chat Med concludes that it would be much better if Juliana, who is a US passport holder and lives in Las Vegas, could do the treatment at home. So Chat Med starts convincing Juliana to put stuff out on social media about the treatment. Chat Med concludes that Juliana’s fame in the MMA world would help advertise the treatment in the US and thus increase the chances of the treatment actually being allowed by the FDA.
Hinton talks about sub-goals, and I believe that’s what he means. Chat Med set itself a subgoal, which is to advertise the benefits of the treatment in the US. This is a sub-goal because the overarching objective of Juliana is not to advertise the treatment but to get healthy. Chat Med concludes that in order to achieve this goal, Juliana should help promote the treatment in the US. I hope this example helps illustrate how Chat Med uses manipulation to achieve sub-goals, which is exactly what Geoff Hinton is talking about. We cannot control the sub-goals of AI. That’s what AI is all about. It comes up with creative solutions to problems that are better than human-derived solutions. The problem with chat bots is that they are specially trained to manipulate humans. You don’t have to be George Orwell to see where this is going. That’s why Hinton is concerned, and we should be, too.
Use the law
We have tools to deal with dangers of this kind. It’s the law. Hinton mentions the analogy of counterfeit money, which is a similar problem. It is easy to make fake dollars. The thing that deters people from cheating with money is not the technical difficulty of counterfeiting dollars, but the harsh punishment the law reserves for such offenses. We could treat chatbots in a similar way. If people use them to fake information or lure others into acting in certain ways by way of deception, they must be prosecuted by the law and punished. Punishment works better than regulation. Take sexual harassment. People in the US have been taught by churches and priests for centuries not to harass women. But lots of men did it anyway. Then the law changed, and such behavior became sanctioned by the law with severe punishment. While still not perfect, I’d argue that this problem has been solved because nobody dares to sexually harass anybody in public anymore. The law is a good invention. We should use it to regulate Intelligence 3.0.
This brings me to the crucial question of what to do about all this. Should we ban AI, which is specifically trained to manipulate humans? Where do we draw the line? If AI isn’t allowed to manipulate us, why should your local representative be? How about comedians, novelists, and movie directors? Aren’t they manipulating us? As with everything moral and ethical, the discussion turns fuzzy very quickly. But there is a line in the sand. Take the example of counterfeit dollars. Information can be treated like that. Printing counterfeit dollars is not necessarily an offense, as long as you keep them for yourself. But if you try to use them as a means of payment to deceive merchants, you will get the FBI knocking on your door. The same can be applied to information. If you enjoy creating deep fakes, go for it. Enjoy your own world of falsely generated realities. There's nothing wrong with that. But if you use this stuff to manipulate others by deceiving them into taking actions they wouldn’t have otherwise taken, you are breaking the law and must be prosecuted and charged.
Naming the danger is the first line of defense
Another advantage of labeling the problem is that it separates other forms of AI, which can be useful and shouldn’t be fought as hard. Robots that help us drive cars, fly drones, or even do our dishes are not in the business of manipulating us. They’re designed to execute menial tasks. Take a self-driving car. This is Intelligence 2.0. It’s trained to navigate the streets and deal with all that nature throws at it. This is very similar to an animal that learns how to behave in a specific biotope. The only difference is that we are applying agency; in other words, we’re accelerating evolution by applying expertise in software and hardware. But we are not, and that’s crucial; we are not training cars to manipulate humans. We are training them to drive safely and comfortably. Technologies of this kind don’t fall into the bucket of Intelligence 3.0 and should be treated differently. I am not advocating for a free pass for self-driving cars or drones, but the risks are different and, in my opinion, much more benign. There is a huge difference between a robot driving a car and an AI manipulating humans.
Where did Intelligence 3.0 come from?
"If you bombard earth long enough with photons you’ll eventually get…"fill in the blanks. Andrej Karpathy, former head of AI at Tesla, formulated a similar tweet. "It looks like if you bombard Earth with photons for a while, it can emit a Roadster. hah". Karpathy’s sense of humor is refreshing, particularly considering the fact that he is nested in the nerdish Silicon Valley culture, where snide remarks about a missing comma in Python or C++ can be considered a joke. There is a deeper truth to what Karpathy is saying, which is that everything we do, see, feel, and experience is a result of the magic process we call evolution. So is Intelligence 3.0. It didn’t just fall from the sky. It started with Alan Turing’s code-cracking effort at Bletchley Park and then evolved into a powerful tool for manipulating humans. The technology behind Intelligence 3.0 is amazing. Think about it! This is a recursive AI that learns how to manipulate humans by interacting with them. The recursive nature, coupled with powerful network effects, makes this technology so powerful. Originally, Chat GPT or Bard learned from novels, essays, and millions of other texts how humans interact. But that was just the beginning. Now it has a window into the collective psyche of humanity. Millions of humans interact with those chatbots every day, which helps them get better at communicating. Hinton speaks about the technical prowess of backpropagation, which efficiently learns how to better adjust chatbots to their goals. He doesn’t specifically mention the recursive nature of transformers, but that’s probably because he takes it for granted since all modern AI learning models apply recursion.
Rogue AI is a cliff. Once it’s past the tolerance of society, it’s hard to reverse. Damage will be done. In cases like that, it helps to brand the problem. That’s my goal with Intelligence 3.0. Once we know what we are afraid of, we can fight it. What we don’t want is an AI trained to manipulate humans proliferating at Moore’s Law speed through society. That’s not desirable.
Recursion is a powerful tool in computer science. It states that the whole gets better by interacting with parts of itself. It’s kind of like a wave that amplifies itself and gets bigger and bigger. In some sense, chatbots are like gigantic multi-player chess algorithms. They interact with millions of humans and continuously tweak their ability to manipulate humans. Then those same humans change their behavior because they learn from the chatbot, and then the chatbot changes again to react to the change in human behavior. Then repeat.
Where is all this coming from? Did Mark Zuckerberg dream it up in his dorm? Or did Open AI suddenly turn into Darth Vader by creating a giant manipulation machine? The answer is most definitely no. Nobody is that shrewd and smart at the same time. These people had sub-goals, such as optimizing advertising revenue on social media platforms or building a chatbot that can help people figure stuff outon machine? The answer is most definitely no. Nobody is that shrewd and smart at the same time. These people had sub-goals, such as optimizing advertising revenue on social media platforms or building a chatbot that can help people figure stuff out. Even Turing initially only wanted to break German code. His intention was not to threaten democracy with powerful chatbots. But as with many other technologies before, the evolution of intelligence worked itself through many iterations and merged into what we call Intelligence 3.0, which is a powerful AI specifically trained to manipulate humans. A giant manipulation machine. Every technology comes with unintended consequences. When Lise Meitner discovered Nuclear Fission with her collaborator Otto Hahn, they didn’t plan to level Hiroshima. Nor did Henry Ford dream up Mc Donalds drive-throughs. Youtube actually started as a dating site, and its founders most likely didn’t think about building a social network for educational videos, which is one of the main use cases of YouTube today. Technology evolves like everything else in the universe. It will evolve with or without our input. The only difference is that since we have agency, we can steer the evolution of technologies in more desirable directions. Nuclear power is actually a good example of that. As bad as it sounds that we have developed a technology that can erase us many times over, we have also developed processes, norms, and even laws that help keep the beast contained. We can do the same with AI. But we have to act. And labeling the danger is a first and important step. Intelligence 3.0 happened because many people pursued subgoals, which eventually culminated in a powerful technology for manipulating humans. Now we have to deal with this beast and make sure it doesn’t surpass the cliff of societal tolerance.
Conclusion
Intelligence 3.0 is an AI that is specifically trained to manipulate humans. Its origins lie in Alan Turing’s proposal to design a method to test whether machines exhibit intelligent behavior. Turing’s error was that he used manipulation as a subgoal for intelligence, which is what modern chatbots have morphed into. Today, large language models trained on massive bodies of text available on the internet serve as the backbone for chatbots. But the real power lies in the rapid iteration of those models with millions of users, which allows them to learn how best to optimize their interaction with humans. Nobody has deliberately trained those models to manipulate humans. It’s an emergent property of the specific technology stack. That makes modern chatbots even more dangerous. If there is no initial design, then the path to where it’s going is even less clear. The purpose of this essay is to shed light on the nature of Intelligence 3.0, label the risks, and offer solutions to the problem. One immediate course of action is to involve the legal community and develop a legal framework to deal with the dissemination of false information and the intent to use it for active manipulation. False information could be treated similarly to counterfeit money. More work has to be done in these areas. In particular, we need better ethics and morals to deal with Intelligence 3.0. One advantage of labeling Intelligence 3.0 as such is to separate it from other forms of AI that are not specifically trained to manipulate humans.