mircea

Who’s Afraid of AGI?

or, Why you shouldn’t be afraid of mathematics

February, 2023

TL;DR; version:

AGI stands for Artificial General Intelligence.

It is theoretized to be the level of AI at which an “algorithm” exposes general inteligence on par with a human. It is the “phylosopher’s stone” of the modern techno-alchemist*.

Many people, and sometimes even smart ones, are afraid that we are on the verge of inventing an AGI that will be more “intelligent” (whatever that may mean) than humans. Reportedly, Elon Musk can’t sleep at night because of his worries about AI and AGI.

But why be afraid of the AGI?

Well, because that “more intelligent” than humans AGI will start improving itself and will turn itself into a super-AGI which will be so much more intelligent than all of us that it will be like a God compared to us. And then, it might just decide to dispose with our species. Or make us into its pets. The techno-apocalypse, also called the “singularity”, is an idea originally proposed by Ray Kurzweil, which seems to not let some people sleep at night. The fear of AI has flared these days, probably because of the popularity of the recent advances in AI and the associated hopes and journalistic and marketing amplifications.

Note that the field of AI is nowhere near to any such level of generality: most of the algorithms are narrowly focused on a single domain: playing chess, playing go, generating images, generating text, classifying images, etc. (And it should be like that. Algorithms are tools, and tools are there to enhance our capabilities, not to replace us).

Moreover, most of those who are afraid of AI seem to forget that all current AI is just advanced applied mathematics. Even the recently impressive ChatGPT can be seen in some sense as a giant mathematical function that takes as input the last 1000 tokens of text, and generates text. You give it some input. It provides some output. You should’t lose sleep over it. (Also, if you pay attention at the text it generates, you’ll realize it looks smarter than it is… In fact, the more impressed you are by it, the more likely you’re not paying attention to the details; but that’s another story.)

Surely, there is a clear increase in complexity when one goes from linear regression, to logistic regression, to neural networks, to deep neural networks, and to large language models, but in the end, each one of these models is, nothing more than a very complicated mathematical function defined as a statistical approximation from very large quantities of example data. A stochastic mathematical function that you don’t understand is still a … mathematical function.

By being afraid of AI, one is literally afraid of mathematical functions. Yes, it’s true, we’re all a bit afraid of mathematics, but that’s because we’re too lazy to put the time and understand it, not because mathematics is dangerous in itself.

In fact, until recently, those who were afraid of mathematics would rather not mention it. It is really a new activity to express your fears about it, and it is definitely a silly idea to try to stop everybody else from doing mathematics for six months.

Thus, don’t lose sleep over fears of AGI-apocalypse: God is not a mathematical function computed in the Sillicon Valley.

And instead of proposing that people stop doing research in AI, I think we’d all be better served by a ban on people who do not work in AI (e.g. journalists, marketing, pundits, Elon Musk): for six months they should not be allowed to worry in public about the techno-apocalypse. It’d be a better, quieter, more rational world.

* I’m not the only one who sees the parallel with alchemy: Rodney Brooks, the robotics pioneer, says in a tweet: “My take on LLM/CHATters. We’re going to transmute lead to gold! Why now? We just pumped massive energy in and got Pb to turn to liquid. First time ever. It logically follows that soon we will have AgI(silver incalculable) and even AuI. We (my in group) are going to be RIIICCCHHH!” (source)

Further Reading

• Rodney Brooks: What will Transformers Transform
• Andrew NG in a Letter to The Batch from Oct 25, 2023 covers very similar points to mine, including the kinds of people who might benefit from stirring fear around AI. Also he says: “My greatest fear for the future of AI is if overhyped risks (such as human extinction) lets tech lobbyists get enacted stifling regulations that suppress open-source and crush innovation.” link to tweet

Further Notes

• There’s probably a small cohort of “fearful” – the behavior of which can be explained by another dictum: “You can not make somebody understand something, if their job depends on not understanding it”. These are the pundits, the journalists, the researchers who by hyping up the “potential” of the technology benefit themselves in one way or another.
• There’s also the fear of some of these algorithms taking over our jobs. It might take over some jobs and it will help us get better at other jobs. Washing dishes has been taken over by robots and nobody misses it. Chess has been automated to the point where no human can beat an algorithm, and that does not prevent us to still enjoy the game, admire and reward the talented human players (reportedtly Magnus Carlsen has a net worth of 50M). In fact, the chess engines are just helping everybody to become better at playing the game. This second is the future that I see of language models, e.g. we’ll all get better at writing, we won’t be “replaced by a writing” algorithm, whatever that may mean.
• Sure, if we put statstical functions that we don’t fully understand inside systems that can harm humans, than we should be afraid. But that’s a stupid thing to do in the first place; and in that case the fear should not be of the AI component itself but of the unpredictibility of the whole resulting system. My rule of thumb is this: use AI if you can “undo” its action (spam classification, code completion) or if you don’t give a speck about mistakes (image search that returns a muffin in between dogs) then use AI as much as you can. However, don’t let an AI algorithm drive your car or shoot weapons. Driving over a child can’t be undone . And you do care about mistsakes in weapon-related situations too.
• The whole discussion about LLMs and the fear thereof, reminds of of the Arthur C. Clarke saying “Any sufficiently advanced technology is indistinguishable from magic”. And if something is magic, you have all the rights to be afraid of it, I guess.
• Unlikely to have AGI w/o a sense of self eliot miranda on linkedin . And a sense of self is not well-defined without a body is my conjecture.
• Elon musk can’t sleep at night because he’s afraid of AI; Bostrom is at least capitalizing on his fears by writing a book on the topic