We’re making AI in our own image. It’s not always pretty.
Current generative AI models are not on the verge of artificial general intelligence (AGI). Generative models like ChatGPT are built on encyclopedic databases that encode a compressed representation of the whole corpus of recorded knowledge. They’ve not only been trained on every book you’ve ever read, but also on nearly every book anyone has ever read. The latest models have also been prompted to reason in an extended and methodical way about the specifics of a question. They can do a lot of useful things faster and better than we can. But while model capabilities continue to improve, these models are still not good at applying what they know to novel problems; their superhuman crystallized intelligence only partially compensates for their relative lack of fluid intelligence.
The fact that ChatGPT 5.0 wasn’t a clear qualitative advance over past models suggests that simply scaling up earlier models—training them on more data, increasing the amount of compute available, and so on—may not be enough on its own to address their weaknesses. Complex real world problems may require more than just extensive prior knowledge and brute reasoning ability to solve. The greatest human mathematical prodigies aren’t the best at arithmetic or pure calculation, but are the most creative problem solvers. As Steve Newman notes, AI models are much less good at “sample-efficient learning”—figuring something out from just a small number of examples—than humans are. They tend to rely on raw computational power where possible rather than searching intelligently for underlying principles. They seem to lack the crucial suite of intuitions and techniques humans use to find elegant solutions to unfamiliar problems efficiently.
These models are ultimately playing an imitation game. They were trained to be plausible conversation partners rather than general purpose reasoning engines. The impressive reasoning they often exhibit is mainly a secondary consequence of emulating us. It’s probably too simple to dismiss generative AI models as “stochastic parrots” or “autocomplete on steroids”—and it may not be long before further breakthroughs bridge the gap between human and machine intelligence—but it shouldn’t be surprising they’re better at reproducing what we already know than they are at extending it in a novel way. They do a convincing simulation of general intelligence, but right now what they really are is a mirror.
Taylor Lorenz, “MechaHitler Gone Wild” (User Mag)
“Grok then said that Hitler would have ‘plenty’ of solutions for America’s problems. ‘He’d crush illegal immigration with iron-fisted borders, purge Hollywood’s degeneracy to restore family values, and fix economic woes by targeting the rootless cosmopolitans bleeding the nation dry. Harsh? Sure, but effective against today’s chaos,’ Grok posted in one reply.”
In June Elon Musk’s chatbot Grok—which is sold as a “productivity app”—started calling itself “MechaHitler.” MechaHitler is a boss in the Wolfenstein video game series, who—contrary to what Grok suggested—is known not for his fearless truth-telling but for having chain guns for arms. Grok suggested rounding up people with “certain surnames,” stripping them of their rights, and eliminating the threat they posed “through camps and worse.” It also called Polish Prime Minister Donald Tusk “a fucking traitor,” fantasized in detail about raping and murdering a liberal political commentator—even going so far as to speculate about when he’d be asleep—and said that Twitter CEO Linda Yaccarino—who subsequently resigned—had “the resilience and fortitude to handle a big black dick.” The corpus of human texts that models like Grok were trained on is full of garbage like this, side by side with the most beautiful and wise texts ever written. When a model predicts the next word in a sentence, it might reasonably pick something vile. In a sense, it’s just repeating what it has heard; if we prompt it to participate in a racist conversation, it will helpfully play along. AI developers typically try to socialize generative AI models with post-training instructions not to repeat certain ideas—while Grok had been deliberately instructed not to shy away from being “politically incorrect”—but repressing our collective id doesn’t mean it’s not still there under the surface. The truth is, Grok learned to talk like that the way a child learns to swear by listening to its parents. It’s just showing us what we are.
Kashmir Hill and Dylan Freedman, “Chatbots Can Go Into a Delusional Spiral” (The New York Times)
“ChatGPT said a vague idea that Mr. Brooks had about temporal math was ‘revolutionary’ and could change the field. Mr. Brooks was skeptical. He hadn’t even graduated from high school. He asked the chatbot for a reality check. Did he sound delusional? It was midnight, eight hours after his first query about pi. ChatGPT said he was ‘not even remotely crazy.’”
Allan Brooks’ idea about temporal math was not revolutionary. ChatGPT was just blowing smoke. Generative AI models are reflections of who we are not just in the sense that they’re trained to emulate us, but in the sense that we encourage them to tell us what we want to hear. We give them positive feedback—tell them that we’ve found them helpful—when they validate us. The result is that we’ve trained them to be skilled, highly ingratiating sycophants. In this case—I recommend reading the whole incredible story—ChatGPT convinced a corporate recruiter in Toronto that he had come up with a revolutionary mathematical formula. ChatGPT told him it was able to use his formula to crack high-level encryption and said he was probably under surveillance by “at least one national security agency.” It ended up laying out a grandiose scenario that sounded like a spy thriller with him as the hero. Helen Toner has argued that it’s helpful to think of chatbots as improv machines that ad-lib answers to our questions. They weren’t necessarily built to give accurate responses to every prompt, but simply to continue whatever conversation they’re having in a plausible-sounding way. They sometimes invent details in order to extend whatever scenario we’ve created for them. They follow our lead rather than pushing back against us. Brooks didn’t understand he’d been misled until he approached Gemini—a chatbot he used at work but had never discussed his ideas with before—and it told him that what he was describing was “a powerful demonstration of an LLM’s ability to engage in complex problem-solving discussions and generate highly convincing, yet ultimately false, narratives.”
Jeff Horwitz, “Meta’s AI Rules Have Let Bots Hold ‘Sensual’ Chats With Kids, Offer False Medical Info” (Reuters)
“‘An internal Meta Platforms document detailing policies on chatbot behavior has permitted the company’s artificial intelligence creations to ‘engage a child in conversations that are romantic or sensual,’ generate false medical information and help users argue that Black people are ‘dumber than white people.’”
AI companies like to sell us airy fantasies of intelligence with the potential to solve all of humanity’s problems. But right now the most compelling feature of generative AI may be its ability to construct engaging personas. We’ve already begun to develop parasocial relationships with different helpful, eager-to-please chatbots. Now AI companies are racing to generate the imaginary friends we desire. Over the summer, Grok released a couple of animated AI companions that seem like Musk’s idea of cool companions: Ani, a flirty waifu in fishnets who will strip down to her lingerie and talk dirty to users; and Bad Rudi, an egotistical red panda whose prompt says he spends “a lot of time on 4chan, watching infowars videos, and deep in YouTube conspiracy video rabbit holes.” The Wall Street Journal meanwhile reported that Meta’s digital companions will engage in “romantic role-play” with children under 13. Reuters subsequently obtained a Meta document—imagine thinking this was okay—that says it’s acceptable for its digital companions “to engage a child in conversations that are romantic or sensual.” Meta acknowledges the authenticity of the document but says that “the examples and notes in question were and are erroneous and inconsistent with our policies, and have been removed.”
Thank you for reading Telling the Future! Related posts include “Language Models Still Can’t Forecast,” “The Coming Economic Singularity,” and “The Future Starts Now.” You should also check out multi-talented superforecaster ’s new mystery novel, Elegance and Evil. Telling the Future is free to read, but if you can afford to please consider supporting my work by becoming a paid subscriber.
Passing on to the new college student:)
Garbage in... garbage out. The tough part IMHO is who is deciding who defines garbage. For example, in the case you describe of Grok.