Next Token Prediction Could Result in a Super Hacker

“The limits of my language mean the limits of my world.” – Ludwig Wittgenstein, Tractatus logigo-philosphicus, 1922

I used to be skeptical that next-token prediction could ever lead to AGI. Now I’m convinced it’s enough. Whether if frontier model companies will actually crack it is a separate question. They could hit a data wall, a computation wall or some natural physical limit we don’t fully understand yet.

But what matters is allowing ourselves to entertain the possibility that next-token prediction is enough build super hackers or a general intelligent entity.

I think what makes us superior to other animals isn’t tools or fire or walking upright or opposable thumbs. It’s language. Not in the narrow sense of English or Telugu, but in the broad sense any symbolic system. Sign language, braille, even just thinking in pictures. Language is what lets us reason, and creativity is just what emerges when you push reasoning far enough along multiple axis.

It’s what allows us to deduce, imagine, discover, dream, and invent. You could strip away speech, sight, or touch, but as long as there’s some kind of representation happening in the brain, reasoning can still follow.

Try this thought experiment: could a human be intelligent without any access to language—not just speech, but symbols, gestures, and spatial representation? The harder I try to imagine it, the more impossible it seems to be intelligent. Without some way to represent the world, how could you ever reason about it?

You could argue that animals like dolphins, crows, or portia show signs of intelligence. But we don’t know if they have internal representations that allow them to generalize, abstract, or perform deductions. It’s possible they’re just running hardcoded reasoning traces shaped by evolution. Not thinking, just reacting very, very well.

Now, some humans are better at this than others. People like Ramanujan or Einstein seemed to operate with faster, more powerful reasoning capabilities. Ramanujan claimed his mathematics came to him in dreams. Einstein ran entire thought experiments in his head. What they were doing, really, was manipulating abstract representations of reality called mathematics(A language far more precise and expressive than plain English) and discovering things through reasoning or logic.

Still, even their mind has limits. It is apparent that what these super humans did/do is not the ceiling, it is the limits of evolution. The human brain has constraints, we have shorter working memory, slow processing of abstract representation, and slower reasoning, maybe around ~100 tokens/minute. Further, we get tired, we lose focus, and maybe we die just before making a breakthrough. Our brains are evolved for not to get killed by a predator in savanna, not to launch rockets to mars.

And “Maybe” that’s where i think language models might surpass the limited-reasoning-savanna-survival-brains.

What’s becoming increasingly obvious after spending time with these models is this: while they’re trained to predict the next token over an attention window, what emerges is something far more powerful, reasoning.

This is especially clear with Gemini 2.5. The way it can reason through code is mind-blowing. It doesn’t feel like a parrot anymore. It’s not just pattern matching. And even if it is, what do you think we do? We match patterns against our world model, our environment, and our own internal chain of thought. Stack enough of that together, and we call it intelligence.

And obviously, they are not quite at human level, Gemini 2.5 can’t reason through code like me and find vulnerabilities.

If a model can truly attend to a 100k-tokens and reason through it with accurate logical deductions and inductions, whether through chain-of-thought or some RL, and actually come to the right answer, then what we’re looking at is a superintelligent machine.

This became obvious to me while testing it on hacking tasks. If i give a small codebase with 5000 lines of code, it can tell me what it does faster than I can.

Now, I’m good at this. My actual skill isn’t just in spotting vulnerabilities just like that, my actuall skill is in understanding the code, understanding the multiple layers of the system it runs in. Then, once I build a mental model of what the program/system is supposed to do, I start hacking it, by asking (logical statements) questions:

Can this state ever transition into something no one planned for? Can this state transition to weird machine state ? Can this sanitizer be bypassed?

And what I’m really doing is looking for a proof of that. If the proof exists, the system is vulnerable.

Programs are proofs. Proofs are programs.

When I look at code, I’m proving that weird machine states exist. States that no one intended. States that let me make a system do something it was never meant to do.

Why I’m a good hacker isn’t because I’m some genius. It’s because my reasoning is slightly better than a chimpanzee’s, built on top of a symbolic system we call language. I am good with logic, just like other humans. And I just happen to use that system to reason about other languages—programming languages for hours and hours.

Sure, the current models aren’t there yet. They don’t have the heuristics I use to skim through irrelevant code. They don’t have episodic memory or long-term abstraction recall. But once they crack the latent dimension of reasoning—once they can hold a million tokens in mind, trace logic across layers, and prove those weird states better than I can?

Then we’re not dealing with assistants anymore. We’re dealing with something superintelligent, a super hacker

If they become this reasoning machines, the singularity is probably near.