Karpathy on Software 3.0 and the Attention Breakthrough

Andrej Karpathy has a way of making a decade of AI feel like one straight line. If you build with LLMs, generative AI, agents, or transformers, his breakdown of how programming got here is the framing worth keeping. It is short, concrete, and it explains why the work looks different than it did five years ago.

There is nothing to install here. What you get is the mental model, plus one line you will probably want to remember.

What Happened

Karpathy traces the path in five stops: traditional software, neural networks, transformers, GPTs, and what he calls Software 3.0.

The turning point was "Attention Is All You Need", the 2017 paper that changed the trajectory of AI. The idea sounds almost too plain. Instead of building ever more complex architectures on top of RNNs, you remove the RNN and keep the attention. That opened the door to the transformer era we are living through now.

He then reframes what programming itself is:

• Software 1.0 — design the algorithm.
• Software 2.0 — design the dataset.
• Software 3.0 — design the prompt.

And the line worth bookmarking: "The hottest new programming language is English."

Why It Matters

A prompt can increasingly be treated as a program, with the LLM acting as the system that interprets and runs the instructions. That matters because transformers work across text, images, and speech, and GPTs start to behave less like one tool and more like a general-purpose computer.

If natural language is the new interface, then knowing what you want and saying it precisely is the skill. Frames like this one are how you keep up with where all of it is heading.