Gradient Descent Learned to Code

On Aug. 4, 2017, Andrej Karpathy posted a sentence that read like a joke: “Gradient descent can write code better than you. I’m sorry.” Four months later, his essay Software 2.0 made the serious version of the argument. Neural network weights, he wrote, are a new kind of source code. Instead of writing every rule by hand, programmers curate data and let an optimizer search for a program that works.

In 2017, that was mostly a philosophical claim. Gradient descent fit weights for tasks like image recognition and translation; the output was a set of parameters. The provocative part was the implication: if the weights are the program, then the thing writing the program is no longer a human.

Nine years later, the joke has a benchmark result. Models trained by gradient descent now read a repository, edit multiple files, run tests, inspect failures, and submit patches. On SWE-bench Verified, a set of 500 human-filtered GitHub issues, top published systems resolve around 80% of the tasks. In March 2024, the SWE-agent system scored 12.47%. The direction is clear even if the exact number moves every month.

What gradient descent actually learned

A neural network is a large parametric function. Training adjusts its weights to reduce a loss. In 2017, that process produced specialized models: a detector for pedestrians, a translator, a classifier. The “code” was a matrix of numbers, legible to the machine.

The transformer changed the scale of what could be learned. A language model trained on text and code learns a distribution over sequences. Predict the next token well enough, across enough code, and the model begins to imitate programming patterns: function signatures, error handling, library calls, test structure. Instruction tuning then teaches it to follow a request. Reinforcement learning from human feedback teaches it to prefer useful answers.

The last step is the one that made coding take off. Code can be executed. A compiler can reject it. A test suite can fail it. That gives a model a reward signal that is cheaper and more objective than a human preference. Karpathy made this point in 2025: the most predictive feature of a task is verifiability. If a task can be reset, attempted many times, and rewarded automatically, gradient descent can optimize it.

Coding is almost purpose-built for that loop. The model writes a patch. The harness runs the tests. The failure message becomes the next prompt. The model reads the stack trace, edits again, and runs the suite. This is why agentic coding improved faster than many other knowledge-work tasks: the environment gives feedback, and the feedback is grounded in execution.

The short history of the last few years

The product era began with Codex and GitHub Copilot in 2021. The first widely used benchmark, HumanEval, asked models to complete small Python functions from docstrings. The tasks were useful for measuring syntax and basic reasoning, and they soon saturated. Once models could pass most function-level tests, the benchmark stopped separating them.

SWE-bench, introduced in 2023, raised the unit of work from a function to a repository. The model gets a real GitHub issue and must produce a patch that passes hidden tests. The first systems were weak. SWE-agent’s 12.47% in March 2024 was a breakthrough because it showed that an agent with a shell, a file editor, and a test loop could do real repository work. The scaffold mattered as much as the model. The model proposed; the harness executed; the tests judged.

Then came reasoning models, longer context, better tool use, and cheaper inference. By late 2025, the same benchmark was reporting scores around 80%, with different labs and different agent scaffolds. Terminal-Bench pushed the evaluation further, from repository patches to multi-step command-line work: install packages, configure services, debug builds, and recover from errors. The unit of evaluation moved again, from “write a function” to “finish a job.”

Karpathy’s own language moved too. In February 2025 he coined “vibe coding” for the practice of describing software in natural language and letting the model produce it. In his Software 3.0 talk, he described prompts as a new kind of program. The claim shifted: the interface to the computer had moved up a level, from syntax to intent.

What the benchmarks prove, and what they do not

SWE-bench Verified is a strong benchmark because it uses real issues, real repositories, and hidden tests. It is also narrow. The tasks are Python-heavy, curated to be solvable, and scored by tests. A patch that passes the tests can still be insecure, overfitted to the test, or unmaintainable. The benchmark measures task resolution, not engineering judgment.

The scores also compare systems, not just models. A top result may use a lab’s own agent scaffold, retries, test-time compute, and tooling. Two models with the same name can produce different results under different harnesses. That does not make the progress fake. It makes the claim more precise: the combination of a model, a harness, and a verifier can now solve a large share of well-specified repository tasks.

This is the part Karpathy’s 2017 tweet did not capture. Gradient descent did not learn to write code in the way a developer writes code. It learned a distribution over code, and then a harness turned that distribution into edits, tests, and retries. The model is the generator. The verifier is the editor. The developer is increasingly the person who defines the verifier and owns the result.

The job that remains

A model can beat a developer on a timed, well-specified, test-covered task. It does not beat a team on an ambiguous, high-stakes system. The remaining work is the work that is hard to score automatically: deciding what to build, choosing boundaries, designing interfaces, reasoning about failure modes, reviewing security, and taking responsibility for a production change.

These are the expensive part of software engineering, and they are hard to score automatically. A generated patch that passes a test can still violate a privacy boundary, add a dependency with a bad license, or hide a race condition that the test suite never exercises. The better the model gets at the first draft, the more the bottleneck moves to review, verification, and maintenance.

That is why the developer role is changing rather than disappearing. The person who can specify a problem, write a good test, read a diff, and decide whether the result is safe becomes more valuable. The person whose only contribution is typing the same boilerplate is more exposed. Karpathy’s “vibe coding” is real, but it works best when verification is cheap and failure is reversible. It is a poor fit for a payments ledger or a kernel patch.

What comes next

The next benchmark fight is already visible. Repository patches are becoming saturated. The field is moving to long-horizon tasks: agents that run for hours, use tools, recover from mistakes, and coordinate with other agents. The evaluation is moving toward end-to-end outcomes, cost, and reliability. Security review is moving in the same direction, with skills that force an agent to prove a boundary violation rather than list suspicious patterns.

The hard problem is trust. A model that writes code at near-zero marginal cost creates a verification bill that grows with every line. The organizations that win will be the ones that treat generated code as an untrusted contribution: tests, review, provenance, and a human who can say no.

Karpathy’s 2017 sentence was half joke and half forecast. The forecast was right about the direction. Gradient descent became a competent programmer in the part of programming that can be checked by a machine. The joke is that the rest of programming turned out to be the part that matters.

What gradient descent actually learned

The short history of the last few years

What the benchmarks prove, and what they do not

The job that remains

What comes next

Sources and References