Backpropagation: How a Network Learns Who to Blame
One wrong answer. A billion numbers share the blame. The trick that decides how much each one owes is why every model you use can learn. How We Got Here — Episode 3 of ~26.
I want to start this week with a confession about what surprised me while I was writing it.
I knew the shape of the backpropagation story before I sat down. Werbos in 1974, ignored. Rumelhart, Hinton, and Williams in 1986, celebrated. The algorithm that ended the winter. I had the beats.
What I did not expect was how much the whole thing would turn out to be about a single, very human question — one that has almost nothing to do with calculus and almost everything to do with how we live.
The question is: when something goes wrong at the end of a long chain of decisions, who gets the blame?
That is the question this episode is built on. The technical name for it is the credit assignment problem. And the more time I spent with it, the less it felt like a math problem and the more it felt like a description of, well, most of my life.
What the machine and I have in common
Here is the problem the way the field had it, stripped down.
You have a network. It is a stack of layers, and inside it are millions of little numbers — weights — that all worked together to produce one answer. The answer comes out wrong. You can measure exactly how wrong at the very end. But you cannot see inside. The layers in the middle never get told what they should have done. All you have is the one error at the finish line, and a mountain of numbers that each contributed something to it.
So how do you know which numbers to fix?
I read that and felt a small, uncomfortable click of recognition. Because that is the shape of almost every hard thing I have ever had to figure out. A project ships late, and I have to work backward through a hundred decisions to find which ones actually mattered. A system fails in production, and the error I can see at the end is three layers downstream of the thing that actually broke. A relationship goes quiet, and there is no single log line that tells you where it started to go wrong.
We are, all of us, constantly trying to assign credit backward through chains we cannot fully see. The network’s version is just cleaner, because at least the network can do the math.
The part that moved me
The Medium version I posted this morning walks through how the algorithm actually solves it — the chain rule, the blame flowing backward layer by layer, the whole mechanism. If you want the machinery, that is where it lives. This letter is for the part the machinery cannot hold.
What moved me was the elegance of the answer, and I want to try to say why in plain words.
The hard part was never the last layer, the one touching the outcome. The hard part was the buried middle — the parts of the system that never see the result and never get told what they were supposed to do. And backpropagation’s answer to that is stranger than it sounds. You never have to tell the middle the right answer. You only need one honest signal of how wrong the whole thing was at the end, and a way for each part to know how strongly it is connected to the parts in front of it. That is enough. From just those two things, every buried piece can work out its own share of the blame, on its own, by listening only to its neighbor.
No central authority handing down verdicts. Just local conversations, each one honest, adding up to something globally true.
I have been turning that over for days. Because the way I usually try to fix things is the opposite. I want to stand at the top and assign blame from above, with full visibility, all at once. And the lesson sitting inside this 1986 algorithm is that you do not need the view from the top. You need an honest error signal and a willingness to pass it, gently and accurately, to the thing right next to you.
I am not going to pretend that is a tidy life lesson. But it has been sitting with me.
The two men who paid for it
There are two people in this story I cannot stop thinking about, and neither of them is one of the famous winners.
The first is Paul Werbos. He worked out backpropagation in a Harvard dissertation in 1974 — the actual answer, five years after Minsky’s book had told everyone this whole direction was a dead end. And almost no one read it. Not because it was wrong. Because the field had already been talked out of looking for a solution. By his own account, Werbos went to Minsky directly, and offered — generously, almost tenderly — to let Minsky co-author the correction so he would not be embarrassed when the answer came out. Minsky said no. He had tried to make these systems work, failed, and concluded that if he could not do it, it could not be done.
I keep imagining that conversation. A young researcher holding the answer, offering to share the credit with the man whose book made the answer invisible. And being waved off.
The second is David Rumelhart. He was not even a computer scientist — he was a cognitive psychologist, someone who spent his life trying to understand how human minds reason and remember. He was the one who, more than anyone, drove the 1986 paper that finally made the field listen. And then, in the 1990s, he developed Pick’s disease, a degenerative illness that slowly took the exact faculties he had devoted his career to studying. He died in 2011, before the world he helped build really arrived.
Rosenblatt, from Episode 1, never lived to see his vindication. Rumelhart never lived to see his. I said it on Medium and I will say it here too, because it is the truest thing I found this week: this field is built on a lot of people who never got to read the ending.
What I keep coming back to
The lesson I am carrying out of this episode is uncomfortably close to last week’s, and I think that is the point.
Last week it was Minsky proving a real limit and then over-extrapolating it into a verdict that cost a decade. This week it is the same man, looking straight at the credit assignment problem, naming it himself in 1961, and concluding it could not be solved — while the solution was sitting in a dissertation he declined to read.
What he got wrong was not the math. It was the leap from “I can’t see how to do this” to “this can’t be done.”
I do that. Maybe you do too. When I cannot see the path through something — a bug, a design, a hard conversation — the tempting move is to decide the path is not there. It feels like rigor. It feels like maturity, even, to declare a thing impossible and move on. But there is a quieter, harder posture available, and it is the one Werbos took: I haven’t found it yet.
The whole history of this field seems to bend on which of those two sentences people choose to believe.
I want to be the kind of person, and the kind of engineer, who defaults to the second one. Not out of optimism. Out of honesty about how little I can usually see from where I am standing.
What’s next
Next Tuesday is convolutional networks. Backpropagation gave us a way to train deep networks, but it did not tell us how to shape one for a particular kind of problem. Episode 4 is about a young researcher named Yann LeCun wiring backprop into a network built to actually look at images — and teaching a machine to read handwritten ZIP codes for the postal service in 1989. The first time any of this did a real job in the real world.
If Episode 3 was the algorithm that ended the winter, Episode 4 is the first architecture that put it to work.
I’ll see you Tuesday.
If you’re new here, the longer Medium version of this piece is up — it walks through the actual mechanism of backpropagation, the chain rule made plain, the control-theory roots that predate neural networks entirely, and the 1986 paper that solved XOR with the very architecture Minsky used to bury it. The Substack is the reflection; the Medium is the machinery. And every Tuesday from here, we take one more step in the timeline together.
Cordero is a senior research software engineer at the Scientific Software Engineering Center at the University of Washington’s eScience Institute. If this landed, forward it to someone who has recently decided a problem in front of them is impossible and might just not have found it yet.