Support Vector Machines: The Decade the Neural Networks Lost
For ten years, the cleanest math in machine learning beat neural networks fair and square, and the people who wrote them off were right. How We Got Here — Episode 5 of ~26.
I opened a notebook the other night to prototype a quick classifier. Nothing fancy, just a yes-or-no sorter for a small pile of labeled data I had sitting around. And I did the thing I always do without thinking. I imported scikit-learn, reached for an SVM, called .fit(), and had a working model before my coffee got cold.
It’s such an automatic move that it barely counts as a decision. Small problem, limited data, need an answer tonight. Just use an SVM. Most of us learned it as a reflex and never stopped to ask where the reflex came from.
I stopped this time, because I’d spent the week inside this episode. And what I found underneath that reflex surprised me more than anything in the series so far.
For about ten years, that little instinct wasn’t the convenient choice. It was the frontier. The most advanced thing you could do. The support vector machine was what the smartest people in the field were most excited about. The neural network, the ancestor of everything I now use every day, was the embarrassing cousin nobody wanted to be seen with.
This is the episode where the wheel turns the wrong way. And the part that’s been sitting with me is that, for a decade, it turned the wrong way for good reasons.
The idea that was too beautiful to bet against
I want to give you just enough of the machinery to feel why the field fell in love, because the whole emotional weight of this story rests on the SVM being real, provable, and better. (The Medium version has the full walkthrough. This is the short tour.)
Think back to Episode 1, to the perceptron drawing a line between two groups of dots. The perceptron found a line. Any line that separated the training data would do. The support vector machine asked a better question: of all the lines that could separate these two groups, which one is best? Its answer, once you see it, feels obviously right. The best boundary is the one that leaves the widest empty street between the two groups, the biggest buffer, so that a new, slightly unusual example still has room to land on the correct side. Not just a line. The best possible line.
Then it did something that still strikes me as close to magic. Real data usually can’t be split by a straight line at all. The two groups are tangled together. So the SVM lifts the data into a higher-dimensional space where the tangle pulls apart and a clean boundary becomes possible. The catch is that this space can be enormous, sometimes infinite, far too large to actually compute. The trick (the famous kernel trick) is that you never have to. There’s a shortcut that tells you how similar any two points would be in that giant space while you do all your arithmetic down in the small, cheap, original one. You get the power of the huge space without ever paying to build it.
I sat with that for a while. It’s such an elegant idea. Mathematical guarantees, decades of theory, almost no fiddling. Compared to neural nets, it looked like the adult in the room.
If I’d been there in 1996, I think I’d have walked away from neural networks too.
The wall I’d been promised
There was one more reason the neural nets lost, and it’s the one I’d been half-dreading since Episode 3, because I promised you it was coming.
Backpropagation, remember, teaches a network by sending its error signal backward through the layers, so each one learns its share of the blame. Here’s what nobody tells you until you feel it: every time that signal crosses a layer on its way back, it gets multiplied by a small number. Multiply by a fraction, then another, then another, and after enough layers the signal doesn’t just shrink. It vanishes. By the time it reaches the earliest layers (the ones nearest the input), there’s almost nothing left. They barely learn at all.
That’s the vanishing gradient problem, and it had a brutal consequence. You could train a shallow network (one or two layers) and it would work fine. But stack the layers deep (deep enough to build up the rich, layered understanding that was supposed to make neural networks special) and the whole thing stalled. The signal couldn’t reach the bottom.
Which means the one thing that could have set neural networks apart, depth, was exactly the thing that wouldn’t train. And a shallow neural net had no real edge over an SVM. It was just messier and slower for the same result.
So the field made a completely rational choice. It picked the beautiful thing that worked over the ugly thing that mostly didn’t. I keep wanting to be indignant on the neural network’s behalf, the way I was in the last few episodes. But I can’t be. This time, the people who moved on were right.
Two ideas, one hallway
Here’s the detail that turned this from a technical story into a human one for me.
The man whose name sits on the modern SVM is Vladimir Vapnik — a Soviet émigré, a mathematician who built the theory in Moscow and then, in 1990, came to AT&T Bell Labs. He wrote the famous 1995 paper there, with a colleague named Corinna Cortes.
Bell Labs. That’s where Yann LeCun was, the same years, building the convolutional networks we talked about last week — the ones reading ZIP codes off the mail.
They were in the same building. The idea that would win the decade and the idea that would win the future shared a hallway, and the two men argued about it directly. There’s a well-documented moment from 1995: a bet, made right there, with LeCun as the witness. Vapnik wagered that by 2005 nobody would be using neural networks anymore. He was that sure the brain-inspired stuff was a dead end.
He lost that bet. But I can’t bring myself to laugh at him for it, because for ten years he looked completely right. He wasn’t being arrogant. He was reading the evidence in front of him, and the evidence said his idea was better. It was better. That’s what makes this one hard.
What I keep turning over
The other episodes had a clean shape. An idea was right, and short-sighted people failed to see it, and eventually the truth won out. I knew how to feel about those. Root for the underdog. Trust the buried correctness of the ignored idea.
This one doesn’t give me that. Here the people who walked away from neural networks weren’t fools. They were the smartest people in the field, making the correct call on the best available evidence, and they were right for a decade. The SVM was better (measurably, repeatedly better) and the neural network still couldn’t get past the vanishing-gradient wall.
And that unsettles me more than any of the earlier stories, because it breaks the comfortable version of how progress works. It’s not always the visionaries versus the blind. Sometimes the whole field climbs to the top of a very good idea, looks around, and slowly realizes it’s standing on a hill, not a mountain. And the only way forward is to climb back down — back toward the unfashionable thing everyone already dismissed — and start up a harder slope.
Almost nobody wants to do that. Why would they? They’re standing on the winning idea. It takes a strange kind of stubbornness to walk downhill on purpose.
I’ve been thinking about that with my own work. How often the “obviously right” tool, the one with the cleanest story, the best guarantees, the least friction, turns out to be a local maximum. Good, real, defensible, and still not the highest ground there is. And how the move that actually matters might be the uncomfortable one: setting down the thing that’s working to go wrestle with the thing that’s stuck.
I don’t have a clean rule for telling those apart. I wish I did. But I’ve started noticing when a choice feels too settled, too obviously correct, and treating that feeling as a small flag rather than a comfort.
What’s next
Next Tuesday, the crack of light. In 2006, Geoffrey Hinton — who never stopped believing in the messy stuff, all the way through the winter — found a way to sneak a learning signal past that wall and train a deep network one layer at a time. It wasn’t the final answer. But it was enough to make the field look again. And because “neural networks” had become almost a slur by then, they gave it a new name.
They called it deep learning. That’s where we go next.
I’ll see you Tuesday.
If you’re new here, the deeper Medium version of this piece is up — it walks through the actual machinery: what maximum margin means, how the kernel trick pulls off its sleight of hand, why the vanishing gradient made “deep” impossible, and the full story of the bet in the Bell Labs hallway. The Substack is the reflection; the Medium is the mechanics. And every Tuesday, we take one more step down the timeline together.
Cordero is a senior research software engineer at the Scientific Software Engineering Center at the University of Washington’s eScience Institute. If this landed, forward it to the person on your team who’s convinced the “old” methods are beneath them — the SVM they’d never admit to using was, once, the smartest thing in the room.