The Bitter Lesson

It's not just AI. Our trade's history is chockfull of valiant efforts to solve problems that were overrun by the exponential decline of computing costs by the time they really worked properly. Remember DSEE / ClearCase? They had all sorts of complicated virtual file systems to deliver tagged and branched contents of source code repositories. But drive space expanded with a Moore's-law style curve and now we have "git pull". Far simpler. System administrators don't hate our guts for adopting…

>> In computer chess, the methods that defeated the world champion, Kasparov, in 1997, were based on massive, deep search. "Massive, deep search" that started from a book of opening moves and the combined expert knowledge of several chess Grandmasters. And that was an instance of the minimax algorithm with alpha-beta cutoff, i.e. a search algorithm specifically designed for two-player, deterministic games like chess. And with a hand-crafted evaluation function, whose parameters were filled-in by self-play. But…

> Early methods conceived of vision as searching for edges, or generalized cylinders, or in terms of SIFT features. But today all this is discarded. Modern deep-learning neural networks use only the notions of convolution and certain kinds of invariances, and perform much better. This assessment is a bit off. First, convolution and invariance are definitely not the only things you need. Modern DL architectures use lots of very clever gadgets inspired by decades of interdisciplinary research. Second, architecture…

The question comes down to what the research project is. Are we modelling intelligence, or are we simply trying to find a solution to a specific problem? I take intelligence to be "what you do when you do not know what you're doing", or equivalently, a class of algorithms which build conceptualisations of an environment, of one's capacities, (and so on), which "furnishes the mind" with the appropriate building blocks to begin the traditional narrow/weak AI-style algorithmic techniques. Few, if…