On Chomsky and the Two Cultures of Statistical Learning

Chomsky's one paragraph quote at the beginning of this article is more clear and thoughtful than the rest of this. I feel the author's missing the point. In the case of language, observing and reporting statistical probabilities in written/spoken language output does very little to explain the cognitive systems used in acquiring and using language. Even one statistical anomaly serves to show that statistical learning is NOT the entire picture when it comes to language development. There was another article…

I think that Norvig hits the nail on the head near the beginning of his piece: "I believe that Chomsky has no objection to this kind of statistical model [the Newtonian model of gravitational attraction]. Rather, he seems to reserve his criticism for statistical models like Shannon's that have quadrillions of parameters, not just one or two." This is no more than an objection to problems of fitting your chosen model to data. If you only have a small number…

> Chomsky has focused on the generative side of language The answers to "why" that Chomsky pushes so hard for are very valuable to adult language learners. There are basic syntactic rules to generating broadly correct language. Having these rules discovered and explained in the simplest possible form is irreplaceable by statistical models. Neural networks, much like native speakers can say "well this just sounds right," but adult learners need a mathematical theory of how and why they can generate…

I used to think the same way. But after spending the last few years getting frustrated trying to create more complex linguistic systems, e.g., dialog systems or scientific articles understanding, I am coming to the conclusion that the current statistical approach is a dead end. It’s actually impeding the field because it’s working so well for certain tasks that when people are trying to build systems with real understanding, they can’t match the performance obtained by gigantic language models. But…