The Unreasonable Effectiveness of Recurrent Neural Networks

Karpathy is one of my favourite authors - not only is he deeply involved in technical work (audit the CS231n course for more[1]!), he spends much of his time demystifying the field itself, which is a brilliant way to encourage others to explore it :) If you enjoyed his blog posts, I highly recommend watching his talk on "Automated Image Captioning with ConvNets and Recurrent Nets"[2]. In it he raises many interesting points that he hasn't had a chance to…

If this were written today, Karpathy would have to call it "The Unreasonable Effectiveness of Convolutions". Since 2015, convolutions, causal or dilated convolutions, and especially convolutions with attention like the Transformer, have made remarkable inroads onto RNN territory and are now SOTA for most (all?) sequence-related tasks. Apparently RNNs just don't make very good use of that recurrency & hidden memory, and the non-locality & easy optimization of convolutions allow for much better performance through…

Good article, but ugh, please leave Shakespeare alone if (as you freely admit) you don't know anything about Shakespeare. Just because the real thing sounds strange to you doesn't mean that your "samples" are anything but nonsense. edit To (preemptively) stress that I am not trolling: I find the topic and the writeup interesting and useful, and I appreciate the work that people do to share their experiences. But this is not a good example of the RNN working; it's…

2015