Transformers from Scratch

This article describes positional encodings based on several sine waves with different frequencies, but I've also seen positional "embeddings" used, where the position (the position is an integer value) is used to select an differentiable embedding from an embedding table. Thus, the model learns its own positional encoding. Does anyone know how these compare? I've also wondered why we add the positional encoding to the value, rather than concatenating them? Also, the terms encoding, embedding, projection, and…

This is a great comprehensive deep dive, thank you for sharing! Interestingly, there was a blog post and video in 2019 by Peter Bloem [1] with the same title which I consider to be one of the very best quick intros to transformers. Rather than diving into all of the details right away, Peter focuses on the intuition and “peels the onion” gradually. [1] http://peterbloem.nl/blog/transformers Other notable good intros to transformers are: The Illustrated Transformer by Jay Alammar…