https://deepakbaby.in/tags/self-attention/Recent content in Self-Attention on Deepak BabyHugoenThu, 05 Nov 2026 23:31:18 +0100https://deepakbaby.in/posts/attention-is-all-you-need/Thu, 05 Nov 2026 00:00:00 +0000https://deepakbaby.in/posts/attention-is-all-you-need/<p>In <a href="https://deepakbaby.in/posts/attention-history-bahdanau-to-transformer/">Part 1</a> we saw the key idea of attention: do not compress the whole source sentence into one vector. Keep a vector for each source token, and let the decoder look back at the useful ones. In Bahdanau attention, the decoder state asks a question, the encoder states act like memory, and the answer is a weighted blend of source tokens.</p> <p>Bahdanau attention already had several components that the Transformer would reuse. But the surrounding RNN still processed tokens one by one, which limited parallelism during training. It also passed information through a chain of hidden states, so connecting two distant tokens required many sequential updates.</p>