Multi-head self-attention, one head at a time

Four possible attention patterns over the same sentence, followed by the learned combination. These are illustrative archetypes—not roles assigned to specific heads.
query key score probability value

Nearby context

Guided multi-head self-attention visualization Attention arcs over the sentence The animal did not cross the street because it was tired.