MENTAL MODEL · M0090

Self-Attention Mechanism

TechnicalHigh supportArtificial Intelligence

Included

Version 1.0.0 · Updated 2026-07-30

CORE DEFINITION

Self-attention relates different positions of one sequence to construct contextual representations. In Transformer scaled dot-product attention, Q, K, and V are projections of the same sequence representation. Each query assigns weights to accessible keys and aggregates the corresponding values. Causal self-attention masks future positions and cannot access the entire future context.

SCAFFOLDING EFFECT

psychology

Reduce cognitive load

Information weighting metaphor. When reading or analyzing complex situations, do not exert equal effort. Dynamically compute the relevance between current information and the 'global context', focusing attention on key clues.

anchor

Anchor fast decisions

Scaled dot-product self-attention computes softmax(QKᵀ/√dk + mask)V: dot products measure compatibility, scaling controls magnitude, masks constrain visibility, and softmax supplies row-wise weights. Multiple heads use different projections and combine their outputs. This enables information aggregation without guaranteeing semantic understanding or correct pronoun resolution.

MINIMUM ACTION

In progress 0/1

Practice this model in one real situation:

Use a three-row, two-dimensional input X=[[1,0],[0,1],[1,1]] and initially set all three projection matrices to identity, giving Q=K=V=X. Compute QKᵀ/√2. Set future-position scores to negative infinity before each row’s softmax, then multiply by V. Compare masked and unmasked results and check that the first row can use only itself.

Check to track your progress (stored locally)

Learning progress0%

account_treeGenealogyexpand_more

menu_bookReferencesexpand_more

Source support: Explicit

link
en.wikipedia.orghttps://en.wikipedia.org/wiki/Attention_Is_All_You_NeedEN · Explicit
verified
link
arxiv.orghttps://arxiv.org/html/1706.03762v7EN
verified

RELATED MODELS