Attention Mechanism

CORE DEFINITION

Attention mechanism is a technique in artificial neural networks that mimics cognitive attention. This mechanism can enhance the weights of certain parts of the input data while diminishing others, thereby focusing the network's attention on the most important subset of the data. Which parts of the data are more important than others depends on the context. The attention mechanism can be trained using gradient descent. Architectures similar to the attention mechanism were first proposed in the 1990s, under names such as multiplicative module, sigma pi unit, and hypernetwork. The flexibility of the attention mechanism comes from its 'soft weight' property, meaning that these weights can be changed at runtime, unlike typical weights that must remain fixed during runtime. Uses of the attention mechanism include memory functions in Neural Turing Machines, reasoning tasks in Differentiable Neural Computers, language processing in Transformer models, and multimodal data processing (audio, images, video, and text) in Perceiver models.