MENTAL MODEL · M6292

Rectified Linear Unit

BusinessHigh supportManagement

Included

Version 1.0.0 · Updated 2026-07-30

CORE DEFINITION

The most commonly used activation function. $f(x) = max(0, x)$. When input is less than 0, output is 0 (suppression); when greater than 0, output is linear. Simple, efficient, sparse.

SCAFFOLDING EFFECT

psychology

Reduce cognitive load

One-sided filtering. For negative energy and meaningless noise, directly zero out (ignore) and do not engage; for positive, valuable signals, pass through as is and fully support. This is the most efficient energy management strategy.

anchor

Anchor fast decisions

The activation function determines the neuron output. ReLU = max(0, x): if input ≤ 0, output 0 (suppresses the neuron, creating sparsity); if input > 0, pass through as is (linear, preserving the signal). Computation is extremely simple, alleviates vanishing gradients, and enables training of deep networks. Its 'negative to zero, positive to pass' is metaphorically used for energy management.

MINIMUM ACTION

In progress 0/5

Practice this model in one real situation:

Apply ReLU after each linear transformation in the network.
Negative inputs are directly set to 0 (discard noise / negative signals).
Positive inputs are passed linearly, preserving useful features.
During training, use its sparsity to accelerate convergence (note the dead neuron problem
can switch to Leaky ReLU).

Check to track your progress (stored locally)

Learning progress0%

account_treeGenealogyexpand_more

menu_bookReferencesexpand_more

Source support: Explicit

link
zh.wikipedia.orghttps://zh.wikipedia.org/wiki/%E7%BA%BF%E6%80%A7%E6%95%B4%E6%B5%81%E5%87%BD%E6%95%B0ZH · Explicit
verified

RELATED MODELS