MENTAL MODEL · M7561

Information Gain

DecideHigh supportDecision Science

Included

Version 1.0.0 · Updated 2026-07-30

CORE DEFINITION

Entropy reduction: the reduction in classification uncertainty due to a feature. Formula: IG = H(parent) - weighted average H(children). Feature selection: choose the feature with the largest information gain to split.

SCAFFOLDING EFFECT

psychology

Reduce cognitive load

Quantify "how much value this information has". Not all information is equally useful; information gain helps identify the most discriminative features.

anchor

Anchor fast decisions

Information gain = entropy of parent node minus the weighted average entropy of child nodes after splitting by a feature, quantifying how much uncertainty the feature reduces. Based on Shannon entropy: the purer the partition (the purer the child nodes), the larger the gain. Decision trees (ID3/C4.5) use this to select the optimal feature for each split.

MINIMUM ACTION

In progress 0/6

Practice this model in one real situation:

Calculate the entropy of the current node H(parent)
For each candidate feature, partition the subset by its values and compute the weighted average entropy
Split on the feature with the largest gain
Recurse until pure or reaching depth/sample thresholds
Note that gain is biased towards features with many values
use gain ratio (C4.5) to correct.

Check to track your progress (stored locally)

Learning progress0%

account_treeGenealogyexpand_more

menu_bookReferencesexpand_more

Source support: Explicit

link
zh.wikipedia.orghttps://zh.wikipedia.org/wiki/%E7%9B%B8%E5%AF%B9%E7%86%B5ZH · Explicit
verified

RELATED MODELS