MENTAL MODEL · M0025

Grokking

SystemsHigh supportComplexity Science

Included

Version 1.0.0 · Updated 2026-07-28

CORE DEFINITION

In some experiments on small algorithmic datasets, neural networks first fit the training set while generalization remains near chance. Generalization improves substantially only after much more optimization. This delayed improvement is called grokking; it does not mean training accuracy stays low or that every learning process must pass through this stage.

SCAFFOLDING EFFECT

psychology

Reduce cognitive load

Distinguish memorization of training examples from generalization to unseen examples. Track their different time scales instead of treating a training fit as mastery or every plateau as an imminent breakthrough.

anchor

Anchor fast decisions

Fitting training data and generalizing to unseen data can occur on different time scales. Power and colleagues observed generalization long after overfitting in some networks. This observation alone establishes neither a single causal mechanism for all models nor a guarantee that plateaus end.

MINIMUM ACTION

In progress 0/2

Practice this model in one real situation:

Choose a reproducible small algorithmic task, fix the training and validation split, and log accuracy and loss on both. Set a compute budget in advance and check for a phase with a good training fit but poor validation performance. Continue within that budget and record any validation improvement
evaluate finally on a test set excluded from tuning.

Check to track your progress (stored locally)

Learning progress0%

account_treeGenealogyexpand_more

menu_bookReferencesexpand_more

Source support: Explicit

link
en.wikipedia.orghttps://en.wikipedia.org/wiki/GrokEN · Explicit
verified
link
arxiv.orghttps://arxiv.org/abs/2201.02177EN
verified

RELATED MODELS