Cognitive Scaffold

Preparing your thinking workspace

arrow_back_ios_new
MENTAL MODEL · M6093

Markov Decision Process, MDP

Markov Decision Process, MDP
DecideHigh supportDecision Science
Included
account_tree

Version 1.0.0 · Updated 2026-07-28

CORE DEFINITION

In mathematics, a Markov decision process (MDP) is a discrete-time stochastic control process. It provides a mathematical framework for modeling decision-making in situations where outcomes are partly random and partly under the control of a decision maker. MDPs are useful for studying optimization problems solved via dynamic programming. MDPs were known at least as early as the 1950s; a core research on MDPs is Ronald Howard's 1960 book "Dynamic Programming and Markov Processes." They are used in many fields, including robotics, automation, economics, and manufacturing. The name comes from Russian mathematician Andrey Markov, as they are a generalization of Markov chains. At each time step, the stochastic process is in some state s. The decision maker may choose any action a that is available in state s.

SCAFFOLDING EFFECT

psychology

Reduce cognitive load

In mathematics, a Markov decision process (MDP) is a discrete-time stochastic control process. It provides a mathematical framework for modeling decision-making in situations where outcomes are partly random and partly under the control of a decision maker. MDPs are useful for studying optimization problems solved via dynamic programming. MDPs were known at least as early as the 1950s; a core research on MDPs is Ronald Howard's 1960 book "Dynamic Programming and Markov Processes."

anchor

Anchor fast decisions

MDP characterizes sequential decisions with states, actions, transition probabilities, and immediate rewards: an agent in state s takes action a, transitions to state s' with a probability, and receives a reward, with the goal of maximizing cumulative discounted reward. It formalizes planning under uncertainty.

MINIMUM ACTION

In progress 0/4

Practice this model in one real situation:

Check to track your progress (stored locally)
Learning progress0%
account_treeGenealogyexpand_more
menu_bookReferencesexpand_more

Source support: Explicit

  • link
    zh.wikipedia.orghttps://zh.wikipedia.org/wiki/%E9%A6%AC%E5%8F%AF%E5%A4%AB%E6%B1%BA%E7%AD%96%E9%81%8E%E7%A8%8BZH · Explicit
    verified

RELATED MODELS

Related model

Segal's Law

An adage using two disagreeing watches to illustrate uncertainty from conflicting sources.

arrow_forwardRelated model

The Idea Maze

A good idea is not a momentary inspiration, but a maze map containing multiple paths, dead ends, turning points, and treasures.

arrow_forwardRelated model

Double Diamond Model

Proposed by the British Design Council.

arrow_forward