MENTAL MODEL · M7645

Meta-Evaluation

TechnicalmediumArtificial Intelligence

Included

Updated 2026-08-11

Loading revision record…

INTRODUCTION

English translation pending.

CORE DEFINITION

A practice from program evaluation and measurement theory, closely connected to Goodhart's law. The core proposition is that evaluation is itself a fallible product subject to bias, method error, and gaming, so it requires its own quality control. The key qualification is that the check must cover use as well as design, since a sound measurement can still cause harm when applied to decisions it was never meant to support.

SCAFFOLDING EFFECT

psychology

Reduce cognitive load

- Criterion check: confirm the metrics actually measure the intended construct. - Gaming scan: ask how people would behave if they optimized the metric directly. - Use review: check whether the results are being applied to decisions they can support.

anchor

Anchor fast decisions

Metrics are proxies for the construct of interest, and the gap between proxy and construct creates room for distortion. Once a measure becomes a target, effort shifts toward the measure rather than the underlying goal, so the indicator improves while the outcome does not. Meta-evaluation closes the loop by auditing the proxy, the incentives it creates, and the decisions it feeds.

MINIMUM ACTION

In progress 0/1

Practice this model in one real situation:

Restate the purpose and criteria of the original evaluation. 2. Check whether the indicators measure the intended construct and whether the data is reliable. 3. Identify biases in the evaluators and the process. 4. Ask how the results will be used and whether that use is legitimate. 5. Revise the evaluation design and rerun it before acting on the results.

Check to track your progress (stored locally)

Learning progress0%

account_treeGenealogyexpand_more

menu_bookReferencesexpand_more

Source support: Explicit

link
eric.ed.govhttps://eric.ed.gov/?id=EJ916544ZH · Explicit
verified

RELATED MODELS