Inter-Rater Reliability
Updated 2026-08-15
INTRODUCTION
English translation pending.
CORE DEFINITION
A psychometric property measuring the consistency with which two or more raters assign scores to the same subjects, quantified by statistics such as Cohen's kappa for categorical judgments and the intraclass correlation coefficient for continuous ones. The core proposition is that an instrument whose scores depend on who is rating is measuring the rater rather than the subject, so reliability must be established before scores can be interpreted. The key qualification is that high agreement does not imply validity: a group of raters can be consistently wrong when they share the same bias.
SCAFFOLDING EFFECT
Reduce cognitive load
- Independent scoring: have several raters score the same sample independently without conferring first. - Agreement measure: compute kappa or an intraclass correlation coefficient on the resulting scores. - Calibration loop: when agreement is low, clarify the rubric and then rescore the same sample.
Anchor fast decisions
A score is only informative if it is a property of the subject rather than of the person assigning it, and disagreement between raters is direct evidence that the instrument leaves something to interpretation. Measuring agreement makes that ambiguity visible as a number, which forces the vague criteria to be specified rather than left to individual judgment. The mechanism also reveals its own limit, since raters who share a common bias will agree closely while all being wrong, which is why reliability must be paired with evidence of validity.
MINIMUM ACTION
In progress 0/1Practice this model in one real situation:
account_treeGenealogyexpand_more
menu_bookReferencesexpand_more
Source support: Explicit
- en.wikipedia.orghttps://en.wikipedia.org/wiki/Inter-rater_reliabilityverified
PRIVATE NOTES · Only visible to you
SAVED Q&A
ENTRY Q&A · Private saving available
Ask with a clear boundary
thinkingmodels answers from published entry context only.
Your question is sent to thinkingmodels. The answer uses public entry context only.
RELATED MODELS