Survey Article: Inter-Coder Agreement for Computational Linguistics
Source: https://aclanthology.org/J08-4004/
Read URL: https://aclanthology.org/J08-4004.pdf
Read date: 2026-10-30
Depth: Abstract and §§2.1–2.2; discussion of coefficient assumptions

Paraphrased evidence: The review distinguishes agreement, reliability and validity. High agreement cannot establish truth when observers share a bias; coefficients also depend on category distributions and assumptions.
Scope: Reliability of annotation
Limit: Internal AI checking is not independent human annotation or human IRR
Application (project inference): Do not invent kappa/alpha; preserve disagreement and validation scope

This note is an original paraphrase, not a saved original document. SHA256 below identifies this note, not the remote page.
