Beyond Kappa: Estimating Inter-Rater Agreement with Nominal Classifications

Abstract

Cohen’s Kappa and a number of related measures can all be criticized for their definition of correction for chance agreement. A measure is introduced that derives the corrected proportion of agreement directly from the data, thereby overcoming objections to Kappa and its related measures.