Specific agreement on ordinal and multiple nominal outcomes can be calculated for more than two raters

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

OBJECTIVE: The concept of specific agreement has been proposed for dichotomous outcomes for two and more raters. We aim to extend this concept for variables with more than two ordinal or nominal categories and more than two raters.

STUDY DESIGN AND SETTING: We used two data sets: 4 plastic surgeons classifying photographs after breast reconstruction on a 5 point ordinal scale; and 6 raters classifying psychiatric patients into 5 diagnostic categories. For m raters, all (i.e. m(m-1)/2) pairwise agreement tables were summed to calculate the observed agreement, specific agreement and conditional probabilities. The 95% confidence intervals were obtained by bootstrapping.

RESULTS: Specific agreement was calculated for each ordinal or nominal category to examine when one of the raters scored in a specific category, what is the probability that the other raters scored in that same category. And suppose one of the raters scored X1, what is the probability that the other raters scored X1 or any of the other categories (conditional probability). It appeared for example that among the psychiatric disorders, depression and personality disorders were often mixed up, while neurosis was rarely mixed up with schizophrenia.

CONCLUSION: The concept of specific agreement for variables with ordinal and multiple nominal categories provides relevant clinical information. The extension to conditional probabilities of alternative categories broadens the clinical application with examining which categories are most often mixed up.

Original languageEnglish
Pages (from-to)47-53
JournalJournal of Clinical Epidemiology
Volume96
Early online date4 Dec 2017
DOIs
Publication statusPublished - Apr 2018

Cite this

@article{00487787e1ee4424889fbfba46db4be2,
title = "Specific agreement on ordinal and multiple nominal outcomes can be calculated for more than two raters",
abstract = "OBJECTIVE: The concept of specific agreement has been proposed for dichotomous outcomes for two and more raters. We aim to extend this concept for variables with more than two ordinal or nominal categories and more than two raters.STUDY DESIGN AND SETTING: We used two data sets: 4 plastic surgeons classifying photographs after breast reconstruction on a 5 point ordinal scale; and 6 raters classifying psychiatric patients into 5 diagnostic categories. For m raters, all (i.e. m(m-1)/2) pairwise agreement tables were summed to calculate the observed agreement, specific agreement and conditional probabilities. The 95{\%} confidence intervals were obtained by bootstrapping.RESULTS: Specific agreement was calculated for each ordinal or nominal category to examine when one of the raters scored in a specific category, what is the probability that the other raters scored in that same category. And suppose one of the raters scored X1, what is the probability that the other raters scored X1 or any of the other categories (conditional probability). It appeared for example that among the psychiatric disorders, depression and personality disorders were often mixed up, while neurosis was rarely mixed up with schizophrenia.CONCLUSION: The concept of specific agreement for variables with ordinal and multiple nominal categories provides relevant clinical information. The extension to conditional probabilities of alternative categories broadens the clinical application with examining which categories are most often mixed up.",
keywords = "Journal Article",
author = "{de Vet}, {Henrica C W} and Mullender, {Margriet G} and Iris Eekhout",
note = "Copyright {\circledC} 2017 Elsevier Inc. All rights reserved.",
year = "2018",
month = "4",
doi = "10.1016/j.jclinepi.2017.11.024",
language = "English",
volume = "96",
pages = "47--53",
journal = "Journal of Clinical Epidemiology",
issn = "0895-4356",
publisher = "Elsevier USA",

}

TY - JOUR

T1 - Specific agreement on ordinal and multiple nominal outcomes can be calculated for more than two raters

AU - de Vet, Henrica C W

AU - Mullender, Margriet G

AU - Eekhout, Iris

N1 - Copyright © 2017 Elsevier Inc. All rights reserved.

PY - 2018/4

Y1 - 2018/4

N2 - OBJECTIVE: The concept of specific agreement has been proposed for dichotomous outcomes for two and more raters. We aim to extend this concept for variables with more than two ordinal or nominal categories and more than two raters.STUDY DESIGN AND SETTING: We used two data sets: 4 plastic surgeons classifying photographs after breast reconstruction on a 5 point ordinal scale; and 6 raters classifying psychiatric patients into 5 diagnostic categories. For m raters, all (i.e. m(m-1)/2) pairwise agreement tables were summed to calculate the observed agreement, specific agreement and conditional probabilities. The 95% confidence intervals were obtained by bootstrapping.RESULTS: Specific agreement was calculated for each ordinal or nominal category to examine when one of the raters scored in a specific category, what is the probability that the other raters scored in that same category. And suppose one of the raters scored X1, what is the probability that the other raters scored X1 or any of the other categories (conditional probability). It appeared for example that among the psychiatric disorders, depression and personality disorders were often mixed up, while neurosis was rarely mixed up with schizophrenia.CONCLUSION: The concept of specific agreement for variables with ordinal and multiple nominal categories provides relevant clinical information. The extension to conditional probabilities of alternative categories broadens the clinical application with examining which categories are most often mixed up.

AB - OBJECTIVE: The concept of specific agreement has been proposed for dichotomous outcomes for two and more raters. We aim to extend this concept for variables with more than two ordinal or nominal categories and more than two raters.STUDY DESIGN AND SETTING: We used two data sets: 4 plastic surgeons classifying photographs after breast reconstruction on a 5 point ordinal scale; and 6 raters classifying psychiatric patients into 5 diagnostic categories. For m raters, all (i.e. m(m-1)/2) pairwise agreement tables were summed to calculate the observed agreement, specific agreement and conditional probabilities. The 95% confidence intervals were obtained by bootstrapping.RESULTS: Specific agreement was calculated for each ordinal or nominal category to examine when one of the raters scored in a specific category, what is the probability that the other raters scored in that same category. And suppose one of the raters scored X1, what is the probability that the other raters scored X1 or any of the other categories (conditional probability). It appeared for example that among the psychiatric disorders, depression and personality disorders were often mixed up, while neurosis was rarely mixed up with schizophrenia.CONCLUSION: The concept of specific agreement for variables with ordinal and multiple nominal categories provides relevant clinical information. The extension to conditional probabilities of alternative categories broadens the clinical application with examining which categories are most often mixed up.

KW - Journal Article

U2 - 10.1016/j.jclinepi.2017.11.024

DO - 10.1016/j.jclinepi.2017.11.024

M3 - Article

VL - 96

SP - 47

EP - 53

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

SN - 0895-4356

ER -