Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy: A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool

Brooke Levis, Andrea Benedetti, Alexander W. Levis, John P.A. Ioannidis, Ian Shrier, Pim Cuijpers, Simon Gilbody, Lorie A. Kloda, Dean McMillan, Scott B. Patten, Russell J. Steele, Roy C. Ziegelstein, Charles H. Bombardier, Flavia De Lima Osório, Jesse R. Fann, Dwenda Gjerdingen, Femke Lamers, Manote Lotrakul, Sonia R. Loureiro, Bernd LöweJuwita Shaaban, Lesley Stafford, Henk C.P.M. Van Weert, Mary A. Whooley, Linda S. Williams, Karin A. Wittkampf, Albert S. Yeung, Brett D. Thombs*

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review


In studies of diagnostic test accuracy, authors sometimes report results only for a range of cutoff points around data-driven "optimal" cutoffs. We assessed selective cutoff reporting in studies of the diagnostic accuracy of the Patient Health Questionnaire-9 (PHQ-9) depression screening tool. We compared conventional meta-analysis of published results only with individual-patient-data meta-analysis of results derived from all cutoff points, using data from 13 of 16 studies published during 2004-2009 that were included in a published conventional metaanalysis. For the "standard" PHQ-9 cutoff of 10, accuracy results had been published by 11 of the studies. For all other relevant cutoffs, 3-6 studies published accuracy results. For all cutoffs examined, specificity estimates in conventional and individual-patient-data meta-analyses were within 1% of each other. Sensitivity estimates were similar for the cutoff of 10 but differed by 5%-15% for other cutoffs. In samples where the PHQ-9 was poorly sensitive at the standard cutoff, authors tended to report results for lower cutoffs that yielded optimal results. When the PHQ-9 was highly sensitive, authors more often reported results for higher cutoffs. Consequently, in the conventional metaanalysis, sensitivity increased as cutoff severity increased across part of the cutoff range-an impossibility if all data are analyzed. In sum, selective reporting by primary study authors of only results from cutoffs that perform well in their study can bias accuracy estimates in meta-analyses of published results.

Original languageEnglish
Pages (from-to)954-964
Number of pages11
JournalAmerican Journal of Epidemiology
Issue number10
Publication statusPublished - 15 May 2017

Cite this