The Hospital Anxiety and Depression Scale (HADS) is a scale originally developed for the assessment of anxiety and depression in hospitalized patients. Despite its wide diffusion, research on its factorial structure has displayed inconsistent results, leaving doubts about its use in chronic musculoskeletal pain. The purpose of this study was to thoroughly assess the factorial structure of the HADS in patients with chronic pain and to give guidance for a potential refinement. Data from 2,522 patients with chronic pain from the Amsterdam Pain (AMS-PAIN) cohort were analyzed through: 1) exploratory bifactor analysis based on a Schmid-Leiman orthogonalization, 2) confirmatory factor analysis comparing a unidimensional model, the original correlated factors model and a bifactor model, 3) item response theory (IRT) analysis based on the graded response model. The results of the confirmatory factor analysis and of the IRT analysis were then cross-validated in an independent sample of patients with chronic pain (n = 8,604). Both exploratory and confirmatory analyses revealed the presence of a strong general emotional distress factor, suggesting that the HADS can be used as a unidimensional scale. The IRT analysis led to the exclusion of three items and to the recoding of one item. The refined 11-item HADS scale was successfully cross-validated and confirmed as a unidimensional, locally independent, monotonic, and reliable scale. PERSPECTIVE: An 11-item shorter version of the HADS could be used to measure emotional distress in patients with chronic musculoskeletal pain. Given its unidimensionality, the use of its total score seems appropriate.