Although analgesics are widely recommended in current guidelines, underuse and inadequate prescription of analgesics seem to result in suboptimal treatment effects in patients with knee and/or hip osteoarthritis (OA). This study aimed (i) to describe the use of analgesics; and (ii) to determine factors that are related to analgesic use in patients with knee and/or hip OA referred to an outpatient center. A cross-sectional study with data from 656 patients with knee and/or hip OA referred to an outpatient center (Amsterdam Osteoarthritis (AMS-OA) cohort) was conducted. Self-reported use of analgesic (yes/no) was administered and subdivided into acetaminophen, non-steroidal anti-inflammatory drugs (NSAIDs, including coxibs) and opioids. Logistic regression analyses were performed to analyze the association between analgesic use and disease-related, predisposing and enabling factors. Analgesic use was reported by 63% of the patients, with acetaminophen, NSAIDs and opioid use reported by 50, 30 and 12%, respectively. Factors related to analgesic use were higher pain severity, longer duration of symptoms, higher radiographic hip OA severity, overweight/obesity and psychological distress. These factors explained 21% of the variance of analgesic use. More than one-third of patients with established knee and/or hip OA referred to an outpatient center did not use any analgesics. Although multiple, mostly disease-related associated factors were found, analgesic use remained predominantly unexplained. Our study seems to indicate that prescription of analgesics should be guided more dominantly by clinical symptoms and needs, and preceded by a thorough shared decision-making process between patient and physician.