Background: Physical inactivity in hospitalized older adults is highly prevalent and associated with detrimental health outcomes. Understanding its determinants is important for prognosis and tailoring interventions in geriatric rehabilitation inpatients. Methods: Within the REStORing health of acutely unwell adulTs (RESORT) observational, longitudinal cohort, geriatric rehabilitation inpatients wore an inertial sensor (ActivPAL4) for one week to objectively assess instrumented sedentary behavior (i-SB) and physical activity (i-PA). Determinants were grouped in five geriatric domains: morbidity, cognition/psychology, physical performance, functional performance, and nutritional status. Their association with i-SB (mean sitting, lying, non-upright time) and i-PA (mean number of steps, sit-to-stand transitions and upright time) quintiles were examined using multivariate ordinal logistic regression analyses with Bonferroni correction (p < 0.006). Results: A total of 145 inpatients were included (mean age 83.0, SD 7.7 years; 55.9% females). More comorbidities were associated with a lower daily number of steps (OR:0.91, 95%CI: 0.86–0.96) and lower upright time (OR:0.93, 95%CI: 0.88–0.98). Depressive symptoms (higher Hospital Anxiety and Depression Scale score) were associated with higher non-upright time (OR: 1.12, 95%CI: 1.03–1.21) and lower upright time (OR: 0.89, 95%CI: 0.83–0.96). Better physical performance (higher Functional Ambulation Classification, gait speed, and Short Physical Performance Battery score) was associated with lower i-SB measures (OR range: 0.07–0.78, p < 0.0005) and higher i-PA measures (OR range: 1.35–19.50, p < 0.0005). Higher functional performance (Katz index of Activities of Daily Living score) was associated with lower i-SB measures (OR range: 0.61–0.69, p ≤ 0.003) and higher i-PA measures (OR range: 1.60–3.64, p < 0.0005). Being malnourished was associated with lower i-PA measures (OR range: 0.29–0.32, p ≤ 0.004). Conclusions: Worse morbidity, depressive symptoms, worse physical and functional performance, and worse nutritional status were associated with higher i-SB and lower i-PA. These determinants should be taken into account while designing and promoting multidisciplinary physical activity interventions.