Introduction Stereotactic ablative radiotherapy (SABR) is currently not the guideline-recommended treatment for lung tumors measuring more than 5 cm. However, improvements in radiotherapy techniques have led to increasing use of SABR for larger tumors. Methods We analyzed the clinical outcomes in patients with a primary or recurrent NSCLC measuring more than 5 cm and treated with five or eight fractions of SABR at our center. Patients who had prior thoracic radiotherapy were excluded. Results A total of 63 consecutive patients with a median tumor diameter of 5.8 cm (range 5.1–10.4) were identified; 81% had T2N0 disease and 18% had T3N0 disease. The median Charlson comorbidity index was 2 (range 0–6). After a median follow-up of 54.7 months, median survival was 28.3 months. Disease-free survival at 2 years was 82.1%, and the local, regional, and distant control rates at 2 years were 95.8%, 93.7%, and 83.6%, respectively. An out-of-field distant recurrence at one or more sites was the most common pattern of failure (10%). Grade 3 or higher toxicity was recorded in 30% of patients, with radiation pneumonitis being the most common toxicity (19%). A likely (n = 4) or possible (n = 8) treatment-related death was scored in 19% of patients. There was preexisting interstitial lung disease in eight patients (13%), with fatal toxicity developing in five of them (63%). Conclusions Lung SABR in tumors larger than 5 cm resulted in high local control rates and acceptable survival outcomes in a patient population with appreciable comorbidity. Patients with interstitial lung disease should be considered a very high-risk population for SABR.