Introduction: The relative importance of structural magnetic resonance imaging (MRI) and tau positron emission tomography (PET) to predict diagnosis and cognition in Alzheimer's disease (AD) is unclear. Methods: We tested 56 cognitively unimpaired controls (including 27 preclinical AD), 32 patients with prodromal AD, and 39 patients with AD dementia. Optimal classifiers were constructed using the least absolute shrinkage and selection operator with 18F-AV-1451 (tau) PET and structural MRI data (regional cortical thickness and subcortical volumes). Results: 18F-AV-1451 in the amygdala, entorhinal cortex, parahippocampal gyrus, fusiform, and inferior parietal lobule had 93% diagnostic accuracy for AD (prodromal or dementia). The MRI classifier involved partly the same regions plus the hippocampus, with 83% accuracy, but did not improve upon the tau classifier. 18F-AV-1451 retention and MRI were independently associated with cognition. Discussion: Optimized tau PET classifiers may diagnose AD with high accuracy, but both tau PET and structural brain MRI capture partly unique information relevant for the clinical deterioration in AD.