Background: Alzheimer’s disease is characterized by aggregated β-amyloid and tau proteins, but the clinical presentations and patterns of brain atrophy vary substantially. A part of this heterogeneity may be linked to the risk allele APOE ε4. The spread of tau pathology is related to atrophy and cognitive decline, but little data exist on the effects of APOE ε4 on tau. The objective of this preliminary study was therefore to test if tau load and brain structure differ by APOE ε4 in Alzheimer’s disease. Methods: Sixty-five β-amyloid-positive patients at the prodromal and dementia stages of Alzheimer’s disease were enrolled, including APOE ε4-positive (n = 46) and APOE ε4-negative (n = 19) patients. 18F-AV-1451 positron emission tomography was used to measure tau and brain magnetic resonance imaging (MRI) was used to measure cortical thickness. Results: Compared with their APOE ε4-positive counterparts, APOE ε4-negative patients had greater tau load and reduced cortical thickness, with the most pronounced effects for both in the parietal cortex. Relative to the overall cortical tau load, APOE ε4-positive patients had greater tau load in the entorhinal cortex. APOE ε4-positive patients also had slightly greater cortical β-amyloid load. There was an interaction between APOE ε4 and 18F-AV-1451 on cortical thickness, with greater effects of 18F-AV-1451 on cortical thickness in APOE ε4-negative patients. APOE ε4 and 18F-AV-1451 were independent predictors of cognition, but showed distinct associations with different cognitive tests. Conclusions: APOE genotype may be associated with differences in pathways in Alzheimer’s disease, potentially through differential development and spread of tau, as well as through effects on cognitive outcomes involving non-tau-related mechanisms.