Neuroimaging studies of apolipoprotein E (ApoE4) have implicated its association with brain atrophy in Alzheimer's disease. To date, few studies have used automated morphological analysis techniques to assess ApoE4-related brain structure change in both gray and white matter in nondemented older adults. Nondemented (CDR = 0, n = 53) subjects over 60 had MRI, diffusion tensor imaging, and neurocognitive assessments. We assessed differences in cognition and brain structure associated with ApoE4 genetic variation using voxel-based morphometry techniques, and tract-based spatial statistics of fractional anisotropy change. In nondemented older adults with the E4 allele, cognitive performance was reduced, and atrophy was present in the hippocampus and amygdala compared to ApoE4 negative participants. We also report that E4 carriers have decreased fractional anisotropy in the left parahippocampal gyrus white matter. In conclusion, the presence of an ApoE4 allele in nondemented older adults is associated with decreases in cognition and gray and white matter changes in the medial temporal cortex. Overall we provide further evidence of the effects of genetic variance related to imaging and cognitive measures of risk for Alzheimer's disease.