In one of the largest brain MRI studies to date, we used tensor-based morphometry (TBM) to create 3D maps of structural atrophy in 676 subjects with Alzheimer's disease (AD), mild cognitive impairment (MCI), and healthy elderly controls, scanned as part of the Alzheimer's Disease Neuroimaging Initiative (ADNI). Using inverse-consistent 3D non-linear elastic image registration, we warped 676 individual brain MRI volumes to a population mean geometric template. Jacobian determinant maps were created, revealing the 3D profile of local volumetric expansion and compression. We compared the anatomical distribution of atrophy in 165 AD patients (age: 75.6+/-7.6 years), 330 MCI subjects (74.8+/-7.5), and 181 controls (75.9+/-5.1). Brain atrophy in selected regions-of-interest was correlated with clinical measurements--the sum-of-boxes clinical dementia rating (CDR-SB), mini-mental state examination (MMSE), and the logical memory test scores - at voxel level followed by correction for multiple comparisons. Baseline temporal lobe atrophy correlated with current cognitive performance, future cognitive decline, and conversion from MCI to AD over the following year; it predicted future decline even in healthy subjects. Over half of the AD and MCI subjects carried the ApoE4 (apolipoprotein E4) gene, which increases risk for AD; they showed greater hippocampal and temporal lobe deficits than non-carriers. ApoE2 gene carriers--1/6 of the normal group--showed reduced ventricular expansion, suggesting a protective effect. As an automated image analysis technique, TBM reveals 3D correlations between neuroimaging markers, genes, and future clinical changes, and is highly efficient for large-scale MRI studies.