We sought to identify magnetic resonance- (MR)-imaged structures associated with declarative memory in a community-dwelling sample of elderly Mexican-American individuals with a spectrum of cognitive decline. Measured structures were the hemispheric volumes of the hippocampus (HC), parahippocampal gyrus, and remaining temporal lobes, as well as severity of white matter signal hyperintensities (WMH). Participants were an imaged subsample from the Sacramento Area Latino Study of Aging (SALSA), N = 122. Individuals were categorized as normal, memory impaired (MI), cognitively impaired non-demented (CIND), or demented. We show that WMH was the strongest structural predictor for performance on a delayed free-recall task (episodic memory) in the entire sample. The association of WMH with delayed recall was most prominent in elderly normals and mildly cognitively impaired individuals with no dementia or impairment of daily function. However, the left HC was associated with verbal delayed recall only in people with dementia. The right HC volume predicted nonverbal semantic-memory performance. We conclude that WMH are an important pathological substrate that affects certain memory functions in normal individuals and those with mild memory loss and discuss how tasks associated with WMH may rely upon frontal lobe function.