Cognitive deficits affecting memory, attention and speed of information processing are common in multiple sclerosis (MS). The mechanisms of cognitive impairment remain unclear. Here, we examined the association between neuropsychological test performance and brain atrophy in a group of mildly disabled patients with relapsing-remitting MS. We applied voxel-based morphometry (SPM2) to investigate the distribution of brain atrophy in relation to cognitive performance. Patients had lower scores than control subjects on tests of memory and executive function, including the PASAT, Digit Span Backward and a test of short-term verbal memory (Memo). Among patients, but not healthy controls, performance on the PASAT, a comprehensive measure of cognitive function and reference task for the cognitive evaluation of MS-patients, correlated with global grey matter volume as well as with grey matter volume in regions associated with working memory and executive function, including bilateral prefrontal cortex, precentral gyrus and superior parietal cortex as well as right cerebellum. Compared to healthy subjects, patients showed a volume reduction in left temporal and prefrontal cortex, recently identified as areas predominantly affected by diffuse brain atrophy in MS. A comparison of low performers in the patient group with their matched control subjects showed more extensive and bilateral temporal and frontal volume reductions as well as bilateral parietal volume loss, compatible with the progression of atrophy found in more advanced MS-patients. These findings indicate that MS-related deficits in cognition are closely associated with cortical atrophy.