Although the relationship between structural differences within the prefrontal cortex (PFC) and executive function (EF) has been widely explored in cognitively impaired populations, little is known about this relationship in healthy young adults. Using optimized voxel-based morphometry (VBM), surface-based morphometry (SBM), and fractional anisotropy (FA) we determined the association between regional PFC grey matter (GM) morphometry and white matter tract diffusivity with performance on tasks that tap different aspects of EF as drawn from Miyake et al.'s three-factor model of EF. Reductions in both GM volume (VBM) and cortical folding (SBM) in the ventromedial PFC (vmPFC), ventrolateral PFC (vlPFC), and dorsolateral PFC (dlPFC) predicted better common EF, shifting-specific, and updating-specific performance, respectively. Despite capturing different components of GM morphometry, voxel- and surface-based findings were highly related, exhibiting regionally overlapping relationships with EF. Increased white matter FA in fiber tracts that connect the vmPFC and vlPFC with posterior regions of the brain also predicted better common EF and shifting-specific performance, respectively. These results suggest that the neural mechanisms supporting distinct aspects of EF may differentially rely on distinct regions of the PFC, and at least in healthy young adults, are influenced by regional morphometry of the PFC and the FA of major white matter tracts that connect the PFC with posterior cortical and subcortical regions.