As the Apolipoprotein E (APOE) ɛ4 allele is a major genetic risk factor for sporadic Alzheimer's disease (AD), which has been suggested as a disconnection syndrome manifested by the disruption of white matter (WM) integrity and functional connectivity (FC), elucidating the subtle brain structural and functional network changes in cognitively normal ɛ4 carriers is essential for identifying sensitive neuroimaging based biomarkers and understanding the preclinical AD-related abnormality development. We first constructed functional network on the basis of resting-state functional magnetic resonance imaging and a structural network on the basis of diffusion tensor image. Using global, local and nodal efficiencies of these two networks, we then examined (i) the differences of functional and WM structural network between cognitively normal ɛ4 carriers and non-carriers simultaneously, (ii) the sensitivity of these indices as biomarkers, and (iii) their relationship to behavior measurements, as well as to cholesterol level. For ɛ4 carriers, we found reduced global efficiency significantly in WM and marginally in FC, regional FC dysfunctions mainly in medial temporal areas, and more widespread for WM network. Importantly, the right parahippocampal gyrus (PHG.R) was the only region with simultaneous functional and structural damage, and the nodal efficiency of PHG.R in WM network mediates the APOE ɛ4 effect on memory function. Finally, the cholesterol level correlated with WM network differently than with the functional network in ɛ4 carriers. Our results demonstrated ɛ4-specific abnormal structural and functional patterns, which may potentially serve as biomarkers for early detection before the onset of the disease.