To explore the effects of white matter in the absence of auditory input in the early deaf, we conducted a tract-based statistical analysis of the diffusion tensor anisotropy and the voxel-based morphometry in the white matter of 13 early deaf and 29 hearing individuals. Deaf individuals showed significant decreases in diffusion anisotropy and in regional volume reductions within the temporal white matter. Decreased anisotropy was also found at the internal capsule, superior longitudinal fasciculus, and the inferior frontal white matter. In contrast, the forceps major of the corpus callosum, where interhemispheric connections between visual cortices exist, showed increased anisotropy. We interpreted these white matter alterations in terms of both disuse-driven atrophy and compensatory plasticity in the early deaf.