A visual task for semantic access involves a number of brain regions. However, previous studies either examined the role of each region separately using univariate approach, or analyzed a single brain network using covariance connectivity analysis. We hypothesize that these brain regions construct several functional networks underpinning a word semantic access task, these networks being engaged in different cognitive components with distinct temporal characters. In this paper, multivariate independent component analysis (ICA) was used to reveal these networks based on functional magnetic resonance imaging (fMRI) data acquired during a visual and an auditory word semantic judgment task. Our results demonstrated that there were three task-related independent components (ICs), corresponding to various cognitive components involved in the visual task. Furthermore, ICA separation on the auditory task showed consistency of the results with our hypothesis, regardless of the input modalities.