This review traces the structural maturation of the human auditory system, and compares the timeline of anatomical development with cotemporaneous physiological and behavioral events. During the embryonic period, there is formation of basic structure at all levels of the system, i.e. the inner ear, the brainstem pathway, and the cortex. The second trimester is a time of rapid growth and development, and by the end of this period, the cochlea has acquired a very adult-like configuration. During the perinatal period, the brainstem reaches a mature state, and brainstem activity is reflected in behavioral responses to sound, including phonetic discrimination, and in evoked brainstem and early middle latency responses. The perinatal period is also the time of peak development of brainstem input to the cortex through the marginal layer, and of the long latency cortical potentials, the N(2) and mismatch negativity. In early childhood, from the sixth post-natal month to age five, there is progressive maturation of the thalamic projections to the cortex and of the longer latency Pa and P(1) evoked potentials. Later childhood, from six to twelve years, is the time of maturation of the superficial cortical layers and their intracortical connections, accompanied by appearance of the N(1) potential and improved linguistic discriminative abilities. Some consideration is given to the potential negative effects of deafness-induced sound deprivation during the perinatal period and childhood.