A common problem in the analysis of functional magnetic resonance imaging (fMRI) data is quantifying the statistical reliability of an estimated activation map. While visual comparison of the classified active regions across replications of an experiment can sometimes by informative, it is typically difficult to draw firm conclusions by inspection; noise and complex patterns in the estimated map make it easy to be misled. Here, several statistical models, of increasing complexity, are developed, under which "test-retest" reliability can be meaningfully defined and quantified. The method yields global measures of reliability that apply uniformly to a specified set of brain voxels. The estimates of these reliability measures and their associated uncertainties under these models can be used to compare statistical methods, to set thresholds for detecting activation, and to optimize the number of images that need to be acquired during an experiment.