In this paper, methods for assessing the accuracy and the reliability of functional magnetic resonance imaging techniques are presented. First, a modified receiver operating characteristic analysis is described for evaluating the accuracy of fMRI studies. With this modified approach, the true positives or the activated pixels are estimated based on highly averaged experimental data acquired with the same stimulation/task. Unlike ROC analysis based on simulated activation data, the present approach can be applied to experimentally acquired data without simplifying the activation related changes. To assess the reliability of fMRI studies, the kappa statistic was adopted for evaluating the overall agreement of functional activation maps from repeated experiments in individual subjects. To demonstrate the utility of these techniques, both the ROC analysis and the reliability assessment were applied to quantitatively evaluate the improvement in accuracy and reliability of a retrospective technique for physiological noise reduction in fMRI.