ReviewMultiple testing corrections, nonparametric methods, and random field theory
Introduction
In the whimsically titled letter “Holmes & Watson reply to Sherlock” (Holmes et al., 1998) my colleagues and I made a serious critique of Halber et al. (1997), a paper evaluating thresholding methods for PET activation data. The paper directly compared a nonparametric permutation method (named “Sherlock”), which provided inferences fully corrected for multiple testing, to uncorrected P < 0.05 inference, finding that the latter method was to be preferred for its power. In response to our letter, the paper's authors defended the uncorrected approach as the (then) default setting in the SPM1 software and claimed that it had been used in “approximately 1200 publications”.
Over a decade later, and one “Voodoo correlations” (Vul et al., 2009) imbroglio and post-mortem ichthyological fMRI study (Bennett et al., 2011) later, it seems everyone agrees that (a) correcting inferences for the search over the brain is essential and (b) such corrections are not consistently utilized in fMRI. Hopefully some historical perspective can strengthen the discipline's resolve to uphold good statistical practice.
What follows is a highly selective review of the literature on the multiple testing problem in fMRI and its antecedents (PET and M/EEG). I have tried to capture the major landmark publications, and while this selection is inevitably quirky and personal, I hope it will provide a useful perspective in this important aspect of fMRI data analysis. See Holmes (1994) and Petersson et al. (1999) for more careful and detailed reviews of early work in this area.
Section snippets
The problem
Whether studying brain structure or brain function, using MRI, PET or M/EEG modalities, the end result of an experiment is typically a set of statistic values (e.g. T or F values) that comprises an image. This “image” may be a 2D surface, a 3D volume, or even a 4D movie of statistics over time. Call T = {Ti} the statistic image, with Ti the value at voxel i. Before even mentioning “multiple testing” we must define the objects under inference. There are in fact a variety of ways of summarizing a
Early days
Many “fMRI statistical methods” are in fact generic procedures developed first for PET. Hence we start with seminal work by Fox and Mintun (1989), who showed that non-quantitative H215O PET8 could be used to map brain function. As part of that paper they proposed “Change Distribution Analysis” to determine if there were any effects in the image. They used the distribution of all local extrema, that is, the value of
The future
Looking ahead, there is renewed enthusiasm for resampling-based test as GPU's make order-of magnitude speed-ups (Eklund et al., 2011), and in particular which make local multivariate methods attractive (Eklund et al., 2011, Nandy and Cordes, 2007).
Predictive analyses and “brain reading” distill inference to a single accuracy number (Haynes and Rees, 2006) and seem to be a step away from “brain mapping”. But in practice investigators wish to determine which brain regions are responsible for the
References (66)
- et al.
False discovery rate revisited: FDR and topological inference using Gaussian random fields
NeuroImage
(2009) - et al.
Topological FDR for neuroimaging
NeuroImage
(2010) - et al.
Resampling fMRI time series
NeuroImage
(2005) - et al.
Detecting activations in PET and fMRI: levels of inference and power
NeuroImage
(1996) - et al.
Classical and Bayesian inference in neuroimaging: theory
NeuroImage
(2002) - et al.
Thresholding of statistical maps in functional neuroimaging using the false discovery rate
NeuroImage
(2002) - et al.
A topographic latent source model for fMRI data
NeuroImage
(2011) - et al.
Validating cluster size inference: random field and permutation methods
NeuroImage
(2003) - et al.
Combining voxel intensity and cluster extent with permutation test framework
NeuroImage
(2004) - et al.
Nonstationary cluster-size inference with random field and permutation methods
NeuroImage
(2004)
Robust smoothness estimation in statistical parametric maps using standardized residuals from the general linear model
NeuroImage
Simple group fMRI modeling and inference
NeuroImage
A semi-parametric approach to estimate the family-wise error rate in fMRI using resting-state data
NeuroImage
Combining spatial extent and peak intensity to test for activations in functional imaging
NeuroImage
Statistical tests for fMRI based on experimental randomization
NeuroImage
Threshold-free cluster enhancement: addressing problems of smoothing, threshold dependence and localisation in cluster inference
NeuroImage
Tract-based spatial statistics: voxelwise analysis of multi-subject diffusion data
NeuroImage
An improved theoretical p value for SPMs based on discrete local maxima
NeuroImage
Unified univariate and multivariate random field theory
NeuroImage
Regional homogeneity approach to fMRI data analysis
NeuroImage
Cluster mass inference via random field theory
NeuroImage
Discovering the false discovery rate
J. R. Stat. Soc. B
Controlling the false discovery rate: a practical and powerful approach to multiple testing
J. R. Stat. Soc. B Methodol.
Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: an argument for proper multiple comparisons correction
J. Seren. Unexpected Results
Statistical methods of estimation and inference for functional MR image analysis
Magn. Reson. Med.
Global, voxel, and cluster tests, by theory and permutation, for a difference between two groups of structural MR images of the brain
IEEE Trans. Med. Imaging
Colored noise and computational inference in neurophysiological (fMRI) time series analysis: resampling methods in time and wavelet domains
Human Brain Mapp.
Fast random permutation tests enable objective evaluation of methods for single-subject fMRI analysis
Int. J. Biomed. Imaging
Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): use of a cluster-size threshold
Magn. Reson. Med.
Noninvasive functional brain mapping by change-distribution analysis of averaged PET images of H215O tissue activity
J. Nucl. Med.
Comparing functional (PET) images: the assessment of significant change
J. Cereb. Blood Flow Metab.
Cited by (160)
Mediation analysis for high-dimensional mediators and outcomes with an application to multimodal imaging data
2023, Computational Statistics and Data AnalysisIntegrating MR imaging with full-surface indentation mapping of femoral cartilage in an ex vivo porcine stifle
2023, Journal of the Mechanical Behavior of Biomedical MaterialsThe human brain networks mediating the vestibular sensation of self-motion
2022, Journal of the Neurological SciencesTheta activity in semantic priming under visual crowding as revealed by magnetoencephalography
2022, Neuroscience Research