Elsevier

NeuroImage

Volume 62, Issue 2, 15 August 2012, Pages 811-815
NeuroImage

Review
Multiple testing corrections, nonparametric methods, and random field theory

https://doi.org/10.1016/j.neuroimage.2012.04.014Get rights and content

Abstract

I provide a selective review of the literature on the multiple testing problem in fMRI. By drawing connections with the older modalities, PET in particular, and how software implementations have tracked (or lagged behind) theoretical developments, my narrative aims to give the methodological researcher a historical perspective on this important aspect of fMRI data analysis.

Introduction

In the whimsically titled letter “Holmes & Watson reply to Sherlock” (Holmes et al., 1998) my colleagues and I made a serious critique of Halber et al. (1997), a paper evaluating thresholding methods for PET activation data. The paper directly compared a nonparametric permutation method (named “Sherlock”), which provided inferences fully corrected for multiple testing, to uncorrected P < 0.05 inference, finding that the latter method was to be preferred for its power. In response to our letter, the paper's authors defended the uncorrected approach as the (then) default setting in the SPM1 software and claimed that it had been used in “approximately 1200 publications”.

Over a decade later, and one “Voodoo correlations” (Vul et al., 2009) imbroglio and post-mortem ichthyological fMRI study (Bennett et al., 2011) later, it seems everyone agrees that (a) correcting inferences for the search over the brain is essential and (b) such corrections are not consistently utilized in fMRI. Hopefully some historical perspective can strengthen the discipline's resolve to uphold good statistical practice.

What follows is a highly selective review of the literature on the multiple testing problem in fMRI and its antecedents (PET and M/EEG). I have tried to capture the major landmark publications, and while this selection is inevitably quirky and personal, I hope it will provide a useful perspective in this important aspect of fMRI data analysis. See Holmes (1994) and Petersson et al. (1999) for more careful and detailed reviews of early work in this area.

Section snippets

The problem

Whether studying brain structure or brain function, using MRI, PET or M/EEG modalities, the end result of an experiment is typically a set of statistic values (e.g. T or F values) that comprises an image. This “image” may be a 2D surface, a 3D volume, or even a 4D movie of statistics over time. Call T = {Ti} the statistic image, with Ti the value at voxel i. Before even mentioning “multiple testing” we must define the objects under inference. There are in fact a variety of ways of summarizing a

Early days

Many “fMRI statistical methods” are in fact generic procedures developed first for PET. Hence we start with seminal work by Fox and Mintun (1989), who showed that non-quantitative H215O PET8 could be used to map brain function. As part of that paper they proposed “Change Distribution Analysis” to determine if there were any effects in the image. They used the distribution of all local extrema, that is, the value of

The future

Looking ahead, there is renewed enthusiasm for resampling-based test as GPU's make order-of magnitude speed-ups (Eklund et al., 2011), and in particular which make local multivariate methods attractive (Eklund et al., 2011, Nandy and Cordes, 2007).

Predictive analyses and “brain reading” distill inference to a single accuracy number (Haynes and Rees, 2006) and seem to be a step away from “brain mapping”. But in practice investigators wish to determine which brain regions are responsible for the

References (66)

  • S. Kiebel et al.

    Robust smoothness estimation in statistical parametric maps using standardized residuals from the general linear model

    NeuroImage

    (1999)
  • Jeanette A. Mumford et al.

    Simple group fMRI modeling and inference

    NeuroImage

    (2009)
  • Rajesh Nandy et al.

    A semi-parametric approach to estimate the family-wise error rate in fMRI using resting-state data

    NeuroImage

    (2007)
  • Jean-Baptiste Poline et al.

    Combining spatial extent and peak intensity to test for activations in functional imaging

    NeuroImage

    (1997)
  • Jonathan Raz et al.

    Statistical tests for fMRI based on experimental randomization

    NeuroImage

    (2003)
  • Stephen M. Smith et al.

    Threshold-free cluster enhancement: addressing problems of smoothing, threshold dependence and localisation in cluster inference

    NeuroImage

    (2009)
  • Stephen M. Smith et al.

    Tract-based spatial statistics: voxelwise analysis of multi-subject diffusion data

    NeuroImage

    (2006)
  • Keith J. Worsley et al.

    An improved theoretical p value for SPMs based on discrete local maxima

    NeuroImage

    (2005)
  • Keith J. Worsley et al.

    Unified univariate and multivariate random field theory

    NeuroImage

    (2004)
  • Yufeng Zang et al.

    Regional homogeneity approach to fMRI data analysis

    NeuroImage

    (2004)
  • Hui Zhang et al.

    Cluster mass inference via random field theory

    NeuroImage

    (2009)
  • Yoav Benjamini

    Discovering the false discovery rate

    J. R. Stat. Soc. B

    (2010)
  • Y. Benjamini et al.

    Controlling the false discovery rate: a practical and powerful approach to multiple testing

    J. R. Stat. Soc. B Methodol.

    (1995)
  • Craig M. Bennett et al.

    Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: an argument for proper multiple comparisons correction

    J. Seren. Unexpected Results

    (2011)
  • R.C. Blair et al.
  • Edward T. Bullmore et al.

    Statistical methods of estimation and inference for functional MR image analysis

    Magn. Reson. Med.

    (1996)
  • Edward T. Bullmore et al.

    Global, voxel, and cluster tests, by theory and permutation, for a difference between two groups of structural MR images of the brain

    IEEE Trans. Med. Imaging

    (1999)
  • Edward T. Bullmore et al.

    Colored noise and computational inference in neurophysiological (fMRI) time series analysis: resampling methods in time and wavelet domains

    Human Brain Mapp.

    (2001)
  • Anders Eklund et al.

    Fast random permutation tests enable objective evaluation of methods for single-subject fMRI analysis

    Int. J. Biomed. Imaging

    (2011)
  • Flitney, David E., & Jenkinson, Mark. 2000. Cluster Analysis Revisited. FMRIB Technical Report...
  • S.D. Forman et al.

    Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): use of a cluster-size threshold

    Magn. Reson. Med.

    (1995)
  • Peter T. Fox et al.

    Noninvasive functional brain mapping by change-distribution analysis of averaged PET images of H215O tissue activity

    J. Nucl. Med.

    (1989)
  • Karl J. Friston et al.

    Comparing functional (PET) images: the assessment of significant change

    J. Cereb. Blood Flow Metab.

    (1991)
  • Cited by (160)

    View all citing articles on Scopus
    View full text