Much nano-research contains serious statistical error

Much neuroscientific research contains a serious statistical error, cognitive psychologist Sander Nieuwenhuis has discovered. This is even the case if the research has been published in top journals such as Nature and Science.

Nature Neuroscience

Sander Nieuwenhuis

Sander Nieuwenhuis

In an article in Nature Neuroscience, Nieuwenhuis and colleagues from the University of Amsterdam report the results of extensive literature research that has shown that neuroscientists have great difficulty reporting statistical interactions.

A typical example of the statistical error is, ‘Practising the task increased the neural activity in the genetically modified mice (P < 0.05) but not in the control mice (P > 0.05).’


Incorrect argument

The researchers actually want to claim that one effect is greater than another effect: the practise effect in genetically modified mice is greater on brain activity than the practise effect in control mice. In order to substantiate this claim they should report a statistically significant interaction (between amount of practice and type of mouse). However, they instead report that one effect is statistically significant whereas the other effect is not. Although this claim may appear convincing at first sight, it is an incorrect argument because the difference between ‘significant’ and ‘not significant’ is not necessarily significant itself. 

Significance
Statistical significance (expressed as a p-value on a scale from 0 to 1) means that a result is probably not a coincidence. If  p<0.05, then the effect is statistically significant: we can assume with enough certainty that the effect is not coincidental and that there really is a difference.

Alarm bells

Nieuwenhuis and his colleagues discovered that in the cognitive neurosciences this statistical error occurs at least as often as the correct procedure, even in top journals such as Nature and Science. Studies are therefore published in these journals of which one of the main conclusions is based on an incorrect statistical procedure. The literature research showed that the error occurs even more often in the basic neurosciences. A previous publication contended that the error is also made much too often in the social sciences (Gelman & Stern, 2006, American Statistician). Nieuwenhuis and his colleagues hope that their publication will set alarm bells ringing for researchers and reviewers. Nieuwenhuis, S., Forstmann, B.U., & Wagenmakers, E.-J. (2011). Erroneous analyses of interactions in neuroscience: A problem of significance. 1105-1107

European subsidy

Dr Sander Nieuwenhuis is attached to the Leiden Institute for Brain and Cognition (LIBC) and to the Faculty of Social Sciences as a cognitive psychologist. In July 2011 he received a Starting Investigator Grant from the European Research Council to conduct research on the effect of the noradrenaline neurotransmitter on the brain.

Links


Research area
‘The brain itself is multidisciplinary, and should therefore be studied as such’
Brain function and dysfunction over the lifespan


Dossier

Brain and cognition

Last Modified: 25-01-2012