Posts tagged ‘null hypothesis testing’

September 24, 2012

More wrong interpretations of P values – “repeated sampling”

by James Thorniley

A while ago I wrote a little rant on the (mis)interpretation of P-values. I’d like to return to this subject having investigated a little more. First, this post, I’m going to point to an interesting little subtlety pointed out by Fisher that I hadn’t thought about before, in the second post, I will argue why P-values aren’t as bad as they are sometimes made out to be.

So, last time, I stressed the point that you can’t interpret a P-value as a probability or frequency of anything, unless you say “given that the null hypothesis is true”. Most misinterpretations, e.g. “the probability that you would accept the null hypothesis if you tried the experiment again”, make this error. But there is one common interpretation that is less obviously false: “A P-value is the probability that the data would deviate as or more strongly from the null hypothesis in another experiment, than they did in the current experiment, given that the null hypothesis is true”. This is something that you might think is a more careful statement, but the problem is that in fact when we calculate P values we take into account aspects of the data not necessarily related to how strongly they deviate from the prediction of the null hypothesis. This could be misleading, so we’ll build it up more precisely in this post.

December 5, 2011

When should you use a null hypothesis test? Probably never

by James Thorniley

I came across a nice paper called “The Null Ritual” by Gerd Gigerenzer et al recently, it’s an excellent read in my opinion, and sums up a lot of the things that are wrong with null-hypothesis testing. This process is pervasive in many areas of science, particularly psychology (which is what these authors are mainly talking about), and it’s flawed in too many ways to count. Gigerenzer’s paper is worth reading, I’m going to attempt to summarise it, focussing on the things that really bother me. A good point that this paper makes is that its not actually the test itself that is intrinsically “wrong”, it’s more to do with the way that it has permeated in scientific culture. This is what they are calling the “Null Ritual” – the process of more or less automatically doing a null hypothesis test, without there necessarily being a good reason to, except perhaps that some journals or reviewers seem to require it. Before reading this, you might want to try filling in the Jellymatter poll on the subject (which is taken from “The Null Ritual”), before I discuss the correct answer below.


Get every new post delivered to your Inbox.

Join 483 other followers