Accepting the Null
Can we accept the null?
You might have heard we don’t “accept” the null hypothesis. We merely “find no evidence” to reject it.
There’s a version of the world where this framing of statistical inference makes sense: where the chief task of the researcher is to gather information about the world and conclude whether it can tell us anything, and if not? Oh well. We’ll get ’em next time.
This world exists, and we’ll vacation there in this blog, but I’ve never lived there, and you probably don’t either.
When you are the policymaker — even the policymaker’s assistant’s understudy — you can’t avoid… making policy. You either ship A or you ship B.
If you “found no evidence to reject the null,” and shipped A, then you looked at the data and decided that A was better than B by some metric. Why else wouldn’t you go with B? So, there’s a very real sense that you “accepted” the null.
The distinction is between decision-making and fact-finding.
Suppose we’ve got T = E[Y(B)] — E[Y(A)], a standard average treatment effect, and, if we knew T, we would want to do action B if T > 0, and action A if T < 0.
T > 0 is either true or false. It’s a fact. So, one framing of statistical inference: do we have enough evidence to conclude T > 0 is true? If we take this approach, we set up a hypothesis test with the null hypothesis that T ≤ 0 and evaluate whether to reject it. From this perspective, it really does make sense to say, “Oh well.” In fact, we must be willing to because we can’t assume there’s enough evidence to prove T > 0.
But this is an unusual way to state the problem if we’re deciding between A and B. We can’t avoid making a choice, so it doesn’t make sense to talk about the data “not having enough evidence” for us to decide. Some datasets support choice A, and others support choice B.
That isn’t to say the hypothesis testing framework and statistical significance aren’t useful. They are. They’re a nice way of quantifying how much status quo bias we want to incorporate into our decision (and we should have some: we have more experience with the past and more knowledge of its risks — better the devil you know…).
Statistical inference, like art and morality, requires us to draw the line somewhere. The hypothesis testing framework helps us find where to draw it.
But the key is to recognize that we aren’t actually testing a hypothesis. Not really. We’re not finding facts. We’re making decisions. So, we do accept nulls and alternatives as we please.
A side-benefit of treating statistical inference as a decision problem is that it’s much easier to explain. We’re looking at this dataset and deciding whether it favors A or B. You don’t need to try to explain the subtle asymmetries of nulls and alternatives.
Nice!
Thanks for reading!
Zach
Connect at: https://linkedin.com/in/zlflynn
Udemy course: https://www.udemy.com/course/identifying-causal-effects-for-data-scientists/?couponCode=928E40B177B5DB6F5266
If you would like my help with something (involving statistics, I should clarify): https://zflynn.com/consulting.html