Trending towards significance
You may have heard that the result of an experiment is “trending” towards significance despite currently being insignificant. It isn’t. But the reason why is difficult to find by Googling. This short post is my attempt to put the info out there!
For a math-free intuition of the main point, suppose it was possible to predict whether a currently insignificant result was “trending” towards becoming significant. Wouldn’t this make the result significant today? i.e., if you could tell, based on current information, whether the result would be significant in the future, it would be significant now!
Math-y version:
Suppose we run a t-test to test whether the mean of some normally distributed iid random variables is 0.
Let’s look at the difference between the test statistics at n and n+m, where m is small-ish relative to n:
For small m relative to n, (1/sqrt(n+m) — 1/sqrt(n)) is small.
So:
i.e., future changes in the test statistic are approximately independent of all the previous data. This means we cannot predict the future change in the test statistic from the prior values. So, under the null, we have no idea whether the trend we are seeing will continue. Since we haven’t ruled out the null today, there’s no reason to think any trend will continue tomorrow. Random walks can generate graphs that look like trends — but they ain’t!
Zach
LinkedIn: https://www.linkedin.com/in/zlflynn/
If you want my help with any Experimentation, Analytics, etc. problem, click here.
[A little more precisely: W(n) is a random walk. For large n, W(n) is approximately a Weiner process, and increments in a Weiner process are independent of their historical values.]