Journal of Data Science

Abstract: Frequentist and Bayesian hypothesis testing are often viewed as “two separate worlds” by practitioners. While theoretical relationships of course exist, our goal here is to demonstrate a practical example where one must be careful conducting frequentist hypothesis testing, and in that context illustrate a practical equivalence between Bayesian and frequentist testing. In particular, if the sample size is random (hardly unusual in prac tical problems where the sample size may be “all available experimental units”), then choosing an α level in advance such as 0.05 and using it for every possible sample size is inadmissible. In other words, one can find a dif ferent overall procedure which has the same overall type I error but greater power. Not coincidentally, this alternative procedure is based on Bayesian testing procedures.