Journal of Data Science

Abstract: Motivated by a situation encountered in the Well Elderly 2 study, the paper considers the problem of robust multiple comparisons based on K independent tests associated with 2K independent groups. A simple strategy is to use an extension of Dunnett’s T3 procedure, which is designed to control the probability of one or more Type I errors. However, this method and related techniques fail to take into account the overall pattern of p-values when making decisions about which hypotheses should be rejected. The paper suggests a multiple comparison procedure that does take the overall pattern into account and then describes general situations where this alternative approach makes a practical difference in terms of both power and the probability of one or more Type I errors. For reasons summarized in the paper, the focus is on 20% trimmed means, but in principle the method considered here is relevant to any situation where the Type I error probability of the individual tests can be controlled reasonably well.