Post hoc testing (p-value adjustment) when exploring unexpected patterns

[Andrea Martin]

Hi,

I am running a study on 8 subjects. Before testing all 8 subjects were assumed to be drawn from the same population. However, subjects 1-6 and subjects 7-8 show very different results in my data. I also have reason to believe that there is a small theoretical possibility that they are from different populations. I have therefore run my analysis (friedman’s ANOVA) in three ways; (1) on all subjects, (2) on subjects 1-6, and (3) on subjects 7-8.

Since the last two statistical tests are post-hoc, do I need to adjust the p-value accordingly? If so, how do I do this?

[A Yu]

I assume these are data from a repeated- measures design? Would suggested Wilcoxon T test for the post-test.

[Jerry Frieman]

Another alternative is to use a randomization test. Here you do not have to make any assumptions about populations. An excellent source is Dugard, File, & Todman’s Single-case and Small-n Experimental Designs, Second Edition (2012)

[A Yu]

That’s interesting. Thanks!

[Anonymous]

R provide a function for multiple comparison after friedman-test:

http://rss.acs.unt.edu/Rdoc/library/pgirmess/html/friedmanmc.html

Or you follow this algoithm:

1) If you don’t know which groups are different:Do the friedman test with all groups. If it shows significance: test the single groups against each other with Wilcoxon-signed test (wst) and divied p-value through the number of wst you want to perform (Bonferroni correction).

2) If you know already which group you want to test: test like in 1) but without Bonferroni correction.

[Author]

Posts