I am running a study on 8 subjects. Before testing all 8 subjects were assumed to be drawn from the same population. However, subjects 1-6 and subjects 7-8 show very different results in my data. I also have reason to believe that there is a small theoretical possibility that they are from different populations. I have therefore run my analysis (friedman’s ANOVA) in three ways; (1) on all subjects, (2) on subjects 1-6, and (3) on subjects 7-8.
Since the last two statistical tests are post-hoc, do I need to adjust the p-value accordingly? If so, how do I do this?
Another alternative is to use a randomization test. Here you do not have to make any assumptions about populations. An excellent source is Dugard, File, & Todman’s Single-case and Small-n Experimental Designs, Second Edition (2012)
1) If you don’t know which groups are different:Do the friedman test with all groups. If it shows significance: test the single groups against each other with Wilcoxon-signed test (wst) and divied p-value through the number of wst you want to perform (Bonferroni correction).
2) If you know already which group you want to test: test like in 1) but without Bonferroni correction.