Post hoc testing (p-value adjustment) when exploring unexpected patterns

Home Forums Methodspace discussion Post hoc testing (p-value adjustment) when exploring unexpected patterns

Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
  • #1608
    Andrea Martin


    I am running a study on 8 subjects. Before testing all 8 subjects were assumed to be drawn from the same population. However, subjects 1-6 and subjects 7-8 show very different results in my data. I also have reason to believe that there is a small theoretical possibility that they are from different populations. I have therefore run my analysis (friedman’s ANOVA) in three ways; (1) on all subjects, (2) on subjects 1-6, and (3) on subjects 7-8.

    Since the last two statistical tests are post-hoc, do I need to adjust the p-value accordingly? If so, how do I do this?

    A Yu

    I assume these are data from a repeated- measures design? Would suggested Wilcoxon T test for the post-test.

    Jerry Frieman

    Another alternative is to use a randomization test. Here you do not have to make any assumptions about populations. An excellent source is Dugard, File, & Todman’s Single-case and Small-n Experimental Designs, Second Edition (2012)

    A Yu

    That’s interesting. Thanks!


    R provide a function for multiple comparison after friedman-test:

    Or you follow this algoithm:

    1) If you don’t know which groups are different:Do the friedman test with all groups. If it shows significance: test the single groups against each other with Wilcoxon-signed test (wst) and divied p-value through the number of wst you want to perform (Bonferroni correction).

    2) If you know already which group you want to test: test like in 1) but without Bonferroni correction.

Viewing 5 posts - 1 through 5 (of 5 total)
  • You must be logged in to reply to this topic.