- This topic has 9 replies, 3 voices, and was last updated 5 years, 1 month ago by
.
-
Posts
-
Do you find it hard to understand statistical analysis, or perhaps never really thought about it? Maybe the problem does not lie with you.
Everything you were taught about analysing numeric data (the ‘quantiative’ paradigm) was probably wrong. See this new open access paper:
http://onlinelibrary.wiley.com/doi/10.1002/rev3.3041/abstract
With more journals banning or restricting the use of significance tests there is a chance to make some progress in 90-year-old struggle to get scientists and social scientists to wake up to the reality of probability calculations.
“What do we instead?” I hear you ask.
A bit of this:
http://www.workingoutwhatworks.com/en-GB/Magazine/2015/1/Trustworthiness_of_research
A bit of this:
https://www.dur.ac.uk/resources/education/Numberneedtodisturb.pdf
But mostly nothing at all. See this:
https://www.dur.ac.uk/resources/education/research/Findingswithandwithoutsigtests.pdf
and this:
https://www.dur.ac.uk/resources/education/research/Regressionwithandwithoutsigtests.pdf
I attended an interesting presentation on this topic at the 2015 American Evaluation Association conference by Bill Pate. The best approach seems to be to include a significance test if appropriate for the data, but also use other ways of determining if your results are significant and meaningful.
No. This Bill Pate is completely wrong. Just ditch the sig tests. You don’t know what they mean. Nor do I. Nor does Bill Pate.
Stephen,
I’m curious what makes you say we don’t know what the statistical significance tests mean? Tell me more. Thanks!
Bernadette
My understanding is that a significance test tells you the probability that you would get the results you got if your null hypothesis is true.
By the way, I just saw this American Statistical Association statement on p values announced on another forum this morning. http://www.amstat.org/newsroom/pressreleases/P-ValueStatement.pdf
And what use would that probability be? What do you do with it in your studies?
ASA has been fudging this since they nearly banned p values in 1999. They are making some progress against a lot of uninformed opposition.
Great question. Come to think of it, in the few studies I’ve been involved in where we’ve included p values (or statistical significance tests), I can’t think of any way that the p values were useful or anything that we did with that information other than report it in case anyone asked. Basically, the p values didn’t tell us anything we didn’t already know from eye-balling the data. Heh.
I certainly agree that oftentimes too much emphasis is put on testing statistical significance and not enough emphasis on other aspects of research quality. By the way, useful links in your original message, thank you.
Speaking for myself only…
Thanks for the articles Stephen. I have just joined the forum.
I have always had a question about sample size estimation and power analysis that might be related to what you have posted.
In sample size estimation using power analysis (let’s say for a t-test) we specify the gain or change in the scores that makes sense to us (or effect size instead) and Type I and Type II error, etc. and then estimate what sample size we need to be able to reject the null hypothesis if the gain we have specified is observed. Isn’t this fooling ourselves?
If the gain that makes sense is observed then the intervention is effective. Why should we insist on having a large sample just to be able to show that gain is statistically significant? Suppose we have observed the gain but it’s not statistically significant. Is it really important?
I don’t say that large samples aren’t good. I mean what’s the point in selecting a large sample just for the purpose of obtaining a statistically significant result. Because we always know that with larger sample the result will be significant.
Hans
Hi,
I think what you say is substantially correct. ‘Power’ calculations are predicated on sig tests. If we reject use fo sig tests we must also reject power calculations. It is still the case that the larger a sample, ceteris paribus, the better the study and the more trustworthy the findings. But power no. It is contradictory surely. It argues if there is an effect of non-zero size (x) we need this many cases (y) assuming, as null hypothesis, that x is zero to find x!
- You must be logged in to reply to this topic.
