Quantitative research

Add a Comment

Comment by MJ on May 17, 2012 at 13:11

Hi everyone,

I am doing ROC Curve analysis to establish cut-off points for a scale. I calculated Youden's index to establish cut-off points, however to my despair, the Youden's index is equal for two of the scores:

Let's say my subjects could score between 10-20 on a scale and the Youden's index is the highest at 124.5 for scores 15 and 16 on this scale. I wonder what would be the best cut-off: 15 and 16?!
obviously sensitivity is higher on one score and the specificity for the other

Thank you for your help

 

Comment by Oscar on April 17, 2012 at 7:22

well, there's a very simple reason for that: because that is not our purpose. methods and techniques developed by statisticians usually trickle down into other sciences to inform their practice, but then there's this entity called "methodologist" which acts a "knowledge translator" of sorts trying to shape methods so that they become adequate to their substantive area of research. the problem is that methodologsists in the social/behavioural/health sciences are rarely trained in formal mathematics (with some notable exceptions of course, such as biostatisticians). most people go through the standard graduate-level courses on applied statistics which emphasize software use, with almost exclusive reliance on hand-waving arguments and metaphors to try and explain what goes on the little black box we call SAS, SPSS, JMP, Minitab, etc...  as a result, misconceptions generate misconceptions that perpetuate themselves over and over again until one looks at the literature in disbelief of what people are actually doing.

the statistical science has become a cornerstone of the research enterprise, creating a demand of people proficient on these methods. nevertheless, the number of PhDs in statistics (whether pure or applied) is still relatively small (math has never won the popularity contest of things people like to study) so simply out of sheer need statisticians "delegate" the tasks of data analysis to methodologists, quantiative analysts, etc who may or may not be able to fully grasp what they're supposed to be doing.... and as you can imagine these people later become professors or instructors who teach the same things to their students who then add a little mix of their own and that keeps on rolling down so we end up having (even in published, reputable introductory textbooks in statistics) strange claims such as:

- regression needs all variables to be normally distributed (WRONG, the distributional assumptions on any linear model are only on the RESIDUALS)

- analysis of covariance using the pre-test as covariate takes care of pre-existing conditions to make the groups you are comparing more similar (WRONG, ANCOVA cannot fix lack of randomization regardless of how many times people say so)

- a statistically significant ANOVA implies significance of the post-hoc tests (WRONG, ANOVA can very well detect a linear combination among the predictors that is not expressively tested in the usual Tukey's or whatever things SPSS spits out as an output)

.

.

.

.

and dont even get me started on hypothesis testing!! although Andy has already written a couple of posts that address this issue in a much more detailed way

Comment by MJ on April 16, 2012 at 22:16

Hi that is very true, as you said statistician publish the facts for themselves while clinicians want you to do things that are outdated

I wonder why statisticians go through too much pain to develop new methods then if they are not going to be used at some point in medical research?!

Comment by Oscar on April 16, 2012 at 18:54

it seems like that's what (s)he's after. it would have been nice for you to do a power calculation before and include that in your stats section. now if you did it and you did not include it i think it would be good for you to do so. but it seems like this person is attempting to use power analysis as a tool for data analysis through these post-hoc/post-experiment power calculations that for reasons beyond my understanding are very popular...

 

... in any case, i would skip the question. when i've been faced with questions like that i direct them towards that citation or quite a few other ones that deal with the same issue. i also tend to casually comment that it seems interesting to me that journals that deal with applied research methodology tend to advocate this procedure sometimes (because there are published articles out there that think this is still worthwhile to do) whereas the journals where statisticians or mathematical statisticians publish (American Statistician, Journal of the American Statistical Association, Biometrika, etc.) uniformly have condemened doing this over the years. food for thought i think.

 

in any case, yes, GPower should be able to handle it...i think. i havent used it in a while, heh. :)

Comment by MJ on April 16, 2012 at 13:27

Hi Oscar

Thanks for your attention. I quickly looked at the paper, so the reviewer wants me to do a "post-experiment" power calculation, which this paper calls "abuse of power". Is that what you mean

does that mean I can just skip his question, so if I ever want to do this I would have to use the GPower right

thanks

Comment by Oscar on April 16, 2012 at 7:31

Hoenig, J.M. & Heisy, D. M.(2001) The Pervasive Fallacy of Power Calculations for Data Analysis. The American Statistician, 55(1), 19-24.

 

 

      so yeah, i added the extra bolded and underlined terms so you can kindly direct whoever told you that to a very well-written paper where both analytic and simulation results show that what this person is asking you to do is essentially a waste of time. this is once again one of those situations where people who dont understand how things work mindlessly repeat misconceptions over and over again in some bizarre attempt to make them sound legit.

Comment by MJ on April 15, 2012 at 22:34

Hello, I am back with another Power question!
This time I have been asked a general power question and I am really confused about how to do this. here is the Q:

Given your small study size, and the fact that adherence to physical activity was low in both cases and controls, a type 2 error could occur. A power calculation should therefore be provided in the statistical methods section. 

 

Comment by Oscar on April 3, 2012 at 18:38

the interaction seems like a more straight-forward idea to me as well... i guess it also depends a little bit on what the person who's asking you to perform this may have in mind... i mean for me, from a very pragmatic point of view, i would only care to see whether there is a differential effect of eating habits by gender in its ability to predict the logit-odds of cancer risk... but if this person has some sort of theory in mind that says that, i dunno... smoking should also be considered then you'd need to test eating habits*gender and eating habits*smoking status... and the more (s)he would like you to control for to test for the homogeneity of linear trends the more interactions you're gonna need to include and the more you're gonna start running into weird issues of multicollinearity or unequal-contributions across predictors and so on...

 

....so i guess fingers-crossed i'm thinking (hoping) just one interaction should be enough for him/her... if (s)he wants more then i guess the simultaneous contrasts would be the way to go...

Comment by MJ on April 3, 2012 at 17:24

Thanks alot Oscar I will order that from library and will see what I can do,,,I may end up doing a simple interaction analysis thought

:)

Comment by Oscar on April 3, 2012 at 17:16

well... i really, really like my "Experiments: Planning, Analysis and Optimization" by Wu & Hamada published on the Wiley's series collection in probability and statistics. i rarely use any other book outside from that one anymore...

 

if you're familar with SAS you may wanna check out the 'multicomp' R package (i'm an R user) and you should be able to translate their code for contrasts into SAS, in case you need some reference there...

• ‹ Previous
• 1
• 2
• 3
• …
• 10
• Next ›
• Page