Home › Forums › Methodspace discussion › Comparing two sets of ordinal data (4-likert scale)
- This topic has 12 replies, 4 voices, and was last updated 6 years ago by
Stephen Gorard.
-
AuthorPosts
-
15th April 2015 at 2:35 am #612
Niveen Elshawa
MemberHi, every one.
I am a PhD candidate conducting a quantitative study. I used two questionnaires (4-likert scale) to collect data; one for the assessment beliefs of tertiary English instructors and the other for their assessment practices. The two 67 item- questionnaires are identical item by item (e.g. An item in the beliefs questionnaire > Assessment helps to focus teaching > is paralleled by an item in the practices questionnaire > I use assessment to help focus my teaching>.
According to my readings, both are ordinal data and they are the DVs in my study.
My question is:
What is the appropriate statistical test to use to answer my second research question:
To what extent do English language instructors’ assessment beliefs correspond to their practices?
I do really appreciate your help.
Thank you.
16th April 2015 at 11:55 am #624Stephen Gorard
ParticipantAgain – you are making a widespread mistake encouraged by poor methods resources/teaching. There is no test to assess how well your data on beliefs and practices match, and you do not need one.
Simply compare the matched responses on the two sets of items. Perhaps use crosstabs with percentages (no idea how many pairs there are).
16th April 2015 at 5:01 pm #623Dave Collingridge
ParticipantGenerally speaking Likert scales are ordinal if they follow the usual SD D N A SA format. In this case they should be analyzed with nonparametric statistics. The exception is when multiple questions are aggregated (find the mean response for multiple items). It is statistically acceptable to use more powerful parametric statistics when comparing aggregated responses. But before aggregating questions one must make sure that the questions being aggregated load onto the same factor using principal components analysis. PCA will ensure that questions being aggregated actually measure the same underlying construct.
Likert scales that are anchored on both ends like this SD | | | | SA are interval and may be analyzed individually with parametric statistics. People often analyze ordinal data with parametric statistics and some reviewers don’t seem to care. It is probably fine but not correct strictly speaking in a statistical sense.
17th April 2015 at 10:48 am #622Seth Ansah
MemberYou can use Spearman’s correlation coefficient.
17th April 2015 at 2:38 pm #621Niveen Elshawa
MemberThanks Prof. Stephen for your response.
Actually, I used percentages to report the results for my first two questions:
1.What are the assessment beliefs of English language instructors?
2.What are the assessment practices of English language instructors?
However, I think that comparing the matched responses using cross-tabs with percentages would be a tedious process since I have 67 pairs altogether. I need to report the results according to the constructs of the study.
What do you think Prof.?
Waiting for your feed forward. tq
17th April 2015 at 3:17 pm #620Niveen Elshawa
MemberThanks Dave for your clarifications.
Since I plan to aggregate multiple questions in my four-section two-questionnaires, then it would be fine to use parametric statistics. But, what is the suitable test to compare the two sets of items?
Can I use parametric statistics with a sample of only 83 respondents?
Waiting for your feed forward. tq
17th April 2015 at 3:28 pm #619Niveen Elshawa
MemberThanks Seth for your suggestion.
Do you think Spearman’s correlation coefficient is suitable when my data is normally distributed?
As far as I know, Spearman’s correlation coefficient should be used if the data is NOT normally distributed. Otherwise, Pearson correlation coefficient should be used, instead.
Correct me if I am mistaken.
tq
17th April 2015 at 6:59 pm #618Stephen Gorard
ParticipantI am very suspicious of ‘constructs’ – see my reasons, attached. Spearman’s Rank CC (as suggested below) would be fine. It does not depend on the distribution of the data. But ignore anything to do with sig testing and p values. Just report the ‘effect’ sizes. However, you will lose info. If the 67 items matter (and I can’t believe you really asked people 67 things!) then you have to analyse the 67. Crosstabs is better, easy to understand and makes no assumptions. You can’t really go wrong.
For future reference, work out how to analyse responses before collecting data to ensure that you collect it appropriately. It is a bad sign to finish data gathering and only then think what to do with it. See my book on Research Design.
18th April 2015 at 12:22 am #617Seth Ansah
MemberHi Niveen,
You mentioned earlier that your data is ordinal, which is non-parametric, and as such normality of distribution doesn’t matter. It should be Spearman’s CC (Coolican, 1994: p378).
20th April 2015 at 1:35 am #616Niveen Elshawa
MemberThanks Seth for your clarifications.
Since I plan to aggregate multiple questions in my four-section two-questionnaires, then it would be fine to use parametric statistics,right?
But, can I use parametric statistics with a sample of only 83 respondents?
tq
20th April 2015 at 2:45 am #615Niveen Elshawa
MemberThanks again Prof. Stephen for your guidance.
Since I plan to aggregate multiple questions in my four-section two-questionnaires, then would it still be fine to use Spearman’s Rank CC (non- parametric)?
Can I use parametric statistics with a sample of only 83 respondents?
Waiting for your feed forward.
Thank you in advance.
20th April 2015 at 9:17 am #614Stephen Gorard
ParticipantDid you read my chapter about the dangers of aggregation like this? You can ‘use’ parametric’ with 83. But they have to be selected completely at random with no non-response and no missing data. My guess is that you do not have that. So ditch the idea of a ‘test’.
Really – what did you plan to do when you collected this data or did you not bother to think ahead?
20th April 2015 at 9:56 am #613Seth Ansah
MemberI agree with Stephen on randomly sampling 83. On the other hand, if the questionnaire items are measured at ordinal level (nonparametric), why do you want to aggregate them into constructs which are parametric?
-
AuthorPosts
- You must be logged in to reply to this topic.