Comparing two sets of ordinal data (4-likert scale)

[Niveen Elshawa]

Hi, every one.

I am a PhD candidate conducting a quantitative study. I used two questionnaires (4-likert scale) to collect data;  one for  the assessment beliefs  of tertiary English instructors and the other for their assessment practices. The two 67 item- questionnaires are identical item by item (e.g. An item in the beliefs questionnaire > Assessment helps to focus teaching > is paralleled by an item in the practices questionnaire > I use assessment to help focus my teaching>.

According to my readings, both are ordinal data and they are the DVs in my study. 

My question is:

What is the appropriate statistical test to use to answer my second research question:

 To what extent do English language instructors’ assessment beliefs correspond to their practices?

I do really appreciate your help.

Thank you.

[Stephen Gorard]

Again – you are making a widespread mistake encouraged by poor methods resources/teaching. There is no test to assess how well your data on beliefs and practices match, and you do not need one. 

Simply compare the matched responses on the two sets of items. Perhaps use crosstabs with percentages (no idea how many pairs there are). 

[Dave Collingridge]

Generally speaking Likert scales are ordinal if they follow the usual SD  D   N   A   SA format. In this case they should be analyzed with nonparametric statistics. The exception is when multiple questions are aggregated (find the mean response for multiple items). It is statistically acceptable to use more powerful parametric statistics when comparing aggregated responses. But before aggregating questions one must make sure that the questions being aggregated load onto the same factor using principal components analysis. PCA will ensure that questions being aggregated actually measure the same underlying construct.

Likert scales that are anchored on both ends like this SD   |      |      |      |     SA are interval and may be analyzed individually with parametric statistics. People often analyze ordinal data with parametric statistics and some reviewers don’t seem to care. It is probably fine but not correct strictly speaking in a statistical sense. 

[Seth Ansah]

You can use Spearman’s correlation coefficient. 

[Niveen Elshawa]

Thanks Prof. Stephen for your response.

 Actually, I used percentages to report the results for my first two questions:

1.What are the assessment beliefs of English language instructors?

2.What are the assessment practices of English language instructors?

 However, I think that comparing the matched responses using cross-tabs with percentages would be a tedious process since I have 67 pairs altogether. I need to report the results according to the constructs of the study. 

What do you think Prof.?

Waiting for your feed forward.  tq

[Niveen Elshawa]

Thanks Dave for your clarifications.

Since I plan to aggregate multiple questions in my four-section two-questionnaires, then it would be fine to use parametric statistics. But, what is the suitable test to compare the two sets of items? 

Can I use parametric statistics with a sample of only 83 respondents? 

Waiting for your feed forward. tq

 

[Niveen Elshawa]

Thanks Seth for your suggestion.

Do you think Spearman’s correlation coefficient is suitable when my data is normally distributed? 

As far as I know, Spearman’s correlation coefficient should be used if the data is NOT normally distributed. Otherwise, Pearson correlation coefficient should be used, instead.

Correct me if I am mistaken.

tq

[Stephen Gorard]

I am very suspicious of ‘constructs’ – see my reasons, attached. Spearman’s Rank CC (as suggested below) would be fine. It does not depend on the distribution of the data. But ignore anything to do with sig testing and p values. Just report the ‘effect’ sizes. However, you will lose info. If the 67 items matter (and I can’t believe you really asked people 67 things!) then you have to analyse the 67. Crosstabs is better, easy to understand and makes no assumptions. You can’t really go wrong. 

For future reference, work out how to analyse responses before collecting data to ensure that you collect it appropriately. It is a bad sign to finish data gathering and only then think what to do with it. See my book on Research Design. 

[Seth Ansah]

Hi Niveen, 

You mentioned earlier that your data is ordinal, which is non-parametric,  and as such normality of distribution doesn’t matter. It should be Spearman’s CC (Coolican,  1994: p378).

[Niveen Elshawa]

Thanks Seth for your clarifications.

Since I plan to aggregate multiple questions in my four-section two-questionnaires, then it would be fine to use parametric statistics,right?

But, can I use parametric statistics with a sample of only 83 respondents? 

 tq

[Niveen Elshawa]

Thanks again Prof. Stephen for your guidance.

Since I plan to aggregate multiple questions in my four-section two-questionnaires, then would it still be fine to use Spearman’s Rank CC (non- parametric)?

Can I use parametric statistics with a sample of only 83 respondents? 

Waiting for your feed forward. 

Thank you in advance.

 

[Stephen Gorard]

Did you read my chapter about the dangers of aggregation like this? You can ‘use’ parametric’ with 83. But they have to be selected completely at random with no non-response and no missing data. My guess is that you do not have that. So ditch the idea of a ‘test’.

Really – what did you plan to do when you collected this data or did you not bother to think ahead?

[Seth Ansah]

I agree with Stephen on randomly sampling 83. On the other hand, if the questionnaire items are measured at ordinal level (nonparametric), why do you want to aggregate them into constructs which are parametric? 

[Author]

Posts