Methodological question – help needed – please!

[JanePowell]

I’m trying to work out a methodology for a series of usability tests. We’ll be comparing 6 different information systems against each other and against google using a number of different short tasks which respondents will perform on the systems presented to them. At the moment we’re hoping to use 36 respondents.

We want each respondent to perform 2 tasks on each system and we only have time to test 4 out of the 6 systems in each session + tasks we set using google.

I’m trying to work out how many tasks I need to set up so that each task is allocated to each system the same number of times e.g. 3 respondents have done task 1 on system 1, 3 have done task 2 on system 1 etc… If we didn’t have to use google in every session this would be straightforward but I’m really confused by having to use google each time.

My mind is boggling and I know there’s a lot of expertise out there if anyone can help me I would be truly grateful!

Thank you,

Jane

 

[Jeremy Miles]

Hi Jane

I think a repeated measures design will give you the most bang for your buck in terms of people, but it’s a tricky thing to analyse when you have data which are missing by design. Traditionally you’d use a repeated measures anova, but if you do that, you’ll get listwise deletion and no subjects left, so you need to use a multilevel model to analyze these data (or a generalized estimating equation).

If each person is going to do 4/6 tasks then you could assign randomly, but you haven’t really got enough people to ensure balance. Instead, you can say that each person misses two tasks, therefore combinations for missing are:

1, 2
1, 3
1, 4
1, 5
1, 5
2, 3

There are 15 possible combinations for missing, so I’d use 30 participants if I were you. Then I’d try to order them so everyone did a different order (google latin square designs to find orders).

Having google in each time doesn’t make much difference if you’re using multilevel. One additional thing is that if you have data that are missing by design, you know that your data are missing completely at random, so you know that the assumptions are satisfied.

Jeremy

P.S. Are you Jane from York? If so, hello.

[JanePowell]

Thank you so much Jeremy and everyone who has helped me with this. I’m not Jane from York by the way but hello anyway!

[Author]

Posts