27th February 2012 at 8:56 pm #2595David MorganParticipant
TWO PROBLEMS WITH RANDOM SAMPLING AND TWO ALTERNATIVES.
David L. Morgan
I can’t count the number of time that I have read qualitative articles where the Discussion section begin with an “admission” that the study was limited by an inability to generalize the results. All too often, the implication is that we could have generalized, if only we had a random sample. As someone who took several courses on survey research, I want to argue that it is time we got past our obsession with random samples and generalizability. In particular, I will explain two issues in random sampling that making it inappropriate for the kinds of research, then I will suggest two alternatives that should guide our conversations instead.
Two Problems with Random Sampling and Generalizability
The first problem that we need to consider is whether we can meaningfully generalize data from a small sample, even if it is a random sample. Here is an example that I remember vividly from graduate school. One of my fellow students was giving a presentation about his dissertation, which involved five detailed case studies from a set of several hundred organizations. When he proudly announced that he had drawn a random sample, the professor in charge of our survey research program was astounded. “How could you possibly want a random sample with such a small sample? Your margin of error is enormous!”
What the survey researcher was saying is that you can’t get a usable estimate from a small random sample — where “small” means something like 50 or less. The formal terminology for this problem involves the margin of error around the results from a small sample, regardless of whether it was a random sample. Fortunately, this problem is easy to illustrate without any technical discussion.
Most of us have already encountered the margin of error in reports of polls on political campaigns. For example, suppose one candidate led another by fifteen percent in a poll a sample size of 500 and a margin of error of five percent.. To use the margin of error, you simply add and subtract it from the results that appeared in the sample. In this case, in this case, fifteen percent plus or minus five means that the candidate’s lead was somewhere between twenty percent and ten percent.
There is an intuitive source for this margin of error. Because the result comes from a sample rather than a count of the whole population, there is bound to be some potential for inaccuracy, and survey researchers. Another intuitive point is that smaller samples are going to be less accurate than larger ones, and there are formulas to determine this difference in accuracy. In my example, suppose that this statewide poll had relied on a sample of 50 instead of the 500. In that case, the margin of error is plus or minus thirteen percent, so what appears to be a fifteen point lead is actually somewhere between two percent and twenty-eight percent. Put simply, the accuracy of the sample drop rapidly as the size of the sample decreases.
Table 1 shows how inaccurate the estimates from small random samples are. The table is based on the standard assumption in survey research, that the value from some characteristic is evenly split in the population as a whole, so that margin of error is based on a “true” value of fifty percent. Gender would be an obvious example, so that we would be drawing a random from a population that is half men and half women.
Sample Size Margin of Error Range for Estimate
5 44% 6 – 94%
10 31% 19 – 81%
20 22% 28 – 72%
30 17% 33 – 67%
40 15% 35 – 65%
50 13% 37 – 63%
The fundamental point from this table is that even a random sample is almost useless for all practical purposes when that sample is as small as the sizes we typically work with in qualitative research. Given our goal of understanding the topics that we study in depth and detail, we absolutely need to focus on small samples. The cost of this choice, however, is the inability to generalize in any meaningful way, regardless of whether we have a random sample.
Let’s go back to my friend with his random sample of five cases. It should now be clear what the professor of survey research meant when he said that the margin of error would be “enormous”. But a detailed study of five cases is not that unusual in qualitative research. What would be unusual is a study that collected data from as many as 50 participants, yet we have just seen how poor our estimate would be even with a sample of size 50.
Now, on to the second problem with random samples which is much easier to summarize. The problem is that random samples would be impractical for almost everything that we actually study, regardless of whether we were doing a qualitative or a quantitative study.
In particular, to generate a random sample we need to be able to do two things. First, we need to be able to enumerate every member of the population from which the sample will be drawn. Second, we need to be able to contact every member of your potential sample in order to solicit their participation. Now think about the topics and the categories of participants that we study — how realistic would it be for any researcher, either qualitative or quantitative, to draw a random sample under those circumstances? So, given the kinds of things that we study, we are not alone in our inability to generalize.
Two Alternatives to Random Sampling and Generalizability
If we need to get over our problems with random sampling, what kinds of issues should we be talking about instead? I will make two suggestions. First, we all know (or should know) that the appropriate alternative to random sampling is purposive sampling. What we need to be able to do is give a strong defense for the choices we make when we use a particular strategy for purposive sampling.
Most of the time, our colleagues in qualitative research will know how to evaluate these choices, using the shared standards in our field. Sadly, too many quantitative researchers do believe in the magical properties of random samples, so the first thing to do is present them with the arguments against using random sampling. Based on what I said above, I would recommend something like the following wording: “Given both the large margin of error associated with a small sample size and the great difficulties in drawing a random sample from this population, we have chosen to use a purposive sampling strategy.” So, start by talking to quantitative researchers in their own language and explain to them in their own terms, why a random sample would be a poor choice, then educate them about the kind of purposive sampling that is especially appropriate for the current study.
My second suggestion involves what we ought to be talking about in our conversations with other qualitative researchers. If we are going to drop the routine statements about lack of generalizability from our Conclusions sections, then what should replace it? In my opinion, Lincoln and Guba (1985) supplied the answer to this question: We should be thinking about transferability rather than generalizability.
Too often, we rely on the implicit assumption that our results must have some relevance for other settings and other groups, rather than addressing the factors that might increase or decrease transferability. We almost never think that our research is just about these particular people in this particular setting; more often, we think that our results have broader implications. Thus, when we talk about the limitations of our research, we should speak in terms that matter to other qualitative researchers (as well as practitioners who might want to make use of our research).
The bottom line is that we need to change the conversation. We need to spend far less time worrying about our inevitable problems with random samples and generalizability. Instead, we need to concentrate on the things that really matter to us as qualitative researchers, namely purposive sampling and transferability.5th March 2012 at 9:21 pm #2597Roger GommMember
Lincoln and Guba are a major source of misunderstandings in methodology. In the paper making their absurd claim that ‘the only generalisation is that no generalisation is possible’ they (rightly) argue that one form of generalisation cannot be derived from qualitative research/ small samples, using much the same argument as David Morgan. This is empirical generalization, which is the drawing of frequency estimates about a population from the study of a sample.Doing this accurately is not possible with small samples. But hardly any qualitative researchers hazard such generalisations. What they more usually do is to propose generalizations of a different kind: theoretical generalisations, which take the form of a proposition such that what happened in the case studied will very likely also happen in other (as yet) unstudied cases with the same characteristics.Lincoln and Guba pull a trick /or demonstrate their ignorance by failing to identify this as ‘generalization’. They then call it ‘transferability’ and are able thus to claim that ‘no generalization is possible’.
On a broader front, of course if ‘no generalization’ were possible we wouldn’t be able to use the same words for different instances; for example calling one organisation a ‘factory’ entails matching it against some criteria which identify factories in general.
Of course even though it is possible to propose theoretical generalisations/transferability from one or a small number of cases, such generalizations have to be confirmed by the study of a much larger number. Specification rather than generalization then becomes a problem in deciding what exactly are the conditions under which the theoretical generalization is true; which is the same as asking what exactly is the population of cases to which the proposal will transfer.20th March 2012 at 9:03 pm #2596David MorganParticipant
First of all, I prefer to limit the term “generalizability” to the use of representative samples from well-defined populations. Perhaps I wasn’t clear enough about that, but my main point was that small samples are seldom worth the trouble, regardless of whether they are theoretically generalizable.
I’m certainly not alone in arguing for this carefully limited form of generalizability. In particular, Shadish, Cook & Campbell make much the same argument in their book on Experimental and Quasi-Experimental Designs. The point here is that issues of generalizability are hardly limited to qualitative studies, since most experiments can also be viewed as N = 1 case studies.
With regard to the concept of transferability, I definitely prefer this approach as a way to think about the relevance of qualitative results. Of course this kind of reasoning is speculative, but the basic point is to “generate hypotheses” and let others determine their relevance. The same goes for high-level quantitative research such as Random Controlled Trials, which may maximize internal validity at the expense of external validity, so that their transferability is likewise limited.
Finally, I’ll give one example of transferability with regard to surveys. Is a study done with a random sample in Britain generalizable to the U.S.? Certainly not, because these are two different populations. But could the results be transferable? Depending on the subject matter, there would be stronger connections in some cases than others, and it would be up to scholars in the U.S. to determine the relevance (i.e., “transferability”) of those results for their own situation.
- The forum ‘Default Forum’ is closed to new topics and replies.