The matter of “generalizability” in qualitative researches

Home Forums Default Forum The matter of “generalizability” in qualitative researches

Viewing 15 posts - 1 through 15 (of 16 total)
  • Author
    Posts
  • #5468

    Hello everyone,

    Generalization or generalizability is one of the most important aims of any research; however you find that in qualitative researches no one expects it. As a journal reviewer I have found that some researchers state the generalization matter as a limitation for their studies. As you know this is one of qualitative researches internal characteristics not a limitation.
    Is not it odd to you?
    Please, let me know your perspectives on this issues.

    Thanks

    #5483
    Katie Metzler
    Participant

    An interesting topic… This quote on the difference between generalization in qual and quant research comes from HORSBURGH, D. Evaluation of qualitative research, Journal of Clinical Nursing 2003; 12: 307–312

    ‘Whilst it is often argued that generalizability is not the
    purpose of qualitative research, Morse (l999a) states that
    if qualitative research is not considered to be generalizable,
    then it is arguably of little use (and is unlikely to be
    funded). She compares quantitative and qualitative
    approaches, in order to differentiate between the concept
    of generalizability as applicable to each. In quantitative
    work generalizability is statistical, i.e. the study sample is
    matched to the study population at large to ensure
    comparability of demographic characteristics and, if this
    is done correctly, then it is assumed that the findings
    from the sample are generalizable. In qualitative work,
    however, participants are selected by means of theoretical
    sampling, i.e. for their ability to provide information
    (and consequent theory development) about the area
    under investigation. Situational, rather than demographic,
    representativeness is what is sought. It may be said,
    then, that generalizability in qualitative research refers to
    the extent to which theory developed within one study
    may be exported (K.M. Melia, personal communication)
    to provide explanatory theory for the experiences of
    other individuals who are in comparable situations. This
    position is supported by the comments of Popay et al.
    (1998) on the subject of generalizability, who emphasize
    that, ‘…the aim is to make logical generalizations to a
    theoretical understanding of a similar class of phenomena
    rather than probabilistic generalizations to a population’’.

    Some other interesting papers on this issue!!
    http://soc.sagepub.com/cgi/content/abstract/39/2/295

    http://www.nova.edu/ssss/QR/QR4-3/myers.html

    #5482
    Nadia Ahouari_Idri
    Participant

    Dear Mojtaba:
    To the best of my knwoledge, qualtitative research cannot be generalisable because it treats data qualitatively. That is, the sample generally is samll because the researcher needs detailed information about a number of individuals/group. Hence, the sample’s historical/social/cultural background are to be different. I give here the example of anxiety in EFL. We can conduct research using the interview as a tool of data collection. Let’s take Egypt, Algeria and Japan. Results from the interview are more likely to be different in such different situations. Egypt is a welfare country where English can be learned easily compared to Algeria. The latter are more likely to have difficulties because they can be influenced by the French culture. However, both samples can learn English better than Japanese whose culture, societ, etc are totally different from the anglosaxon ones. My point here is that the three samples can relate anxiety to socio-cultural and affective considerations. My question, then, is: Can the results found in Egypt generalisable? If yes, how can different experiences, societies and cultures give the same data? i don’t think so.
    That’s why, I believe that research in human and social sciences is qualitative in its nature, but it’s better to use mixed methods and the quantitative method to generalize. Add to this the aspect of validity and reliability. The same research should be undertakedn several times, in different contexts and the analysis should be treated by different researchers before clainming its generalisability.
    All the best:
    Nadia

    #5481

    Dear Katie

    Thank you for the comments. And how nice it is to hear from a person who is familiar with nursing literature. Yes, you are right. Those researchers interested in qualitative research are not required to think on genaralisability as expected in quantitative one.

    Please read below scrap from a book:

    Generalizability and transferability
    A final issue in qualitative analysis is that of generalizability. Generalizability
    refers to the extent to which findings from a study apply to a wider population
    or to different contexts. In a sample survey, random sampling allows generalizability
    through the principle that the study sample is likely to be statistically
    representative of the larger population of interest, so findings can be extrapolated
    to that population. In qualitative work, study participants are rarely
    randomly sampled in this way, and the logic of generalizability is rather different.
    Some argue that it is not an issue in qualitative work, which properly
    aims to provide ‘thick’ description, or to address particularities, rather than to
    provide ‘typical’ accounts or generalizable findings.
    There are, though, two reasons why researchers in the field of health do
    have to address the issue of generalizability. First, if researchers are to make
    claims to their findings being useful, at whatever level, to health practice, they
    do have to consider the theoretical import of their findings: the extent to
    which they refer to some setting or population wider than that of the research
    itself. Second, and more pragmatically, the credibility of qualitative findings in
    non-social science fields is often fragile, and qualitative research is easily marginalized
    as ‘interesting, but not research evidence’ because the generalizability
    is questionable. Addressing these concerns does not mean adopting, or even
    adapting, the procedures of quantitative approaches and attempting to imitate
    the kinds of random samples drawn, or comparing the study population to
    wider ones. Instead, it involves thinking through what kind of relationship the
    study findings have to other populations and settings, and unpacking exactly
    what inferences can be drawn from the data analysis.
    Essentially, though, these all refer to the same question that generalizabiltity
    addresses in quantitative work; namely, how far can the findings of this
    particular study be extrapolated? There are various ways in which findings
    from qualitative work can be supported as more widely relevant. These
    include:
    , ……

    I uploaded the file of the book in here for you to give you the chance to read all information on generalisability presented in the book.

    I hope you find it useful. Thank you for the file you kindly uploaded for me.

    With my best regards.

    Siavash
    A PhD candidate of nursing from Persia.

    #5480

    Dear Nadia

    Thank you for the comments. I am in agreement with you on the matter of generalisability in qualitative research.
    I hope you find the book uploaded in response to the previous comment helpful.
    Please, do not forget that Mixed methods design is employed to increase our depth of understanding on a subject rather provide us with the opportunity to increase the chance for generalization.
    The method has been introduced to bridge the gap between qualitative (naturalistic paradigm) and quantitative (experimental) designs.
    The discussion is lengthy and the related materials should be read by yourself.
    Please read below book in order to find more on Mixed methods design:

    Mixed Methods Research for Nursing and the Health Sciences
    Sharon Andrew, Elizabeth J. Halcomb
    ISBN: 978-1-4051-6777-2
    Paperback
    248 pages
    January 2009, Wiley-Blackwell

    With my regards

    Siavash
    A PhD candidate of nursing from Persia.

    #5479

    Hello Mojtaba,
    A research in any field else than literature is a process of understanding as how things/systems work. This adds to over understanding and we would be able to build upon this understanding. It was simply in modern art, literature etc any creation was for self satisfaction and only those who shared the same experience could understand and explain it. But, it is not with sciences, whether social or hard sciences, we have to make our experiences and findings general. A research without generalisation may be a piece of literature not science.
    However, in social sciences as empirically rigorous as “economics” findings and generalisation based on them is generally no more than recreation of a situation which we perceive to be true. Generalisation in any science depends on his religious and political ideology, and also “pretended” level of objectivity. Give any set of social or economic data to two ideologically diametrically opposite experts, their finding, conclusions, hence, generalisations are bound to be diametrically opposite. Hard sciences, sometimes, are no exception to this.
    However, still researchers in all fields tend to generalize. So far as, qualitative research is concerned I don’t think it is worthless but so far as generalisability is concerned, it is more true in this case than empirical research, ” beauty lies in the eyes of beholder”.
    It is not to say that generalisation can not be made in qualitative research. Examples of Marx and Weber are eminent ones. Both of them mostly using qualitative methods not only generalised but also gave certain laws which are still valid in many cases. It should be remembered that even in hard sciences like physics, “laws” need not be unassailable.
    In qualitative research, worth of generalisation depends on the researcher,s understanding of the object of research, be it nursing, his/her critical thinking and ability to deduce from his/her experience or experiences of others who have gone or going through same or similar experiences. “Deduction” as against “induction” is key of generalisability of qualitative research.

    #5478

    Hello Mojtaba,
    A research in any field else than literature is a process of understanding as how things/systems work. This adds to over understanding and we would be able to build upon this understanding. It was simply in modern art, literature etc any creation was for self satisfaction and only those who shared the same experience could understand and explain it. But, it is not with sciences, whether social or hard sciences, we have to make our experiences and findings general. A research without generalisation may be a piece of literature not science.
    However, in social sciences as empirically rigorous as “economics” findings and generalisation based on them is generally no more than recreation of a situation which we perceive to be true. Generalisation in any science depends on his religious and political ideology, and also “pretended” level of objectivity. Give any set of social or economic data to two ideologically diametrically opposite experts, their finding, conclusions; hence, generalisations are bound to be diametrically opposite. Hard sciences, sometimes, are no exception to this.
    However, still researchers in all fields tend to generalize. So far as, qualitative research is concerned I don’t think it is worthless but so far as generalisability is concerned, it is more true in this case than empirical research, “beauty lies in the eyes of beholder”.
    It is not to say that generalisation can not be made in qualitative research. Examples of Marx and Weber are eminent ones. Both of them mostly using qualitative methods not only generalised but also gave certain laws which are still valid in many cases. It should be remembered that even in hard sciences like physics, “laws” need not be unassailable.
    In qualitative research, worth of generalisation depends on the researcher’s understanding of the object of research, be it nursing, his/her critical thinking and ability to deduce from his/her experience or experiences of others who have gone or going through same or similar experiences. “Deduction” as against “induction” is key for generalisation in qualitative research.

    #5477

    Hello dear Prof. Mohammad Firoz Khan

    Thank you for the comments. Your kind words show me that you believe in experimental paradigm not the natural one. Just for your knowledge, I am a nurse and studied my major which is based in experimental paradigm. However in the PhD degree I got familiar with qualitative researches and natural paradigm. I reached to the conclusion that our world is not consisted of just sensible facts, something that can be sensed. The majority of facts has natural and insensible quality. Interestingly, theses kinds of facts direct us to sensible and experimental realities. Also, I found that our world is so narrow in the eyes of an experimental scientists.

    We need not to think of generalisability of qualitative researches. This is a fallacy presented by experimental philosophers that generalisation should be followed by qualitative researchers if the research is to get worth.

    Qualitative researchers need not to follow the rules of experimental paradigm, but they think of some other characteristics of their research, which can not be found in experimental designs.

    To encounter the criticism of quantitative researchers, The word “transferability” was suggested. as explained in the previous comments.

    We should not forget that the model and theory made by qualitative researchers give the direction to quantitative researchers to make a hypothesis and conduct a research. As you can see qualitative research feed quantitative research, and some criticism of gerenalisability, etc does not reduce the qualitative researches worth.

    #5476
    Dr Lubna Almenoar
    Participant

    Generalizability is only solid when the sample taken is highly representative of the larger group as a whole. If not, its not even right to generalize. That’s why there is qualitative and quantitative types of research where a good researcher knows which one to go for because the results are going to be valuable to any kind of problem solution process.

    #5475
    David Morgan
    Participant

    Transferability is more relevant than generalizability

    As soon as someone mentions the word “generalizability,” that creates images of statistical generalization from samples to larger populations, but that is not the goal that we are pursuing in qualitative research. In particular, when we do applied qualitative research, one of our key goals should be to make that research as relevant as possible. For that reason, I far prefer the term “transferability” rather than “generalizability” when we talk about what we have learned from our research.

    Transferability essentially refers to the other situations and contexts where we believe our results are most likely to be relevant and applicable. Of course, that kind of claim requires some specific justification, as well as an recognition that going beyond our own research sites and participants always involves some speculation. But if we don’t speculate about where our research will be most useful, then how we will get people to investigate its value for them?

    FYI, to the best of my knowledge, therm “transferability” originated in Lincoln & Guba’s 1985 book on “Naturalistic Inquiry.”

    #5474
    David Morgan
    Participant

    A second comment, this time on the statistical requirements for generalizability

    As someone who took several courses on survey methods in graduate school, I am painfully aware of how little many qualitative researchers know about the concept of generalizability. First of all, lets start this discussion with the assumption that you do indeed have a random sample. Then, there are two main things that affect the generalizability of a sample: the size of the sample and the variance in the population. We can handle the variance issue with the intuitive recognition that the more diverse the population is, the larger the sample will need to be in order to capture that diversity effectively.

    But sample size is almost always the more important factor in generalizability, and as your sample size gets smaller and smaller, you drop below the point where you can any kind of useful generalization, due to the mathematics involved. Sample size below 50 are seldom accurate enough be worthwhile, and sample sizes below 20 are simply not worth the effort.

    And remember, everything I just said was based on the assumption that you have a random sample. So if a random sample of less than 20 is virtually useless from a statistical point of view, then that is a very strong argument for relying on a carefully thought out purposive sample as the most effective way to do any kind of research that involves “small Ns”.

    The biggest mistake that I see is the assumption that qualitative research could solve the whole problem of generalizability simply by using random samples. The confusion seems be between the inescapable fact that non-random samples are not generalizable, without the additional knowledge that small random samples are so inaccurate that they do not produce a useful level of generalizability.

    So, the text time some committee member tries to make a graduate student do a random sample on an N of 6 or 12 as a way to create generalizability, someone needs to tell that person as firmly as possible that they need to review the formulas on samples size and generalizability (or, in more technical terms, sampling error and the determination of confidence intervals).

    By the way, if you are wondering how much difference the POPULATION size makes, the answer is: surprising little. That is why a sample of 1500 people can be sufficient for a study of the whole U.S. population.

    And if anyone would like to read a somewhat more detailed discussion of these issues, they can check out the several entries on sampling that I wrote for the Sage Encyclopedia. of Qualitative Research Methods — and those entries are nice cross-referenced to each other.

    #5473

    This discussion seems to be very useful. I make a small request. Either the person who started this discussion or some learned professor may provide a basic definition of ‘qualitative research’. If it is difficult to define, give some description.

    #5472
    Paul Kiff
    Member

    Dear Mojtaba

    The point of qualitative methods is that they are not really designed for situations where you need or can possibly achieve generalisability.

    Such methods are used to acquire certain types of data.

    It is actually possible to achieve some genralisability from qualitative surveys provided that enough cases are collected but the very nature of qualitative methods (e.g in-depth interviews) makes them very labour-intensive and very susceptible to variation in interviewer performance.

    It is usually the case that a soundly designed quantiative survey requires a prior piece of qualitative data collection in order to determine what data is to be collected and what data classifications are to be used. e.g such research might tell you whether ethnicity or race is a data item you need to collect. It should also tell you what categories are appropriate.

    Qualitative methods can also be used to explore the meanings and interpretations to be placed on quantitative data analysis results and what analyses to undertake.

    Even highly scientific controlled experiments may require qualitative methods to be used to gather data to measure the impact of a trialled intervention.

    Best wishes

    Paul Kiff

    Graduate Student Research Skills Programme leader

    University of East London

    UK

    #5471

    I agree. This is an interesting discussion. However, there are a conceptual difference between “qualitative research” and “qualitative methods”. You can easily study qualitative data with quanitative techniques, i.e., statistical techniques such as latent class analysis (of Lazarsfeld) or logistic regression. In such techniques all data is qualitative (that is, discrete). You can also answer qualititative questions such as “Is the gender wage gap a result of the family workload?”. I believe that statistical techniques would be superior to qualitative techniques in answering such a question. But I am certainly not sure.

    Generally, then, I find the term “qualitative research” ambigious. Is the data qualitative, the method or the nature of the question? I rather prefer to focus on the research question: Do you have a hypothesis that deals with a qualitative question or a hypothesis about quantities? The subsequent exercise is to find out which method to use – in order to falsify the stated hypothesis. Thus my point is that one should start with theory. It is not the method that defines the analysis, it is – hopefully – the nature of the theoretical question that you want to answer.

    A particularly good discussion of this way of thinking is presented in a paper by nobel prize winner Herbert Simon, Rationality as Process and as Product of Thought (http://www.jstor.org/stable/1816653). Simon’s point is (among many) that most questions that are of interest to social scientists (including economists) are qualitative of nature.

    A brilliant sociological study that answers some “puzzling facts” in social mobility research is Boudon’s early work “Education, opportunity, and social inequality; changing prospects in Western society” from 1974, published on Wiley. Using quantitative techniques for qualititative data (mobility tables) he tries to show (among other things) that increased access to education (i.e., the democratization of education) does not necessarily lead to increased social mobility – a widely held belief in government circles and the academic world.

    Hope I added something to the discussion.

    #5470

    I consider the term ‘Qualitative Research’ in more than one way. The term may refer to any of the following:
    1. A research involving certain concepts, where the concepts are considered for their inherent relations using arguments consistent with the accepted notions of logical reasoning so as to explain and establish certain propositions. It may be noted that some of the concepts considered may be attributes having some qualitative characteristics.
    2. A research involving certain concepts in the form of attributes, which are measured using their characteristics that are qualitative from among respondents in a sample and results are inferred as in the case of usual quantitative methods.
    3. A research involving a sample survey, where the sample selected may not be a probability sample (Random sample) but a sample chosen out of convenience using judgment or otherwise. The qualitative nature of the study will refer to the quality of the sample.
    4. A research that is qualitative, i.e. an excellent work (Qualitative has a dictionary meaning of ‘having to do with quality).

    Considering the first one, it is to be understood that the topic of research, which generates a set of questions are to be answered in an appropriate sequence, constructing a reasonable (Logical) succession among the concepts so that the objectives are fulfilled using narration in a story telling framework. This may be stated as ‘substantiating and validating’ the associated hypotheses through arguments. In early days such a type of inquiry was called as ‘substantive’ research. Generally, one uses ‘deduction’ in an intelligent way to reach the goal. Naturally, there is no place for generalization as the results are deduced using deductive arguments.

    In the second case, the research with qualitative measurements is expected to be carried out in the manner in which a study based on survey research involving quantitative measurements. The problem of generalization will be the same as in the case of quantitative studies using inference with the accepted notions of errors. One has to choose appropriate methods, as the qualitative characteristics are related to attributes and not expressed in the form of numbers. If numbers are used to code them, they will be in Nominal Scale. A number of techniques have come into focus now for this purpose, like ‘Log-linear’ models. In many cases researchers stop with cross tabulation with two attributes. The same can be extended to include more categories and results obtained as in the case of quantitative research. There will be the same problems of generalizations as in a research using quantitative methods.

    The third one refers to a survey, where the sample selected will not be a random sample in the usual sense. They may be convenient sample, judgment sample, quota sample and the like. These studies may have problems of generalizations both when the concepts are measured quantitatively or qualitatively. The acceptability of results may not be universal. However, if the researcher is interested only in a segment and has a good judgment the results will be useful for his/her objectives.

    The fourth one is the one that is most desired in every study. A large majority of the studies, which are quantitative, use a wide spectrum of statistical methods as they are all available for a press of a button. Sophisticated software are available in many ways either for a price or for free. One has to repeatedly ask himself/herself before using these high end techniques, whether the basis for using such methods is known to them to produce qualitative research. A sitting with an experienced statistician before finalizing the methods will be highly rewarding.

Viewing 15 posts - 1 through 15 (of 16 total)
  • The forum ‘Default Forum’ is closed to new topics and replies.