qualitative research methods creswell

• Find My Rep

You are here

The Sage website, including online ordering services, may be unavailable due to system maintenance on August 17th between 2:00 am and 8:00 am BST. If you need assistance, please  visit our Contact us page .

Thank you for your patience and we apologise for the inconvenience.

Qualitative Inquiry and Research Design Choosing Among Five Approaches

• John W. Creswell - Department of Family Medicine, University of Michigan
• Cheryl N. Poth - University of Alberta, Canada
• Description

Supplements

"Creswell & Poth's text is clear, comprehensive, and compelling. The students loved this text."

"After almost 6 years of teaching this class using this book, I have yet to find a student who does not like it. This is unusual in textbooks, and it shows me that the writing is clear, yet at a level of complexity that matches my students' needs. My students consider this a kind of 'formula' book, where they can follow the examples and conduct their own study with some guidance from me. I respect what Dr. Creswell has attempted to do for instructors and students, and I appreciate the approaches he takes (both methodologically and text-wise!)."

"The writing is clear and to the point. The use of jargon is minimal. Focusing on five different types of research designs is a friendly way to introduce this topic to students interested in qualitative research."

"The vivid and extended examples of the different approaches to research that is a theme throughout the entire book is a huge strength. Those examples really help learning. In addition, the charts and tables which help the reader make distinctions among the approaches are helpful."

Creswell and Poth provide clear guidelines for qualitative research, the book is an ideal resource for developing the framework for a successful qualitative study and is easy to read.

For instructors

Select a purchasing option.

• Electronic Order Options VitalSource Amazon Kindle Google Play eBooks.com Kobo

Related Products

• Request new password
• Create a new account

Research Design: Qualitative, Quantitative, and Mixed Methods Approaches

Student resources, welcome to the companion website.

Welcome to the companion website for Research Design , Fourth Edition, by John W. Creswell. The resources on the site have been specifically designed to support your study.

Open Access Resources Include:

• Group Activities
• Suggested Studies
• Class Activities
• Study Questions
• Journal Articles with Activities

Instructors Can Log In to Access:

• PowerPoint® Slides
• Lecture Notes
• Course Syllabi
• Discussion Questions

About the book:

The eagerly anticipated Fourth Edition of the title that pioneered the comparison of qualitative, quantitative, and mixed methods research design is here! For all three approaches, Creswell includes a preliminary consideration of philosophical assumptions, a review of the literature, an assessment of the use of theory in research approaches, and reflections about the importance of writing and ethics in scholarly inquiry. He also presents the key elements of the research process, giving specific attention to each approach. The Fourth Edition includes extensively revised mixed methods coverage, increased coverage of ethical issues in research, and an expanded emphasis on worldview perspectives.

Acknowledgments

We gratefully acknowledge John W. Creswell for writing an excellent text. Special thanks are also due to Karen Thomas-Brown, Angela Heads, and Rosemary Gillett-Karam for updating and creating the ancillaries on this site

Disclaimer:

This website may contain links to both internal and external websites. All links included were active at the time the website was launched. SAGE does not operate these external websites and does not necessarily endorse the views expressed within them. SAGE cannot take responsibility for the changing content or nature of linked sites, as these sites are outside of our control and subject to change without our knowledge. If you do find an inactive link to an external website, please try to locate that website by using a search engine. SAGE will endeavour to update inactive or broken links when possible.

For instructors

Access resources that are only available to Faculty and Administrative Staff.

Want to explore the book further?

Order Review Copy

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Creswell, J. W. (2014). Research Design: Qualitative, Quantitative and Mixed Methods Approaches (4th ed.). Thousand Oaks, CA: Sage

The book Research Design: Qualitative, Quantitative and Mixed Methods Approaches by Creswell (2014) covers three approaches-qualitative, quantitative and mixed methods. This educational book is informative and illustrative and is equally beneficial for students, teachers and researchers. Readers should have basic knowledge of research for better understanding of this book. There are two parts of the book. Part 1 (chapter 1-4) consists of steps for developing research proposal and part II (chapter 5-10) explains how to develop a research proposal or write a research report. A summary is given at the end of every chapter that helps the reader to recapitulate the ideas. Moreover, writing exercises and suggested readings at the end of every chapter are useful for the readers. Chapter 1 opens with-definition of research approaches and the author gives his opinion that selection of a research approach is based on the nature of the research problem, researchers' experience and the audience of the study. The author defines qualitative, quantitative and mixed methods research. A distinction is made between quantitative and qualitative research approaches. The author believes that interest in qualitative research increased in the latter half of the 20th century. The worldviews, Fraenkel, Wallen and Hyun (2012) and Onwuegbuzie and Leech (2005) call them paradigms, have been explained. Sometimes, the use of language becomes too philosophical and technical. This is probably because the author had to explain some technical terms.

Related Papers

Daniel Ortiz

Research Design: Qualitative, Quantitative and Mixed Methods

Dr. Esra Öztürk Çalık

Conducting a well-established research requires deep knowledge about the research designs. Doing research can be likened to jumping into the sea which may transform into a huge ocean if the researcher is not experienced. As a PhD candidate and a novice researcher, I believe that the book "Research Design: Qualitative, Quantitative and Mixed Methods Approaches" by J.W. Creswell is a true reference guide for novice researchers since it is the most comprehensive and informative source with its reader-friendly structure.

International Journal of Social Sciences Interdisciplinary Research

Prashant Astalin

rhoda taller

Yan-yi Chang

John W. Creswell was previously a professor in educational psychology in the University of Nebraska–Lincoln. He moved to the University of Michigan in 2015 as a professor in the Department of Family Medicine. He has published many articles and close to 27 books on mixed methods. Professor Creswell is also one of the founding members of the Journal of Mixed Methods Research. He was a Fulbright scholar in South Africa in 2008 and Thailand in 2012. In 2011, he served as a visiting professor in the School of Public Health of Harvard University. In 2014, he became the Chairman of the Mixed Methods International Research Association. Professor Creswell has a personal website called “Mixed Methods Research” at http://johnwcreswell.com/. The site contains the information about his background, his own blog, consulting works and published books. He also posted replies questions from academic researchers and practitioners in the blog.

Chisomo Mgunda

Gloria Thakane Leutle

Richard Baskas, Ed.D. Candidate

Nurul Gasmi

To understand educational research, you now have the map (the steps that exist in the process of research) and the different paths you can take (quantitative and qualitative). Now we will explore some distinguishing features along the qualitative research design. These features are the research designs you can use to collect, analyze, and interpret data using quantitative and qualitative research. Some of the research designs may be familiar; others may be new, such as how these paths can converge with two designs called mixed methods research and action research. The discussion of designs will provide a more advanced understanding of educational research on your journey.

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED PAPERS

IRJET Journal

The SAGE Handbook of Applied Social Research Methods

Global Academic Journal of Linguistics and Literature

Dr. Lok R A J Sharma

IJIP Journal

Education for Health: Change in Learning & Practice

Kelly Devers

Proceedings of the Sixth International Conference on Technological Ecosystems for Enhancing Multiculturality

ANA MARIA PINTO-LLORENTE

International Journal of Academic Multidisciplinary Research (IJAMR

Ramatu Ussif

Kathryn Pole

iosrjournals.org

Presented at Academic Business World International Conference. Nashville, Tennessee. May, 2007

Greg L Lowhorn

Changsong Wang

Mid-Western Educational Researcher

Mixed Methods Research for Improved Scientific Study

Mette Baran

Bostley Asenahabi

Studies in Comparative International Development

Gerardo Munck

International Research Journal of MMC

Umesh Raj Aryal

Villia Jefremovas

Seda Khadimally

shiroashi fullbuster

Nova Southeastern University

Addisalem Tadesse

Sarah Baloyi

koek Comfort

Dhea Savella Segui

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

• Technical Support
• Find My Rep

You are here

The Sage website, including online ordering services, may be unavailable due to system maintenance on August 17th between 6:00 pm and 12:00 am PDT. If you need assistance, please  visit our Contact us page for further information. 

Thank you for your patience and we apologise for the inconvenience.

Five Qualitative Approaches to Inquiry

1. Narrative research

2. Phenomenology research

3. Grounded theory research

4. Ethnographic research

5. Case study research

VIEW APPROACH CompariSON

The Author Behind the Five Approaches

John W. Creswell , PhD is an Professor of Family Medicine and Co-Director, Michigan Mixed Methods Research and Scholarship Program at the University of Michigan. He has authored numerous articles and 27 books on mixed methods research, qualitative research and research design. While at the University of Nebraska-Lincoln, he held the Clifton Endowed Professor Chair, served as director of a mixed methods research office, founded the SAGE journal, the Journal of Mixed Methods Research , and was an Adjunct Professor of Family Medicine at the University of Michigan and a consultant to the Veterans Administration health services research center. He was a Senior Fulbright Scholar to South Africa in 2008 and to Thailand in 2012. In 2011 co-led a national working group on mixed methods practices at the National Institute of Health, served as a Visiting Professor at Harvard’s School of Public Health, and received an honorary doctorate from the University of Pretoria, South Africa.  In 2014 he was the President of the Mixed Methods International Research Association.  In 2015 he joined the staff of Family Medicine at the University of Michigan.

Titles by Best-selling Author John W. Creswell

Qualitative Inquiry and Research Design

Research Design

Designing and Conducting Mixed Methods Research

30 Essential Skills for the Qualitative Researcher

A Concise Introduction to Mixed Methods Research

• Request new password
• Create a new account

Qualitative Inquiry and Research Design: Choosing Among Five Approaches

Student resources, welcome to the companion website.

Welcome to the SAGE edge site for Qualitative Inquiry and Research Design, Fourth Edition.

The SAGE edge site for Qualitative Inquiry and Research Design by John Creswell and Cheryl Poth offers a robust online environment you can access anytime, anywhere, and features an impressive array of free tools and resources to keep you on the cutting edge of your learning experience.

Acknowledgements

We gratefully acknowledge John Creswell and Cheryl N. Poth for writing an excellent text. Special thanks are also due to Tim Guetterman for developing the ancillaries on this site.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Neurol Res Pract

How to use and assess qualitative research methods

Loraine busetto.

1 Department of Neurology, Heidelberg University Hospital, Im Neuenheimer Feld 400, 69120 Heidelberg, Germany

Wolfgang Wick

2 Clinical Cooperation Unit Neuro-Oncology, German Cancer Research Center, Heidelberg, Germany

Christoph Gumbinger

Associated data.

Not applicable.

This paper aims to provide an overview of the use and assessment of qualitative research methods in the health sciences. Qualitative research can be defined as the study of the nature of phenomena and is especially appropriate for answering questions of why something is (not) observed, assessing complex multi-component interventions, and focussing on intervention improvement. The most common methods of data collection are document study, (non-) participant observations, semi-structured interviews and focus groups. For data analysis, field-notes and audio-recordings are transcribed into protocols and transcripts, and coded using qualitative data management software. Criteria such as checklists, reflexivity, sampling strategies, piloting, co-coding, member-checking and stakeholder involvement can be used to enhance and assess the quality of the research conducted. Using qualitative in addition to quantitative designs will equip us with better tools to address a greater range of research problems, and to fill in blind spots in current neurological research and practice.

The aim of this paper is to provide an overview of qualitative research methods, including hands-on information on how they can be used, reported and assessed. This article is intended for beginning qualitative researchers in the health sciences as well as experienced quantitative researchers who wish to broaden their understanding of qualitative research.

What is qualitative research?

Qualitative research is defined as “the study of the nature of phenomena”, including “their quality, different manifestations, the context in which they appear or the perspectives from which they can be perceived” , but excluding “their range, frequency and place in an objectively determined chain of cause and effect” [ 1 ]. This formal definition can be complemented with a more pragmatic rule of thumb: qualitative research generally includes data in form of words rather than numbers [ 2 ].

Why conduct qualitative research?

Because some research questions cannot be answered using (only) quantitative methods. For example, one Australian study addressed the issue of why patients from Aboriginal communities often present late or not at all to specialist services offered by tertiary care hospitals. Using qualitative interviews with patients and staff, it found one of the most significant access barriers to be transportation problems, including some towns and communities simply not having a bus service to the hospital [ 3 ]. A quantitative study could have measured the number of patients over time or even looked at possible explanatory factors – but only those previously known or suspected to be of relevance. To discover reasons for observed patterns, especially the invisible or surprising ones, qualitative designs are needed.

While qualitative research is common in other fields, it is still relatively underrepresented in health services research. The latter field is more traditionally rooted in the evidence-based-medicine paradigm, as seen in " research that involves testing the effectiveness of various strategies to achieve changes in clinical practice, preferably applying randomised controlled trial study designs (...) " [ 4 ]. This focus on quantitative research and specifically randomised controlled trials (RCT) is visible in the idea of a hierarchy of research evidence which assumes that some research designs are objectively better than others, and that choosing a "lesser" design is only acceptable when the better ones are not practically or ethically feasible [ 5 , 6 ]. Others, however, argue that an objective hierarchy does not exist, and that, instead, the research design and methods should be chosen to fit the specific research question at hand – "questions before methods" [ 2 , 7 – 9 ]. This means that even when an RCT is possible, some research problems require a different design that is better suited to addressing them. Arguing in JAMA, Berwick uses the example of rapid response teams in hospitals, which he describes as " a complex, multicomponent intervention – essentially a process of social change" susceptible to a range of different context factors including leadership or organisation history. According to him, "[in] such complex terrain, the RCT is an impoverished way to learn. Critics who use it as a truth standard in this context are incorrect" [ 8 ] . Instead of limiting oneself to RCTs, Berwick recommends embracing a wider range of methods , including qualitative ones, which for "these specific applications, (...) are not compromises in learning how to improve; they are superior" [ 8 ].

Research problems that can be approached particularly well using qualitative methods include assessing complex multi-component interventions or systems (of change), addressing questions beyond “what works”, towards “what works for whom when, how and why”, and focussing on intervention improvement rather than accreditation [ 7 , 9 – 12 ]. Using qualitative methods can also help shed light on the “softer” side of medical treatment. For example, while quantitative trials can measure the costs and benefits of neuro-oncological treatment in terms of survival rates or adverse effects, qualitative research can help provide a better understanding of patient or caregiver stress, visibility of illness or out-of-pocket expenses.

How to conduct qualitative research?

Given that qualitative research is characterised by flexibility, openness and responsivity to context, the steps of data collection and analysis are not as separate and consecutive as they tend to be in quantitative research [ 13 , 14 ]. As Fossey puts it : “sampling, data collection, analysis and interpretation are related to each other in a cyclical (iterative) manner, rather than following one after another in a stepwise approach” [ 15 ]. The researcher can make educated decisions with regard to the choice of method, how they are implemented, and to which and how many units they are applied [ 13 ]. As shown in Fig.  1 , this can involve several back-and-forth steps between data collection and analysis where new insights and experiences can lead to adaption and expansion of the original plan. Some insights may also necessitate a revision of the research question and/or the research design as a whole. The process ends when saturation is achieved, i.e. when no relevant new information can be found (see also below: sampling and saturation). For reasons of transparency, it is essential for all decisions as well as the underlying reasoning to be well-documented.

Iterative research process

While it is not always explicitly addressed, qualitative methods reflect a different underlying research paradigm than quantitative research (e.g. constructivism or interpretivism as opposed to positivism). The choice of methods can be based on the respective underlying substantive theory or theoretical framework used by the researcher [ 2 ].

Data collection

The methods of qualitative data collection most commonly used in health research are document study, observations, semi-structured interviews and focus groups [ 1 , 14 , 16 , 17 ].

Document study

Document study (also called document analysis) refers to the review by the researcher of written materials [ 14 ]. These can include personal and non-personal documents such as archives, annual reports, guidelines, policy documents, diaries or letters.

Observations

Observations are particularly useful to gain insights into a certain setting and actual behaviour – as opposed to reported behaviour or opinions [ 13 ]. Qualitative observations can be either participant or non-participant in nature. In participant observations, the observer is part of the observed setting, for example a nurse working in an intensive care unit [ 18 ]. In non-participant observations, the observer is “on the outside looking in”, i.e. present in but not part of the situation, trying not to influence the setting by their presence. Observations can be planned (e.g. for 3 h during the day or night shift) or ad hoc (e.g. as soon as a stroke patient arrives at the emergency room). During the observation, the observer takes notes on everything or certain pre-determined parts of what is happening around them, for example focusing on physician-patient interactions or communication between different professional groups. Written notes can be taken during or after the observations, depending on feasibility (which is usually lower during participant observations) and acceptability (e.g. when the observer is perceived to be judging the observed). Afterwards, these field notes are transcribed into observation protocols. If more than one observer was involved, field notes are taken independently, but notes can be consolidated into one protocol after discussions. Advantages of conducting observations include minimising the distance between the researcher and the researched, the potential discovery of topics that the researcher did not realise were relevant and gaining deeper insights into the real-world dimensions of the research problem at hand [ 18 ].

Semi-structured interviews

Hijmans & Kuyper describe qualitative interviews as “an exchange with an informal character, a conversation with a goal” [ 19 ]. Interviews are used to gain insights into a person’s subjective experiences, opinions and motivations – as opposed to facts or behaviours [ 13 ]. Interviews can be distinguished by the degree to which they are structured (i.e. a questionnaire), open (e.g. free conversation or autobiographical interviews) or semi-structured [ 2 , 13 ]. Semi-structured interviews are characterized by open-ended questions and the use of an interview guide (or topic guide/list) in which the broad areas of interest, sometimes including sub-questions, are defined [ 19 ]. The pre-defined topics in the interview guide can be derived from the literature, previous research or a preliminary method of data collection, e.g. document study or observations. The topic list is usually adapted and improved at the start of the data collection process as the interviewer learns more about the field [ 20 ]. Across interviews the focus on the different (blocks of) questions may differ and some questions may be skipped altogether (e.g. if the interviewee is not able or willing to answer the questions or for concerns about the total length of the interview) [ 20 ]. Qualitative interviews are usually not conducted in written format as it impedes on the interactive component of the method [ 20 ]. In comparison to written surveys, qualitative interviews have the advantage of being interactive and allowing for unexpected topics to emerge and to be taken up by the researcher. This can also help overcome a provider or researcher-centred bias often found in written surveys, which by nature, can only measure what is already known or expected to be of relevance to the researcher. Interviews can be audio- or video-taped; but sometimes it is only feasible or acceptable for the interviewer to take written notes [ 14 , 16 , 20 ].

Focus groups

Focus groups are group interviews to explore participants’ expertise and experiences, including explorations of how and why people behave in certain ways [ 1 ]. Focus groups usually consist of 6–8 people and are led by an experienced moderator following a topic guide or “script” [ 21 ]. They can involve an observer who takes note of the non-verbal aspects of the situation, possibly using an observation guide [ 21 ]. Depending on researchers’ and participants’ preferences, the discussions can be audio- or video-taped and transcribed afterwards [ 21 ]. Focus groups are useful for bringing together homogeneous (to a lesser extent heterogeneous) groups of participants with relevant expertise and experience on a given topic on which they can share detailed information [ 21 ]. Focus groups are a relatively easy, fast and inexpensive method to gain access to information on interactions in a given group, i.e. “the sharing and comparing” among participants [ 21 ]. Disadvantages include less control over the process and a lesser extent to which each individual may participate. Moreover, focus group moderators need experience, as do those tasked with the analysis of the resulting data. Focus groups can be less appropriate for discussing sensitive topics that participants might be reluctant to disclose in a group setting [ 13 ]. Moreover, attention must be paid to the emergence of “groupthink” as well as possible power dynamics within the group, e.g. when patients are awed or intimidated by health professionals.

Choosing the “right” method

As explained above, the school of thought underlying qualitative research assumes no objective hierarchy of evidence and methods. This means that each choice of single or combined methods has to be based on the research question that needs to be answered and a critical assessment with regard to whether or to what extent the chosen method can accomplish this – i.e. the “fit” between question and method [ 14 ]. It is necessary for these decisions to be documented when they are being made, and to be critically discussed when reporting methods and results.

Let us assume that our research aim is to examine the (clinical) processes around acute endovascular treatment (EVT), from the patient’s arrival at the emergency room to recanalization, with the aim to identify possible causes for delay and/or other causes for sub-optimal treatment outcome. As a first step, we could conduct a document study of the relevant standard operating procedures (SOPs) for this phase of care – are they up-to-date and in line with current guidelines? Do they contain any mistakes, irregularities or uncertainties that could cause delays or other problems? Regardless of the answers to these questions, the results have to be interpreted based on what they are: a written outline of what care processes in this hospital should look like. If we want to know what they actually look like in practice, we can conduct observations of the processes described in the SOPs. These results can (and should) be analysed in themselves, but also in comparison to the results of the document analysis, especially as regards relevant discrepancies. Do the SOPs outline specific tests for which no equipment can be observed or tasks to be performed by specialized nurses who are not present during the observation? It might also be possible that the written SOP is outdated, but the actual care provided is in line with current best practice. In order to find out why these discrepancies exist, it can be useful to conduct interviews. Are the physicians simply not aware of the SOPs (because their existence is limited to the hospital’s intranet) or do they actively disagree with them or does the infrastructure make it impossible to provide the care as described? Another rationale for adding interviews is that some situations (or all of their possible variations for different patient groups or the day, night or weekend shift) cannot practically or ethically be observed. In this case, it is possible to ask those involved to report on their actions – being aware that this is not the same as the actual observation. A senior physician’s or hospital manager’s description of certain situations might differ from a nurse’s or junior physician’s one, maybe because they intentionally misrepresent facts or maybe because different aspects of the process are visible or important to them. In some cases, it can also be relevant to consider to whom the interviewee is disclosing this information – someone they trust, someone they are otherwise not connected to, or someone they suspect or are aware of being in a potentially “dangerous” power relationship to them. Lastly, a focus group could be conducted with representatives of the relevant professional groups to explore how and why exactly they provide care around EVT. The discussion might reveal discrepancies (between SOPs and actual care or between different physicians) and motivations to the researchers as well as to the focus group members that they might not have been aware of themselves. For the focus group to deliver relevant information, attention has to be paid to its composition and conduct, for example, to make sure that all participants feel safe to disclose sensitive or potentially problematic information or that the discussion is not dominated by (senior) physicians only. The resulting combination of data collection methods is shown in Fig.  2 .

Possible combination of data collection methods

Attributions for icons: “Book” by Serhii Smirnov, “Interview” by Adrien Coquet, FR, “Magnifying Glass” by anggun, ID, “Business communication” by Vectors Market; all from the Noun Project

The combination of multiple data source as described for this example can be referred to as “triangulation”, in which multiple measurements are carried out from different angles to achieve a more comprehensive understanding of the phenomenon under study [ 22 , 23 ].

Data analysis

To analyse the data collected through observations, interviews and focus groups these need to be transcribed into protocols and transcripts (see Fig.  3 ). Interviews and focus groups can be transcribed verbatim , with or without annotations for behaviour (e.g. laughing, crying, pausing) and with or without phonetic transcription of dialects and filler words, depending on what is expected or known to be relevant for the analysis. In the next step, the protocols and transcripts are coded , that is, marked (or tagged, labelled) with one or more short descriptors of the content of a sentence or paragraph [ 2 , 15 , 23 ]. Jansen describes coding as “connecting the raw data with “theoretical” terms” [ 20 ]. In a more practical sense, coding makes raw data sortable. This makes it possible to extract and examine all segments describing, say, a tele-neurology consultation from multiple data sources (e.g. SOPs, emergency room observations, staff and patient interview). In a process of synthesis and abstraction, the codes are then grouped, summarised and/or categorised [ 15 , 20 ]. The end product of the coding or analysis process is a descriptive theory of the behavioural pattern under investigation [ 20 ]. The coding process is performed using qualitative data management software, the most common ones being InVivo, MaxQDA and Atlas.ti. It should be noted that these are data management tools which support the analysis performed by the researcher(s) [ 14 ].

From data collection to data analysis

Attributions for icons: see Fig. ​ Fig.2, 2 , also “Speech to text” by Trevor Dsouza, “Field Notes” by Mike O’Brien, US, “Voice Record” by ProSymbols, US, “Inspection” by Made, AU, and “Cloud” by Graphic Tigers; all from the Noun Project

How to report qualitative research?

Protocols of qualitative research can be published separately and in advance of the study results. However, the aim is not the same as in RCT protocols, i.e. to pre-define and set in stone the research questions and primary or secondary endpoints. Rather, it is a way to describe the research methods in detail, which might not be possible in the results paper given journals’ word limits. Qualitative research papers are usually longer than their quantitative counterparts to allow for deep understanding and so-called “thick description”. In the methods section, the focus is on transparency of the methods used, including why, how and by whom they were implemented in the specific study setting, so as to enable a discussion of whether and how this may have influenced data collection, analysis and interpretation. The results section usually starts with a paragraph outlining the main findings, followed by more detailed descriptions of, for example, the commonalities, discrepancies or exceptions per category [ 20 ]. Here it is important to support main findings by relevant quotations, which may add information, context, emphasis or real-life examples [ 20 , 23 ]. It is subject to debate in the field whether it is relevant to state the exact number or percentage of respondents supporting a certain statement (e.g. “Five interviewees expressed negative feelings towards XYZ”) [ 21 ].

How to combine qualitative with quantitative research?

Qualitative methods can be combined with other methods in multi- or mixed methods designs, which “[employ] two or more different methods [ …] within the same study or research program rather than confining the research to one single method” [ 24 ]. Reasons for combining methods can be diverse, including triangulation for corroboration of findings, complementarity for illustration and clarification of results, expansion to extend the breadth and range of the study, explanation of (unexpected) results generated with one method with the help of another, or offsetting the weakness of one method with the strength of another [ 1 , 17 , 24 – 26 ]. The resulting designs can be classified according to when, why and how the different quantitative and/or qualitative data strands are combined. The three most common types of mixed method designs are the convergent parallel design , the explanatory sequential design and the exploratory sequential design. The designs with examples are shown in Fig.  4 .

Three common mixed methods designs

In the convergent parallel design, a qualitative study is conducted in parallel to and independently of a quantitative study, and the results of both studies are compared and combined at the stage of interpretation of results. Using the above example of EVT provision, this could entail setting up a quantitative EVT registry to measure process times and patient outcomes in parallel to conducting the qualitative research outlined above, and then comparing results. Amongst other things, this would make it possible to assess whether interview respondents’ subjective impressions of patients receiving good care match modified Rankin Scores at follow-up, or whether observed delays in care provision are exceptions or the rule when compared to door-to-needle times as documented in the registry. In the explanatory sequential design, a quantitative study is carried out first, followed by a qualitative study to help explain the results from the quantitative study. This would be an appropriate design if the registry alone had revealed relevant delays in door-to-needle times and the qualitative study would be used to understand where and why these occurred, and how they could be improved. In the exploratory design, the qualitative study is carried out first and its results help informing and building the quantitative study in the next step [ 26 ]. If the qualitative study around EVT provision had shown a high level of dissatisfaction among the staff members involved, a quantitative questionnaire investigating staff satisfaction could be set up in the next step, informed by the qualitative study on which topics dissatisfaction had been expressed. Amongst other things, the questionnaire design would make it possible to widen the reach of the research to more respondents from different (types of) hospitals, regions, countries or settings, and to conduct sub-group analyses for different professional groups.

How to assess qualitative research?

A variety of assessment criteria and lists have been developed for qualitative research, ranging in their focus and comprehensiveness [ 14 , 17 , 27 ]. However, none of these has been elevated to the “gold standard” in the field. In the following, we therefore focus on a set of commonly used assessment criteria that, from a practical standpoint, a researcher can look for when assessing a qualitative research report or paper.

Assessors should check the authors’ use of and adherence to the relevant reporting checklists (e.g. Standards for Reporting Qualitative Research (SRQR)) to make sure all items that are relevant for this type of research are addressed [ 23 , 28 ]. Discussions of quantitative measures in addition to or instead of these qualitative measures can be a sign of lower quality of the research (paper). Providing and adhering to a checklist for qualitative research contributes to an important quality criterion for qualitative research, namely transparency [ 15 , 17 , 23 ].

Reflexivity

While methodological transparency and complete reporting is relevant for all types of research, some additional criteria must be taken into account for qualitative research. This includes what is called reflexivity, i.e. sensitivity to the relationship between the researcher and the researched, including how contact was established and maintained, or the background and experience of the researcher(s) involved in data collection and analysis. Depending on the research question and population to be researched this can be limited to professional experience, but it may also include gender, age or ethnicity [ 17 , 27 ]. These details are relevant because in qualitative research, as opposed to quantitative research, the researcher as a person cannot be isolated from the research process [ 23 ]. It may influence the conversation when an interviewed patient speaks to an interviewer who is a physician, or when an interviewee is asked to discuss a gynaecological procedure with a male interviewer, and therefore the reader must be made aware of these details [ 19 ].

Sampling and saturation

The aim of qualitative sampling is for all variants of the objects of observation that are deemed relevant for the study to be present in the sample “ to see the issue and its meanings from as many angles as possible” [ 1 , 16 , 19 , 20 , 27 ] , and to ensure “information-richness [ 15 ]. An iterative sampling approach is advised, in which data collection (e.g. five interviews) is followed by data analysis, followed by more data collection to find variants that are lacking in the current sample. This process continues until no new (relevant) information can be found and further sampling becomes redundant – which is called saturation [ 1 , 15 ] . In other words: qualitative data collection finds its end point not a priori , but when the research team determines that saturation has been reached [ 29 , 30 ].

This is also the reason why most qualitative studies use deliberate instead of random sampling strategies. This is generally referred to as “ purposive sampling” , in which researchers pre-define which types of participants or cases they need to include so as to cover all variations that are expected to be of relevance, based on the literature, previous experience or theory (i.e. theoretical sampling) [ 14 , 20 ]. Other types of purposive sampling include (but are not limited to) maximum variation sampling, critical case sampling or extreme or deviant case sampling [ 2 ]. In the above EVT example, a purposive sample could include all relevant professional groups and/or all relevant stakeholders (patients, relatives) and/or all relevant times of observation (day, night and weekend shift).

Assessors of qualitative research should check whether the considerations underlying the sampling strategy were sound and whether or how researchers tried to adapt and improve their strategies in stepwise or cyclical approaches between data collection and analysis to achieve saturation [ 14 ].

Good qualitative research is iterative in nature, i.e. it goes back and forth between data collection and analysis, revising and improving the approach where necessary. One example of this are pilot interviews, where different aspects of the interview (especially the interview guide, but also, for example, the site of the interview or whether the interview can be audio-recorded) are tested with a small number of respondents, evaluated and revised [ 19 ]. In doing so, the interviewer learns which wording or types of questions work best, or which is the best length of an interview with patients who have trouble concentrating for an extended time. Of course, the same reasoning applies to observations or focus groups which can also be piloted.

Ideally, coding should be performed by at least two researchers, especially at the beginning of the coding process when a common approach must be defined, including the establishment of a useful coding list (or tree), and when a common meaning of individual codes must be established [ 23 ]. An initial sub-set or all transcripts can be coded independently by the coders and then compared and consolidated after regular discussions in the research team. This is to make sure that codes are applied consistently to the research data.

Member checking

Member checking, also called respondent validation , refers to the practice of checking back with study respondents to see if the research is in line with their views [ 14 , 27 ]. This can happen after data collection or analysis or when first results are available [ 23 ]. For example, interviewees can be provided with (summaries of) their transcripts and asked whether they believe this to be a complete representation of their views or whether they would like to clarify or elaborate on their responses [ 17 ]. Respondents’ feedback on these issues then becomes part of the data collection and analysis [ 27 ].

Stakeholder involvement

In those niches where qualitative approaches have been able to evolve and grow, a new trend has seen the inclusion of patients and their representatives not only as study participants (i.e. “members”, see above) but as consultants to and active participants in the broader research process [ 31 – 33 ]. The underlying assumption is that patients and other stakeholders hold unique perspectives and experiences that add value beyond their own single story, making the research more relevant and beneficial to researchers, study participants and (future) patients alike [ 34 , 35 ]. Using the example of patients on or nearing dialysis, a recent scoping review found that 80% of clinical research did not address the top 10 research priorities identified by patients and caregivers [ 32 , 36 ]. In this sense, the involvement of the relevant stakeholders, especially patients and relatives, is increasingly being seen as a quality indicator in and of itself.

How not to assess qualitative research

The above overview does not include certain items that are routine in assessments of quantitative research. What follows is a non-exhaustive, non-representative, experience-based list of the quantitative criteria often applied to the assessment of qualitative research, as well as an explanation of the limited usefulness of these endeavours.

Protocol adherence

Given the openness and flexibility of qualitative research, it should not be assessed by how well it adheres to pre-determined and fixed strategies – in other words: its rigidity. Instead, the assessor should look for signs of adaptation and refinement based on lessons learned from earlier steps in the research process.

Sample size

For the reasons explained above, qualitative research does not require specific sample sizes, nor does it require that the sample size be determined a priori [ 1 , 14 , 27 , 37 – 39 ]. Sample size can only be a useful quality indicator when related to the research purpose, the chosen methodology and the composition of the sample, i.e. who was included and why.

Randomisation

While some authors argue that randomisation can be used in qualitative research, this is not commonly the case, as neither its feasibility nor its necessity or usefulness has been convincingly established for qualitative research [ 13 , 27 ]. Relevant disadvantages include the negative impact of a too large sample size as well as the possibility (or probability) of selecting “ quiet, uncooperative or inarticulate individuals ” [ 17 ]. Qualitative studies do not use control groups, either.

Interrater reliability, variability and other “objectivity checks”

The concept of “interrater reliability” is sometimes used in qualitative research to assess to which extent the coding approach overlaps between the two co-coders. However, it is not clear what this measure tells us about the quality of the analysis [ 23 ]. This means that these scores can be included in qualitative research reports, preferably with some additional information on what the score means for the analysis, but it is not a requirement. Relatedly, it is not relevant for the quality or “objectivity” of qualitative research to separate those who recruited the study participants and collected and analysed the data. Experiences even show that it might be better to have the same person or team perform all of these tasks [ 20 ]. First, when researchers introduce themselves during recruitment this can enhance trust when the interview takes place days or weeks later with the same researcher. Second, when the audio-recording is transcribed for analysis, the researcher conducting the interviews will usually remember the interviewee and the specific interview situation during data analysis. This might be helpful in providing additional context information for interpretation of data, e.g. on whether something might have been meant as a joke [ 18 ].

Not being quantitative research

Being qualitative research instead of quantitative research should not be used as an assessment criterion if it is used irrespectively of the research problem at hand. Similarly, qualitative research should not be required to be combined with quantitative research per se – unless mixed methods research is judged as inherently better than single-method research. In this case, the same criterion should be applied for quantitative studies without a qualitative component.

The main take-away points of this paper are summarised in Table ​ Table1. 1 . We aimed to show that, if conducted well, qualitative research can answer specific research questions that cannot to be adequately answered using (only) quantitative designs. Seeing qualitative and quantitative methods as equal will help us become more aware and critical of the “fit” between the research problem and our chosen methods: I can conduct an RCT to determine the reasons for transportation delays of acute stroke patients – but should I? It also provides us with a greater range of tools to tackle a greater range of research problems more appropriately and successfully, filling in the blind spots on one half of the methodological spectrum to better address the whole complexity of neurological research and practice.

Take-away-points

Acknowledgements

Abbreviations.

Authors’ contributions

LB drafted the manuscript; WW and CG revised the manuscript; all authors approved the final versions.

no external funding.

Availability of data and materials

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

• Open access
• Published: 16 August 2024

Going virtual: mixed methods evaluation of online versus in-person learning in the NIH mixed methods research training program retreat

• Joseph J. Gallo 1 ,
• Sarah M. Murray 1 ,
• John W. Creswell 2 ,
• Charles Deutsch 3 &
• Timothy C. Guetterman 2  

BMC Medical Education volume  24 , Article number:  882 ( 2024 ) Cite this article

37 Accesses

Metrics details

Despite the central role of mixed methods in health research, studies evaluating online methods training in the health sciences are nonexistent. The focused goal was to evaluate online training by comparing the self-rated skills of scholars who experienced an in-person retreat to scholars in an online retreat in specific domains of mixed methods research for the health sciences from 2015–2023.

The authors administered a scholar Mixed Methods Skills Self-Assessment instrument based on an educational competency scale that included domains on: “research questions,” “design/approach,” “sampling,” “analysis,” and “dissemination” to participants of the Mixed Methods Research Training Program for the Health Sciences (MMRTP). Self-ratings on confidence on domains were compared before and after retreat participation within cohorts who attended in person ( n  = 73) or online ( n  = 57) as well as comparing across in-person to online cohorts. Responses to open-ended questions about experiences with the retreat were analyzed.

Scholars in an interactive program to improve mixed methods skills reported significantly increased confidence in ability to define or explain concepts and in ability to apply the concepts to practical problems, whether the program was attended in-person or synchronously online. Scholars in the online retreat had self-rated skill improvements as good or better than scholars who participated in person. With the possible exception of networking, scholars found the online format was associated with advantages such as accessibility and reduced burden of travel and finding childcare. No differences in difficulty of learning concepts was described.

Conclusions

Keeping in mind that the retreat is only one component of the MMRTP, this study provides evidence that mixed methods training online was associated with the same increases in self-rated skills as persons attending online and can be a key component to increasing the capacity for mixed methods research in the health sciences.

Peer Review reports

Introduction

The coronavirus pandemic accelerated interest in distance or remote learning. While the acute nature of the pandemic has abated, changes in the way people work have largely remained, with hybrid conferences and trainings more commonly implemented now than during the pre-pandemic period. Studies of health-related online teaching have focused on medical students [ 1 , 2 , 3 ], health professionals [ 4 , 5 ], and medical conferences [ 6 , 7 , 8 ] and have touted the advantages of virtual training and conferences in health education, but few studies have assessed relative growth in skills and competencies in health research methods for synchronous online vs. in-person training.

The National Institutes of Health (NIH)-funded Mixed Methods Research Training Program (MMRTP) for the Health Sciences provided training to faculty-level investigators across health disciplines from 2015–2023. The NIH is a major funder of health-related research in the United States. Its institutes span diseases and conditions (e.g., mental health, environmental health) in addition to focus areas (e.g., minority health and health disparities, nursing) and developing research capacity. Scholars in the MMRTP seek to develop skills in mixed methods research through participation in a summer retreat followed by ongoing mentorship for one year from a mixed methods expert matched to the scholar to support their development of a research proposal. Webinars leading up to the retreat include didactic sessions taught by the same faculty each year, and the retreat itself contains multiple interactive small group sessions in which each scholar presents their project and receives feedback on their grant proposal. Due to pandemic restrictions on gatherings and travel, in 2020 the MMRTP retained all components of the program but transitioned the in-person retreat to a synchronous online retreat.

The number of NIH agencies funding mixed methods research increased from 23 in 1997–2008 to 36 in 2009–2014 [ 9 ]. The usefulness of mixed methods research aligns with several Institutes’ strategic priories, including improving health equity, enhancing feasibility, acceptability, and sustainability of interventions, and addressing patient-centeredness. However, there is a tension between growing interest in mixed methods for health sciences research and a lack of training for investigators to acquire mixed methods research skills. Mixed methods research is not routinely taught in doctoral programs, institutional grant-writing programs, nor research training that academic physicians receive. The relative lack of researchers trained in mixed methods research necessitates ongoing research capacity building and mentorship [ 10 ]. Online teaching has the potential to meet growing demand for training and mentoring in mixed methods, as evidenced by the growth of online offerings by the Mixed Methods International Research Association [ 11 ]. Yet, the nature of skills and attitudes required for doing mixed methods research, such as integration of quantitative and qualitative data collection, analysis, and epistemologies, may make this type of training difficult to adapt to an online format without compromising its effectiveness.

Few studies have attempted to evaluate mixed methods training [ 12 , 13 , 14 , 15 ] and none appear to have evaluated online trainings in mixed methods research. Our goal was to evaluate our online MMRTP by comparing the self-rated skills of scholars who experienced an in-person retreat to an online retreat across specific domains. While the MMRTP retreat is only one component of the program, assessment before and after the retreat among persons who experienced the synchronous retreat online compared to in-person provides an indication of the effectiveness of online instruction in mixed methods for specific domains critical to the design of research in health services. We hypothesized that scholars who attended the retreat online would exhibit improvements in self-rated skills comparable to scholars who attended in person.

Participants

Five cohorts with a total of 73 scholars participated in the MMRTP in person (2015–2019), while four cohorts with a total of 57 scholars participated online (2020–2023). Scholars are faculty-level researchers in the health sciences in the United States. The scholars are from a variety of disciplines in the health sciences; namely, pediatrics, psychiatry, general medicine, oncology, nursing, human development, music therapy, nutrition, psychology, and social work.

The mixed methods research training program

Formal program activities include two webinars leading up to a retreat followed by ongoing mentorship support. The mixed methods content taught in webinars and the retreat is informed by a widely used textbook by Creswell and Plano Clark [ 18 ] in addition to readings on methodological topics and the practice of mixed methods. The webinars introduce mixed methods research and integration concepts, with the goal of imparting foundational knowledge and ensuring a common language. Specifically, the first webinar introduces mixed methods concepts, research designs, scientific rigor, and becoming a resource at one’s institution, while the second focuses on strategies for the integration of qualitative and quantitative research. Retreats provide an active workshop blending lectures, one-on-one meetings, and interactive faculty-led small workgroups. In addition to scholars, core program faculty who serve as investigators and mentors for the MMRTP, supplemented with consultants and former scholars, lead the retreat. The retreat has covered the state-of-the-art topics within the context of mixed methods research: rationale for use of mixed methods, procedural diagrams, study aims, use of theory, integration strategies, sampling strategies, implementation science, randomized trials, ethics, manuscript and proposal writing, and becoming a resource at one’s home institution. In addition to lectures, the retreat includes multiple interactive small group sessions in which each scholar presents their project and receives feedback on their grant proposal and is expected to make revisions based on feedback and lectures.

Scholars are matched for one year with a mentor based on the Scholar’s needs, career level, and area of health research from a national list of affiliated experienced mixed methods investigators with demonstrated success in obtaining independent funding for research related to the health sciences and a track record and commitment to mentoring. The purpose of this arrangement is to provide different perspectives on mixed methods design while also providing specific feedback on the scholar's research proposal, reviewing new ideas, and together developing a strategy and timeline for submission.

From 2015–2019 (in-person cohorts) the retreat was held over 3 days at the Johns Hopkins University Bloomberg School of Public Health (in 2016 Harvard Catalyst, the Harvard Clinical and Translational Science Center, hosted the retreat at Harvard Medical School). Due to pandemic restrictions, from 2020–2023 the retreat activities were conducted via Zoom with the same number of lecture sessions (over 3 days in 2020 and 4 days thereafter). We made adaptations for the online retreat based on continuous feedback from attendees. We had to rapidly transition to online in 2020 with the same structure as in person, but feedback from scholars led us to extend the retreat to 4 days online from 2021–2023. The extra day allowed for more breaks from Zoom sessions with time for scholars to consider feedback from small groups and to have one-on-one meetings with mentors. Discussion during interactive presentations was encouraged and facilitated by using breakout rooms at breaks mid-presentation. Online resources were available to participants through CoursePlus, the teaching and learning platform used for courses at the Johns Hopkins Bloomberg School of Public Health, hosting publications, presentation materials, recordings of lectures, sharing proposals, email, and discussion boards that scholars have access to before, during, and after the retreat.

Measurement strategy

Before and after the retreat in each year, we distributed a self-administered scholar Mixed Methods Skills Self-Assessment instrument (Supplement 1) to all participating scholars [ 15 ]; we have reported results from this pre-post assessment for the first two cohorts [ 14 ]. The Mixed Methods Skills Self-Assessment instrument has been previously used and has established reliability for the total items (α = 0.95) and evidence of criterion-related validity between experiences and ability ratings [ 15 ]. In each year, the pre-assessment is completed upon entry to the program, approximately four months prior to the retreat, and the post-assessment is administered two weeks after the retreat. The instrument consists of three sections: 1) professional experiences with mixed methods, including background, software, and resource familiarity; 2) a quantitative, qualitative, and mixed methods skills self-assessment; and 3) open-ended questions focused on learning goals for the MMRTP. The skills assessment contains items for each of the following domains: “research questions,” “design/approach,” “sampling,” “analysis,” and “dissemination.” Each skill was assessed via three items drawn from an educational competency ratings scale that ask scholars to rate: [ 16 ] “My ability to define/explain,” “My ability to apply to practical problems,” and “Extent to which I need to improve my skill.” Response options were on a five-point Likert-type scale that ranged from “Not at all” (coded ‘1’) to “To a great extent” (coded ‘5’), including a mid-point [ 17 ]. We took the mean of the scholar’s item ratings over all component items within each domain (namely, “research questions,” “design/approach,” “sampling,” “analysis,” and “dissemination”).

Open-ended questions

The baseline survey included two open-ended prompts: 1) What skills and goals are most important to you?, and 2) What would you like to learn? The post-assessment survey also included two additional open-ended questions about the retreat: 1) What aspects of the retreat were helpful?, and 2) What would you like to change about the retreat? In addition, for the online cohorts (2020–2023), we wanted to understand reactions to the online training and added three questions for this purpose: (1) In general, what did you think of the online format for the MMRTP retreat?, 2) What mixed methods concepts are easier or harder to learn virtually?, and 3) What do you think was missing from having the retreat online rather than in person?

Data analysis

Our evaluation employed a convergent mixed methods design [ 18 ], integrating an analysis of ratings pre- and post-retreat with analysis of open-ended responses provided by scholars after the retreat. Our quantitative analysis proceeded in 3 steps. First, we analyzed item-by-item baseline ratings of the extent to which scholars thought they “need to improve skills,” stratified into two groups (5 cohorts who attended in-person and 4 cohorts who attended online). The purpose of comparing the two groups at baseline on learning needs was to assess how similar the scholars in the in-person or online groups were in self-assessment of learning needs before attending the program. Second, to examine the change in scholar ratings of ability to “define or explain a concept” and in their ability to “apply to practical problems,” from before to after the retreat, we conducted paired t-tests. The goal was to compare the ratings before and after the retreat among scholars who attended the program in person to scholars who attended online. Third, we compared post-retreat ratings among in-person cohorts to online cohorts to gauge the effectiveness of the online training. We set statistical significance at α  < 0.05 as a guide to inference. We calculated Cohen’s d as a guide to the magnitude of differences [ 19 ]. SPSS Version 28 was employed for all analyses.

We analyzed qualitative data using a thematic analysis approach that consisted of reviewing all open-ended responses, conducting open coding based on the data, developing and refining a codebook, and identifying major themes [ 20 ]. We then compared the qualitative results for the in-person versus online cohorts to understand any thematic differences concerning retreat experiences and reactions.

Background and experiences of scholars

Scholars in the in-person ( n  = 59, 81%) and online ( n  = 52, 91%) cohorts reported their primary training was quantitative rather than qualitative or mixed methods, and scholars across cohorts commonly reported at least some exposure to mixed methods research (Table  1 ). However, most scholars did not have previous mixed methods training with 17 (23%) and 16 (28%) of the in-person and online cohorts, respectively, having previously completed a mixed methods course. While experiences were similar across in-person vs. online cohorts, there were two areas in which the scholars reported a statistically significant difference: a larger portion of the online cohorts reported writing a mixed methods application that received funding ( n  = 35, 48% in person; n  = 46, 81% online), and a smaller proportion of the online cohorts had given a local or institutional mixed methods presentation ( n  = 32, 44% in person; n  = 15, 26% online).

Self-identified need to improve skills in mixed methods

At baseline, scholars rated the extent to which they needed to improve specific mixed methods skills (Table  2 ). Overall, scholars endorsed a strong need to improve all mixed methods skills. The ratings between the in-person and online cohorts were not statistically significant for any item.

Change in self-ratings of skills after the retreat

Within cohorts.

For all domains, the differences in pre-post assessment scores were statistically significant for both the in-person and online cohorts in ability to define or explain concepts and to apply concepts to practical problems (left side of Table  3 ). In other words, on average scholars improved in both in-person and online cohorts.

Across cohorts

Online cohorts had significantly better self-ratings after the retreat than did in-person cohorts in ability to define or explain concepts and to apply concepts to practical problems (in sampling, data collection, analysis, and dissemination) but no significant differences in research questions and design / approach (rightmost column of Table  3 ).

Scholar reflections about online and in-person retreats

Goals of training.

In comparing in-person to online cohorts, discussions of the skills that scholars wanted to improve had no discernable differences. Scholars mentioned wanting to develop skills in the foundations of mixed methods research, how to write competitive proposals for funding, the use of the terminology of mixed methods research, and integrative analysis. In addition, some scholars expressed wanting to become a resource at their own institutions and providing training and mentoring to others.

Small group sessions

Scholars consistently reported appreciating being able to talk through their project and gaining feedback from experts in small group sessions. Some scholars expressed a preference for afternoon small group sessions, “The small group sessions felt the most helpful, but only because we can apply what we were learning from the morning lecture sessions” (online cohort 9). How participants discussed the benefits of the small group sessions or how they used the sessions did not depend on whether they had experienced the session in person or online.

Online participants described a tradeoff between the accessibility of a virtual retreat versus advantages of in-person training. One participant explained, “I liked the online format, as I do not have reliable childcare” (online cohort 8). Many of the scholars felt that there was an aspect of networking missing when the retreat was held fully online. As one scholar described, when learning online they, “miss getting to know the other fellows and forming lasting connections” (online cohort 9). However, an equal number of others reported that having a virtual retreat meant less hassle; for instance, they were able to join from their preferred location and did not have to travel. Some individuals specifically described the tradeoff of fewer networking opportunities for ease of attendance. One scholar wrote, being online “certainly loses some of the perks of in person connection building but made it equitable to attend” (online cohort 8).

Learning online

No clear difference in ease of learning concepts was described. A scholar explained: “Learning most concepts is essentially the same virtually versus in person” (online cohort 8). However, scholars described some concepts as easier to learn in one modality versus the other, for example, simpler concepts being more suited to learning virtually while complex concepts were better suited to in-person learning. There was notable variation though in the topics which scholars considered to be simple versus complex. For instance, one scholar noted that “I suppose developing the joint displays were a bit tougher virtually since you were not literally elbow to elbow” (online cohort 7) while another explained, “joint displays lend themselves to the zoom format” (online cohort 8).

Integrating survey responses and scholar reflections

In-person and online cohorts were comparable in professional experiences and ratings of the need to improve skills before attending the retreat, sharpening the focus on differences in self-rated skills associated with attendance online compared to in person. If anything, online attendees rated skills as good or better than in-person attendees. Open-ended questions revealed that, for the most part, scholar reflections on learning were similar across in-person and online cohorts. Whether learning the concept of “mixed methods integration” was more difficult online was a source of disagreement. Online attendance was associated with numerous advantages, and small group sessions were valued, regardless of format. Taken together, the evidence from nine cohorts shows that the online retreat was acceptable and as effective in improving self-rated skills as meeting in person.

Mixed methods have become indispensable to health services research from intervention development and testing [ 21 ] to implementation science [ 22 , 23 , 24 ]. We found that scholars participating in an interactive program to improve mixed methods skills reported significantly increased confidence in their ability to define or explain concepts and in their ability to apply the concepts to practical problems, whether the program was attended in-person or synchronously online. Scholars who participated in the online retreat had self-rated skill improvements as good or better than scholars who participated in person, and these improvements were relatively large as indicated by the Cohen’s d estimates. The online retreat appeared to be effective in increasing confidence in the use of mixed methods research in the health sciences and was acceptable to scholars. Our study deserves attention because the national need is so great for investigators with training in mixed methods to address complex behavioral health problems, community- and patient-centered research, and implementation research. No program has been evaluated as we have done here.

Aside from having written a funded mixed methods proposal, the online compared to earlier in person cohorts were comparable in experiences and need to improve specific skills. Within each cohort, scholars reported significant gains in self-rated skills on their ability to “define or explain” a concept and on their ability to “apply to practical problems” in domains essential to mixed methods research. However, consistent with our hypothesis that online training would be as effective as in person we found that online scholars reported better improvement in self-ratings in ability to define or explain concepts and to apply concepts to practical problems in sampling, data collection, analysis, and dissemination but no significant differences in research questions and design / approach. Better ratings in online cohorts could reflect differences in experience with mixed methods, secular changes in knowledge and availability of resources in mixed methods, and maturation of the program facilitated by continued modifications based on feedback from scholars and participating faculty [ 13 , 14 , 15 ].

Ratings related to the “analysis” domain, which includes the central concept of mixed methods integration, deserve notice since scholars rated this skill well below other domains at baseline. While both in-person and online cohorts improved after the retreat, and online cohorts improved substantially more than in-person cohorts, ratings for analysis after the retreat remained lower than for other domains. Scholars consistently have mentioned integration as a difficult concept, and our analysis here is limited to the retreat alone. Continued mentoring one year after the retreat and work on their proposal is built in to the MMRTP to enhance understanding of integration.

Several reviews point out the advantages of online training including savings in time, money, and greenhouse emissions [ 1 , 7 , 8 ]. Online conferences may increase the reach of training to international audiences, improve the diversity of speakers and attendees, facilitate attendance of persons with disabilities, and ease the burden of finding childcare [ 1 , 8 , 25 ]. Online training in health also appears to be effective [ 2 , 4 , 5 , 25 ], though studies are limited because often no skills were evaluated, no comparison groups were used, the response rate was low, or the sample size was small [ 1 , 6 ]. With the possible exception of networking, scholars found the online format was associated with advantages, including saving travel, maintaining work-family balance, and learning effectively. As scholars did discuss perceived increase in difficulty networking, deliberate effort needs to be directed at enhancing collaborations and mentorship [ 8 ]. The MMRTP was designed with components to facilitate networking during and beyond the retreat (e.g., small group sessions, one-on-one meetings, working with a consultant on a specific proposal).

Limitations of our study should be considered. First, the retreat was only one of several components of a mentoring program for faculty in the health sciences. Second, in-person and online cohorts represent different time periods spanning 9 years during which mixed methods applications to NIH and other funders have been increasing [ 9 ]. Third, the pre- and post-evaluations of ability to explain or define concepts, or to apply the concepts to practical problems, were based on self-report. Nevertheless, the pre-post retreat survey on self-rated skills uses a skills self-assessment form we developed [ 15 ], drawing from educational theory related to the epistemology of knowledge [ 26 , 27 ].

Despite the central role of mixed methods in health research, studies evaluating online methods training in the health sciences are nonexistent. Our study provides evidence that mixed methods training online was associated with the same increases in self-rated skills as persons attending online and can be a key component to increasing the capacity for mixed methods research in the health sciences.

Availability of data and materials

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Mixed Methods Research Training Program

Wilcha RJ. Effectiveness of Virtual Medical Teaching During the COVID-19 Crisis: Systematic Review. JMIR Med Educ. 2020;6(2):e20963.

Article   Google Scholar  

Pei L, Wu H. Does online learning work better than offline learning in undergraduate medical education? A systematic review and meta-analysis. Medical Education Online. 2019;24(1) https://doi.org/10.1080/10872981.2019.1666538

Barche A, Nayak V, Pandey A, Bhandarkar A, Nayak K. Student perceptions towards online learning in medical education during the COVID-19 pandemic: a mixed-methods study. F1000Res. 2022;11:979. https://doi.org/10.12688/f1000research.123582.1 .

Ebner C, Gegenfurtner A. Learning and Satisfaction in Webinar, Online, and Face-to-Face Instruction: A Meta-Analysis. Frontiers in Education. 2019;4(92) https://doi.org/10.3389/feduc.2019.00092

Randazzo M, Preifer R, Khamis-Dakwar R. Project-Based Learning and Traditional Online Teaching of Research Methods During COVID-19: An Investigation of Research Self-Efficacy and Student Satisfaction. Frontiers in Education. 2021;6(662850) https://doi.org/10.3389/feduc.2021.662850

Chan A, Cao A, Kim L, et al. Comparison of perceived educational value of an in-person versus virtual medical conference. Can Med Educ J. 2021;12(4):65–9. https://doi.org/10.36834/cmej.71975 .

Rubinger L, Gazendam A, Ekhtiari S, et al. Maximizing virtual meetings and conferences: a review of best practices. Int Orthop. 2020;44(8):1461–6. https://doi.org/10.1007/s00264-020-04615-9 .

Sarabipour S. Virtual conferences raise standards for accessibility and interactions. Elife. Nov 4 2020;9 https://doi.org/10.7554/eLife.62668

Coyle CE, Schulman-Green D, Feder S, et al. Federal funding for mixed methods research in the health sciences in the United States: Recent trends. J Mixed Methods Res. 2018;12(3):1–20.

Poth C, Munce SEP. Commentary – preparing today’s researchers for a yet unknown tomorrow: promising practices for a synergistic and sustainable mentoring approach to mixed methods research learning. Int J Multiple Res Approaches. 2020;12(1):56–64.

Creswell JW. Reflections on the MMIRA The Future of Mixed Methods Task Force Report. J Mixed Methods Res. 2016;10(3):215–9. https://doi.org/10.1177/1558689816650298 .

Hou S. A Mixed Methods Process Evaluation of an Integrated Course Design on Teaching Mixed Methods Research. Int J Sch Teach Learn. 2021;15(2):Article 8. https://doi.org/10.20429/ijsotl.2021.150208 .

Guetterman TC, Creswell J, Deutsch C, Gallo JJ. Process Evaluation of a Retreat for Scholars in the First Cohort: The NIH Mixed Methods Research Training Program for the Health Sciences. J Mix Methods Res. 2019;13(1):52–68. https://doi.org/10.1177/1558689816674564 .

Guetterman T, Creswell JW, Deutsch C, Gallo JJ. Skills Development and Academic Productivity of Scholars in the NIH Mixed Methods Research Training Program for the Health Sciences (invited publication). Int J Multiple Res Approach. 2018;10(1):1–17.

Guetterman T, Creswell JW, Wittink MN, et al. Development of a Self-Rated Mixed Methods Skills Assessment: The NIH Mixed Methods Research Training Program for the Health Sciences. J Contin Educ Health Prof. 2017;37(2):76–82.

Harnisch D, Shope RJ. Developing technology competencies to enhance assessment literate teachers. AACE; 2007:3053–3055.

DeVellis RF. Scale development: Theory and applications. 3rd ed. Sage; 2012.

Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. 3rd ed. Sage Publications; 2017.

Cohen J. Statistical power analysis for the behavioral sciences. 3rd ed. Academic Press; 1988.

Boeije H. A purposeful approach to the constant comparative method in the analysis of qualitative interviews. Qual Quant. 2002;36:391–409.

Aschbrenner KA, Kruse G, Gallo JJ, Plano Clark VL. Applying mixed methods to pilot feasibility studies to inform intervention trials. Pilot Feasibility Stud. 2022;8(1):217–24. https://doi.org/10.1186/s40814-022-01178-x .

Palinkas LA. Qualitative and mixed methods in mental health services and implementation research. J Clin Child Adolesc Psychol. 2014;43(6):851–61.

Albright K, Gechter K, Kempe A. Importance of mixed methods in pragmatic trials and dissemination and implementation research. Acad Pediatr Sep-Oct. 2013;13(5):400–7. https://doi.org/10.1016/j.acap.2013.06.010 .

Palinkas L, Aarons G, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed methods designs in implementation research. Adm Policy Ment Health. 2011;38:44–53.

Ni AY. Comparing the Effectiveness of Classroom and Online Learning: Teaching Research Methods. J Public Affairs Educ. 2013;19(2):199–215. https://doi.org/10.1080/15236803.2013.12001730 .

Harnisch D, Shope RJ. Developing technology competencies to enhance assessment literate teachers. presented at: Society for Information Technology & Teacher Education International Conference; March 26, 2007 2007; San Antonio, Texas.

Guetterman TC. What distinguishes a novice from an expert mixed methods researcher? Qual Quantity. 2017;51:377–98.

Download references

Acknowledgements

The Mixed Methods Research Training Program is supported by the Office of Behavioral and Social Sciences Research under Grant R25MH104660. Participating institutes are the National Institute of Mental Health, National Heart, Lung, and Blood Institute, National Institute of Nursing Research, and the National Institute on Aging.

Author information

Authors and affiliations.

Johns Hopkins University, Baltimore, MD, USA

Joseph J. Gallo & Sarah M. Murray

University of Michigan, Ann Arbor, MI, USA

John W. Creswell & Timothy C. Guetterman

Harvard University, Boston, MA, USA

Charles Deutsch

You can also search for this author in PubMed   Google Scholar

Contributions

All authors conceptualized the design of this study. TG analyzed the scholar data in evaluation of the program. TG and JG interpreted results and were major contributors in writing the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Timothy C. Guetterman .

Ethics declarations

Ethics approval and consent to participate.

The program was reviewed by the Johns Hopkins Institutional Review Board and was deemed exempt as educational research under United States 45 CFR 46.101(b), Category (2). Data were collected through an anonymous survey. Consent to participate was waived.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Gallo, J.J., Murray, S.M., Creswell, J.W. et al. Going virtual: mixed methods evaluation of online versus in-person learning in the NIH mixed methods research training program retreat. BMC Med Educ 24 , 882 (2024). https://doi.org/10.1186/s12909-024-05877-2

Download citation

Received : 15 January 2024

Accepted : 08 August 2024

Published : 16 August 2024

DOI : https://doi.org/10.1186/s12909-024-05877-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Research training
• Mixed methods research
• Research capacity building
• Online education
• Teaching methods

Sorry, there was a problem.

Download the free Kindle app and start reading Kindle books instantly on your smartphone, tablet, or computer - no Kindle device required .

Read instantly on your browser with Kindle for Web.

Using your mobile phone camera - scan the code below and download the Kindle app.

Image Unavailable

• To view this video download Flash Player

Follow the author

Research Design: Qualitative, Quantitative, and Mixed Methods Approaches 5th Edition

This bestselling text pioneered the comparison of qualitative, quantitative, and mixed methods research design. For all three approaches, John W. Creswell and new co author J. David Creswell include a preliminary consideration of philosophical assumptions; key elements of the research process; a review of the literature; an assessment of the use of theory in research applications, and reflections about the importance of writing and ethics in scholarly inquiry. New to this Edition

• Updated discussion on designing a proposal for a research project and on the steps in designing a research study.  
• Additional content on epistemological and ontological positioning in relation to the research question and chosen methodology and method. 
• Additional updates on the transformative worldview. 
• Expanded coverage on specific approaches such as case studies, participatory action research, and visual methods. 
• Additional information about social media, online qualitative methods, and mentoring and reflexivity in qualitative methods. 
• Incorporation of action research and program evaluation in mixed methods and coverage of the latest advances in the mixed methods field
• Additional coverage on qualitative and quantitative data analysis software in the respective methods chapters. 
• Additional information about causality and its relationship to statistics in quantitative methods. 
• Incorporation of writing discussion sections into each of the three methodologies. 
• Current references and additional readings are included in this new edition.
• ISBN-10 1506386709
• ISBN-13 978-1506386706
• Edition 5th
• Publication date January 2, 2018
• Language English
• Dimensions 7 x 0.75 x 10 inches
• Print length 304 pages
• See all details

Editorial Reviews

About the author.

John W. Creswell, PhD, is a Professor of Family Medicine and Senior Research Scientist of

the Michigan Mixed Methods Program. He has authored numerous articles and 34 books on

mixed methods research, qualitative research, and research design. While at the University of

Nebraska–Lincoln, he held the Clifton Endowed Professor Chair, served as Director of the

Mixed Methods Research Office, co-founded SAGE’s Journal of Mixed Methods Research , and

was an Adjunct Professor of Family Medicine at the University of Michigan and a consultant to

the Veterans Administration Health Services Research Center in Ann Arbor, Michigan. He was

a Senior Fulbright Scholar to South Africa in 2008 and to Thailand in 2012. In 2011, he co-led

a National Institutes of Health working group on the “best practices of mixed methods research

in the health sciences,” served as a Visiting Professor at Harvard’s School of Public Health and

received an honorary doctorate from the University of Pretoria, South Africa. In 2014, he was

the founding President of the Mixed Methods International Research Association. In 2015, he

joined the staff of Family Medicine at the University of Michigan to Co-Direct the Michigan

Mixed Methods Program. In 2017, he coauthored the American Psychological Association

“standards” on qualitative and mixed methods research. The fourth edition of this book on

Qualitative Inquiry & Research Design won the 2018 McGuffey Longevity Award from the U.S.

Textbook & Academic Authors Association. During the COVID-19 pandemic, he gave virtual

keynote presentations to many countries from his office in Osaka, Japan. Updates on his work

can be found on his website at johnwcreswell.com.

Product details

• Publisher ‏ : ‎ SAGE Publications, Inc; 5th edition (January 2, 2018)
• Language ‏ : ‎ English
• Paperback ‏ : ‎ 304 pages
• ISBN-10 ‏ : ‎ 1506386709
• ISBN-13 ‏ : ‎ 978-1506386706
• Item Weight ‏ : ‎ 1.2 pounds
• Dimensions ‏ : ‎ 7 x 0.75 x 10 inches
• #6 in Social Sciences Methodology
• #12 in Social Sciences Research
• #25 in Core

About the author

John w. creswell.

John W. Creswell is a Professor of Educational Psychology at Teachers College, University of Nebraska-Lincoln. He is affiliated with a graduate program in educational psychology that specializes in quantitative and qualitative methods in education. In this program, he specializes in qualitative and quantitative research designs and methods, multimethod research, and faculty and academic leadership issues in colleges and universities.

Customer reviews

• 5 star 4 star 3 star 2 star 1 star 5 star 76% 14% 6% 1% 3% 76%
• 5 star 4 star 3 star 2 star 1 star 4 star 76% 14% 6% 1% 3% 14%
• 5 star 4 star 3 star 2 star 1 star 3 star 76% 14% 6% 1% 3% 6%
• 5 star 4 star 3 star 2 star 1 star 2 star 76% 14% 6% 1% 3% 1%
• 5 star 4 star 3 star 2 star 1 star 1 star 76% 14% 6% 1% 3% 3%

Customer Reviews, including Product Star Ratings help customers to learn more about the product and decide whether it is the right product for them.

To calculate the overall star rating and percentage breakdown by star, we don’t use a simple average. Instead, our system considers things like how recent a review is and if the reviewer bought the item on Amazon. It also analyzed reviews to verify trustworthiness.

Customers say

Customers find the reference book great and clear, making everything clear. However, opinions are mixed on readability, with some finding it well-written, easy to read, and detailed, while others say it's nearly impossible to read in some sections and some concepts are not thoroughly explained.

AI-generated from the text of customer reviews

Customers find the reference book great, excellent, and straightforward. They also say it's appropriate for graduate students, teachers, and researchers of many stripes.

"...term clarifications in textbooks, but this title just makes everything so clear that my confusion around commonly synonymously-used terminology..." Read more

"This book gives an excellent overview of quantitative , qualitative, and mixed methods research...." Read more

"...It is appropriate for graduate students , teachers, and researchers of many stripes – especially in medicine, education, psychology, and the social..." Read more

"An excellent resource on the research process , stepping through the entire sequence step by step, including selection an approach, literature review..." Read more

Customers are mixed about the readability of the book. Some mention it's well written, easy to read, and simple. They also say the authors are well known in the field of social research design. However, some find the language overcomplicated, some concepts are not thoroughly explained, and the green colored text doesn't align properly with the black and white text.

"...cover the practical nuances of thesis writing in such a clean, simple , and thorough way that I actually couldn't put the book down and read it cover..." Read more

"This book gives an excellent overview of quantitative, qualitative , and mixed methods research...." Read more

"... Some concepts are not thoroughly explained in one particular section, so readers should be patient in going back and forth to firmly grasp those..." Read more

"...It is accessible and readable enough that anyone engaged in critical inquiry can gain from this work...." Read more

Reviews with images

• Sort reviews by Top reviews Most recent Top reviews

Top reviews from the United States

There was a problem filtering reviews right now. please try again later..

Top reviews from other countries

• About Amazon
• Investor Relations
• Amazon Devices
• Amazon Science
• Sell products on Amazon
• Sell on Amazon Business
• Sell apps on Amazon
• Become an Affiliate
• Advertise Your Products
• Self-Publish with Us
• Host an Amazon Hub
• › See More Make Money with Us
• Amazon Business Card
• Shop with Points
• Reload Your Balance
• Amazon Currency Converter
• Amazon and COVID-19
• Your Account
• Your Orders
• Shipping Rates & Policies
• Returns & Replacements
• Manage Your Content and Devices

• Conditions of Use
• Privacy Notice
• Consumer Health Data Privacy Disclosure
• Your Ads Privacy Choices