*The ideas and statements herein do not reflect or represent the official views of the Government Accountability Office in any way.

Andrew begins…

Hey Darby, since the release of ChatGPT about a year and a half ago, developments in AI have come so fast, it’s like a firehose coming at us all the time. What are some of the ways that AI is being used in qualitative research?

Andrew, you and I have been attending conferences together like QRCA and QUAL360 where a dizzying array of companies have been exhibiting and presenting their latest AI tools designed for qualitative research. The market research industry seems to be the primary audience for many of these tools, with their LLMs (Large Language Models) being trained to understand consumer behavior, brands, and marketplace decision making. For the field of social science, however, we may not be so quick to incorporate these tools, nor may they be quite ready for our needs.

Regardless, here are some ways we’ve seen companies and organizations leveraging AI to support their qualitative research.

Discussion Guides

• Generating literature reviews and conducting social listening to support hypothesis generation
• Drafting moderator guides
• Pretesting moderator guides with personas (sometimes called “digital twins” or “synthetic respondents”)

Recruitment

• Using “Digital twins” or “synthetic respondents” to pretest screeners
• Targeting potentially eligible participants

Data Collection

• AI moderation
• AI-suggested probe
• Replacing human participants with “Digital twins” or “synthetic respondents”
• Live sentiment coding
• AI-assisted translation
• AI-generated transcripts/notes

Analysis

• Coding data
• Thematic analysis
• Summaries

These are the AI-supported tasks I’ve become aware of in the past year, but I’m assuming there are many more types of offerings out there and on the horizon.

Andrew, what concerns does the federal government have around using AI, specifically applied to the issues that the QUALPOR community should be keeping in mind as we explore using AI tools with our clients and research partners?

Darby, there is an increasingly long list of use cases for using AI in the government, and we can talk about those in another conversation. The list of government concerns when it comes to using AI is also long. It’s one of the few policy areas where there is broad bi-partisan support for addressing AI related issues. There are over 100 AI related bills on the Federal legislative docket, and many states have also introduced their own AI legislation. This past fall, President Biden released an Executive Order (EO) focusing on AI safety; and in March, the Office of Management and Budget (OMB) released its implementation memo to the EO that details specific actions that federal agencies must take when developing and/or deploying AI to ensure the safety and protect the rights of Americans. In the spirit of the government’s efforts to ensure the safe and responsible use of AI, here are some of the AI-related issues that the QUALPOR community should keep in mind as we explore AI tools with our clients and research partners:

1. Safety & Security
2. Privacy
3. Ethical Considerations
4. Transparency
5. Equity & Bias Mitigation
6. Accountability & Responsibility
7. Human-in-the-Loop integration (which is the combination of AI and human expertise)

Darby, what are some of the AI tools you have been exploring?

We have dabbled with ChatGPT to see how it does in generating questions for moderator guides and creative projective techniques. My conclusion is that ChatGPT is somewhat like an intern. It can generate basic ideas that may stimulate our thinking, but ultimately cannot replace our experience and contextual understanding of our client’s needs and the population we are studying.

At SSRS, we’ve also been exploring how AI features can be leveraged in asynchronous online bulletin boards. For example, we recently conducted a multi-language study using a platform that integrated AI-driven translations (basically, Google Translate was baked into the board itself). Because of this capability, we were able to pose questions, understand responses, ask probes, and read translations of video uploads across participants speaking 8 different languages while the board was live. This feature worked well for us in some languages, but not as well in others. The platform also suggested different AI-generated probes tailored to each person’s responses. Like the ChatGPT example above, however, we found the AI probes to be somewhat basic and often not exactly relevant to our research goals, so we tended not to use them. Bottom line, we are curious to continue exploring these capabilities with a heavy dose of caution.

Finally, we’ve also been exploring tools to help us with qualitative data analysis. There are a few important factors to consider when exploring AI analysis tools. First and foremost is data security.  We need to be certain that our data is being stored safely and securely, that it is not accessible to anyone outside of our research team, and that it is not being used to train LLMs. So far, we’ve found several platforms that meet these requirements. Another important factor is whether the platform links the thematic codes and findings back to the source data. This helps us assess the accuracy of the AI conclusions. In all, we are seeing somewhat positive results so far with the platforms we’ve been testing – when we compare the AI analysis to our own human analysis, it seems to do a reasonably good job identifying the themes. We are going to continue testing platforms, because we do believe that if we can create efficiencies in the coding phase, it will give us more time to spend generating thoughtful, insightful reports.

Well, Andrew, how have you been using AI at GAO and what are you exploring?

Darby, at GAO, we are involved with and using AI in various ways.

First, as a Congressional oversight agency, we are auditing agency use of AI and looking at the AI inventory lists of Federal agencies. One of our findings is that agencies don’t have a full understanding of what is and is not AI, and they don’t yet have the workforce capacity to keep up with AI demands. One of the Biden Administration’s Executive Order recommendations was to increase the Federal AI workforce and for each agency to have a Chief AI Officer.

A second way we are involved with AI is in our role as a convener and in conducting technology assessments of emerging technology. We have a growing AI portfolio and we’ve conducted recent technology assessments of AI in Healthcare, of AI in improving climate disaster modeling, and looking at both the commercial development and human and environmental impacts of generative AI. For each of these technology assessments, I’ve moderated a 3-day virtual expert panel with about 20 of the world’s leading experts on the relevant area of AI.

Finally, our innovation lab has developed its own LLM and is currently developing use cases. One of which, as you mentioned, is using generative AI to analyze unstructured text – which could be anything from analyzing public comments on a proposed regulation, summarizing current proposed Congressional Bills, or analyzing transcripts from interviews, focus groups or expert panels. This use case is the holy grail for me at GAO!  Darby, what most excites you and concerns you about AI as you look to the future of AI for qualitative researchers?

I must say, I’m really excited about the prospects for AI to help our qualitative analysis process be more efficient. I am confident that AI will never replace humans in qualitative research, but if it can support us with the manual work of sifting through transcripts and sorting quotes into thematic areas, that could save us a lot of time and free us up for higher-level thinking. Honestly, I am NOT as excited about these new platforms that use AI chatbots to ask the questions as if they are humans. The beauty of qualitative research is in our human interactions, rapport building, and connections we make with our participants, and I just don’t see AI ever being able to do that well.

Andrew, you’ve told us about the concerns of the government, but what excites you about AI, particularly from a qualitative research perspective?

Darby, I agree with your insight about AI helping qualitative researchers become more efficient in all aspects of the research process. Despite my reservations about certain AI applications — like real- time moderation — I’m most excited about the integration of AI in the post-data collection phase, which would include incorporating AI transcriptions and using AI to analyze text.

Expanding on what I mentioned in my previous answer, at GAO we are currently testing and integrating two AI methodologies for analyzing text. The first combines traditional NLP with human discernment, leveraging topic modeling to construct initial categories followed by human analysts expertly coding the data. Additionally, we are testing the use of our own LLMs for both the labeling and coding steps with humans in a quality control role. The second method uses generative AI to create summaries and themes of the data with the ability to click into the themes and validate the underlying data that is driving the themes – a critical step for organizations that prioritize accountability and transparency.

You and I have tested platforms that weave AI into every stage of research. And I have been impressed with the advancements represented by these platforms. However, I urge fellow qualitative researchers to exercise caution and not to become overly reliant on these tools. Although the synergy of AI’s precision and human expertise can lead us to more informed and impactful research outcomes, we must diligently verify and interpret the AI’s output because the responsibility for decisions based on AI-generated insights rests on our shoulders.