Overview

At the 79th Annual Conference of the American Association for Public Opinion Research (AAPOR), SSRS Vice President of Innovations and Solutions, Darby Steiger, led a three-hour collaborative session with qualitative researchers across the public opinion industry to discuss the future of qualitative research and how artificial intelligence (AI) has the potential to, and is already, transforming the way we conduct qualitative research. Darby also presented the key findings of this session at the recent DC-AAPOR conference during a session entitled “AI and Advanced Text Analysis in Survey Research.”

This blog post summarizes the main takeaways from this industry-wide session, and does not necessarily reflect our own internal explorations of AI for qualitative research at SSRS.

Background

As part of a new initiative at AAPOR’s 79th annual conference, researchers were encouraged to submit abstracts to form “Idea Groups,” for small groups of attendees to gather for focused conversations about issues challenging our industry. Darby and four other colleagues across AAPOR convened an Idea Group on the “Future of Qualitative Research.” The goal of this sharing session was to create a space for qualitative researchers to brainstorm the benefits and downsides of incorporating AI into our research, share what AI tools our organizations have been testing, and to explore the implications of AI on AAPOR, QUALPOR (the qualitative affinity group of AAPOR), and the future of qualitative research. A total of 25 qualitative researchers convened for this session, representing federal agencies, academia, private research organizations, and independent contractors.

Principal Findings

During the session, we collaboratively generated an extensive list of how AI is transforming the way we conduct qualitative research, from the proposal phase of research through to the analysis and reporting phase.

DS Blog Timeline

Proposal Phase

At the proposal phase, participants suggested a number of ways that AI could support proposal activity:

  • Review the RFP and identify the requirements
  • Create a timeline for the proposal response
  • Draft the proposal outline
  • Draft a project schedule
  • Conduct a preliminary literature review & collect background information
  • Review the draft proposal against the RFP to critique the response

Some participants urged caution in sharing proprietary information from an organization’s proposal with an open-source platform such as ChatGPT.

Design Phase

At the design phase of qualitative research studies, we often focus on generating project timelines, drafting protocols and research materials that will address our research questions in engaging and productive manners, and creating screener questionnaires to identify the types of participants we wish to study. Participants in the Idea Group discussed several ways that AI can help support this phase of qualitative research.

Many spoke about using open AI tools as a sort of “intern” to perform the following types of tasks:

  • Create initial drafts of protocols
  • Generating first drafts of stimuli, messaging, icons, or other materials that may be tested with qualitative participants.
  • Drafting screener questionnaires
  • Using synthetic respondents (based on detailed personas) to programmatically test the efficacy of the screener.
  • Tailoring language for informed consent documents to be more appropriate for the populations being studied

Recruitment

At the recruitment stage, participants had several ideas for leveraging AI tools to publicize the study, identify potential qualified participants and prepare recruitment materials. Ideas included:

  • Targeting respondents within a large database
  • Generating a list of key social media influencers or top 10 users on Reddit to help promote the study
  • Asking ChatGPT for suggestions on local organizations or sources to turn to find rare populations
  • Supporting translation of recruitment materials

Participants noted that AI or natural language processing could be helpful in processing open-ended answers on recruitment screeners to identify fake respondents, but that users should take care not to accidentally weed out non-native English speakers.

Data Collection

Participants were aware of many different ways that AI could be used to support qualitative data collection, though few organizations reported actually using these tools, other than for AI transcription.

  • AI-moderating using a tool that does all of the moderating and probing
  • AI-probe assistance in either asynchronous bulletin boards or in live focus groups
  • Live transcription
  • Live summaries, sentiment coding
  • Live translation
  • Live behavior coding (For example, if there are long pauses, providing suggestions of what to say next)
  • Using synthetic participants to fill in the gaps in the study design
  • For cognitive testing, using AI to help improve questions between rounds, or to identify emerging themes or gaps

Analysis

At the analysis phase, several participants mentioned that their organizations have been developing in-house NLP or AI tools within their organizations to comb through transcripts or recordings to identify themes, code data, to summarize results, or to draft recommendations or implications. A few participants mentioned their organizations do not have this capacity so are testing outside AI platforms that can perform these functions. However, they mentioned the importance of carefully vetting these platforms to ensure they are closed systems that are protecting the data and not using it to train other models.

Caveats: Ethical and Compliance Issues

During the Idea Group, participants discussed several ethical and compliance issues that should be considered when deciding whether and how to use AI to support qualitative research. These included:

  • Data Privacy and Security:  How do AI tools protect participant data, keep it confidential, and prevent it from being shared outside of the platform?
  • Ethical Considerations:  How do organizations convey to participants how their data is being stored and protected when using AI tools? What are our responsibilities of what we communicate to participants?
  • Equity and Bias Mitigation:  Recognizing inherent biases in AI algorithms and LLMs, how do we mitigate that bias and assess AI results through an equity-focused lens?
  • Trust and Fraud: How do we know the AI results can be trusted, and from the opposite perspective, how can we make sure we are detecting when participants or respondents are fake or using AI to generate answers?
  • Maintaining Human Element:  With all of the AI temptations that are emerging, how do we keep humans at the helm of qualitative research?
  • Nuance and Culture: How well can AI pick up on human emotions like humor and sarcasm, and understand nuance and cultural differences?

Conclusion

SSRS is proud to have led the formation and execution of this important session, laying the groundwork for many future conversations within the AAPOR and QUALPOR communities about the appropriateness of using AI tools in our qualitative research.