Darby Steiger and Nashay Lowe Present:

Qualit-AI-tive Analysis: How Does AI Stack Up Against Humans?

AI tools are emerging at a rapid pace in all aspects of humanity, alluring us with their capabilities and speed, and providing a vision of the future where we can have answers to anything and everything with the snap of our fingers in the form of a well-crafted prompt. How this impacts us as qualitative researchers is exciting and terrifying all at once. How can we trust that AI is generating valid and reliable answers? How can AI address the thorny issues of baked-in bias based on the language models it uses to train itself? And as much as we’d like to say the answer is no, are there any tasks that AI might actually be able to do better than us as expert qualitative researchers?

SSRS, a social science research firm, has been testing a number of AI platforms and tools to explore how we might leverage AI in the analysis phase for qualitative research studies including focus groups, IDIs, and online boards. This presentation reviews our experiences with one such platform that has shown promising results for our needs. We will present our experiences testing CoLoop on a variety of studies and populations, including focus groups and IDIs with Native Americans, online bulletin boards with sports fans in 15 countries, and focus groups with people with Parkinson’s Disease. We will review our experiences with the platform, results of a comparison to human-analysis, and provide recommendations for researchers considering using AI-enabled analysis platforms.