SSRS recently conducted a novel qualitative study for the Washington Post-Schar School 2024 Deciders Study in which we were asked to gather real-time qualitative feedback during the only 2024 presidential debate held between Donald Trump and Kamala Harris.

The goal of the study was to be able to capture immediate feedback on the candidates’ statements throughout the debate from a large and diverse group of potential voters who had not yet solidified their vote preference and for the Washington Post to be able to publish the results instantly.

The Challenge

In designing our methodological approach, we found that our traditional qualitative toolkit did not seem to be able to meet the needs of this study, including allowing for nearly real-time publication of findings per the client’s request.

A live focus group would have had several disadvantages, including:

  • A small group size
  • The innate disruption of having to interrupt the debate to ask participants to share their reactions
  • The inability to collect reactions from every participant to every probe, and
  • The challenges of providing the client with instant access to the data.

A dial test would have been able to capture reactions in a numerical format (for example, capturing moment-by-moment reactions on a 100-point scale) but would have only allowed us to gather the reasons behind their reactions in a post-survey focus group, at which point participants may have forgotten why they felt that way in the moment.

And an asynchronous board would have required us to pre-set the questions, which would not have allowed us to probe on what was actually being discussed during the debate.

The Solution

Our solution involved using a novel methodology that allowed for larger-scale qualitative insights in real-time. Using Sago’s Crowd Survey platform, we were able to pose tailored questions that were created in-the-moment, thus allowing us to gather simultaneous feedback from dozens of prescreened potential voters and to view and download their responses in real-time.

In practice, this meant that as soon as the candidates finished discussing a particular issue, we could launch a question on the platform asking which candidate made better arguments about that specific issue and to explain why they chose their answer. Similarly, we could ask for reactions to a specific statement made by a candidate, such as whether the statement made sense (and why) or whether they agreed or disagreed with the statement (and why).

 

 

 

Participants remained logged into the platform throughout the entire debate, and each time a question was posed, we typically received 100% response within about 8 minutes. Data were available to the Washington Post team for immediate download, displaying the responses along with participants’ demographic characteristics.

Since this was such a new methodology, one of the challenges we faced was making sure that the participants understood how to log onto the platform and what would be happening throughout the evening. To address this, we designed a communication “campaign” in the days leading up to the debate, sending daily emails and texts to participants to give them basic tips about what to expect. When they arrived on the platform a few minutes before the start of the debate, this information was repeated to them on the landing page of the platform. This active communication was helpful for building rapport between the research team and the participants and contributed to a 100% show rate and a 96% completion rate.

Potential Applications

We are excited by the possibilities that this new methodology offers for gathering large-scale qualitative feedback during live events. Whether the results are made public or used internally for strategic insights, the potential to capture instant reaction – and “the why” – in the moment offers a variety of potential applications for future consideration. Applications could include:

  • Live feedback from fans during sporting events, drafts, online gaming
  • Live feedback during State of the Union, political conventions, debates, town halls, film screenings
  • Instant reactions during usability testing, product testing, mystery shopping