On January 23, 2025, Darby Steiger presented an AAPOR Webinar on Cognitive Interviewing to nearly 200 participants across the AAPOR community. This post includes excerpts from her presentation. For more information about how cognitive testing can help ensure your next survey is ready for launch, contact us. 

What is a Cognitive Interview?

We conduct surveys to better understand what is on the minds of the public, consumers, voters, and others. We want to make sure that when we are writing our questions for surveys, that our respondents are going to be able to understand and mentally process and respond to our questions. Cognitive interviewing is a cost-effective question testing method that we often use to evaluate and improve survey questions, by understanding how respondents interpret, process, and answer them.

During the cognitive interview, participants are asked to complete a survey or a set of questions while the researcher observes and asks probing questions to gain insight into their thought processes.

Even questions that may seem relatively straightforward can reveal a whole host of issues when we put them through cognitive testing. Here is an example of what a cognitive interview might address.

How many times did you go to the dentist in the past year?

This seems like a reasonable question to ask, but cognitive testing might identify the following types of issues:

• Does “the past year” mean in 2024, or in the past 12 months?
• Should I include visits when I just saw the hygienist? Does “dentist” include an oral surgeon or an orthodontist?
• My usual pattern is twice a year, but I can’t really remember how many times I actually made it in the past year.
• I took my kids to the dentist a few times too, do you want to know how many times I was at the dentist’s office, or just visits for myself?

In identifying these issues during a cognitive interview, we might end up revising our survey question to look something like this:

In the past 12 months, did you have a dental examination or cleaning? Include visits to all types of dental care providers, such as dentists, orthodontists, oral surgeons, and all other dental specialists.

Key Features of a Cognitive Interview

Focus: During the cognitive interview, our goal is to understand the respondent’s thought processes and the potential for response error in a questionnaire.

Timing: We conduct cognitive interviews sometime between the initial drafting of the questionnaire and the timing of the survey launch.

Interviewers: Cognitive interviewers are specially trained researchers that are trained in active listening techniques, and who also have a grounding in the principles of questionnaire design so that they know what to be listening for.

Participants: The participants are those who share characteristics of interest to the survey. For example, if we were preparing to launch a survey with teenagers, we would cognitively test the survey with teens.

Method: We typically conduct 8-15 interviews in a round of testing, and might conduct multiple rounds of testing depending on what we learn. If so, we might pause after a round of testing, tweak the questions based on what we have learned, and go back out for additional testing.

Content: We aren’t always testing the entire questionnaire. We might just focus the interview on a subset of questions, such as:

• New questions
• “Borrowed” questions
• Revisions to existing items
• Existing questions asked of different populations or in different modes
• Translations of existing items

Probing: The Heart of a Cognitive Interview

Probing is the heart of a cognitive interview. There are three traditional ways we ask probes in a cognitive interview:

Concurrent: Concurrent probes are those we ask immediately after a question is asked. As soon as the participant answers the question, we pause and ask them to reflect on what the question was asking and how they arrived at their answer. We tend to use this approach earlier in the questionnaire design process so that we can get feedback on how those questions are performing before we take the time to program a questionnaire.

• Advantages: Captures real-time reactions to a question
• Disadvantages: Interrupts the normal questionnaire flow and may condition participants to overthink

Retrospective: Retrospective probing is when we wait to ask our probes until after a participant has completed a section of the questionnaire, or at the very end of the survey. We tend to use retrospective probes when the questionnaire is closer to its final form because it allows us to get a sense of the actual questionnaire flow. We may also combine concurrent and retrospective probing together to harness the advantages of both approaches.

• Advantages: Captures a more authentic respondent experience without interrupting the flow, can produce a more realistic estimate of survey length
• Disadvantages: Respondents may forget what they were thinking in the moment as they interpreted the question or decided on their answer.

Think-aloud: With the think-aloud approach, participants are asked to verbalize their thought processes as they come up with their answer to the question. This approach allow you to have active insight into the participant’s understanding of the question and their decisions they make as they decide on their answer. This approach is less commonly used.

• Advantages: Avoids the potential bias of the interviewer by influencing the participant with the probes they ask; requires minimal training for interviewers
• Disadvantages: Can be unnatural and burdensome for participants; easy to get off track; results can be more difficult to interpret

Probing Strategies

Regardless of the probing approach, there are two critical types of probing that happen in every cognitive interview: scripted probes and spontaneous probes.

Scripted probes: Scripted probes are generally designed in advance of the cognitive interview in a written protocol. Their focus is on testing predetermined hypotheses and assumptions about what we think might be challenging for survey respondents. Scripted probes might include questions such as:

• How easy or hard was the question to answer?
• What does the term “_____” mean to you?
• How did you decide on your answer?
• How easy or hard was it to find a response choice that fit for you?
• Were any response options missing?
• How certain are you of your answer?

Spontaneous probes: The cognitive interviewer needs to be prepared to administer spontaneous probes based on active listening and observing non-verbal cues such as puzzled expressions or long pauses. If it’s evident that the participant is having difficulty with the question, we will pose spontaneous probes such as:

• “What was going through your mind as you tried to answer the question?”
• “You took a little while to answer that question. What were you thinking about?”
• “You seem to be somewhat unsure about your answer. Can you tell me why?”
• “What caused you to change your answer?”
• “How certain are you of your answer?”

Conclusion

Cognitive testing is an important tool in our questionnaire design toolkit that helps us ensure the questions we ask on surveys will be easy to understand, include clear instructions, have exhaustive response options that capture the full range of perspectives on a topic, and will minimize bias and respondent burden. By conducting a small number of cognitive interviews with participants who represent the types of individuals who will be responding to the survey, cognitive testing is a cost-efficient methodology that can expose potential difficulties that can be fixed prior to survey fielding.

To learn more about cognitive testing, here are some helpful references:

• Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public opinion quarterly, 71(2), 287-311.
• Boeije, H., & Willis, G. (2013). The cognitive interviewing reporting framework (CIRF). Methodology.
• Miller, K., Chepp, V., Willson, S., & Padilla, J. L. (Eds.). (2014). Cognitive interviewing methodology. John Wiley & Sons.
• Tourangeau, R., Maitland, A., Steiger, D., & Yan, T. (2020). A framework for making decisions about question evaluation methods. Advances in questionnaire design, development, evaluation and testing, 47-73.
• Tourangeau, R. (2000). The psychology of survey response. University of Cambridge.
• Willis, G. B. (2004). Cognitive interviewing: A tool for improving questionnaire design. Sage publications.