Research Objectives

Probability panels like the SSRS Opinion Panel have a lot of advantages. Like online opt-in panels, they provide access to a relatively low-cost pool of online survey respondents. Unlike opt-in panels, they have known coverage properties, calculable response rates, the ability to include non-Internet-users, and much less risk of contamination by fraudulent respondents or bots.

Of course, no sample is perfect; like other probability-based sample sources, probability panels are affected by nonresponse. One metric to which we pay particular attention is the recruitment response rate: the percentage of people who joined the Panel as a percentage of those invited to do so. Higher recruitment response rates are desirable because they reduce the chance that those who join the Panel are different from the broader population in some way that might change survey estimates.

In several recent recruitment waves for the SSRS Opinion Panel, we tested the use of an intensive Nonresponse Follow-Up (NRFU) protocol to increase the recruitment response rate. We wanted to know:

• By how much does the NRFU protocol increase the recruitment response rate to the SSRS Opinion Panel?
• Once on the Panel, do the additional panelists recruited via the NRFU protocol actually respond to surveys when invited to do so?
• Are the additional panelists recruited via the NRFU protocol demographically or otherwise different from those recruited via the standard protocol?

Study Design

The standard recruitment protocol for the SSRS Opinion Panel involves three mailings: an initial invitation letter, a reminder postcard, and then a final reminder letter. All of these mailings offer the respondent the opportunity to complete the recruitment survey either online or by calling a toll-free number to speak with a live interviewer (“inbound phone”).
Within the 2024 recruitment waves, a random treatment group (the “NRFU treatment”) was designated to receive the additional NRFU protocol after the standard three-mailing protocol. The remaining “non-NRFU control” received only the standard three-mailing protocol.

The NRFU protocol included two additional mailings for those who did not respond to the first three:

• A fourth mailing, sent in a USPS Priority Mail® envelope, that included a reminder letter, a paper version of the recruitment survey, and a prepaid return envelope for those who wished to complete the paper version.
• A final “last ditch” reminder letter, sent in a differently-shaped envelope from the prior mailings.

Additionally, the NRFU protocol included an attempt to reach sampled households by phone (“outbound phone”) and a follow-up refusal conversion mailing to those that completed the recruitment survey but declined to join the Panel.

Key Findings

Response to Recruitment Survey

Figure 1 shows the recruitment response rate in the non-NRFU control group vs. the NRFU treatment group in the April and May 2024 recruitment waves for the SSRS Opinion Panel. For each group, the larger percentage shows the percent of initially sampled households in which a person completed the Panel recruitment survey, regardless of whether they ultimately joined the Panel. The lower percentage shows the percent who both completed the survey and agreed to join the Panel, which is the final recruitment response rate.

Figure 1: Effect of NRFU protocol on SSRS Opinion Panel recruitment response rate

In both April and May, the extra contact attempts in the NRFU treatment approximately doubled the final recruitment response rate to the SSRS Opinion Panel.

Completion of Panel Surveys

While a higher recruitment response rate is desirable, the ultimate objective is to recruit panelists who will complete surveys. Increasing the recruitment response rate could become counterproductive if the additional panelists are less likely to respond to surveys after joining the Panel.

Fortunately, there is no evidence that this is the case. Figure 2 tracks the completion rates to the first five surveys to which new recruits from the April and May 2024 waves were invited. So far, new panelists who joined from the NRFU treatment have shown very similar survey-level completion rates to those who joined from the non-NRFU control.

Figure 2: Completion rate to first five SSRS Opinion Panel surveys, non-NRFU control vs. NRFU treatment

Demographics of Recruited Panelists

Figure 3 shows the unweighted demographic composition of new panelists from the April and May 2024 waves, again comparing the NRFU and non-NRFU groups. The NRFU treatment was particularly effective at improving the representation of Hispanic adults who were born outside the U.S. and of adults who use the Internet less frequently. This is a positive finding, because both of these subpopulations are often underrepresented in online surveys.

Figure 3: Demographics of SSRS Opinion Panel joiners, non-NRFU control vs. NRFU treatment

It is important to ensure that the demographic composition of the Panel approximately matches the U.S. adult population (adjusting for response rate differences) to ensure that we can obtain adequately-sized samples of various subpopulations. Ultimately, however, small demographic differences can be usually corrected by weighting the sample to population benchmarks from sources like the Census. Therefore, an important follow-up question is whether, after adjusting for demographic differences, the NRFU condition differs with respect to attitudes and behaviors that do not have Census benchmarks.

To help answer this question, we can look at some attitudinal and behavioral questions that we included on the recruitment survey, after reweighting the panelists recruited from each treatment group to correct for the known demographic differences.

As shown in Figure 4, most of these measures did not show statistical or practical differences between the treatments. However, there are some exceptions.

For example, panelists recruited via the NRFU protocol were less likely to say they voted in the prior presidential election. This is a useful finding, because prior research suggests that online survey panels (even those that, like the SSRS Opinion Panel, are recruited from random samples) tend to overrepresent adults who are more civically engaged. Increasing the share of non-voters among new panelists should help to reduce this kind of overrepresentation. This suggests that the NRFU protocol may be useful at reducing nonresponse bias in non-demographic variables that are hard to correct with weighting.

Figure 4: Weighted outcome measures from SSRS Opinion Panel joiners, non-NRFU control vs. NRFU treatment

Conclusions

Overall, our initial testing of the extended NRFU protocol for SSRS Opinion Panel recruitment found that it approximately doubled the recruitment response rate. Thus far, we have seen no negative impact on survey-level completion rates, though we will continue to monitor this as the NRFU recruits receive more surveys. Differences in the characteristics of panelists recruited via the NRFU protocol are generally small, but suggestive of improved representativeness in some respects (e.g., better representation of immigrant adults and non-voters).

As we continue to experiment with the role of the NRFU protocol in SSRS Opinion Panel recruitment, one goal is to identify which aspects of the NRFU were most effective at obtaining these results and which could potentially be omitted. For example, can we obtain a similar result if we continue to send the fourth mailing in a Priority envelope, but do not offer the hardcopy survey? These insights will help us to continue to hone the key advantage of probability panels—the ability to obtain high-quality data at an affordable cost.