Public Agenda, a nonprofit research and public engagement organization, engaged SSRS to conduct a survey in the spring of 2022 to understand perceptions about the quality and value of for-profit institutions of higher learning in the U.S. The survey’s target population consisted of five groups: (1) current students at for-profit institutions, (2) alumni of for-profit institutions, (3) lapsed students of for-profit institutions, (4) current community college students, and (5) alumni of distance education programs.

The unique and relatively low-incidence target population for this study raised two challenges. First, while probability samples generally yield more accurate estimates than nonprobability samples, collecting the entire sample from probability-based source would have been cost- and time- prohibitive. Second, there are no externally available weighting benchmarks for these five groups.

To obtain a sufficient sample size and allow for valid weighting in a cost-effective manner, the study was conducted using a hybrid sample design that blended:

• A probability sample of 336 respondents from the SSRS Opinion Panel.[1]
• A nonprobability sample of 882 respondents from several opt-in online panel vendors.

[1] To identify the target population, screening and demographic questions were administered to a larger probability sample of 9,361 panelists with a high school diploma through a Bachelor’s degree. This larger sample was weighted to demographic benchmarks for U.S. adults with a high school diploma through a Bachelor’s degree and then subset to the 336 respondents who were in one of the five target groups.

The hybrid sample was weighted using SSRS’s Encipher® Hybrid calibration solution. Encipher® Hybrid is an SSRS-developed methodology that corrects for both demographic and non-demographic differences between probability and nonprobability samples, reducing any selection bias associated with the inclusion of nonprobability respondents. The hybrid design allowed us to use the probability sample to derive demographic and non-demographic weighting benchmarks for the target population, to which the full hybrid sample could then be weighted.

Figures 1 and 2 illustrate how the use of a calibrated hybrid sample allowed this study to obtain more accurate estimates than would have been obtained by relying solely on nonprobability sample sources.

As shown in Figure 1, the distribution of the nonprobability sample across the five groups differs substantially from the population distribution estimated from the probability sample. For example, the probability sample estimates that current students of for-profit institutions make up about 13% of the target population, but such students make up about 22% of the nonprobability sample.

As noted above, there is no external source for these estimates; therefore, without the concurrent probability sample, there would have been no way to correct these imbalances in the nonprobability sample. The inclusion of the SSRS Opinion Panel sample provided a probability-based estimate of the population distribution, to which the combined sample was calibrated.

As shown in Figure 2, differences between the probability and nonprobability samples carried through to some of the substantive measures collected on the survey; but after calibration, the hybrid estimates were much more closely aligned with the probability-based estimates.

Specifically, about half of the nonprobability respondents reported a positive impression of for-profit colleges, compared to the probability-based estimate of about 17%; and about 68% of nonprobability respondents reported that paying for college was easy, compared to the probability-based estimate of about 59%.

This suggests that the nonprobability sample overrepresented persons with these views, relative to the target population. However, with SSRS’s Encipher® Hybrid calibration, the combined sample—blending the probability and nonprobability respondents to allow for a larger total sample—produced substantive estimates that were comparable to what would have been obtained from a purely probability-based sample.

These results demonstrate how hybrid samples, weighted using SSRS’s Encipher® Hybrid methodology, can cost effectively collect data from relatively rare populations, obtaining larger samples than would be feasible from probability-only sources while minimizing selection bias.