To validate the Encipher® Nonprob methodology, we used several surveys that were fielded simultaneously to (1) probability samples selected from the SSRS Opinion Panel and (2) nonprobability samples obtained from an opt-in panel vendor.

These surveys included calibrators from the Calibration Item Bank as well as some selected outcome items covering various topic areas. All items were administered to both the probability and nonprobability samples, allowing us to assess the effectiveness of calibration at reducing selection bias in the outcomes.

In the results that follow, we compare estimated outcomes between four hypothetical designs:

• Probability: using only the probability sample from the SSRS Opinion Panel, weighted on standard demographics. This provides the benchmark against which we measure selection bias in the nonprobability designs.
• Nonprobability – demo: using only the nonprobability sample, weighted only on standard demographics (i.e., not using Encipher® Nonprob).
• Nonprobability – Encipher®: using only the nonprobability sample, applying Encipher® Nonprob to calibrate both on standard demographics and on non-demographic calibrators.
• Pseudo-hybrid: using only the nonprobability sample but applying the hybrid version of Stepwise Calibration—that is, optimizing on the actual bias in outcomes, rather than on the model-based prediction of bias. We call this a “pseudo-hybrid” design because (unlike in a true hybrid design) we still produce the final estimates only using the nonprobability sample; but we use the probability-based estimates to help identify the calibration model that minimizes bias in the nonprobability estimates. We include this design as a point of comparison to assess how well the Encipher® Nonprob solution (which assumes that we do not have probability-based data for outcomes) replicates the informational advantage afforded by a hybrid design (in which we do have probability-based data for outcomes)

Example Outcome: E-cigarette Use

Figure 1 shows one of the outcome estimates—the percent of adults who use e-cigarettes or other vaping products—under each of the four designs.

If we relied solely on standard demographic weighting, a nonprobability sample (Nonprobability – demo) would overestimate this percentage by about 6 percentage points, relative to the probability-based benchmark. Encipher® Nonprob reduces this selection bias to about 3 percentage points, offering a substantial improvement over simple demographic-only weighting of the nonprobability sample.

Figure 1: Percent using e-cigarettes, by sampling/weighting methodology

As illustrated by the Pseudo-hybrid results, we could further reduce selection bias by using the hybrid version of our calibration procedure, which optimizes on the actual rather than predicted bias. This additional bias reduction illustrates the value of the parallel probability sample within a hybrid design (relative to a nonprobability-only design). Specifically, we can develop a more effective weighting model when we have a probability-based benchmark that allows us to measure the actual selection bias in the outcomes. However, if this is not an option, the Encipher® Nonprob solution, optimizing on the model-based prediction of bias, still improves upon simple demographic-only weighting.

Figure 2 generalizes these results, plotting the observed selection bias (i.e., the difference from the probability-based benchmark) for all outcomes collected in the validation study. The figure reports the mean and (due to the presence of one outlier) median selection bias across all outcomes. Overall, although not quite as effective as a hybrid design, Encipher® Nonprob reduces the median selection bias by over 30% relative to standard demographic-only weighting.

Figure 2: Selection bias in validation study outcomes, by sampling/weighting methodology

The validation study reported here demonstrates that, by leveraging SSRS benchmarking samples and advanced predictive modeling techniques, Encipher® Nonprob can improve upon standard demographic-only weighting for studies that rely entirely on nonprobability samples.

That said, this study also reinforces that hybrid designs should usually be preferred over nonprobability-only designs where feasible, since the availability of probability-based estimates for key study outcomes allows for the development of more effective calibration models. In a real-world study, without a parallel probability sample in which outcome items were asked, we ultimately cannot know whether the selected calibration model succeeded at meaningfully reducing selection bias.

SSRS’s team of methodologists and data scientists can provide consultation to help researchers consider the tradeoffs among the available options for working with nonprobability samples. For many studies, the SSRS Opinion Panel is a feasible source of affordable probability-based samples, making a full hybrid design accessible at a reasonable price point. For researchers who do not wish to administer a full questionnaire to a parallel probability sample, but still want to take advantage of the benefits of a hybrid design, another option would be to run a small sub-selection of critical outcomes and topic-customized calibration items on the SSRS Opinion Panel Omnibus.

For more information about how Encipher® Hybrid or Encipher Nonprob could be useful for your study, contact us today.