SSRS ESOMAR 28
01 What experience does your company have in providing online samples for market research?
Context: This answer might help you to form an opinion about the relevant experience of the sample provider. How long has the sample provider been providing this service and do they have, for example, a market research, direct marketing or more technological background? Are the samples solely provided for third party research, or does the company also conduct proprietary work using their panels?
SSRS is a leading full-service, non-partisan public opinion research firm. For more than 30 years, SSRS has specialized in healthcare, public and social policy, and non-partisan political research studies conducted via all data collection modes. Our clients are comprised of various public and private sector entities including major academic institutions, leading national media organizations, and top research institutes.
The SSRS Opinion Panel is a nationally representative panel of U.S. adults age 18 or older. The hallmarks of the SSRS Opinion Panel are methodological rigor, accuracy, affordability, mode flexibility, and representativeness. Our panel is actively used by major academic institutions, media organizations, and other private sector entities – both in the U.S. and abroad.
As a charter member of the AAPOR (American Association for Public Opinion Research) Transparency Initiative, SSRS is committed to full compliance of AAPOR standards and transparency.
SAMPLE SOURCES AND RECRUITMENT
02 Please describe and explain the type(s) of online sample sources from which you get respondents. Are these databases? Actively managed research panels? Direct marketing lists? Social networks? Web intercept (also known as river) samples?
Context: The description of the types of sources a provider uses for delivering an online sample will provide insight into the quality of the sample.
SSRS Opinion Panel members are recruited randomly from a dual-frame random digit dial (RDD) sample, through the SSRS Omnibus survey platform. The SSRS Omnibus survey is a nationally representative (including Hawaii and Alaska) bilingual telephone survey designed to meet standards of quality associated with custom research studies. The SSRS Omnibus completes more than 50,000 surveys annually. In addition, SSRS is in the process of recruiting panelists via ABS (Address Based Sampling) approach.
The advantage of this recruiting design is that it relies on an existing, high-quality sampling frames resulting in an affordable probabilistically-sourced panel. In addition, it affords the flexibility of recruiting a panel that includes adequate representation of typically under-represented groups in public opinion polls such as Hispanics, African Americans, lower educated, or lower income populations.
03 If you provide more than one type of sample source: How are the different sample sources blended together to ensure validity? How can this be replicated over time to provide reliability? How do you deal with the possibility of duplication of respondents across sources?
Context: The variation in data coming from different sources has been well documented. Overlap between different panel providers can be significant in some cases and de-duplication removes this source of error and frustration for respondents.
As mentioned above, SSRS recruits its panel via RDD phone and ABS (Address Based Sampling). They are both probability-based sampling frames. In order to prevent duplication, we de-dupe the panel by email address. In addition, we also use other sources – such as RelevantID, a digital fingerprint technology to prevent duplicate respondents from completing the survey – to make sure our panel is comprised of unique respondents.
04 Are your sample source(s) used solely for market research? If not, what other purposes are they used for?
Context: Combining respondents from sources set up primarily for different purposes (like direct marketing, for example) may cause undesirable survey effects.
The SSRS Opinion Panel is actively used by major academic institutions, media organizations, and other private sector entities – both in the U.S. and abroad. It is used exclusively for research purposes and never for direct marketing.
05 How do you source groups that may be hard to reach on the internet?
Context: Ensuring the inclusion of hard-to-reach groups on the internet (like ethnic minority groups, young people, seniors, etc.) may increase population coverage and improve the quality of the sample provided.
SSRS is sensitive to ensuring that our hard-to-reach groups such as lower-educated respondents, African Americans, and Hispanics are adequately represented in the panel. Our two-pronged RDD-ABS approach uniquely positions us to target hard-to-reach demographic groups effectively. The probability-based Omnibus survey platform screens over 50,000 households annually thereby allowing us to target almost any demographic group. ABS framework allows us to conduct general population recruitment at a high response rate.
The second segment of the population that is inherently not reachable via internet are non-internet respondents (approximately 10% of the population). The SSRS Opinion Panel is a multi-mode panel. All internet households participate via web and all non-internet households participate via phone. SSRS’s own research on non-internet respondents suggests that more than 8 in 10 non-internet respondents are unwilling to take surveys via web. SSRS has therefore made a decision not to provide internet access to non-internet households as providing internet access does not effectively address the coverage bias of non-internet households. By definition, non-internet households need to be contacted via a non-internet mode.
For studies that need completion via a singular web mode yet seek to ensure national representation, the SSRS Advanced Methods team has developed a proprietary propensity weighting solution that takes estimates derived for the U.S. adult internet population and adjusts them to mirror the full U.S. adult population.
06 If, on a particular project, you need to supplement your samples with samples from other providers, how do you select those partners? Is it your policy to notify a client in advance when using a third-party provider?
Context: Many providers work with third parties. This means that the quality of the sample is also dependent on the quality of sample providers that the buyer did not select. Transparency is essential in this situation. Overlap between different providers can be significant in some cases and de-duplication removes this source of error and frustration for respondents. Providers who observe process standards like the ISO standards are required to give you this information.
As a charter member of AAPOR’s (American Association for Public Opinion Research) Transparency Initiative, SSRS is committed to full transparency with our clients and the general public. In the event we need to supplement our panel with sample from other providers, SSRS discloses this upfront to the client.
We have strategic relationships with high-quality sample providers. Our technology infrastructure supports use of sample from multiple partners. One of our de-duping mechanisms is RelevantID, a digital fingerprint technology that prevents duplicate respondents from completing the survey. Please note that SSRS may use other de-duping technology solutions at its discretion.
Sampling and project management
07 What steps do you take to achieve a representative sample of the target population?
Context: The sampling processes (i.e., how individuals are selected or allocated from the sample sources) used are the main factor in sample provision. A systematic approach based on market research fundamentals may increase sample quality.
The SSRS Opinion Panel uses nationally representative probability-based sampling frames to recruit the panelists (dual-frame RDD phone survey and ABS). During the recruitment, we monitor the recruitment by various demographic benchmarks (Age, Gender, Ethnicity, Education, Geography) to ensure representation. These benchmarks are derived from U.S. Census Bureau data (American Community Survey or Current Population Survey). Therefore, our panel statistically represents the U.S. adult population, with a known margin of error.
08 Do you employ a survey router?
Context: A survey router is a software system that allocates willing respondents to surveys for which they are likely to qualify. Respondents will have been directed to the router for different reasons, perhaps after not qualifying for another survey in which they had been directly invited to participate, or maybe as a result of a general invitation from the router itself. There is no consensus at present about whether and how the use of a router affects the responses that individuals give to survey questions.
SSRS does not use survey routers for the SSRS Opinion Panel.
09 If you use a router: Please describe the allocation process within your router. How do you decide which surveys might be considered for a respondent? On what priority basis are respondents allocated to surveys?
Context: Biases of varying severity may arise from the prioritization in choices of surveys to present to respondents and the method of allocation.
SSRS does not use survey routers for the SSRS Opinion Panel.
10 If you use a router: What measures do you take to guard against, or mitigate, any bias arising from employing a router? How do you measure and report any bias?
Context: If Person A is allocated to Survey X on the basis of some characteristic then they may not be allowed to also do Survey Y. The sample for Survey Y is potentially biased by the absence of people like Person A.
SSRS does not use survey routers for the SSRS Opinion Panel.
11 If you use a router: Who in your company sets the parameters of the router? Is it a dedicated team or individual project managers?
Context: It may be necessary to try to replicate your project in the future with as many of the parameters as possible set to the same values. How difficult or easy will this be?
SSRS does not use survey routers for the SSRS Opinion Panel.
12 What profiling data is held on respondents? How is it done? How does this differ across sample sources? How is it kept up-to-date? If no relevant profiling data is held, how are low incidence projects dealt with?
Context: The usefulness to your project of pre-profiled information will depend on the precise question asked and may also depend on when it was asked. If real-time profiling is used, what control do you have over what question is actually asked?
As a part of enrollment process, a variety of socio-demographic and household composition information is collected and saved as part of the panelist profile. This information is collected consistently from all respondents who enroll in the panel. Panelist profile information can then be used to target low incidence populations. Panelist profile information is updated every year via a panel engagement survey.
13 Please describe your survey invitation process. What is the proposition people are offered to take part in individual surveys? What information about the project itself is given in the process? Apart from direct invitations to specific surveys (or to a router), what other means of invitation to surveys are respondents exposed to? You should note that not all invitations to participate take the form of emails.
Context: The type of proposition (and associated rewards) could influence the type of people who agree to take part in specific projects and can therefore influence sample quality. The level of detail given about the project may also influence response.
When a new survey is available, an invitation email plus email reminders are sent to sampled SSRS Opinion Panel members. In addition, all panel members who have provided us consent to receive text messages may also receive text reminders. Non-internet respondents receive an outbound call. Respondents also have the ability to log into their member profile to see if they have any outstanding survey invites. Our standard operating practice is not to provide much project-specific information at the survey invitation stage in order to preserve the unbiased nature of the survey effort.
14 Please describe the (various) incentives that respondents are offered for taking part in your surveys. How does this differ by sample source, by interview length, by respondent characteristics?
Context: The reward or incentive system may have an impact on the reasons why people participate in a specific project and these effects can cause bias to the sample.
All respondents are offered a $5 reward for every web survey they complete. They then have the option of redeeming their rewards as a cash equivalent gift card (such as a Visa gift card) or as a gift card to one of many online or in-store retailers. For surveys exceeding 15 to 20 minutes on average, or surveys involving hard-to-reach populations, we may recommend a higher incentive depending on the subject matter of research.
15 What information about a project do you need in order to give an accurate estimate of feasibility using your own resources?
Context: The “size” of any panel or source may not necessarily be an accurate indicator that your specific project can be completed or completed within your desired time frame.
In order to provide accurate cost and feasibility, it is most helpful to receive the following information at the proposal stage: Target population, incidence rate, sample size, survey length and complexity, subject matter of research, Spanish language requirement and any special requirements or deliverables such as inclusion of video and graphics, openend coding, data tabulation, questionnaire design support, and data analysis support.
16 Do you measure respondent satisfaction? Is this made available to clients?
Context: Respondent satisfaction may be an indicator of willingness to take future surveys. Respondent reactions to your survey from self-reported feedback or from an analysis of suspend points might be very valuable to help understand survey results.
All survey-specific respondent communication contains contact information for the SSRS Opinion Panel manager. We carefully review survey-specific respondent feedback and take necessary action promptly in the event an anomaly is detected. In addition, we can also add survey-specific respondent feedback questions. This information is used for our internal troubleshooting and not a part of data delivered to our clients.
17 What information do you provide to debrief your client after the project has finished?
Context: One should expect a full sample provider debrief report, including gross sample, start rate, participation rate, drop-out rate, the invitations/contact text, a description of the field work process and so on. Sample providers should be able to list the standard reports and metrics that they make available.
As a charter member of AAPOR’s (American Association for Public Opinion Research) Transparency Initiative, SSRS reporting is fully compliant with AAPOR’s transparency standards. Upon completion of the study, in addition to a fully labeled SPSS data file with demographic weights, SSRS also provides a methods report containing detailed information on design, sampling, AAPOR response rate calculations, weighting, and data processing. In addition, this report can also contain various performance metrics including but not limited to: margin of error, design effect, survey start rate, survey completion rate, drop-out rate, item non-response rate, length of interview. We can look into the feasibility of providing additional para-data as needed.
18 Who is responsible for data quality checks? If it is you, do you have in place procedures to reduce or eliminate undesired within survey behaviors, such as (a) random responding, (b) illogical or inconsistent responding, (c) overuse of item non-response (e.g. “Don’t Know”), or (d) speeding (too rapid survey completion).
Please describe these procedures.
Context: The use of such procedures may increase the reliability and validity of the survey data.
Unlike non-probability panels, SSRS Opinion Panel respondents are scientifically selected based on probability-based methods. The level of panel engagement is therefore significantly higher than non-probability panels and undesirable behaviors such as straight-lining, speeding and high item non-response are minimized. We are however sensitive to these data quality issues. The SSRS project management team is responsible for quality control of all survey data. The project team examines the data for common signs of maleficence, such as straight-lining, unusually fast lengths of interview, and high item non-response.
19 How often can any individual be contacted to take part in a survey whether they respond to the contact or not? How does this vary across your sample sources?
Context: Over-solicitation may have an impact on respondent engagement or on self-selection and non-response bias.
The SSRS Opinion Panel is a relatively low burden panel in that our panelists do not receive more than 2 to 3 surveys per month. This keeps the engagement level and data quality high.
20 How often can any individual take part in a survey? How does this vary across your sample sources? How do you manage this within categories and/or time periods?
Context: Frequency of survey participation may increase the risk of undesirable conditioning effects or other potential biases.
Respondents may complete as many surveys as they are invited to. Length in field is usually project-specific but is generally no more than a few weeks. On an average SSRS Opinion Panel members do not receive more than 2 to 3 surveys per month. Panelists typically complete an average of 1 to 2 surveys per month.
21 Do you maintain individual level data, such as recent participation history, date of entry, source, etc., on your survey respondents? Are you able to supply your client with a project analysis of such individual level data?
Context: This type of data per respondent, including how the total population is defined and how the sample was selected and drawn, may increase the possibilities for analysis of data quality.
SSRS maintains individual participation data including participation rate, number of surveys completed, dates of completed surveys, and registration date. We analyze and use this data for panel management purposes. This data is not a part of our standard deliverables. We can provide some of this data upon request.
22 Do you have a confirmation of respondent identity procedure? Do you have procedures to detect fraudulent respondents? Please describe these procedures as they are implemented at sample source registration or at the point of entry to a survey or router. If you offer B2B samples what are the procedures there, if any?
Context: Confirmation of identity can increase quality by decreasing multiple entries, fraudulent panelists, etc.
Our probability-based recruitment methods ensure that only respondents with a confirmed identity are invited into the panel. In addition, we implement a double opt-in methodology at the panel registration stage to confirm respondent identity. We also monitor panelists for undesirable behaviors such as consistent speeding, straight-lining, illogical response patterns, inconsistent panel participation and consistently high item non-response. Such panelists may be removed from the panel as a quality control measure.
Policies and compliance
23 Please describe the ‘opt-in for market research’ processes for all your online sample sources.
Context: The opt-in process indicates the respondents’ relationship with the sample source provider. The market generally makes a distinction between single and double opt-in. Double opt-in refers to the process by which a check is made to confirm that the person joining a panel or database wishes to be a member and understands what to expect (in advance of participating in an actual survey for a paying client).
SSRS implements a double opt-in process for all panelists at the recruitment stage. This ensures that all panelists joining the panel are fully aware of what to expect when participating in panel surveys. In addition, the SSRS Opinion Panel website contains a detailed list of FAQs for panelists: https://ssrsopinionpanel.com/faq/
25 Please describe the measures you take to ensure data protection and data security.
Context: The sample provider usually stores sensitive and confidential information on panelists and clients in databases. These data need to be properly secured and backed-up, as does any confidential information provided by the client. The sample provider should be able to provide you with the latest date at which their security has been evaluated by a credible third-party.
All panelist profile information is stored in highly secure servers. To ensure respondent anonymity, SSRS does not include Personally Identifiable Information (PII) in client datasets. All cases in client data are given a unique ID which they cannot tie back to individual respondents. Internally, SSRS maintains a database that can match these unique IDs to individual respondents for management purposes. All panelists have the ability to opt-out of the panel by simply contacting the panel manager.
26 What practices do you follow to decide whether online research should be used to present commercially sensitive client data or materials to survey respondents?
Context: There are no foolproof methods for protecting audio, video, still images or concept descriptions in online surveys. In today’s social media world, clients should be aware that the combination of technology solutions and respondent confidentiality agreements are “speed bumps” that mitigate but cannot guarantee that a client’s stimuli will not be shared or described in social media.
SSRS acknowledges that it is not possible to guarantee complete confidentiality of data sent over the internet. SSRS has extensive experience in handling studies involving confidential information. SSRS initiatives to safeguard commercially sensitive data includes data use agreements with clients and vendors we work with, in addition our internal processes and IT security infrastructure are geared towards minimizing this risk.
27 Are you certified to any specific quality system? If so, which one(s)?
Context: Being certified may require the supplier to perform tasks in a pre-determined manner and document procedures that should be followed.
SSRS maintains high project quality standards by following rigorous guidelines for sample design, questionnaire programming, quality control, data processing and reporting, as well as weighting post data collection.
SSRS uses Confirmit software and data warehousing solution for SSRS Opinion Panel. Confirmit is the world’s leading provider of survey and reporting software, delivering the most complete and robust market research platform available. Confirmit is a sophisticated web-based data collection tool with capabilities to deliver tailored, fully-branded, and highly-structured surveys for all types of feedback and market research programs.
- Survey Design: Easily create custom-branded surveys that accommodate any level of complexity.
- Data Collection: Collect feedback through virtually any channel including CATI, web, email, mobile, SMS, IVR, face-to-face, and paper.
- Reporting: Drill down into structured and unstructured feedback and provide reports tailored to your customer needs.
- Panel Management: Manage your panels efficiently, increase participation, and deliver accurate and representative feedback.
For decades Confirmit has maintained one of the highest levels of security in our industry, allowing them to earn the trust of the most demanding companies, and meet their stringent security and privacy requirements. Confirmit’s operations undergo an annual comprehensive SOC 2 Type II (SSAE 18 / AT 101) examination by a highly-reputable AICPA accredited firm. Their SaaS production environment, where client data is stored and processed, is ISO 27001 certified, SOC 2 Type II audited and PCI-DSS certified.
In addition to the comprehensive third-party SOC 2 Type II audits of their operations, Confirmit performs a battery of additional testing to ensure the effectiveness of their security posture. These include weekly penetration testing by information security team which are validated by annual third-party penetration tests by independent security professionals (McAfee). Confirmit also performs static code-scanning of the software, which is validated by third-party white-hat application testing performed by highly trained cyber-security professionals (Veracode). Confirmit has always completed the third-party testing with the highest security grade available, “A”.
Confirmit’s full security features can be reviewed here: https://extranet.confirmit.com/Extranet/media/PublicDocuments/SaaS/Confirmit_Horizons_SaaS_Security-January_2019.pdf
28 Do you conduct online surveys with children and young people? If so, do you adhere to the standards that ESOMAR provides? What other rules or standards, for example COPPA in the United States, do you comply with?
Context: The ICC/ESOMAR International Code requires special permissions for interviewing children. These are described in the ESOMAR Guideline for Online Research. In the USA, researchers must adhere to the requirements of the Children’s Online Privacy Protection Act (COPPA). Further information on legislation and codes of practice can be found in Section 6 of the ESOMAR Guideline for Online Research.
The SSRS Opinion Panel does not include respondents under the age of 18.