Reliability and accuracy of online panel derived data
Session Title
Session 1-4-A: Methods in Problem Gambling Research
Presentation Type
Event
Location
Caesars Palace, Las Vegas, Nevada
Start Date
28-5-2019 3:30 PM
End Date
28-5-2019 4:55 PM
Disciplines
Clinical Psychology | Other Psychology
Abstract
There is an increasing trend for researchers to use online panels and crowdsource websites as a source of participants for survey and questionnaire based studies. The driver for this is convenience and capacity to recruit large sample sizes at low cost. There is a stated assumption that online panel respondents are representative of the general population, and that responses are more accurate given their anonymity. Data derived from these samples are used to guide policy decisions, public health initiatives, and treatments. Although sample demographics are often described as representative of the general population, the claim that the responses are more accurate and generalizable is increasingly being drawn into question. A rapid review of the literature since 2012 identified 18 online panel and 7 crowdsource studies, with most published in the previous two years. Results indicated that 60% of these studies failed to report response, or reported participation rather than true response rates. Sixty percent applied a priori or post hoc adjustments to improve data quality. Concerns relate to the high number of participants excluded (30-50%) on the basis of inconsistent responses, rapid completion of questionnaires suggesting inadequate reading of items, and systematic pattern responses. Comparison with reported prevalence rates of gambling disorder ranging from 0.2-1.2% (standardized rate of 2.3%), revealed significantly higher rates of 16.7% to 23.2% for online panels. The limitations of online panels and crowdsource sampling methods will be highlighted and recommendations on procedures to be adopted for data integrity purposes will be described.
Keywords
Gambling disorder; problem gambling; crowdsource; online panel; data quality; Amazon MTurk; Qualtrics
Funding Sources
Nil funding for this study.
Competing Interests
Declarations: Alexander Blaszczynski For the period 2015 – 2018, Professor Blaszczynski has conducted research funded directly by Australian or international government, or government-related funding agencies, and industry operators. These include Gambling Research Exchange Ontario, ClubsNSW, Dooleys Club Lidcombe, Aristocrat Leisure Industries, Australian Communications Media Authority, Gaming Technologies Association, Gambling Research Australia, Responsible Wagering Australia, Commonwealth Bank, NSW Department of Trade and Investment (NSW Office of Liquor, Gaming and Racing), La Loterie Romande (Switzerland), Camelot (United Kingdom), La Française des Jeux (France), Loto-Quebec (Canada), and National Lottery (Belgium), and the National Association for Gambling Studies. He has received honorariums from Manitoba Gambling Research Program and GambleAware (formerly UK Responsible Gambling Trust) for grant reviews, and royalties from several publishers for books and book chapters. He has also received travel and accommodation expenses from Leagues Clubs, Gambling Research Exchange Ontario, USA National Council on Problem Gambling, Japan Medical Society for Behavioural Addiction, Le Comité d'organisation Congrès international sur les troubles addictifs, Victorian Responsible Gambling Foundation, North American Association of State and Provincial Lotteries, and New Horizons (British Columbia Lottery Corporation to attend conferences and meetings. All professional dealings have been conducted with the aim of enhancing responsible gambling and harm minimisation policies and practices, training counsellors in the treatment interventions, and advancing our understanding of the psychology of gambling. Declaration for Dylan Pickering:Funding: Over the last three years, DP has received salary and conference travel costs based on the deed of gift provided by ClubsNSW. DP has received funding for research projects from Gambling Research Exchange Ontario (GREO), and National Association for Gambling Studies (NAGS).
Reliability and accuracy of online panel derived data
Caesars Palace, Las Vegas, Nevada
There is an increasing trend for researchers to use online panels and crowdsource websites as a source of participants for survey and questionnaire based studies. The driver for this is convenience and capacity to recruit large sample sizes at low cost. There is a stated assumption that online panel respondents are representative of the general population, and that responses are more accurate given their anonymity. Data derived from these samples are used to guide policy decisions, public health initiatives, and treatments. Although sample demographics are often described as representative of the general population, the claim that the responses are more accurate and generalizable is increasingly being drawn into question. A rapid review of the literature since 2012 identified 18 online panel and 7 crowdsource studies, with most published in the previous two years. Results indicated that 60% of these studies failed to report response, or reported participation rather than true response rates. Sixty percent applied a priori or post hoc adjustments to improve data quality. Concerns relate to the high number of participants excluded (30-50%) on the basis of inconsistent responses, rapid completion of questionnaires suggesting inadequate reading of items, and systematic pattern responses. Comparison with reported prevalence rates of gambling disorder ranging from 0.2-1.2% (standardized rate of 2.3%), revealed significantly higher rates of 16.7% to 23.2% for online panels. The limitations of online panels and crowdsource sampling methods will be highlighted and recommendations on procedures to be adopted for data integrity purposes will be described.