Submission Title

Reliability and accuracy of online panel derived data

Session Title

Session 1-4-A: Methods in Problem Gambling Research

Presentation Type

Event

Location

Caesars Palace, Las Vegas, Nevada

Start Date

28-5-2019 3:30 PM

End Date

28-5-2019 4:55 PM

Disciplines

Clinical Psychology | Other Psychology

Abstract

There is an increasing trend for researchers to use online panels and crowdsource websites as a source of participants for survey and questionnaire based studies. The driver for this is convenience and capacity to recruit large sample sizes at low cost. There is a stated assumption that online panel respondents are representative of the general population, and that responses are more accurate given their anonymity. Data derived from these samples are used to guide policy decisions, public health initiatives, and treatments. Although sample demographics are often described as representative of the general population, the claim that the responses are more accurate and generalizable is increasingly being drawn into question. A rapid review of the literature since 2012 identified 18 online panel and 7 crowdsource studies, with most published in the previous two years. Results indicated that 60% of these studies failed to report response, or reported participation rather than true response rates. Sixty percent applied a priori or post hoc adjustments to improve data quality. Concerns relate to the high number of participants excluded (30-50%) on the basis of inconsistent responses, rapid completion of questionnaires suggesting inadequate reading of items, and systematic pattern responses. Comparison with reported prevalence rates of gambling disorder ranging from 0.2-1.2% (standardized rate of 2.3%), revealed significantly higher rates of 16.7% to 23.2% for online panels. The limitations of online panels and crowdsource sampling methods will be highlighted and recommendations on procedures to be adopted for data integrity purposes will be described.

Keywords

Gambling disorder; problem gambling; crowdsource; online panel; data quality; Amazon MTurk; Qualtrics

Author Bios

Alex Blaszczynski is a professorial research fellow at the Faculty of Science, Brain and Mind Centre, School of Psychology, University of Sydney. He has conducted research into the psychology of gambling, gambling disorders and responsible gambling. He is Editor-in-Chief of International Gambling Studies.

Dylan Pickering received his BA Psychology Honours in 2013 and was awarded the Australian Psychological Society Prize for gaining first place in his cohort. Since graduating, Dylan has worked as a researcher at the Gambling Treatment and Research Clinic directed by Professor Alex Blaszczynski. He is currently undertaking a PhD at the University of Sydney investigating approaches to measuring recovery in gambling disorders.

Funding Sources

Nil funding for this study.

Competing Interests

Declarations: Alexander Blaszczynski For the period 2015 – 2018, Professor Blaszczynski has conducted research funded directly by Australian or international government, or government-related funding agencies, and industry operators. These include Gambling Research Exchange Ontario, ClubsNSW, Dooleys Club Lidcombe, Aristocrat Leisure Industries, Australian Communications Media Authority, Gaming Technologies Association, Gambling Research Australia, Responsible Wagering Australia, Commonwealth Bank, NSW Department of Trade and Investment (NSW Office of Liquor, Gaming and Racing), La Loterie Romande (Switzerland), Camelot (United Kingdom), La Française des Jeux (France), Loto-Quebec (Canada), and National Lottery (Belgium), and the National Association for Gambling Studies. He has received honorariums from Manitoba Gambling Research Program and GambleAware (formerly UK Responsible Gambling Trust) for grant reviews, and royalties from several publishers for books and book chapters. He has also received travel and accommodation expenses from Leagues Clubs, Gambling Research Exchange Ontario, USA National Council on Problem Gambling, Japan Medical Society for Behavioural Addiction, Le Comité d'organisation Congrès international sur les troubles addictifs, Victorian Responsible Gambling Foundation, North American Association of State and Provincial Lotteries, and New Horizons (British Columbia Lottery Corporation to attend conferences and meetings. All professional dealings have been conducted with the aim of enhancing responsible gambling and harm minimisation policies and practices, training counsellors in the treatment interventions, and advancing our understanding of the psychology of gambling. Declaration for Dylan Pickering:Funding: Over the last three years, DP has received salary and conference travel costs based on the deed of gift provided by ClubsNSW. DP has received funding for research projects from Gambling Research Exchange Ontario (GREO), and National Association for Gambling Studies (NAGS).

Share

COinS
 
May 28th, 3:30 PM May 28th, 4:55 PM

Reliability and accuracy of online panel derived data

Caesars Palace, Las Vegas, Nevada

There is an increasing trend for researchers to use online panels and crowdsource websites as a source of participants for survey and questionnaire based studies. The driver for this is convenience and capacity to recruit large sample sizes at low cost. There is a stated assumption that online panel respondents are representative of the general population, and that responses are more accurate given their anonymity. Data derived from these samples are used to guide policy decisions, public health initiatives, and treatments. Although sample demographics are often described as representative of the general population, the claim that the responses are more accurate and generalizable is increasingly being drawn into question. A rapid review of the literature since 2012 identified 18 online panel and 7 crowdsource studies, with most published in the previous two years. Results indicated that 60% of these studies failed to report response, or reported participation rather than true response rates. Sixty percent applied a priori or post hoc adjustments to improve data quality. Concerns relate to the high number of participants excluded (30-50%) on the basis of inconsistent responses, rapid completion of questionnaires suggesting inadequate reading of items, and systematic pattern responses. Comparison with reported prevalence rates of gambling disorder ranging from 0.2-1.2% (standardized rate of 2.3%), revealed significantly higher rates of 16.7% to 23.2% for online panels. The limitations of online panels and crowdsource sampling methods will be highlighted and recommendations on procedures to be adopted for data integrity purposes will be described.