Content Is King: Impact of Task Design for Eliciting Participant Agreement in Crowdsourcing for HRI

Alisha Bevins, Nina McPhaul, Brittany A. Duncan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations


This work investigates how the design of crowdsourced tasks can influence responses. As a formative line of inquiry, this study sought to understand how users would respond either through movement, response, or shift of focus to varying flight paths from a drone. When designing an experiment, running several proto-studies can help with generating a dataset that is actionable, but it has been unclear how differences in things such as phrasing or pre- and post-surveys can impact the results. Leveraging methods from psychology, computer-supported cooperative work, and the human-robot interaction communities this work explored the best practices and lessons learned for crowdsourcing to reduce time to actionable data for defining new communication paradigms. The lessons learned in this work will be applicable broadly within the human-robot interaction community, even outside those who are interested in defining flight paths, because they provide a scaffold on which to build future experiments seeking to communicate using non-anthropomorphic robots. Important results and recommendations include: increased negative affect with increased question quantity, completion time being relatively consistent based on total number of responses rather than number of videos, responses being more related to the video than the question, and necessity of varying question lengths to maintain engagement.

Original languageEnglish (US)
Title of host publicationSocial Robotics - 12th International Conference, ICSR 2020, Proceedings
EditorsAlan R. Wagner, David Feil-Seifer, Kerstin S. Haring, Silvia Rossi, Thomas Williams, Hongsheng He, Shuzhi Sam Ge
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages12
ISBN (Print)9783030620554
StatePublished - 2020
Event12th International Conference on Social Robotics, ICSR 2020 - Golden, United States
Duration: Nov 14 2020Nov 18 2020

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12483 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference12th International Conference on Social Robotics, ICSR 2020
Country/TerritoryUnited States


  • Aerial vehicle
  • Crowdsourced
  • Gesture

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science


Dive into the research topics of 'Content Is King: Impact of Task Design for Eliciting Participant Agreement in Crowdsourcing for HRI'. Together they form a unique fingerprint.

Cite this