The effects of mismatches between survey question stems and response options on data quality and responses

Jolene D. Smyth, Kristen Olson

Research output: Contribution to journalArticlepeer-review

10 Scopus citations


Several questionnaire design texts emphasize a dual role of question wording: the wording needs to express what is being measured and tell respondents how to answer. Researchers tend to focus heavily on the first of these goals, but sometimes overlook the second, resulting in question wording that does not match the response options provided (i.e., mismatches). Common examples are yes/no questions with ordinal or nominal response options, open-ended questions with closed-ended response options, and check-all-that apply questions with forced-choice response options. A slightly different type of mismatch utilizes a question stem that can be read as asking for two different types of answers with no indication of which type should be provided. In this paper, we report the results of twenty-two experimental comparisons of data quality indicators (i.e., item nonresponse and response time) and response distributions across matched and mismatched versions of questions from a postal mail survey and a telephone survey. We find that mismatched items generally have lower data quality than matched items and that substantive results differ significantly across matched and mismatched designs, especially in the telephone survey. The results suggest that researchers should be wary of mismatches and should strive for holistic design.

Original languageEnglish (US)
Pages (from-to)34-65
Number of pages32
JournalJournal of Survey Statistics and Methodology
Issue number1
StatePublished - Mar 1 2019


  • Data quality
  • Mismatches
  • Question wording
  • Questionnaire design
  • Response time

ASJC Scopus subject areas

  • Statistics and Probability
  • Social Sciences (miscellaneous)
  • Statistics, Probability and Uncertainty
  • Applied Mathematics


Dive into the research topics of 'The effects of mismatches between survey question stems and response options on data quality and responses'. Together they form a unique fingerprint.

Cite this