Comparing survey ranking question formats in mail surveys

Although questions that ask respondents to rank-order a list of items can be analytically valuable, responding to ranking questions typically requires a good deal of cognitive effort. This is especially true in mail questionnaires where the advantages of electronic response formats available in web surveys are inaccessible. In this article, we examine two alternative formats for ranking questions in mail surveys. Using a nationally representative mail survey of U.S. adults, this article experimentally compares ranking formats in which respondents write numbers in boxes versus selecting items for the most and second most important issues using a grid layout. Respondents to the numbering format were more likely to provide usable data, although one-third of respondents in this format still did not follow instructions correctly. Substantive responses differed somewhat across formats. Less educated respondents had difficulty with both formats, resulting in substantively different conclusions about preferences across formats for this group. A numbering format is more effective than a most-second most grid format for collecting ranking data in mail surveys.