Measuring Lineup Difficulty By Matching Distance Metrics With Subject Choices in Crowd-Sourced Data

Niladri Roy Chowdhury, Dianne Cook, Heike Hofmann, Mahbubul Majumder

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


Graphics play a crucial role in statistical analysis and data mining. Being able to quantify structure in data that is visible in plots, and how people read the structure from plots is an ongoing challenge. The lineup protocol provides a formal framework for data plots, making inference possible. The data plot is treated like a test statistic, and lineup protocol acts like a comparison with the sampling distribution of the nulls. This article describes metrics for describing structure in data plots and evaluates them in relation to the choices that human readers made during several large Amazon Turk studies using lineups. The metrics that were more specific to the plot types tended to better match subject choices, than generic metrics. The process that we followed to evaluate metrics will be useful for general development of numerically measuring structure in plots, and also in future experiments on lineups for choosing blocks of pictures. Supplementary materials for this article are available online.

Original languageEnglish (US)
Pages (from-to)132-145
Number of pages14
JournalJournal of Computational and Graphical Statistics
Issue number1
StatePublished - Jan 2 2018


  • Cognitive perception
  • Data mining
  • Data science
  • Data visualization
  • Distance metrics
  • Exploratory data analysis
  • Information visualization
  • Statistical graphics
  • Visual inference

ASJC Scopus subject areas

  • Discrete Mathematics and Combinatorics
  • Statistics and Probability
  • Statistics, Probability and Uncertainty


Dive into the research topics of 'Measuring Lineup Difficulty By Matching Distance Metrics With Subject Choices in Crowd-Sourced Data'. Together they form a unique fingerprint.

Cite this