Hinck, M., King, G., Nakamura, K., & Stewart, B. M. (2026). Where’s the Evidence that Respondents Understand Your Survey Questions?.
Summary
This paper mounts a methodological critique of standard survey research practice, arguing that researchers routinely fail to demonstrate that respondents interpret questions in the way the researchers intended. The authors contend that this gap undermines the validity of survey-based inference: when there is no evidence of shared meaning between question designer and respondent, any conclusion drawn from the responses rests on an unverified assumption. Rather than reporting a single empirical study, the paper is a prescriptive argument that calls for explicit tests of question comprehension to become part of standard survey methodology, and sketches approaches for eliciting and evaluating such evidence.
Key Contributions
- Foregrounds respondent comprehension as an underexamined threat to measurement validity in survey research.
- Advocates a shift in disciplinary norms toward routinely providing explicit evidence that questions are understood as intended.
- Sets a research agenda for developing and adopting tools to test question interpretation.
- Reframes long-standing concerns about question wording and cognitive interviewing as a broader claim about evidentiary standards.
Methods
The paper is primarily a methodological critique paired with a prescriptive argument about survey practice. Drawing on the abstract, it discusses approaches for eliciting and evaluating evidence of respondent comprehension rather than testing a specific hypothesis. It does not present a self-contained empirical study; the central observation is the general absence of comprehension evidence in published survey work.
Findings
- No specific empirical results are reported in the abstract.
- The core observation is that evidence of respondent comprehension is largely missing from published survey research.
- Inferences from survey responses therefore frequently depend on untested assumptions about shared meaning between researcher and respondent.
Connections
This paper speaks to broader concerns about the validity and reliability of survey-based measurement in political and social science, a theme also present in work on survey experiment robustness and measurement quality such as DiGiuseppe2025-es and Luhring2025-od. Its emphasis on evidentiary standards for interpreting responses complements methodological scrutiny of respondent behavior and data quality found in Stagnaro2025-pz and Ulloa2024-jm; the remaining papers under this topic appear less directly related to the comprehension-validity argument advanced here.
Podcast
A research-radio episode discusses this paper: 🎧 MP3 · Spotify