Where's the Evidence that Respondents Understand Your Survey Questions?

Musashi, H., Gary, K., Kentaro, N., & M, S. B. (2026). Where’s the Evidence that Respondents Understand Your Survey Questions?.

View paper

Summary

This paper mounts a methodological critique of standard survey research practice, arguing that researchers routinely fail to demonstrate that respondents actually interpret questions in the way researchers intend. Because survey-based inference depends on shared meaning between researcher and respondent, the absence of such evidence leaves a foundational validity gap. The authors call for a shift in disciplinary norms: comprehension should be tested and documented explicitly, not assumed, and they sketch approaches for generating this kind of evidence.

Key Contributions

• Identifies respondent comprehension as an underexamined but foundational validity threat in survey research.
• Argues that unverified assumptions about shared meaning undermine both descriptive and causal inferences drawn from surveys.
• Advocates a normative shift toward requiring explicit evidence of question understanding as part of standard practice.
• Sets an agenda for developing, adopting, and reporting tools that probe question interpretation.

Methods

The paper is a methodological critique and prescriptive argument rather than an empirical study. It discusses approaches for eliciting and evaluating evidence that respondents understand survey questions as intended, extending traditions of cognitive interviewing and question-wording analysis into a broader evidentiary standard.

Findings

• The core observation is diagnostic: published survey research generally does not supply evidence that respondents understood the questions as intended.
• Without such evidence, standard inferences from survey data rest on unverified interpretive assumptions.
• Comprehension testing is feasible and should become part of routine survey reporting.

Connections

No other papers were provided under shared topics, so there are no genuine intellectual links to draw here. The argument connects most naturally to the broader literatures on measurement validity, cognitive interviewing, and question-wording effects in survey methodology.