National Health Outreach and Engagement Conference 2024

Stewart, R, Bawden, M, Wilson, H, Speirs, K, De la Cruz, K, and Meyer, R.

Cognitive interviewing is a process that involves the administration of draft survey questions
while collecting additional verbal information about the respondents’ answers (Beatty, 2003).
This supplemental information helps researchers determine if their questions are effectively
eliciting the intended information sought by its’ author, find out where they might be going
wrong, and come up with ways to fix potential problems that can pop up when using surveys
(Willis, 2004). For instance, when people are asked questions in cognitive interviews, the things
they say can include: 1) how they came up with their answers, 2) what they think the questions
mean, 3) if they had any trouble answering, or 4) if any of the answer choices were confusing
(Beatty and Willis, 2007). Cognitive interviews stand out as an invaluable tool to develop
effective questions crucial to program development and evaluation, needs assessments, and
programmatic instruments.

The University of Arizona Cooperative Extension leveraged cognitive interviews to craft a
comprehensive statewide needs assessment survey for early care and education (ECE)
professionals. This survey aimed to discern their professional development and training needs
around health and wellness topics that could be effectively addressed or supported by
Cooperative Extension. A national workgroup of Extension faculty also utilized cognitive
interviews to pilot test a national survey of Extension professionals to explore how they provide
programming for the ECE workforce and engage with ECE state systems.

During this presentation, the cognitive interview process will be introduced, and the following
steps described and discussed based on prior experiences: survey development, formulation of
cognitive interview questions, selection of the cognitive interview sample (including how to
engage community partners in selecting a sample), determination of the cognitive interview
administration method, actual conduct of the cognitive interview in which the participant is
asked to read the survey questions out loud and verbalize their thinking about the questions and
answer choices, and the crucial step of using the results to refine the survey. This iterative cycle
may be repeated or modified as necessary to fine-tune the survey questions.
Participants will be actively engaged in hands-on cognitive interview practice within small
groups, utilizing a sample survey. Following this exercise, a debrief will be conducted to assess
whether respondents comprehended each question and whether each question effectively
gathered the intended information. Participants will also discuss how the sample survey could
be refined and implemented to collect data.

Subsequent steps will be delved into as a large group, encompassing the revision of the survey
based on the cognitive interview insights. Practical considerations will be addressed, exploring
various options to accommodate time, budget, and personnel constraints.

This presentation will be useful for Extension professionals who use surveys for needs
assessment or program evaluation. The insights shared and methodologies discussed promise to
enhance the effectiveness of surveys, ensuring that they are well-crafted instruments capable of
yielding reliable and meaningful information for informed decision-making in program
evaluation and community-based programming development.

Citations:

1. Beatty, P. (2003). Answerable Questions: Advances in the Methodology for Identifying and Resolving Questionnaire Problems in Survey Research (Doctoral dissertation, University of Michigan). Dissertation Abstracts International, 64(09), 3504A.
2. Willis, G. B. (2004). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Sage.
3. Beatty, P. C., and Willis, G. B. (2007). Research Synthesis: The Practice of Cognitive Interviewing. Public Opinion Quarterly, 71(2), 287–311. https://doi.org/10.1093/poq/nfm006.