Research Design
Designing surveys to reduce measurement error: The application of cognitive
psychology to survey methodology
I received my doctorate from the University of Chicago in 1986 in cognitive
psychology with a dissertation in language comprehension. It was during this
time that I first began to investigate cognitive aspects of survey methods by
exploring the way people estimate quantities. From 1986-1989 I was a
post-doctoral research associate at Carnegie-Mellon University working on issues
of skill acquisition and intelligent tutoring systems. From 1989-1991 I was a
principal software engineer at Digital Equipment Corporation where I worked on
the psychology of programming and maintaining expert systems. From 1991 to the
present I have been a research psychologist at the US Bureau of Labor Statistics
working on reducing survey measurement error by studying its cognitive origins.
This Spring, I taught "social and cognitive foundations of survey measurement" in
the University of Maryland's Joint Program in Survey Methodology. in the Fall I
will teach "cognition and usability: the scientific foundation of software
usability" at George Mason University.
My current research is primarily concerned with (1) the costs and benefits of
conversational and standardized survey interviewing, in particular comparing
response accuracy and interview duration for both techniques, (2) the strategies
that respondents use to answer behavioral frequency questions ("During the last
month, how many times did you ... ?"), the conditions under which respondents use
different strategies, and the direction and size of the error associated with
each strategy; (3) human-computer interaction and the usability of software
throughout the survey process; I have a particular interest in the usability
issues of collecting survey data over the web.
Tentative Curriculum:
- Day 1: overview of sampling techniques and theories of measurement
- simple random sampling, systematic sampling, stratified sampling, cluster
sampling; issues of non-response;
- Exercise: calculating response rates;
- limits of sampling and measurement error
- determining what to measure: concept development and developmental
interviewing
- Exercise: conduct focus group to establish topics for draft questionnaire
- Day 2: Question comprehension and questions about events and behaviors
- Comprehension: word meaning, grammatical complexity and ambiguity,
presupposition, conversational principles;
- Questions about events and behaviors: autobiographical memory, answering
frequency questions; placing events in time and "telescoping";
- techniques for reducing error in behavioral questions, e.g. "bounded
interviews;"
- Exercise: create and critique a set of behavioral questions based on concept
development exercise on Day 1
- Day 3: Questions about subjective phenomena
- Theories of attitude formation, question order effects, response order
effects
- response scales, open versus closed questions, "don't know" and "no opinion"
options, measuring intensity; problems with "agree"-"disagree"
- Exercise: create and critique a set of questions about subjective phenomena
based on concept development exercise on Day 1
- Day 4: Pretesting questionnaires
- focus groups, cognitive inteviews, behavior coding;
- theory of verbal protocols; collecting and analyzing verbal protocol data
for question pretesting
- Exercise: conduct cognitive interviews with other students using questions
developed in previous two days; code problems; develop "repairs" for
problematic questions; present to group
- Day 5: Interviewing techniques
- Theory and history of standardized interviewing; theory and history of
conversational interviewing; empirical status of both methods;
- ethnographic interviewing
- Exercise: conduct mock interviews with both standardized and conversational
techniques; discuss pros and cons of both approaches
Objectives
The point of the workshop is to (1) introduce students to some of the important
methods involved in conducting survey research, (2) make them aware of the
theoretical and practical obstacles -- particularly with respect to measurement
error, and (3) provide them with enough knowledge so that they can get started
with their own research and will know where to get more information if they need
it.
Bibliography
What I really expect them to be able to read and understand are journal
articles reporting research results since, presumably, they are training
to produce this kind of work themselves. So in addition to the book
title, I list a journal article below that is repesentative of the kind
of research report I will cover in the workshop.
Sudman, S., Bradburn, N., & Schwarz, N. (1996). Thinking About
Answers: The Application of Cognitive Processes to Survey Methodology.
San Francisco: Jossey-Bass Publishers.
Rubin, D.C. & Baddeley, A.D. (1989). Telescoping is not time
compression: A model of the dating of autobiographical events. Memory
and Cogntition, 17, 653-661.
Prerequisites
For purposes of the workshop, it would help if they know what a
distribution is, have been exposed to measures of central tendency, and
are familiar with the notion of variance. Additionaly, it might help
them understand the empirical results we cover if they know something
about hypothesis testing -- what t-tests and ANOVAs are. We will
discuss "validity" and "reliability" so some exposure to these concepts
would be appropriate.