Study finds flaw in surveying public opinion about science

A new study by researchers from Arizona State University, North Carolina State University and University of Wisconsin highlights a major flaw in attempting to use a single survey question to assess public opinion on science issues. The study, published recently by Public Understanding of Science, found that people who say the risks posed by new science fields outweigh the benefits often actually perceive more benefits than risks when asked more detailed questions.

“The goal of the study was to explore whether we could use one survey question to accurately measure public opinion on science and technology issues,” said Elizabeth Corley, co-principal investigator with the Center for Nanotechnology in Society at Arizona State University (CNS-ASU) and co-author of the study.  “But we found that these complex science issues do require multiple survey questions about risks and benefits to accurately measure public opinion about them.”

The study, “Measuring risk/benefit perceptions of emerging technologies and their potential impact on communication of public opinion toward science,” was led by Andrew Binder, an assistant professor of communication at North Carolina State, and co-authored by Corley, Lincoln Professor of Public Policy, Ethics and Emerging Technologies at ASU; Dietram Scheufele, co-principal investigator with Corley at CNS-ASU and John E. Ross Professor of Life Sciences Communication at the University of Wisconsin; Michael Cacciatore, a University of Wisconsin doctoral student; and Bret Shaw, also a professor at the University of Wisconsin.

To assess the problematic nature of single-question surveys, the researchers developed two surveys; one focused on nanotechnology and the other on biofuels. In each survey, respondents were asked an overarching question: Do the risks associated with nanotechnology/biofuels outweigh the benefits; do the benefits outweigh the risks; or are the risks and benefits approximately the same? The researchers then asked survey participants a series of questions about specific risks and benefits associated with nanotechnology or biofuels.

The researchers then compared a participant’s response to the overarching question with his or her responses to the specific questions in order to see whether the overarching question accurately captured the opinion of the individual respondent.

They found a problem.

“There was a significant discrepancy among people who responded to the overarching question that the risks of emerging science outweighed the benefits when compared to their responses to the questions about the specific risks and benefits,” says Binder. “Namely, those same people really perceived more benefits than risks when given the opportunity to evaluate these attributes separately.

For example, in the nanotechnology survey, 50 percent of respondents who said risks outweighed benefits actually evaluated nanotechnology positively in the other portion of the survey.  In fact, only 35.4 percent of respondents who thought risks outweighed benefits actually calculated more risks than benefits in the specific section of the survey. The researchers found similar, though less pronounced, results in the biofuels survey.

The study also showed that people who said that benefits outweighed risks in response to the overarching question consistently perceived more benefits than risks in the specific question section of the surveys.

Beyond the methodological implications, this analysis suggests several reasons why researchers in the area of public attitudes toward science must revisit notions of measurement in order to accurately inform the general public, policymakers, scientists and journalists about trends in public opinion toward emerging technologies.

“The bottom line is that social scientists and journalists need to be very careful when relying on data from a single, overarching survey question,” Binder says. “These oversimplified questions can result in misleading poll data and create problems for policymakers who base their decisions on those findings. They can also be problematic because they may contribute to different polls showing widely different results, which weakens the public’s faith in surveys generally.”

Scheufele’s and Corley’s work on the study is part of their focus at CNS-ASU to monitor, among both the public and scientists, the understanding of and values relating to nanoscale science and engineering and their potential societal outcomes, and to track these variables over time.  CNS-ASU is funded by the National Science Foundation. The study also was funded by the U.S. Department of Agriculture.

Matt Shipman, matt_shipman@ncsu.edu
North Carolina State University