ASU professor earns grant to help identify effective STEM teachers

You can measure height with a ruler and weight with a scale. But what if you’re trying to measure something like how effective a teacher is?

Statistician Jennifer Broatch, a faculty member in Arizona State University’s New College of Interdisciplinary Arts and Sciences, has earned a National Science Foundation grant for a project addressing that question, specifically as it relates to teachers of STEM (science, technology, engineering and mathematics) subjects.

“There is no such thing as a direct measurement of the effectiveness of a teacher or a school district’s STEM program, like there is for a person’s height,” said Broatch, an assistant professor in New College, the core college on ASU’s West campus. “So to estimate effectiveness, we typically look at the performance of students, usually how they perform on standardized tests.”

But students’ standardized test scores don’t necessarily paint an accurate picture of how well a teacher or program is doing. Real-world outcomes, like whether students graduate or go on to major in STEM fields in college, may be at least as relevant.

Broatch’s NSF-funded project, which she is conducting with colleagues Jennifer Green and Walter Stroup from the University of Nebraska-Lincoln, aims to incorporate those real-world outcomes into a teacher effectiveness measurement, and to do so in a way that will enable users across the country to utilize the measurement program at no cost.

Broatch and her colleagues will expand on the concept of value added models (VAM) that attempt to measure a teacher’s impact on student achievement beyond what is expected of a student, controlling for the student’s and his or her peers’ prior performance and demographic information. In their project they will develop a multidimensional VAM to show the relationships among contributions of a teacher or program toward different student outcomes, and allow consideration of real-world outcomes, such as graduating or having a career in a STEM field.

“It’s possible that some teachers and programs could have different effects on different student outcomes,” Broatch said. “One teacher’s students might do better on a standardized test, while another’s might be more likely to go on to college and choose a STEM major. Our project will enable evaluators to assess the effectiveness of teachers and programs on a variety of different outcomes simultaneously.”

“The evaluation of teachers’ performance is critical to the improvement of the K-12 education system, and the use of sound statistical methods, like Dr. Broatch is developing, is essential,” said Roger Berger, director of New College’s School of Mathematical and Natural Sciences. “Her research supports ASU’s aspiration to be socially embedded and to respond to our community’s needs.”

There’s no question that STEM education is viewed nationally as a high priority. Programs like Race to the Top, STEM Education Coalition and Educate to Innovate are investing significant resources in the effort to boost STEM education and the country’s ability to maintain excellence in STEM-related research and industries.

“If we are committing significant resources to STEM education, it’s important to be able to measure whether our efforts are effective,” Broatch said. “While no model is perfect, we believe the addition of real-world outcomes will improve the existing teacher accountability measures by providing a more comprehensive metric of a teacher’s influence on student achievement.”

Broatch and her colleagues will apply the program they develop through their project to a data set from a large school district, thereby advancing measurement techniques. Currently it’s not possible to analyze a large data set of real-world outcomes in multiple categories.

The fact that the project will result in an “open source” data analysis tool that may be used at no cost is also crucial to its potential impact.

Broatch and her colleagues have dubbed their project RealVAMS, which refers to their goal of getting real-world value from VAMS.

“Ultimately, this project should help school systems identify the teachers and programs whose students are highly achieving in STEM subjects, not just through their standardized test scores, but through real-world outcomes,” Broatch said.

“Dr. Broatch’s new methods have the potential to inform decision-making and have positive societal impact,” Berger said. “These are the goals of ASU’s commitment to support use-inspired research.”

Broatch earned her doctorate in statistics from ASU in 2009. Prior to arriving at the West campus, she was a visiting assistant professor in the School of Mathematical and Statistical Sciences at the Tempe campus. Broatch has published articles in academic journals about the application of statistical methods to educational data, especially VAMS. In 2012 she was invited to present her contributions to VAM research at the Joint Statistical Meetings, an annual national gathering of the American Statistical Association. She teaches New College classes, including Probability Statistics Research and Introductory Applied Statistics.

New College’s BS degree in statistics is Arizona’s only bachelor’s degree program focused exclusively on statistics.