A relatively obscure but fascinating study titled, “Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors” appeared in the Journal of Political Economy, 2010, vol 118 no 3, also available online HERE.

From the paper’s abstract:

In primary and secondary education, measures of teacher quality are often based on contemporaneous student performance on standardized achievement tests. In the postsecondary environment, scores on student evaluations of professors are typically used to measure teaching quality. We possess unique data that allow us to measure relative student performance in mandatory follow-on classes. We compare metrics that capture these three different notions of instructional quality and present evidence that professors who excel at promoting contemporaneous student achievement teach in ways that improve their student evaluations but harm the follow-on achievement of their students in more advanced classes. (Bolding, ours.)

In the report,

In contrast, the students of more experienced and more highly qualified introductory professors perform significantly better in the follow-on courses.

The authors go even further with respect to the approach of professors who are seen to improve the follow-on achievement of their students:

One potential explanation for our results is that the less experienced professors may adhere more strictly to the regimented curriculum being tested, whereas the more experienced professors broaden the curriculum and produce students with a deeper understanding of the material. This deeper understanding results in better achievement in the follow-on courses.

Unfortunately,

Student evaluations are positively correlated with contemporaneous professor value-added and negatively correlated with follow-on student achievement. That is, students appear to reward higher grades in the introductory course but punish professors who increase deep learning.

What are we to make of this? Many things, probably. One obvious conclusion is that positive student evaluations do not correlate to long-term student success. As Stanly Fish writes,

Students tend to like everything neatly laid out; they want to know exactly where they are; they don’t welcome the introduction of multiple perspectives, especially when no master perspective reconciles them; they want the answers. But sometimes (although not always) effective teaching involves the deliberate inducing of confusion, the withholding of clarity, the refusal to provide answers; sometimes a class or an entire semester is spent being taken down various garden paths leading to dead ends that require inquiry to begin all over again, with the same discombobulating result; sometimes your expectations have been systematically disappointed. … (from Deep in the Heart of Texas, The New York Times, June 21, 2010 )

Seen from the other direction, then, negative student evaluations do not necessarily tell us that we’re not contributing significantly to long-term student success. Setting aside concern about how student evaluations are used by institutions, how do WE use them? What do we glean from them? If there is negative correlation between evaluations and long-term student success, can we rely on that relationship? Do we ignore them as a necessary nuisance? Or do we take from them what we can, while striving to not stray into the ‘positive evaluation = short-term success’ category?

What IS a Process Educator to make of student evaluations in light of this study?