Why incentive pay won’t fix education or health care

It turns out — surprise! — that it’s really hard to measure quality in complex social systems and that employing simplistic quantitative measures can backfire.

That’s the take-home message from a recent talk by UC Berkley economist and public policy professor Jesse Rothstein who came to SFU to present his latest research on using standardized test scores to measure teacher effectiveness in the US. 

Prof. Rothstein was involved in a 3-year pilot project in Tennessee designed as an experiment to check whether offering teachers bonus pay would improve students’ test score performance. Teachers were randomly assigned to two groups, one was offered bonus pay if their students did well on standardized tests (experimental group) and the other one wasn’t (control group).

After 3 years, there weren’t significant differences in student achievement on standardized tests between the two groups, clearly showing that offering teachers bonus pay did not improve student achievement. Yet, the Obama administration (for which Professor Rothstein worked recently) is continuing to explore incentive pay as a way to improve the education system.

Come to think of it, the idea of incentive pay has become the holy grail in governments’ quest for improving the performance of complex social systems like health care and education. On the surface, there’s a certain intuitive appeal to the idea of paying more to those doing a better job. The “economic theory” behind it is that offering to pay people more for doing a good job will lead to increased work effort as rational individuals choose to maximize their pay.

That’s what’s driving US policy makers to test schemes of offering teachers incentive pay to improve school achievement. That’s what’s driving BC Health Minister Kevin Falcon to offer hospitals funding based on the number of surgeries they do (what he calls patient-focused funding).

But when used mechanically — by tying incentives to some quantitative measure of performance, like test scores or number of surgeries done — such schemes are likely to fail.

The devil — as usual — is in the details. And the details here are that before policy-makers can give somebody a bonus for doing a good job, they need to be able to measure what a good job looks like. This is where standardized tests come in along with other quantitative measures such as the number of medical procedures performed, or the length of hospital stay.

But it turns out — surprise! — that it’s really hard to measure quality in complex social systems and employing simplistic quantitative measures can backfire. In fact, Prof Rothstein quoted an obscure scholar of methodology by the name of Donald Campbell, who coined a rather pessimistic “Campbell’s law” in the late 1970s:

“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

In other words, the perverse incentives that pay-for-performance schemes create in complex social systems may well outweigh any positive incentives for real improvement.

Campbell looked at education in particular (see his working paper here), and argued that

“achievement tests may well be valuable indicators of general school achievement under conditions of normal teaching aimed at general competence. But when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways.

What does this tell us about BC’s new incentive-based funding model in health care?

—

Kudos to the SFU Centre for Education Research and Policy for organizing the public event.

Topics: Children & youth, Education, Health care, Privatization, P3s & public services