Economists: Return Your Salaries for Producing Flawed Studies

By Barnett Berry

7/30/12

Would it be outrageous for economists to have to pay back part of their salaries if their theories don’t improve the economy? What if education economists had to return consulting fees to the think tanks that paid them if their methods were flawed? Or if their conclusions ignored other research findings and the realities of teaching?

If you think this would be a good idea, then Harvard professor and MacArthur Genius Grant winner Roland Fryer and Freakonomics co-author and University of Chicago professor Steven Levitt (and colleagues John List and Sally Sadoff) might just have to cancel their vacations this fall. They would have to return the pay they received for their recent paper and recommendation that teachers must return salary bonuses (paid up front) when their students’ test scores do not improve sufficiently from one year to the next.

The Study: What It Said

Funded by the Broad Foundation—and drawing on a study about how Chinese factory workers are paid (see page 2)—the report concludes that teacher-incentive programs should be framed in terms of losses rather than gains.

Frustrated by study after study showing that single-minded merit-pay plans don’t improve student achievement (e.g., see the RAND report on New York City’s program, and the exhaustive longitudinal Nashville experiment), these highly touted scholars are on a mission to prove, as one journalist suggested, that “the promise of a larger paycheck will motivate (teachers) to work harder when they're up in front of the chalkboard.”

Fryer and colleagues analyzed the student test results (from the ThinkLink Assessment System) of 150 K-8 teachers from Chicago Heights, Illinois. The teachers were divvied into one control group and four treatment groups, with opportunities to be rewarded or punished for individual and team gains or losses. The “gain” treatment group worked under a traditional year-end bonus structure, similar to merit pay proposals of past and present. This group could earn up to $8,000 extra based on their students' standardized test scores.

Another treatment group—the “loss” treatment group—was given $4,000 at the beginning of the year and knew up front that if their students scored below expectations on the end-of-year test, they'd have to return part of this payment, proportionate to their students’ results.  The researchers found that the gain group showed no significant differences in student achievement, while the loss-treatment group’s individual and team results were comparable to increasing teacher quality by one standard deviation.

What It Missed

I am certain that this study was conducted by very smart people, but people who know too little about teaching and learning, schooling and community, teachers and students.

Consider these issues:

1.     The methodology is built on the assumption that money makes teachers more effective, but there is a wealth of evidence, including a recent Gates-funded study, to the contrary.

2.     The researchers used a low-stakes standardized diagnostic test that is supposed to be aligned with state standards, but there is no evidence that is the case. Did the researchers read this recent report on how off the mark many standardized tests are?

3.     While the teachers were assigned randomly to the control and treatment groups to control for working conditions, the researchers do not own up to the fact they are making a summative judgment about teacher effectiveness on the basis of a 25-50-question multiple-choice test that may not be aligned with the standards that teachers had been using all year.  

More Considerations

The researchers did show some concern for how other factors might affect the achievement of these students.  They looked at student mobility. Check. They look at cheating. Check. They look at liquidity constraints. (What?)  Fryer and company considered how teachers who received upfront payments might have competitive advantage if they “purchase certain materials that facilitate instruction.” My first reaction was to offer kudos to Professor Fryer. Not so fast.

Fryer was concerned that teachers who had additional dollars up front might go out and buy “new workbooks or dry-erase markers” that would allow them to teach more effectively than those in the control and gain groups. Really? These concerns tell us so much about how little these researchers know about teaching and learning.

Other studies have shown that teachers are more effective when they loop with their students for more than one year, when they teach the same course (in secondary school) for more than one year, when they have high-quality curriculum materials, and when they work with a team to plan curricula and evaluate student work. All these factors contribute far more to effective teaching than the loss-aversion merit pay schemes posed by this study.

More questions arise about how Fryer’s scheme would work in reality: Would districts have to hire more bureaucrats to collect the dollars awarded to teachers when test scores did not increase? How would such a pay scheme affect teacher morale? What about the impact on collaboration, given that the authors’ statistical model promotes competition among teachers?

Economists like Fryer and company—and their funders—are desperate to find some kind of empirical link between short-term test score gains and teacher pay (or firings). The researchers found a model—based on a Chinese factory where “loss” workers saw only a 1 percent increase in productivity—and decided to use it in such a complex enterprise as teaching.

If they were truly interested in improving schools, they might want to take a look at teachers’ own bold ideas about pay. If they were truly interested in improving schools, they might want to take a look at a larger body of research on teacher incentives and student achievement. Perhaps in the future, economists like Fryer and company will have to return their consulting fees if they don’t.

Your rating: None Average: 5 (2 votes)

Comments

Whiteboard markers make me a much better teacher

Submitted by @redhdteacher (not verified) on Mon, 2012-07-30 16:57

Ummmm...no! This part of the paper actually made me chuckle out loud. Actually, diet soda may have come out of my nose. It is so far off the mark to think that something as simple as workbooks, dry erase boards, or even a promise of pay (or having that pay taken away) can improve student achievement.

My heart broke a little to see that one of the authors of Freakonomics coauthored the paper. But then I got to thinking...this is a snapshot of the typical misconception many have about improving our schools (famous economists and authors included). If it were so simple as equating pay to increased achievement, don't you think one of the numerous merit pay or performance pay programs would have worked and we would have replicated it? But it hasn't. And the research shows that it doesn't...regardless of giving money or taking it away.

A couple thoughts I'm chewing on:

-What would this do to recruitment? Man-oh-man...it would not do good things.
-How about morale? Camaraderie? This would breed heinously unhealthy competition. Which would do the opposite of what it intended...trust me.
-Why are we obsessed with stick and carrot approaches? You want to know how to increase student achievement? Here are some levers to making great improvements. How about time to collaborate with colleagues (meaningfully...not just a 30 minute drive-by planning block where I'm wolfing down my yogurt for lunch, returning parent emails, and hopefully seeing a team member that might have the same planning block). Support. Trust. Great materials and great working conditions. The ability to be creative in my classroom and not follow a scripted program that "ensures" increased student gains. The belief that I am enough of a professional to make decisions regarding assessment in my classroom. Time to teach and help my students grow as learners instead of test, after practice test, after practice test. And while we are on the subject, a little support from the media and public instead of constant scrutiny and discussions about "bad teachers." Let's focus on what the good ones are doing right and learn from that. Let's support those teachers who dedicate so much time away from their children to help teach and raise all of OUR children.
-Still want to involve pay? How about pay for things that will increase student performance, such as getting our National Board Certification and meaningful professional development that will make us better teachers? If I refine my craft as a teacher, you WILL see growth with my students.

So, whiteboard markers will not make me a better teacher. Barnett was right on the money above. There are so many ways to increase student achievement, so many ideas coming from the real experts...the teachers. It's time to listen.

Megan Allen, NBCT
2010 FL Teacher of the Year
Teacherpreneur, Proud 5th Grade Teacher, and Edugeek

Openly Sick of Economists

Submitted by Bill F (not verified) on Mon, 2012-07-30 18:53

This kind of stuff only serves to discourage me even more, Double B.

I've been ranting and raving about the negative impact that standardized testing and competitive merit pay programs have on teachers for years -- and was jazzed to see Dan Pink's recent attempts to discredit the "teachers are motivated by cash" clowns because I believed the tide was shifting towards more sensible #edupolicies.

And JUST when my glass goes from being half empty to half full, another economist who hasn't set foot in a classroom churns out another Broad sponsored report that has no merit -- pardon the pun -- that right wing hacks will use as "evidence" that teachers are being lazy.

Rick DuFour has a great line in his keynotes. He says that merit pay assumes that teachers start each year shooting to be half-assed professionals....which is an improvement over last year's quarter-assed performance.

The truth is that teachers are doing the best that they can under the conditions and with the resources that they are given every single day.

#sheesh

Bill

Wow. Wow. Wow. That's all I

Submitted by Rachel Wood (not verified) on Mon, 2012-07-30 21:17

Wow. Wow. Wow. That's all I could say when I finished reading this report. Talk about motivating teachers to teach to the test!

Megan makes a great point about the number of years that these types of merit pay systems have been going on. We truly need to look at completely new ways of thinking in education. I have been reading a book called "Curriculum 21" (by Heidi Hayes Jacobs) and the author points out that our system of scheduling, blocking subject areas, and even physical space, have been the same for centuries. Students walk into a 19th century classroom at the beginning of the day, then walk back out into today's world after school. Why have we looked at everything else over and over again except the inner structure of the school? Jacobs (2010) asks some questions that I think need to be more closely examined:

-What type of both long-term and short-term schedules will best support specific learners

-What various ways of grouping our learners will assist them in their learning experiences?

-How should faculty be configured to best serve students and to assist one another?

-In what ways can both physical and virtual space be created and used to support our work?

A colleague and I agreed that most likely the whole system may need to be uprooted and restructured. We have a long road ahead to reach the deep change that is necessary in education and these "quick fixes", such as merit pay, seem to slow us down even more.

Disincentive

Submitted by Mark Sass (not verified) on Tue, 2012-07-31 10:32

One of the questions Barnett raises at the end of his post is probably the most important for me. How would this approach work to improve what research tells us is one of the most important features of a successful education system: collaboration. We know that resources are very limited and cuts to education are at an all time high. With limited resources comes competition for those resources. What would the incentive be for me, a teacher who did not lose the upfront money, to work with other teaches in their building, district, or state to collaborate to share their success?

I wonder what the results would be if the upfront money was given to a school, or district, as opposed to individual teachers. And instead of a one time "diagnostic" test we used a portfolio of assessments tools that assessed skills across the curriculum over a period of two years. This would encourage a whole school approach to students and discourage teaching to a test. I wonder what the results would be.

Standardized Test Scores

Submitted by James Boutin (not verified) on Tue, 2012-07-31 18:26

You get the sense the researchers are neither aware nor care about the pitfalls of standardized test scores. The media who praise their work (The Atlantic or Wall Street Journal) seem to be equally apathetic about that major problem.

I have a lot more to say about that at the link below:

http://www.anurbanteacherseducation.com/2012/07/loss-aversion-paying-tea...

Genius? {cough cough}

Submitted by David B. Cohen (not verified) on Wed, 2012-08-01 01:46

"Stupid" is the word that comes to mind. These studies just reflect a total misunderstanding of teachers. Bill has already referred to Dan Pink's work, which included good evidence that incentives like this help with tasks that are not cognitively demanding, and not something people are motivated to do already. But going beyond that misunderstanding, they then choose as the indicator of success the least important, least valuable, least meaningful, least sensitive measure they could choose. We've seen great evidence from huge data sets that such ratings are highly variable and the vacillations in year-to-year results are often beyond the teacher's control. Meanwhile, I'm reminded of my friend and colleague Laura Bradley, whose 8th graders all wrote novels in their English class. They finished novels. Some kids sold their work online. Some have done readings. One girl wrote an opinion piece for The New York Times recently. These students learned an invaluable lesson, and were begging for more time and opportunity to work on their writing skills. Why would anyone want to mess with that by putting test scores first and foremost in Laura's mind? These students' test scores may go up or down this year, regardless of the novel writing unit, but I'll bet these economists could come up with a formula that would project the scores they should have had and then leap to conclusions about any variance either way. Oh, for a little more accountability outside of school!

Levitt and Freakonomics

Submitted by David B. Cohen (not verified) on Wed, 2012-08-01 01:49

One more thing: I'm a "Freakonomics" fan, but here's one other time where I think Levitt misunderstood teaching, teachers, and testing:
http://accomplishedcaliforniateachers.wordpress.com/2011/10/18/freakonom...

Economists need to read this

Submitted by BarnettBerry (not verified) on Wed, 2012-08-01 13:37

Thanks Rachel - we know a whole lot about the kinds of conditions taht allow teachers to teach effectively. check out our 2010 TeacherSolutions report:

http://www.teachingquality.org/sites/default/files/TS_TWC_Report.pdf

Teachers need to be paid differently....so look at what TLNers said in 2007:

http://www.teachingquality.org/legacy/TSP4P2008.pdf

what if Fryer and colleagues read these words of wisdom - and even tried to explain their findings in light of the realities of teachung and what effective teachers have to say about performance pay and the conditions that allow them to teach effectively

CONCLUSION: "Pursuit of Profits" Is the Focus

Submitted by Susan Graham (not verified) on Wed, 2012-08-01 13:44

CONCLUSION:
"There may be significant potential for exploiting loss aversion in the pursuit of both optimal public policy and the pursuit of profits."

The research review on which this study was built would imply that the United States should look not to Finland or Singapore for "optimal public policy" models, but rather to the educational practices of rural India and Kenya. It is unlikely that the United States would consider using the practices of Kenya and rural India to inform the improvement of our transportation or defense programs. Perhaps a more relevant study might draw on research that explores the effectiveness of loss aversion in the form of "claw backs" in the financial sector of our own economy. Has it resulted in a higher level of responsible decision making, ethical behaviors, or altruism in the banking and investment world?

Since the research team posits that personal financial reward will motivate teachers, one must wonder why they addressed the possibility that the pre-paid bonus might have a "liquidity" impact, allowing teachers to purchase supplies such as "white board markers and workbooks." They do not reference research that supports access to "white board markers and workbooks" as having a positive impact on student learning; but if it exists, why wouldn't providing every teacher these consumable resources be a controlled variable rather than an experimental variable. Does measuring how much of their own money to equip their classroom address Optimal Public Policy or Pursuit of Profits? But to be fair, the presumption that teachers take responsibility for providing consumables for their classrooms has been codified by granting teachers a $500 tax deduction acknowledging either the commitment of teachers or the underfunding of education. Perhaps the idea is scalable--Would highly effective nurses use a pre-paid bonus to run by Walmart to pick up bandages and hypodermic needles? Might firemen use a pre-paid bonus to fill up the tank of the hook and ladder truck?

"Pursuit of profits," rather than "Optimal Public Policy" appears to be the focus of this study. Loss aversion is based on fear and while this research may prove that fear can be a better motivator than reward, it doesn't make it a solution. In this experiment model subjects accepted relatively high risk with very limited self determination in exchange for relatively low extrensic reward. If offered the same loss aversion option again, the most likely response will be risk aversion--why bother? And then what? Loss aversion assumes lack of motivation is the only issue. It offers no processes to improve teacher quality and it is likely to prove a detriment to teacher retention. The profits in test score performance achieved by loss aversion are most certainly short term gains--a mindset that has resulted in manipulation, corruption, and distortion of the financial industry of our nation. Perhaps it would be more appropriate for economists to refine the implications of loss aversion in the field of economics before applying them to education.

Post new comment

The content of this field is kept private and will not be shown publicly.