Letter: Overemphasis leads to suspect scores

RE: “High payouts can follow test scores,” Sunday, March 17.

In the earlier days of the new educational reform movement in Louisiana, there was a section of the Louisiana Department of Education devoted to monitoring achievement test score gains.

Expected test score gains were calculated based upon the previous year’s test data, and if new scores exceeded these projections by a certain statistical amount, the scores were determined to be suspect.

In other word, statistically these gains could not have occurred without some outside interference (cheating).

When such occurrences were discovered, a complete audit of the test score documents was done along with interviews of those conducting the testing. Analyses of the test documents included such techniques as erasure analysis of the number of answers changed from incorrect to correct answers, comparing documents of children seated next to each other etc.

With the arrival of Education Superintendent Paul Pastorek, the test score monitoring function was abolished in the name of departmental reorganization.

So the problem continues with our new education reform movement that not only factors test score achievement into teachers’ tenure rights but also provides cash rewards to teachers and schools.

Just how much gain is too much gain and where is the accountability to determine such?

If our esteemed education leaders would do a little investigation of their own they would find a bundle of research showing that the more heavily test scores are factored into teacher evaluations and rewards, the greater the increase in “suspect” test scores.

Jim Anderson

former director of accountability/assessment,
New Orleans Public Schools

Ponchatoula

Comments

1) Comment by bourbon-soda - 21/03/2013

I should have looked up "erasure analysis." Thanks.

2) Comment by Noel Hammatt - 21/03/2013

Erasure analysis is fast and regularly used in standardized testing. It is automated, and normally used in contracts for scoring. I am trying to be simple here, since the points are not really about metrics but about policy. As for the scores of schools, they are subject to some shenanigans, but the point I have made over and over is that the School Performance Scores and Letter Grades bear no relationship to the quality of the school or the teaching within it. None.

3) Comment by bourbon-soda - 21/03/2013

I pointed out the misstatements in case some elementary and up kids happened to read them.

4) Comment by Jim - 21/03/2013

Bourbon-soda, didn't think I was writing an article for a statistical journal. It was only a letter to the editor which from my understanding can be read by elementary school kids on up. Thus the lack of description as put forth by Noel regarding the procedures used to analyze scores. It was more important that public realizes that there hasn't been any accountability regarding test score gains since Paul Pastorek, a lawyer by trade, served as state superintendent of education. Louisiana is recognized nationwide for its educational reforms based upon these possibly 'suspect' scores', and they continue to be reported with no data to support their legitimacy. That's the point! By the way, I can assure you I could hold my own with any discussion you might wish to have regarding the use of parametric and non-parametric methods for data analyses. Your wise crack "what is going on there?" adds little to this discussion.

5) Comment by bourbon-soda - 21/03/2013

@Noel, thank your for clarifications. I put no faith at all in grades for schools. There are too many ways to gerry-rig the system. As noted to teacherguy, below, scamming the test business went on before the "reform" movement gained steam and political power in recent years, though at higher levels in the hierarchy. "Erasure analysis" seems a cumbersome, inexact, labor intensive (therefore itself corruptible) compared to doing t-tests on bell curves.

6) Comment by bourbon-soda - 21/03/2013

@teacherguy - thanks for your response. You and coachblades in particular seem to sane people in insane places, and I appreciate your perspective. That higher stakes tempt to more cheating makes sense, but a culture of falsification has existed for a long time in the educational testing business. It seems at one time to have been at higher levels and to have percolated down to some teachers. An interesting and credible story is at http://www.tegr.org/Review/Articles/vol1/Lake_Woebegon__Twe nty_Years_Later_1_.htm which I found by searching "west virginia lake woebegone." I agree with many opinions here, that what is being done to teachers is unfair. If the schools and DOE are not investigating statistical anomalies, including jumps, it would be nice if something like the Atlanta Journal-Constitution investigation took place. One of my points may not have been clear, that if a "former director of accountability assessment" cannot write something more accurate than "statistically these gains could not have occurred without some outside interference (cheating)," what is going on there?

7) Comment by teacherguy - 21/03/2013

Most teachers I know would NOT manipulate scores, BUT with the overemphasis on these test scores...could I see some teachers doing the unthinkable? Yup!

8) Comment by teacherguy - 21/03/2013

@Bourbon-soda...a point being made here is that the emphasis of being fired and getting bonus pay has driven people that would normally have not "cheated" to begin to do extraordinary measures to ensure employment/bonuses. There was a time in my LA teaching career when my principal would say, "I don't care what the tests say, I know you are teaching your hearts out." Now my principal says, "We must get the kids to score their best or our school, and YOU, will be placed under humiliation and scrutiny. [I don't care what it takes but,] those scores must come back with lots of improvement or you could be fired, or maybe you could even get a bonus." Another point being made here is that the group that investigated statistically significant "jumps" in improvement has been abolished...so how do we know that bad things aren't happening out there with score manipulation?

9) Comment by Noel Hammatt - 20/03/2013

Actually, the standard in use was one of "erasure analysis." Tests were examined and if the ratio of "right" erasures over "wrong" erasures exceeded a certain number of standard deviations from the mean then the tests and the testing conditions at the school were examined. Other states do a much better job of examining, but we have seen major problems in charter schools and in large districts where "Deformers" were pushing for rapid changes. NYC, Atlanta, Washington D.C. et cetera. Funny that there has not been enough actual data released on the RSD for the media and others to do a study of changes in scores down there in the "Miracle RSD." Of course, the changes last year in School Performance Scores and Letter Grades were almost totally due to changes in the way scores were determined for schools, in spite of crude denials by those in the Department of Education.

10) Comment by bourbon-soda - 20/03/2013

1) No statistical procedure can determine that "these gains could not have occurred without some outside interference (cheating)." 2) For a good time, search "ajc test cheating scandal." 3) Why would a population willing to cheat on standardized tests be assumed to be entirely truthful about educational results without standardized testing?