Analysis of test results shows mind-boggling irregularities

Nearly 1 in 10 quadrillion - that's the odds of the level of erasures happening naturally on Roosevelt Middle School's 2009 eighth-grade state reading exams. Overwhelmingly, wrong answers were changed to right, and test scores soared.

The erasure level on Wagner Middle School's 2009 eighth-grade reading and math tests had the same astounding odds.

And Strawberry Mansion High's 11th graders had an even more alarming level of erasures on the 2009 math test. The probability of those erasures occurring by chance is 1 in 10 duodecillion, a rarefied mathematical term. A duodecillion is a 1 followed by 39 zeros.

Those were the odds calculated by Data Recognition Corp. (DRC), a Minnesota firm hired by the Pennsylvania Department of Education to perform a statewide forensic analysis of 2009 achievement tests.

The analysis assigned flags to test results that showed abnormal outcomes and devised what it described as a "threat score" - a simplified 0-to-100 scale to make complex numbers like duodecillion more digestible. The higher the number, the less likely the result occurred naturally. Mansion's threat score was 99.9.

In Philadelphia, the School District has used the state data to single out 13 schools it believes bear further investigation based on the 2009 state forensic analysis of standardized-test scores. The report was first obtained by the Philadelphia Public School Notebook.

The district hasn't named the schools, but an Inquirer analysis has identified the 13. At the top of the list are Roosevelt, Wagner, and Strawberry Mansion, whose principals were all recognized as star performers and promoted by former Superintendent Arlene C. Ackerman.

Stefanie Ressler, Roosevelt's principal, has been moved to a larger and better-performing school. Wagner's principal at the time, Penny Nixon, is now associate superintendent. Lois Powell-Mondesire, the longtime principal of Strawberry Mansion, was promoted to be a "turnaround specialist" principal who advises low-performing schools on how to improve. She recently received a $10,000 bonus for helping to build the district's Promise Academy program.

Besides Roosevelt, Wagner, and Strawberry Mansion, the Inquirer analysis identified Catharine, Cayuga, F.S. Edmonds, Lamberton, John Marshall, Thurgood Marshall, McClure, Munoz-Marin, Olney, and Ziegler Elementaries as schools where the district could not rule out cheating.

These schools each had unexpected erasures and dramatic test gains.

Thurgood Marshall, for example, had suspicious erasures in four of the six grades tested. The school had a 42 percentage-point improvement in sixth-grade math and a 41 percentage-point improvement in fourth-grade reading, according to DRC. In all, Marshall received 17 flags and had 33 students with abnormal tests.

At Ziegler, fourth-grade math scores shot up an unexpected 49 percentage points. School-level erasures were also flagged for both math and reading in fourth grade. One student's math test showed 22 erasures, and 18 were wrong-to-right answers, nearly all in a cluster of sequential questions.

And those are just the schools the district has investigated at the state's direction. The Inquirer analysis uncovered a double-digit list of schools flagged for erasures or academic gains that exceeded levels at some of the schools investigated by the district. But because they didn't meet the state cutoff of three flags in one grade, they didn't get deeper scrutiny.

That list includes:

Tilden Middle School, which had 99.9 threat scores for erasures in two grades. The Inquirer found that 36 percent of eighth-grade students had suspicious erasures. Altogether, 103 students at the school had erasure flags.

The now-closed Barratt Middle School, where eighth-grade math erasures had a threat score of 99.9. The school's sixth-grade math and reading erasures were also flagged.

Logan Elementary, where more than a third of the students who took the fourth-grade reading test had suspicious erasures. The investigators found grade-level erasure problems in three grades. The Inquirer found that the number of fourth-grade students who passed reading grew from 36 percent in 2005-06 to 88 percent in 2008-09.

Forrest Elementary, which was flagged eight times and had suspicious erasures for both subjects in each grade. In all, 12 percent of the students had abnormal tests. Among 19 students flagged in third grade, The Inquirer found 299 questions were erased and 287 were wrong-to-right fixes. One busy sixth grader erased 43 math and reading questions.

Anna B. Day Elementary, which had erasure issues in three grades. In all, 28 students had suspicious erasures. A dozen were fourth graders flagged in reading. Most of their erasures were corrections. No flagged student made more than one right-to-wrong erasure.

Samuel B. Huey Elementary, which had 10 flags for grade-level erasures. Huey's fifth-grade reading scores rose an acceptable 25.4 percentage points, according to DRC. But grade-level erasures on fifth-grade reading tests were approaching 1 in 1 trillion odds of happening naturally. In all, 27 percent of fifth-grade students had suspicious math or reading tests.

John B. Kelly Elementary, which had grade-level erasure concerns in two grades, plus several other abnormal test outcomes. In third grade, 17 students were flagged for erasures, most in reading. One third grader made 18 wrong-to-right erasures on the math test, while another student had a combined 19 erasures in math and reading - each one wrong to right.

Emlen Elementary, which was given erasure flags for both subjects in each grade. Of 28 students at the school with suspicious erasures, half were in sixth grade, and most erasures involved the reading test. DRC considered the 20.3 percent improvement on the sixth-grade reading test to be acceptable, but flagged the erasures as having a more than 1 in 100,000 chance of occurring naturally.

Communications Technology High, which was flagged for erasures in both math and reading. Of the students tested, 41 percent were marked for excessive erasures. Nearly all the students were flagged in both subjects.