‘Lite’ investigation of Atlanta test cheating
The Atlanta Journal-Constitution
Something wasn’t right at Towns Elementary.
On standardized tests in 2009, almost two-thirds of the school’s classrooms produced suspiciously improved scores. Then, on the tightly controlled 2010 test, Towns posted a precipitous decline. Of 21 test segments, Towns’ scores fell in 18.
Yet, investigators looking into allegations of widespread cheating on the Criterion-Referenced Competency Test questioned just four employees at Towns. One was a teacher who said she had told two administrators that someone had tampered with a package of CRCT papers. But neither administrator, investigators later wrote, “provided any information regarding this matter.”
Case closed.
Harper Elementary took a different approach. After dubious changes appeared on CRCT answer sheets, school officials questioned the principal, a testing coordinator and two of four teachers involved; the other two had retired. They also pulled student records to analyze the validity of test scores in the suspected classrooms. Only then was Harper cleared of suspicion.
Harper is in Thomasville, part of a 2,900-student, six-school district in South Georgia. Towns is in Atlanta, which has more than 100 schools and 48,000 students. The dissimilar approaches to investigating CRCT scores reflect more than mere geography and size, however. They also highlight what state officials cite as flaws in Atlanta’s examination of cheating allegations at more than two-thirds of its elementary and middle schools, a deconstruction of the investigation shows.
Atlanta questioned just 230 of 3,100 employees assigned to the 58 schools suspected in the cheating scandal; investigators spoke to three or fewer people at 34 of the schools. High-ranking school district officials — among them, the top aide to Superintendent Beverly Hall — conducted or observed 140 of the interviews. And a data analysis commissioned for Atlanta’s investigation appears to have limited the inquiry’s scope.
“I wonder if they were given instructions and directions of how to constrain this investigation rather than to explore it,” Gov. Sonny Perdue said in an interview.
Even if, as Atlanta school officials have said, many district employees refused to speak with district investigators, that does not excuse gaps in the inquiry, Perdue said.
“When people said, ‘We don’t have to talk to you,’ the investigator says, ‘OK, we’ll just go on somewhere else,’” Perdue said.
“If I’m in charge, then I say, ‘Look, you have a choice. We’re going to get to the bottom of this one way or another. You better be telling your side of the story. This is not yet a criminal investigation. There are no Fifth Amendment rights at this point.’ But the actual facilitation of lawyering-up and stonewalling does not seem to have been, to me, discouraged at all.”
Perdue recently appointed Mike Bowers, a former Georgia attorney general, and Bob Wilson, a former DeKalb County district attorney, to investigate irregularities in Atlanta and in Dougherty County, the only other district that conducted a CRCT inquiry deemed unsatisfactory by the state.
Bowers and Wilson have two powerful tools to compel cooperation from teachers and other school employees: Tampering with public documents is a felony, and so is lying to a state investigator. The prospect of criminal prosecution places the scandal in a different realm.
Atlanta school officials and the business leaders who paid for the district’s investigation have defended their inquiry as thorough and fair. They had to work quickly, but still missed a state-imposed deadline. They also had to examine by far more schools than any other district.
Hall recently issued a memorandum to district employees instructing them to cooperate with any investigation. The memo came after the district’s inquiry concluded, but before Perdue announced he would appoint state investigators.
‘Good won’t do’
Towns Elementary fits the profile of the Atlanta schools facing questions over the CRCT. A large majority of its students, about 83 percent, are black. Virtually all, 96 percent, are poor enough to qualify for a free or reduced-price lunch. And like others under scrutiny, Towns has shown great pride in its apparent achievements.
“Good won’t do,” the school’s website says, “when better is possible.”
When the state analyzed erasures that turned answers on the CRCT from wrong to right, it found an unusually high number of such changes at Towns.
In fifth-grade math, for instance, the 15 students in one class made 184 wrong-to-right changes, or more than 12 each on the 70-question test. On average, Georgia fifth-graders made fewer than two such corrections each.
The likelihood of such an extreme deviation from the norm is all but nil. Nevertheless, the state found similar statistical improbabilities in two-thirds of Towns’ classrooms, including the entire fifth grade.
When they ordered Atlanta to investigate the 58 schools, state officials gave directions as clearly as they could, said Kathleen Mathers, executive director of the Governor’s Office of Student Achievement.
Officials wanted the district to establish a chain of custody, Mathers said, by interviewing anyone who could have touched test papers at each of the 58 schools. Principals, assistant principals, testing coordinators, teachers, test proctors: All, she said, should have been questioned.
“That expectation was made clear” in memos and in discussions with school officials, Mathers said. The erasure analysis suggested where to look, “but human work had to take place from there to find out what did and did not happen.”
DeKalb County, for example, which was ordered to investigate 28 schools, listed the names and job titles of each person interviewed, along with his or her role in testing. DeKalb also compiled what Mathers called “contextual information,” such as how well students performed before and since the 2009 CRCT and whether their scores that year challenged credulity.
Atlanta provided nothing near that level of detail, Mathers said, even though 43 of its 58 schools under suspicion were on the state’s list of those causing “severe” concern — meaning that unusual numbers of erasures had boosted scores in at least one-fourth of their classrooms. The rest, with more than 10 percent of classrooms flagged, were of “moderate” concern, officials said.
“Then, when you look at their drops in scores in 2010,” Mathers said, “your concerns are even greater.”
No explanation
No other district brought in outside investigators to examine the CRCT gains. A “blue-ribbon” commission chosen by the Atlanta Board of Education supervised the investigation; a nonprofit group linked to the superintendent paid for it; it was performed by the forensic arm of the accounting firm KPMG and by Caveon Test Security, a firm that specializes in forensic data analysis.
Both firms limited their inquiries, records suggest.
KPMG handled many of the most sensitive interviews — of Hall and others in the district’s central office, with the school board, and with parents and students.
But KPMG questioned just 15 employees in the central office, plus three school board members and 12 parents and students.
In a recent interview, Hall said the firm set its own course.
“They were the experts,” she said. “Far be it for me to tell them how to do their work.”
A KPMG executive who headed the Atlanta investigation did not respond to a request for comment.
Hall said district employees participated in interviews only at the 33 schools that Caveon identified as showing the least suggestion of cheating.
“On the others,” she said, “we were not there.”
The investigation’s report, however, says that district officials took part in a majority of “school-level” interviews with employees, parents and students — 140 of 242 — at as many as 46 schools.
Working from a script developed by KPMG, the officials questioned 65 employees alone. But they also sat with KPMG investigators for 75 additional interviews — a potentially intimidating presence in a district where employees have alleged they faced retribution for speaking out.
Late last week, the district released a list of the 20 officials who helped conduct the interviews. Most were senior-level administrators from the central office: the district’s internal auditor and its chief strategy officer, for instance, along with the executive directors of high schools, math and science, and special programs. The list also included Hall’s special assistant and the top aide to the deputy superintendent for instruction, Kathy Augustine.
Three of the 20 report directly to Hall, according to the district’s organizational chart. At least 10 others answer directly to her chief deputies.
At 13 schools, these officials assisted KPMG by “note taking,” Hall’s chief of staff, Sharon Pitts, said by e-mail Saturday. “District employees did not ask questions,” Pitts wrote, “only took notes.”
The investigation was further restricted by Caveon’s analysis.
In a March memo to the state’s student achievement office, Caveon said it would rule out cheating in classrooms where wrong-to-right erasures had not caused score increases greater than gains throughout the school. “If we detect items where the within-school performance is inconsistent, we will have evidence of systematic tampering,” the memo said.
This approach, however, would tend to focus on teachers as individual cheaters. School-wide cheating involving erasures on papers from multiple classes, raising all the scores, could go undetected. KPMG’s report later said administrators at some schools “cleaned up” stacks of answer sheets from numerous classrooms at a time.
Caveon also tried to determine whether excessive wrong-to-right erasures in some classrooms could have been caused by a single student who corrected a disproportionate number of answers.
“If we can do this,” Caveon’s memo said, “we can clear a lot of classrooms and their associated schools.”
By June, Caveon had narrowed the investigation’s focus to 12 schools where it said cheating was most likely to have taken place.
That analysis created tension between the district and state officials, including Perdue, who assert that the point was to contain the growing scandal.
Caveon executives did not respond to requests for an interview.
The company’s president, John Fremer, recently defended Caveon’s work in an interview with WABE, the public radio station operated by the Atlanta school board.
“It was a very thorough investigation,” Fremer said. “Did we end up with people like in Perry Mason saying, ‘Oh, that’s it, I confess’? No. We didn’t end up with that. But it wasn’t for lack of trying.”
He offered no reasons, however, for the irregularities found in so many Atlanta schools — in the 12 Caveon cited, and beyond.
“It’s kind of puzzling to me why the overall level of wrong-to-right erasures is so great,” Fremer told the radio station. “I don’t have an explanation for that.”