Reading and math scores have long been the currency of American schooling, and never more so than in the past two decades since the No Child Left Behind Act. Today, advocates will describe a teacher as “effective” when what they really mean is that the teacher’s students had big increases in reading and math scores. Politicians say a school is “good” when they mean that its reading and math scores are high.
So, how much do test scores really tell us, anyway? It turns out: A lot less than we’d like.
For all the attention to testing, there’s been a remarkable lack of curiosity about how much tests tell us. Last spring, for instance, researcher Collin Hitt, of the Southern Illinois University School of Medicine, and two coauthors examined the research on school choice and found a striking disconnect between test score gains and longer-term outcomes. They reported, “Programs that produced no measurable positive impacts on achievement have frequently produced positive impacts on attainment” even as “programs that produced substantial test score gains” have shown no impact on high school graduation or college attendance. More generally, they observe:
The growing literature on early childhood education has found that short-term impacts on test scores are inconsistent predictors of later-life impacts . . . Studies of teacher impacts on student outcomes show a similar pattern of results . . . It turns out that teacher impacts on test scores are almost entirely uncorrelated with teacher impacts on student classroom behavior, attendance, truancy, and grades . . . The teachers who produce improvements in student behavior and noncognitive skills are not particularly likely to be the same teachers who improve test scores.
You would think this disconnect would prompt plenty of furrowed brows and set off lots of alarm bells. It hasn’t. And yet the phenomenon that Hitt et al. note isn’t all that surprising if we think about it. After all, test scores may go up for many reasons. Here are a few of them:
- Students may be learning more reading and math and the tests are simply picking that up. All good.
- Teachers may be shifting time and energy from untested subjects and activities (like history or Spanish) to the tested ones (like reading and math). If this is happening, scores can go up without students actually learning any more.
- Teachers may be learning what gets tested and focusing on that. In this case, they’re just teaching students more of what shows up on the test — again, this means that scores can go up without students learning any more.
- Schools may be focusing on test preparation, so that students do better on the test even as they spend less time learning content — meaning scores may go up while actual learning goes down.
- Scores may be manipulated in various ways, via techniques as problematic as cheating or as mundane as starting the school year earlier. Such strategies can yield higher test scores without telling us anything about whether students actually learned more than they used to.
It matters which of these forces are driving rising scores. To say this is not to deny the value of testing. Indeed, this observation is 100% consistent with a healthy emphasis on the “bottom line” of school improvement. After all, results are what matters.
But that presumes that the results mean what we think they do. Consider: If it turned out that an admired pediatrician was seeing more patients because she’d stopped running certain tests and was shortchanging preventive care, you might have second thoughts about her performance. That’s because it matters how she improved her stats. If it turned out that an automaker was boosting its profitability by using dirt-cheap, unsafe components, savvy investors would run for the hills — because those short-term gains will be turning into long-term headaches. In both cases, observers should note that the “improvements” were phantasms, ploys to look good without actually moving the bottom line.
That’s the point. Test scores can convey valuable information. Some tests, such as the National Assessment of Education Progress (NAEP), are more trustworthy than others. The NAEP, for instance, is less problematic because it’s administered with more safeguards and isn’t used to judge schools or teachers (which means they have less cause to try to teach to it). But the NAEP isn’t administered every year and doesn’t produce results for individual schools. Meanwhile, the annual state tests that we rely on when it comes to judging schools are susceptible to all the problems flagged above.
This makes the question of why reading and math scores change one that deserves careful, critical scrutiny. Absent that kind of audit, parents and communities can’t really know whether higher test scores mean that schools are getting better — or whether they’re just pretending to do so.