Studies: Pandemic Aid Lifted Scores, But Not Enough To Make Up for Lost Learning

Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter

Nearly $200 billion in emergency school funding spent during and after the pandemic succeeded in lifting students’ achievement in math and reading, according to two papers released Wednesday. Test score increases in both studies, which were conducted independently of one another, indicate that states and school districts used the money to effectively support children, even as learning in some areas improved faster than in others.

But the social scientists who authored the research argue that federal dollars could have been spent in ways that would have helped scores bounce back faster. The per-dollar returns of ESSER, the Elementary and Secondary School Emergency Relief Fund, measure up poorly in comparison with those of previously studied efforts to boost achievement, from reducing class sizes to implementing more rigorous curricula.

Dan Goldhaber, the lead author of one of the studies and the director of the Center for Analysis of Longitudinal Data in Education Research, said he believed the crisis conditions of the pandemic made it “hard to spend the ESSER funding in thoughtful, effective ways.”

By his own estimate, 35% of the math recovery achieved during the 2022–23 school year was directly attributable to ESSER funding. Fully 87% of English recovery was credited to ESSER, though he found that gains in that subject were statistically insignificant. Still, he said, that upward movement was limited. 

“Candidly, I think the impact was small, and there are some reasons why it wasn’t larger,” Goldhaber said. “Only 20% of ESSER money was even earmarked for learning loss, and I don’t think there was a lot of oversight of whether that 20% was well spent.”

Only 20% of ESSER money was even earmarked for learning loss, and I don't think there was a lot of oversight of whether that 20% was well spent.

Dan Goldhaber, CALDER

The findings offer a split verdict on the post-COVID academic recovery, while somewhat strengthening the case that putting more resources into schools can elevate their results. The advances measured in both studies are virtually identical not only to one another, but also to earlier, wide-ranging estimates of the impact of additional money on schools.

ESSER was one of the best-known and longest-lasting pillars of Washington’s pandemic response. Years after stimulus checks and free nasal swabs stopped arriving in the mail, many districts are still spending down the aid they received through the program. The last of the supplemental aid will not expire until this September, four years after schools first began to reopen for in-person instruction.

Notably, however, both papers project that American students will not have returned to their pre-COVID learning trajectories by then, and that the cost of a full restoration could amount to hundreds of billions more. With no sign of any further assistance coming from Congress, that bill will need to be picked up by states — if it is paid at all. 

In the meantime, ESSER’s backers can point to real, if incomplete, progress.

Stanford sociologist Sean Reardon helps lead the Education Recovery Scorecard, which released a second study on Wednesday. In an interview, he noted that the federal cash injection was the equivalent of only about one-quarter of the country’s annual K–12 spending, spread over multiple years. While it might have been used more efficiently to stem further learning loss, he added, both national and state leaders were simultaneously focused on goals like reopening schools and alleviating the severe emotional distress that many children are still facing.

One can certainly imagine ways to spend the money that would lead to even more learning gains. But that wasn't entirely what was on policymakers' minds.

Sean Reardon, Stanford University

“One can certainly imagine ways to spend the money that would lead to even more learning gains,” Reardon said. “But that wasn’t entirely what was on policymakers’ minds when they sent out the money.”

‘A huge missed opportunity’

To pinpoint the impact of additional money on COVID-era learning, the two studies take advantage of differences in how the federal funding was awarded to individual districts.

The total ESSER expenditure was fueled by three laws setting aside $13 billion in March 2020, $57 billion in December of that year, and a further $122 billion the following March. Because there was no data showing where learning loss was most concentrated at that time, dollars were allocated to school districts based on their pre-pandemic grants from Title I, the Department of Education’s main program benefiting disadvantaged children.  

But not all districts received comparable amounts, even if they served similar numbers of needy students. Instead, a number of regulations governing Title I — including rules that ensure small states receive minimum allotments, as well as larger sums being granted to states with higher per-pupil spending — introduced significant spending gaps between different schools. Those disparities were significantly magnified as each new emergency funding bill was passed, said Harvard economist Thomas Kane, Reardon’s co-author. 

“With the second two ESSER packages, the federal government was essentially pushing $175 billion through pipes that were meant to handle $16 billion in Title I,” Kane said. “So what might have been a $500 or $600 difference per student in Title I dollars became a $5,000 or $6,000 difference in ESSER funding per student.” 

Both Goldhaber and the Education Recovery Scorecard team accessed standardized test results from the Stanford Education Data Archive, which compiles student scores from different local exams to allow for cross-state comparisons. In each of their studies, $1,000 in ESSER spending per student was found to raise math scores by 0.008 of a standard deviation (a scientific measure showing the distance from a statistical mean).

In the world of education research, an improvement of that size is considered small: something like one-tenth of a medium-sized effect. But the average conceals substantial variation across different states, and many school districts received much more than $1,000 per student. 

As an example, Reardon, Kane, and their collaborators identified 704 districts in which over 70% of students were eligible for free and reduced-price lunch — a commonly used proxy for poverty — then compared the results for those that received unusually large ESSER allocations (more than $8,600 per pupil) to those that received much less (less than $4,600 per pupil). 

The differences were striking. The working-class district of Brockton, Massachusetts was awarded $3,224 per student from the second and third ESSER funding bills, and its students’ math achievement improved by the equivalent of .06 grade levels between 2022 and 2023; but in Dayton, Ohio, per-pupil funding increased almost three times as much ($11,444), and math scores jumped by a factor of 10 (.65 grade levels).

Goldhaber argued that figures like those cast considerable doubt on the proposition that the U.S. government’s emergency relief to schools was mostly wasted.

“One of the ideas that’s out there is that we spent $190 billion and got nothing,” he said. “I don’t think that’s the right answer.”

Given what we know now, any new federal dollars for recovery should probably be structured differently.

Marguerite Roza, Georgetown University

Yet he also voiced disappointment that neither Washington nor states had directly measured what kinds of ESSER spending (tutoring programs or school renovations, improved ventilation or increased staffing) were correlated with higher performance. Despite its huge cost and high stakes, Goldhaber concluded, ESSER was simply “not designed to learn from what districts do.”

“To my mind, that makes it a huge missed opportunity. We can see that there are pretty big differences across states and districts in the degree of catch-up.”

‘Who’s going to pick up the reins?’

While the studies can shed little light on the most successful aspects of ESSER, they will be collectively seen as a major contribution to the research on school finance reforms. This is true both because of the scale of the government’s intervention — perhaps the single greatest natural experiment on the effects of windfall cash on schools that has ever been attempted — and the consistency of the papers’ results. 

Not only do the findings of both studies mirror one another, they also hew closely to those of an influential meta-analysis, published in January, that gathered the results of dozens of previous experiments in increased school funding. That paper also pointed to an average test-score increase of about .032 standard deviations per $1,000 spent over four years, or roughly .008 annually.

Marguerite Roza, head of Georgetown University’s finance-focused Edunomics Lab, called the coinciding findings “reassuring.”

Yet she also noted the “wildly expensive” cost of sending operating aid to states that was not specifically dedicated to learning recovery. According to Goldhaber’s calculations, the government would need to spend an additional $450–$650 billion to fund a full return to levels of academic achievement last seen in 2019; Reardon and Kane tallied a likely cost of just over $904 billion. 

Whether or not those figures represent the true price tag, Roza said, states that intend to replace federal dollars should be more consistent in disbursing them and more stringent about what they pay for.

“Why repeat the same strategy given how unevenly the dollars were distributed and how uneven the effects were on districts and states?” Roza asked. “Given what we know now, any new federal dollars for recovery should probably be structured differently.”

Our results are basically saying that there was a positive effect, but it wasn't enough. Now who's going to pick up the reins?

Thomas Kane, Harvard University

But in Kane’s view, that recommendation may be too optimistic. With just a few months left before the deadline to spend ESSER funds, he observed, too few state authorities had even committed to picking up the torch of learning recovery. 

“In most states, there hasn’t even been a discussion started about what the state role will be now that the federal money is running out,” he said. “Our results are basically saying that there was a positive effect, but it wasn’t enough. Now who’s going to pick up the reins?”

Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter