This is the second of a three-part series that looks at how public agencies can invest resources in programs that work.
President Barack Obama this coming Monday will submit his fiscal year 2013 budget request to Congress. These are budget-conscious times, as the political events of 2011 showed. That’s why the White House should work hard to ensure that as much money as possible is directed toward programs that are proven to work rather than just to the places where money was spent in years past. If every public dollar is to stretch further than before, then legislators can make serious efforts to steer funds toward proven programs and interventions—in other words, toward “what works.”
In the educational arena these may be programs that try to improve literacy rates among struggling students in low-income or minority school districts, for instance, or aim to improve outcomes for children learning English as a second language. Even where federal programs support state or local governments, federal agencies should ideally seek to ensure that funds are directed toward approaches that work.
A second, obvious challenge presents itself immediately, though: How will legislators and on-the-ground decision-makers know which programs are proven effective and which aren’t? One solution to finding and funding “what works” already exists in the world of education. A number of online platforms strive to provide lawmakers, school officials, and foundations making philanthropic donations with comprehensive information about the effectiveness of a wide range of education programs.
These “what works” platforms are in their infancy but they represent a promising step toward evidence-based public decision-making in education. Presented below are several that the Obama administration and Congress should examine as they negotiate federal spending on education in the FY 2013 federal budget beginning in October this year.
Three “what works” platforms for education
The Center for American Progress late in 2011 hosted representatives from a number of “what works” platforms alongside several primary and secondary education researchers and Department of Education employees to gauge the efficacy of these platforms. Let’s first look at each of them and then examine their promise and limitations.
What Works Clearinghouse
The What Works Clearinghouse was established in 2002 to be a central, trusted source for “what works” in education. The Department of Education-run clearinghouse exists to “study the studies.” To date, the clearinghouse has reviewed 250 interventions in 15 different topic areas. The site provides a searchable database of studies, a review of the extent of evidence for a given study, and points out gaps in the evidence to “provide educators with the information they need to make evidence-based decisions.”
Top Tier Evidence Initiative
The Top Tier Evidence Initiative, launched in 2008 by the Coalition for Evidence-Based Policy (a nonpartisan organization dedicated to “increasing government effectiveness through the use of rigorous evidence about what works”), follows a legislative standard to identify proven social interventions via well-designed, randomized, controlled trials that have been reproduced in community settings. They evaluate a wide range of social programs, including educational interventions.
Best Evidence Encyclopedia
The Best Evidence Encyclopedia, operated by the Johns Hopkins University School of Education’s Center for Data-Driven Reform in Education, developed a Consumer Reports-esque rating system for evaluating the strength of evidence supporting various education interventions. The BEE’s gold standard of evidence involves randomized studies at least 12 weeks in duration using large groups of students and valid, large measures of success.
Philanthropic “what works” platforms
There are also a number of “what works” platforms that aim to guide philanthropic dollars toward proven social interventions. These include the Center for High Impact Philanthropy, Social Impact Exchange, and the Business-Higher Education Forum. These platforms provide data-driven evaluations of social programs as a service to foundations and individual donors who want to fund proven programs.
Outstanding issues in existing platforms
The “what works” platforms described above are an important first step for steering more money toward proven educational interventions. As in any fledgling field, however, there is still some work to be done. There aren’t yet very many programs proven “effective” under the most rigorous standards of evidence.
For instance, 90 percent of programs reviewed by the Department of Education’s What Works Clearinghouse do not meet their standards of evidence. Similarly, the Top Tier Evidence Initiative has been able to identify only eight “top tier” interventions and four “near top-tier” ones. In part that is because many programs have never been evaluated to determine effectiveness.
There are also two key metrics that the platforms don’t all take into account. The first is implementability, or the likelihood that a program that works in one school or district will work in another school or district. The other is the critical issue of cost-effectiveness, or the colloquial “bang for the buck.”
A recent report by Lisbeth Schorr and Frank Farrow of the Center for the Study of Social Policy underscores the problem of implementability. They find that context matters, with even proven programs working well in one school district but needing to be tweaked to deliver the same results under different conditions in another.
The platforms should seriously consider including data on cost-effectiveness in their evaluations. Principals, superintendents, and school boards need to have this information in order to decide whether they should choose an intervention that has a 90 percent effectiveness rate but costs $1,000 per child, or a program with a 75 percent effectiveness rate that costs just $250 per child.
The bottom line: “What works” platforms are important and useful even without being perfect, but the federal government should be doing more to incentivize their use and to accelerate the creation and testing of evidence-based interventions in education and elsewhere. This will have the added benefit of boosting demand for evidence-based interventions—without a range of options to choose from, school-level decision-makers sometimes resist new data-driven models. In a difficult fiscal environment, every penny that can be directed toward a proven, evidence-based model should be used to achieve better results.
Kristina Costa is a Special Assistant with the Doing What Works project.