This article originally appeared on Duke University’s Center for Strategic Philanthropy and Civil Society blog Intrepid Philanthropist starting on January 11, 2010.

Last April, Jeff Bradach and I wrote an article that appeared in The Chronicle of Philanthropy. Scaling What Works: Implications for Philanthropists, Policymakers and Nonprofit Leaders tried to summarize some of things that Jeff, who is managing partner and co-founder of the Bridgespan Group, and I have learned over the past ten years as the Edna McConnell Clark Foundation and Bridgespan have provided financing and management consulting to nonprofits striving to expand programs with proven impact.

Our article was well received. It also raised a number of questions that I would like to take this opportunity, kindly offered by Duke’s Center for Strategic Philanthropy and Civil Society, to answer. These questions and answers will be of interest, I hope, to nonprofits, their funders, and policymakers who are confronted with the challenge of converting into legislation, regulations, and RFPs the Obama administration’s commitment to funding evidence-based programs that work. There is a growing political consensus to channel taxpayer funds to programs with proven impact, and the current economic crisis, the federal deficit, and limited state and local budgets make it likely this consensus will continue to grow.

What constitutes an "evidence-based" program?
There is no universally accepted definition, but the Clark Foundation and like-minded funders and research organizations such as Public/Private Ventures, the Coalition for Evidence-Based Policy, and Child Trends believe that only a rigorous, independent evaluation can provide persuasive empirical evidence that a program is effective and, equally important, determine what makes it effective. For if we don’t know what a program’s essential elements are—the demographic characteristics of the population it serves, for example, or the dosage and duration of the programming—how can we replicate it with fidelity and expand it with confidence that it will work elsewhere?

Not every promising program needs to be or can be evaluated independently from the very outset. Yet our experience suggests that the earlier an organization embraces performance measurement and evaluation, the sooner it will be prepared and likely to grow. Especially given the severe constraints on public funds and the keen competition for them, government would be doing a disservice to taxpayers and to the vulnerable populations social programs are intended to assist if it did not require rigorous evaluations before investing large amounts of money in expanding such programs on a significant scale.

How an Organization Can Build Its Evidence Base

Most nonprofits, including a majority of the Clark Foundation’s grantees, do not yet have convincing quantitative evidence of their programs’ effectiveness and lack the organizational capacity to muster it. That is why we invest heavily in helping grantees build their organizational capacity—including developing the infrastructure they need to track and improve their performance.

When an organization implements a program, it’s important that it collect data documenting that participants in the program benefit from it. Developing an internal performance measurement system requires investing in information technology to collect the data systematically and in skilled personnel to analyze it. At the Clark Foundation, we call this High-Level Apparent Effectiveness.

While internally collected evidence can help demonstrate a program’s promise, independent, third-party research can marshal more persuasive evidence by systematically collecting and comparing data on outcomes for program participants and outcomes achieved by people the program does not serve. Once an organization can track internally a program’s performance, sees positive apparent results, and feels confident in the program model and its implementation, then it may be ready for an external comparison group evaluation. Reaching this level of evidence, which we call Demonstrated Effectiveness, requires additional investment in infrastructure.

Finally, an organization can build its performance-tracking capacity to the point where it can undertake a rigorous external evaluation that scientifically confirms a program’s impact on participants. This is what the Clark Foundation terms Proven Effectiveness.

We strongly believe that the ability to build a strong evidence base is inextricably linked to an organization’s overall capacity—and that both are prerequisites for achieving long-term sustainability.

A Single Standard of Evidence?

In an ideal world, an experimental evaluation or randomized controlled trial (RCT), which compares participants in a program to a randomly assigned control group, provides the highest standard of proof and level of confidence that a program works. In the real world, an RCT is not always feasible, for a variety of reasons:

• It can be time consuming and costly.
• Denying services to a control group may raise ethical issues.
• A program may not be mature enough operationally or an organization’s performance management and measurement systems insufficiently developed for such robust research.
• A program may be so complex and multifaceted, or its objectives so broad, that designing an RCT is extremely difficult. The Clark Foundation and those of our grantees that provide a single service, be it home visitations by nurses to low-income first-time mothers or an after-school mentoring program, have found RCTs less daunting than have grantees that offer an array of services to the entire community in which they are based. With these multi-service organizations, as we call them, the Clark Foundation is still exploring the best ways to marshal evidence on their programs’ behalf.

Under some of the above circumstances, a third-party, quasi-experimental evaluation that compares participants to a control group that is not randomly assigned may represent the highest proof point a program is capable of reaching. Developing evidentiary standards for effective programs is a work in progress, and the best measure in one domain—healthcare or youth development, for example—may not apply without adjustments to another, such as alternate energy sources or community development.

The Value of Evaluation

In our experience, an independent evaluation benefits an organization in other important ways in addition to assessing a program externally. The data collection and analysis an evaluation requires provide an opportunity and means to improve a program internally. The information an evaluation gleans enables an organization to determine what works and what doesn't, and to adjust its program(s) accordingly. Preparing for and undergoing an evaluation has prompted several of our grantees to design or improve performance measurement systems and to use these as management tools. The discipline of evaluation, understanding a program’s strengths, limitations, and areas needing improvement, makes an organization more realistic, sophisticated, and effective.

It’s important to note that not every program requires an evaluation to prove its effectiveness. Many organizations, including some EMCF grantees, have adopted and implemented models that have already been evaluated. In such instances it is critical to ensure that the model is replicated with fidelity. Funders and nonprofits can use ongoing performance tracking data to document that the program is reproducing its original outcomes. Many a promising program has been compromised by cutting corners.

How Philanthropy and Government Can Work Together

How can philanthropy and government partner to help nonprofits marshal evidence of their programs’ effectiveness?

By setting high evidentiary standards and funding programs that meet them, government can create a powerful incentive for nonprofits to strengthen their evidence and invest in rigorous evaluation. Already the Obama administration has advanced evidence-based policy in a number of commendable ways:

• It established the Social Innovation Fund, in the words of Assistant to the President and Domestic Policy Council Director Melody Barnes, to further “the President’s new governing philosophy: funding and investing in what works.”
• After reviewing the federal budget and eliminating almost $17 billion supporting programs that did not deliver results, Office of Management and Budget Director Peter Orszag stated, “I am trying to put much more emphasis on evidence-based policy decisions here at OMB. Wherever possible, we should design new initiatives to build rigorous data about what works and then act on evidence that emerges—expanding approaches that work best, fine-tuning the ones that get mixed results, and shutting down those that are failing.”
• The Department of Education’s Investing in Innovation (i3) Fund translates this ideal into action by reserving its largest grants for programs “with a strong base of evidence,” smaller grants for those “with good evidence of their impact and … ready to improve their evidence base,” and the smallest for programs with high potential “whose impact should be studied further.”

Government can help an organization get ready for evaluation. It can do so directly by funding an evaluation or preparations for it, or indirectly by including enough overhead in its grants that a recipient can afford a robust performance tracking system or other enhancement of its infrastructure. But given the pressures and skill sets that exist at public agencies, government can go only so far. Philanthropy can play a crucial role in ensuring such readiness, because we can dedicate our resources and time to working with grantees, with fewer of the political constraints that weigh on public funders. We can also support individuals and organizations with appropriate skills, and can experiment with different ways to achieve readiness and evaluate evidence.

Working in tandem with government yet autonomously, some form of philanthropic initiative or intermediary might create:

• a grantmaking program that helps organizations advance their evidence and organizational capacity;
• a rating tool/framework for making a baseline assessment of programs and evaluating their progress periodically;
• a technical assistance and capacity-building program in module form for training nonprofits in groups and individually; or
• a “knowledge platform” available via a website and other communications vehicles that disseminates information to other nonprofits.

In areas like these, philanthropy has a strong comparative advantage—and we can help promising programs meet higher standards of proof so they can grow and make a greater impact on some of America’s most intractable social problems.