Working to find actionable solutions to the nation's key challenges.

How to Make ‘Pay for Success’ Live Up to Its Name

By Tim Shaw, Kody Carmody

Thursday, July 12, 2018

Pay for Success,” an innovation in government contracting, is emerging as a distinct model for using evidence to determine funding of programs. It makes payments only when programs achieve a specific set of outcomes, in order to focus policymakers and social service providers on achieving specific results, ensure more programs are evaluated for success, and involve a wider array of partners in providing social services by incentivizing private companies to provide the initial capital for those programs.

This model was given a boost last year with $100 million in funding specifically for “Pay for Success” grants included in the Social Impact Partnerships to Pay for Results Act (SIPPRA). As the Commission on Social Impact Partnerships launches to advise on how to best allocate those funds, the following are suggestions to maximize the impact of these programs and to support the evidence-based policymaking movement more broadly.

Fund projects with specific and meaningful outcomes.

Many federal and non-profit grants require that their grantees provide clear measurable goals for their programs. However, many grantees struggle to include specific outcomes that capture the meaningful, long-term changes they hope to see in their program participants. While SIPPRA provides a long list of possible outcomes for eligible proposals, such as reducing the number of children returning to foster care, it correctly leaves the ultimate responsibility of defining measurable outcomes to the state or local government proposing a project. The commission should take care in evaluating SIPPRA applications to make sure that proposed program outcomes are specific, impactful, and long-term to determine whether approved programs made a lasting impact. 

Fund adequate evaluation capacity and use a diverse set of rigorous methods.

Evaluations are the cornerstone of “Pay for Success” projects and should be funded adequately to identify whether a program achieves its outcomes. Unfortunately, funding those evaluations sufficiently has historically been a challenge. A 2015 Government Accountability Office report on “Pay for Success” initiatives found that sufficient data collection and analysis capacity was a key concern for communities engaged in those projects.

Evaluate program implementation as well as outcomes.

To fully understand why a program is or is not successful, it is not enough to evaluate the program’s outcomes; evaluators must also assess how well a program was implemented and in what context it was operating. Social programs can succeed or fail for any number of reasons not associated with the program’s design, including unexpected recessions or economic booms, unexpected program participant dropouts, or even simply poor management. A key benefit of SIPPRA will be the evidence produced by the various programs that the initiative funds, and up to 15% of SIPPRA funds may go toward program evaluations to build that evidence. To ensure that evidence is useful, this money can’t just be used to evaluate whether a program succeeded or failed—data need to be also collected on what might have caused those successes and failures.

Structure the contract to match the program’s outcomes, not for political results.

Flashy headlines in a rapid news cycle do not match an evidence-based approach to meaningful outcomes. Consequently, it may be tempting for local and state applicants and the federal council assessing them to focus on short-term, immediately observable data as the basis for program success. Unfortunately, this can lead to the funding of ineffective programs, or the reduction of funding for effective ones.  Programs that try to address many of the outcomes described in legislation, such as increased employment and earnings, early childhood health and development, and recidivism, can take years to determine whether they are ultimately successful. To ensure funding goes to effective programs, the outcomes associated with SIPPRA grants need to be ones that affect the lives of program participants, not what will generate headlines.

Commit to learning from projects and building knowledge.

The environment in which government implements programs is constantly shifting.  A hard truth in trying to find answers to society’s “wicked problems” is that there are often no easy answers. Even with evidence-based approaches, it is possible for a program to encounter new challenges in different environments or have their effectiveness erode over time with new economic changes. For that reason, some of the programs funded through “Pay for Success” will fail, and others succeed for unexpected reasons, and that’s okay. As long as programs are structured to collect evidence on the details of their process and outcomes—which aspects of the program worked and why—that data will be added to the larger body of information and will help policymakers decide how to improve programs and prioritize funding towards the most effective programs.

Evaluate ‘Pay for Success’ effectiveness as an evidence-based funding model.

In addition to reviewing whether the individual programs funded by SIPPRA are successful at achieving their outcomes, research is needed on the effectiveness of “Pay for Success” as a tool itself. While the passage of SIPPRA and other signals of support from Congress and the White House are positive signs, there is still not enough evidence to  determine that the increased use of data improves outcomes. The evidence-based policymaking community continues to build strategies to effectively develop and use evidence, and the rollout of SIPPRA is a prime opportunity to test “Pay for Success” as one of those tools.

Take the best of ‘Pay for Success’ and apply it to other evidence-based practices.

During a June 2018 event, advocates of SIPPRA pointed to a wide array of benefits of the “Pay for Success” model, such as thoughtfully considering the goals of social programs, making sure the data collected on programs align with the outcomes they are trying to achieve, and maximizing taxpayer’s return on investment. These goals are critical and should not just be limited to “Pay for Success”; they are evidence-based goals for government programs overall. As the “Pay for Success” community learns lessons from the implementation of SIPPRA, best practices from those programs should be applied broadly to support evidence-based policymaking.

Understand that ‘Pay for Success’ is not the silver bullet for evidence-based policymaking.

“Pay for Success” is an innovative way to fund social programs that rely on evidence, but it should be a tool in the evidence-based policymaking toolbox, not the only strategy. This model works best when there is already a large evidence base demonstrating the effectiveness of a program, since those with the contracts stand to lose a significant amount of money if the program does not achieve its outcomes. For other promising programs and policies, such as new and innovative approaches with limited existing evidence, or large, historic federal programs that need ongoing data collection and analysis, “Pay for Success” may not be the right tool.

If policymakers keep the above considerations in mind, “Pay for Success” could play a key role in the future of improving federal efforts to implement evidence-based policymaking.

KEYWORDS: 115TH CONGRESS