Pew Public Safety Performance Project

We have agreed to a major partnership with The Pew Charitable Trusts and GiveWell as part of our work on criminal justice reform. Good Ventures will provide $3 million to support and expand the work of Pew’s Public Safety Performance Project (PSPP), which aims “to advance data-driven, fiscally sound policies and practices in the criminal and juvenile justice systems that protect public safety, hold offenders accountable, and control corrections costs” through technical assistance to states, research and public education, and promotion of nontraditional alliances and collaboration around smart criminal justice policies.

We see this partnership as an important step on multiple fronts:

• Criminal justice reform is a current focus area for us, and PSPP appears to be one of the most prominent and effective organizations working toward change on this front. Funding and following its work represents an opportunity for both impact and learning.
• We are also interested in developing a relationship with Pew as a whole. We believe Pew will be a valuable resource as we continue to explore policy-oriented philanthropy. Based on conversations with Pew representatives, we see supporting PSPP as one of the best ways to support Pew as a whole.
• Finally, the process of establishing this partnership has itself been a valuable learning opportunity. With PSPP’s help, we have conducted a brief review of PSPP’s track record, which was our first attempt to assess the track record of a U.S.-policy-focused organization and taught us a fair amount about the criminal justice reform space. We have also dealt with new challenges around how to balance our goal of transparency with the goal of having maximal impact; when working on policy, there can be particular tension between these, and we have established an agreement regarding public discussion of PSPP that may serve as a guide to future grant agreements.

What does PSPP do?

PSPP’s goal is to “work with states to advance data-driven, fiscally sound policies and practices in the criminal and juvenile justice systems that protect public safety, hold offenders accountable, and control corrections costs.”¹ By advancing data-driven policies in their state work and research, PSPP additionally seeks to reshape the framing of the criminal justice debate from “tough on crime v. soft on crime” to a dialogue about finding the best public safety return on investment.²

Activities PSPP engages in to accomplish this include:

1. Research and public education.³ PSPP has published research on:
   • The size and cost of federal and state corrections systems
   • Lengths of stay in prison⁴
   • “The high cost and low return of incarcerating low-level offenders
   • State-by-state recidivism rates
   • Effective strategies to reduce prison populations, recidivism, and criminal justice costs in ways consistent with public safety
   • Public opinion, both on criminal justice reform broadly and on specific policy proposals”⁵
2. Engaging nontraditional allies of criminal justice reform.⁶ PSPP supports (including via financial support) the voices of nontraditional allies of reform such as business leaders, crime victims, law enforcement, and conservative organizations such as the Texas Public Policy Foundation and Prison Fellowship Ministries, and the American Conservative Union Foundation, which together form the Right on Crime initiative^.7 PSPP believes that publicizing public support for criminal justice reform (including the support of conservatives) puts reform on policymakers’ agendas and expands the window of reforms that policymakers believe are possible.⁸ In our conversations with U.S. public policy generalists, criminal justice reform specialists, and policymakers involved in reforms assisted by PSPP, Right on Crime came up as a particularly influential organization.⁹
3. Providing technical assistance to individual states. ¹⁰ We discuss this activity in more detail below.

PSPP’s provision of technical assistance to states

Pew provides technical assistance as part of a formal process in partnership with the U.S. government. It states:

“PSPP supports efforts in select states that want a better public safety return on their corrections spending. The project also helps state officials, practitioners and others across the country share state-of-the-art knowledge and innovative ideas through national research and publications; policy evaluations; public opinion surveys; national, regional, multi-state, and state-level policy forums; and online information about what works to reduce recidivism.

Much of Pew’s work in the states is conducted in a partnership with the U.S. Department of Justice’s Bureau of Justice Assistance (BJA) under what is called the Justice Reinvestment Initiative (JRI). Although the first three years of the justice reinvestment process (2007-2009) relied in part on BJA discretionary funding, the program was formalized with an appropriation from Congress in 2010. It is important to note that much of the public material about JRI (including a recent assessment of JRI by the Urban Institute) refers only to the work conducted under the formal federal program; this accounts for discrepancies in various documents about how many states have participated in the initiative and what the total projected savings are.

Between 2007 and 2009, technical assistance to the states was provided by the Council of State Governments Justice Center (CSG) and the Vera Institute of Justice (Vera), with support from Pew and BJA. As a result of increasing demands for assistance from the states, Pew built the staff and organizational infrastructure to provide technical assistance to the states directly, beginning with South Carolina in 2009 for the 2010 legislative session.

Under the formal JRI process, state technical assistance currently is provided in two phases: Phase I, which encompasses analysis of the state’s criminal justice system and development of reform policies up to and including adoption of those reforms (i.e. passage of new laws, court rules and agency policies); and Phase II, which refers to the implementation of the new policies, practices and programs. States are selected for participation in JRI by BJA and Pew, through close consultation and collaboration but independent assessment and decision-making processes. Currently, Phase I technical assistance is provided by Pew and by CSG (with support from Pew and BJA). Phase II assistance is provided by CSG and Vera with funding from BJA.”¹¹

While a fairly wide array of reforms fall under the rubric of “justice reinvestment,” the approach generally involves using data to improve the efficiency and reduce the costs of the criminal justice system and then using those cost savings to fund evidence based public safety practices and improve state budgets.¹² Procedurally, justice reinvestment focuses on “consensus-based, bipartisan, and interbranch decision-making.”¹³

What is PSPP’s role in the states where it provides technical assistance?

The justice reinvestment process begins with the establishment of an interbranch, bipartisan criminal justice working group composed of policymakers and criminal justice system leaders.¹⁴ PSPP’s technical assistance then includes three phases

• First, PSPP conducts the data analysis and systems assessment needed for the working group to identify and report on the drivers of correctional populations and costs.¹⁵
• Next, using the findings from the first phase, PSPP helps the working group develop policy proposals and is responsible for forecasting the effects of different proposals so that the working group can consider the relative merits of different reforms.¹⁶
• Finally, PSPP provides legislative support and public education, aiming to help policymakers move the bill through the legislative process.¹⁷

While PSPP’s role varies somewhat by state, policymakers in two states where a justice reinvestment bill passed with PSPP’s assistance¹⁸ told us about specific ways PSPP assisted the reform process:

• hosting a conference to educate policymakers about sentencing and corrections research and the experiences of other states¹⁹
• analyzing data to determine what crimes accounted for the majority of the prison population and, in turn, how prison populations would be affected by modifying various criminal justice policies²⁰
• helping the state prioritize among possible reforms and providing information about evidence based practices²¹
• helping to draft a bill to codify reforms²²
• setting up meetings between legislators, the bill’s supporters, and criminal justice system stakeholders, such as law enforcement²³
• bringing experts to the state to build support for the bill and speak about other states’ experiences with reform²⁴
• garnering positive press coverage by engaging with local media²⁵
• estimating the impact of various proposals and amendments on the state’s prison populations during legislative negotiations,²⁶ and meeting with legislators and answering their questions about the bill²⁷

Which states have gone through the justice reinvestment process?

Between 2007 and 2013, twenty-seven states passed justice reinvestment bills and two states went through the process twice for a total of twenty-nine bills.²⁸ PSPP either funded or itself provided technical assistance to each state other than Delaware, whose technical assistance was provided by the Vera Institute of Justice and funded entirely by the federal Bureau of Justice Assistance.²⁹ The content of the bills varied; PSPP classifies twenty-three of the reforms as “comprehensive” and five as limited.³⁰ In 2014, PSPP is providing assistance to two new states (Mississippi and Idaho) and one state in which a bill previously passed (Michigan).³¹ In order to receive justice reinvestment assistance, these states had to demonstrate their need for reform (as evidenced by large increases in corrections populations and costs) and that their leadership was committed to the reform process.³² A table listing the state, year, and characteristics of each justice reinvestment bill is available here.

What types of policy changes have been made in justice reinvestment states?

The reforms generated by justice reinvestment aim to be individualized and heavily influenced by policymakers’ and stakeholders’ preferences, and the technical assistance provider’s analysis of the drivers of correctional populations and costs in that state.³³ PSPP does, however, aim to maintain a focus on evidence-based corrections and sentencing reforms, and does not emphasize other aspects of the criminal justice system. ³⁴ The Urban Institute’s recent assessment of justice reinvestment listed the following common reforms:

• Risk and needs assessments of offenders to improve decisions about detention, incarceration, release, supervision, and treatment
• Accountability measures to ensure the use of evidence based practices, require justification of departures from sentencing guidelines, and/or develop data reporting requirements
• Good time credits that reduce the sentences or length of community supervision for inmates who behave well in prison or participate in programs
• Intermediate and graduated sanctions, often based on Hawaii’s HOPE model, to deter bad behavior
• Increased funding for community based treatment such as substance abuse programs and reentry plans
• Sentencing changes
• Mandatory supervision requirements to ensure that high-risk offenders are supervised after being released from prison
• Problem solving courts, which are often focused on offenders with substance abuse or mental health disorders

• Streamlined and expanded parole to reduce incarceration while protecting public safety by expanding post-release supervision.³⁵

To get an extremely rough sense of the relative magnitude of these policies, we look to the reform package in Arkansas, the state for which we have the most detail because we chose to more closely vet PSPP’s forecasting process there.³⁶ In Arkansas, 74% of the forecasted reduction in prison beds came from three policy reforms:³⁷

• Improving the parole release process to encourage compliance with Arkansas’ presumptive parole system (a system where, absent a disciplinary record or failure to satisfy special conditions, offenders are presumed to be transferred from prison to parole after a certain proportion of their sentence is served) and ensure that parole hearings occur on time³⁸
• Instituting earned discharge for parole so that parolees are incentivized to follow the rules and rewarded for doing so with accelerated discharge from supervision³⁹
• Measures to encourage judges to adhere to sentencing guidelines⁴⁰

Future activities

Pew may broaden its scope in the future, as discussed below.

Track record

Our vet of PSPP’s track record focuses on evaluating the impact of bills associated with its technical assistance activities discussed above. This does not mean we think PSPP’s other activities aren’t valuable. Rather, consistent with the Open Philanthropy Project’s emphasis on “giving as investment”, our investigation of PSPP’s track record was geared towards determining whether PSPP has a generally impressive record of playing an important role in meaningful policy wins, rather than towards determining the precise “amount” of policy gains we can expect per dollar. To the extent that PSPP has such a track record, we believe a grant to PSPP as a whole (including its harder-to-evaluate activities) is likely to be a good investment. We focus on two main (not entirely separable) questions:

1. How meaningful were the bills that passed in the states where PSPP worked?
2. How much of a role did PSPP play in the states where it worked? Would large, legislative reform packages have passed in PSPP’s absence? If so, to what extent did PSPP improve the quality of the bills?

We recognize that one obstacle to evaluating PSPP’s track record is that it may be too soon to evaluate the success of the most recent reforms. This issue is especially important because PSPP’s assistance has evolved over time and these recent reforms may better reflect PSPP’s future plans than work that occurred farther in the past.⁴¹

How do we evaluate the success of criminal justice reforms?

When evaluating reforms to the criminal justice system, we are primarily concerned with three outcomes.

1. First, we value reductions in the number of people incarcerated because of the large human and economic costs of incarceration. Although we primarily discuss the most easily available data, which focuses on overall prison populations, we are especially interested in reducing the number of low-risk and non-violent offenders in prison. (PSPP is currently developing performance measures that account for the composition of the prison population as well as its magnitude.).⁴²
2. Second, we are concerned with criminal justice reform’s impact on public safety and in particular on violent crime. We did not closely vet these bills’ effects on crime rates, but justice reinvestment aims to reduce recidivism by cutting prison costs and redirecting savings to improve reentry and community supervision.⁴³ PSPP is currently developing a risk- and severity-weighted measure of recidivism and we expect this measure will be a decent proxy for public safety.⁴⁴
3. Third, we value criminal justice reform’s positive impact on state budgets. According to a Pew report, total state spending on corrections increased from $12 billion in 1987 to $49 billion in 2007.⁴⁵A more recent report by Pew estimated that, on average, the operating cost to incarcerate a prisoner for one year was almost $29,000 in 2008.⁴⁶

Our examination of PSPP’s track record focuses on the first of these factors – the organization’s impact on prison populations. This is the legislation’s most direct effect; if the bills PSPP assisted have a substantial impact, this is where we would expect it to show up most quickly. The bills’ effects on prison population are also the easiest to measure and the outcome for which we have the best data. Moreover, because decreases in prison population directly cause decreases in correctional spending, we would expect the bills’ effects on these outcomes to be very highly correlated.

A portion of this decrease in correctional spending is reinvested in programs to increase public safety and the rest is used to improve state budgets. We have not closely looked into the amount of savings that are typically reinvested nor have we closely vetted the evidence behind these programs. A full review at the level of depth we investigate our top charities would have looked into these issues and would also have more closely looked into changes in crime rates, and in the composition of the prison population. As noted above, we would guess that there will not be a negative effect on either of these outcomes due to PSPP’s attention to improving community supervision and limiting prison beds for high level offenders.

Were the bills PSPP worked on a big improvement over the status quo?

How big of an impact does PSPP forecast for the bills it assisted?

States seeking to engage PSPP are assessed in advance for their need for reform. This assessment has a particular focus on whether prison populations and correctional costs are rising, so a naïve comparison between pre-reform and post-reform prison populations would be unlikely to yield an unbiased estimate of reform’s effects.⁴⁷ Instead, to get an initial sense of the bills’ magnitude, we look at PSPP’s impact forecasts (while cautiously interpreting the organization’s forecasts of its own impact).⁴⁸ For each state PSPP assists, PSPP (or a contractor) creates or verifies a state’s own a) projections of the amount that prison populations would grow in the absence of reform and b) projections of post-reform populations. PSPP shared with us results of these forecasts from eighteen of the twenty-nine reforms that occurred before 2013.⁴⁹ Prison populations were rising in these states prior to reform. On average, prison populations grew by about 10% over the course of the five years leading up to reform’s passage and PSPP projected further growth of 11% over 5 years if the status quo were to be maintained.⁵⁰ PSPP forecast that, with reforms, prison populations would, instead, decrease by an average of 1.7% over the five years after reform took place. This would lead to prison populations that would on average be 11% lower after reform than they would have been if the status quo were maintained. We believe that these forecasted prison population reductions due to bills that PSPP assisted are PSPP’s most significant claimed successes, so our evaluation of PSPP’s impact focuses on vetting these forecasts.⁵¹ We did not ask PSPP for estimates of the fiscal savings in each state because we would guess that correctional cost savings are roughly proportional to the number of prison beds eliminated. That said, PSPP did send us policy briefs or other research materials with estimates of fiscal savings from five states:

• Arkansas: $875 million over ten years⁵²
• Georgia: at least $264 million over five years⁵³
• Hawaii: $130 million over five years⁵⁴
• Kentucky: $422 million over ten years⁵⁵
• South Carolina: $241 million over five years⁵⁶

We have not carefully vetted these estimates.

Vetting the forecasts

The majority of saved prison beds attributed by PSPP to these bills represent averted potential prison population growth and not absolute declines in a state’s prison population. This complicates our attempts to verify the accuracy of the impact forecasts. Prison populations could drop by the predicted amount in the states where PSPP works, but these small absolute declines only reflect major changes (relative to the counterfactual) if the counterfactual forecasts of continued growth under the status quo are correct. In light of these concerns, we took the following steps to investigate the impact on prison populations of the bills PSPP worked on:

• For a fairly arbitrarily chosen state (Arkansas), we looked into the methods and assumptions behind PSPP’s forecasts and attempted to determine whether they were reasonable, keeping in mind that we are not experts on criminal justice reform.⁵⁷
• We plotted PSPP’s forecasts of post-reform prison population next to actual post-reform prison population in these states. We looked to see whether actual post-reform prison populations were reasonably close to PSPP’s forecasts and to see whether there were noticeable breaks in prison population trends at the time various bills passed.
• We used data from the Bureau of Justice Statistics to graph actual prison population data in each state in which PSPP worked and examined these graphs to look for breaks in the trends at the time of reform.
• We calculated actual post-reform prison population changes in each state, including both states PSPP had worked in and states PSPP had not worked in.

Vetting the model used to forecast the impact of reform in Arkansas

PSPP contracted out the forecast of the impact of SB 750, Arkansas’ justice reinvestment bill.⁵⁸ The impact estimate is composed of two separate forecasts. First, there is a “cost of doing nothing” forecast that projects future prison populations under the status quo. Second, PSPP estimated the reduction in inmates’ time served due to the bill.⁵⁹

Between 2008 and 2010, Arkansas’ prison population grew 9.9% from 14,686 to 16,147.⁶⁰ The forecasts projected that, had no bill passed, Arkansas’ prison population would have continued rising to 19,222 in 2015.⁶¹ Instead, due to SB 750’s passage, PSPP predicted that Arkansas’ prison population would fall to 14,226 in 2011, followed by a slower rise to 16,279 in 2015.⁶²Thus, PSPP forecast that five years after its passage, the bill would be saving Arkansas 2,943 prison beds – a 15.3% decline relative to the counterfactual.⁶³ PSPP reports that they intend to be conservative as they settle on assumptions of the effects of each policy change.⁶⁴ Their level of conservatism depends on three factors:

• How reliable is the data related to that policy change?
• How “hard” is the policy lever used to create change (i.e. how likely are practitioners to implement the reform as intended)?
• How much confidence does PSPP have in the likelihood of quality implementation?⁶⁵

A couple of notes about the overall model behind the Arkansas forecast:

• Prison beds saved are allocated to the year the inmate with reduced length of stay is admitted into prison, not the year in which the bed is actually saved. This led the model to overstate the speed with which the Arkansas reforms would take effect.⁶⁶
• The model does not explicitly account for most of the second order effects of reform. For example, the model does not explicitly model recidivism among the inmates whose sentences are reduced, any change in the crime rate or the composition of crime due to changed incentives, or behavior change among law enforcement in response to reduced lengths of stay. We are unsure whether these omissions are more likely to lead to underestimates or overestimates of the bill’s impact on prison populations.⁶⁷

Our bottom line is that there are clear limitations to the general model PSPP used to forecast the effects of Arkansas’ reform and we would expect to be more confident in PSPP’s more recent microsimulations (which address the above issues) than we are in this older model. Overall, we would guess that the Arkansas model’s limitations would lead PSPP to overestimate the speed with which the bill affected prison populations by up to a couple of years. However, assuming here for the sake of argument that the assumptions of the impact of individual provisions of the policy are correct, we would guess that the model’s limitations would not lead to a large overestimate of overall impact.

Vetting PSPP’s estimates of the impact of individual policy changes.

PSPP estimated that Arkansas’ justice reinvestment bill saved 2,541 prison beds in the bill’s first year.⁶⁸ Because PSPP projected that 74% of these saved beds resulted from three policy changes (see < a href=”#PSPPsprovisionoftechnicalassistancetostates”>above), we focused on understanding how PSPP and Arkansas officials arrived at their projections of these three reforms.

• Improving Arkansas’ parole release process. While analyzing the drivers of increased correctional costs in Arkansas, the state’s criminal justice working group, assisted by PSPP, found that 55% of prisoners were being held an average of 6.2 months beyond the date at which they were presumptively entitled to parole meaning that absent a disciplinary record or failure to satisfy special conditions, there was a presumption that they would be released to parole on that date.⁶⁹ While the state did not record a reason for most of these delays, PSPP believes some of these prisoners were held because (due to crowding) they were being kept not in prisons but in jails (where parole review is often delayed), some were held because of the parole board’s failure to properly implement presumptive parole, and some were held because the inmates had behavioral violations in prison or otherwise did not deserve parole.⁷⁰ SB 750 included provisions to address this issue by:
  • Reemphasizing the expectation that, absent behavioral violations or other circumstances, Arkansas maintains a presumptive parole system.
  • Establishing new reporting requirements for the parole board including the responsibility to track and report on the percent of offenders receiving their parole hearing on time and the percent of offenders approved for parole at their hearings
  • Establishing a system allowing sentenced offenders held in jail to be transferred to parole even if they never entered a prison operated by the state Department of Corrections.
  • Instituting additional training, oversight, and guidelines for parole officers
  • Initiating the process of preparing for an offender’s hearing further ahead of that hearing’s date⁷¹

  PSPP estimated that this set of policies would cut both the percent of prisoners held beyond their presumptive parole date and the length for which those prisoners are held in half – to 25% and three months respectively – equivalent to a 78% reduction in the number of prison beds devoted to prisoners held past their parole dates.⁷² Overall, PSPP estimated that these policy changes would save 967 prison beds in 2011, which is 5.8% of the baseline forecasted prison population.⁷³ PSPP told us that by six months after the bill was passed, the state had progressed halfway to PSPP’s predictions, with 47% of inmates held for an average of 4.2 months beyond their parole date.⁷⁴ We do not have the expertise to determine whether PSPP’s assumption that Arkansas’ attempt to streamline the parole process will reduce the total amount of time prisoners are held past their parole hearing dates by 78%. The impact of the relevant policy changes is far from mechanical and could easily be reversed by a Governor unfriendly to reform. We would therefore guess that PSPP’s assumption is somewhat optimistic but based on the progress made after six months, it seems to be a good faith attempt and unlikely to be a major overestimate.

• Earned discharge from parole. This policy allows Arkansas’ Department of Community Corrections to count each day of parole a parolee serves without misbehavior as two days towards the parole that they owe. In effect, the policy halves parole time for well-behaved parolees.⁷⁵ If an offender commits a new felony while on parole, his/her parole can be revoked and s/he can serve his/her remaining parole time in prison as well as being prosecuted for the new crime as such.⁷⁶ Instituting earned discharge for parole aims to save prison beds by removing parolees from supervision more quickly so that they are less likely to have this additional time added to their sentences if they commit another felony.⁷⁷ Using the total number of parolees who had their parole revoked for committing a new crime during the second half of their parole in 2009 as a baseline, PSPP estimated that, in 2011, this policy change would save 396 beds.⁷⁸ 396 beds saved represents 2.4% of the baseline forecasted prison population in 2011.⁷⁹ We note that the prison beds saved due to this policy come from reducing the sentences of repeat offenders who may be more likely than others to commit future offenses. However, the calculation of this policy’s impact strikes us as fairly mechanical and it is hard for us to imagine it as being a large overestimate.⁸⁰
• Encouraging adherence to sentencing guidelines. PSPP forecast that Arkansas would save 674 beds in 2011 (4% of the forecasted prison population under baseline conditions) by encouraging judges to adhere to Arkansas’ sentencing guidelines.⁸¹ The bill encourages judges to sentence within the guidelines by a) merging the form where the sentence is recorded with a second form the judge must use to justify any deviation from the guidelines – so that any deviation from the guidelines is salient to the judge at the time the sentencing decision is made, and b) mandating the public reporting of which counties deviate from sentencing guidelines most frequently.⁸² PSPP assumed that this policy would have the effect of reducing the sentences for lower-level crimes to their total level in 2002 (a 20% reduction).⁸³ While we are highly uncertain about the extent to which judges are affected by this type of behavioral nudge, PSPP’s estimate, which is just 16.8% of the beds that would be saved by complete adherence to guidelines, strikes us as reasonable.⁸⁴

Overall, we do not have the subject matter expertise to come to a confident conclusion about whether PSPP’s assumptions about the effects of each policy change in Arkansas’ legislative package are reasonable. However, while flaws in PSPP’s model may exist, PSPP’s approach to forecasting the effects of Arkansas’ reform strikes us as reasonable and it appears to us that PSPP had thought of and addressed most of the concerns we could come up with. We believe there is a high risk of relying too heavily on an organization’s own forecasts of its impact, but our vet of PSPP’s forecasting process left us reasonably impressed. We thus would guess that PSPP’s forecasts are reasonable representations of the projected impacts of the reforms passed.

Comparing PSPP’s forecasts to actual post-reform prison population data

Looking in depth at post-reform events in Arkansas Informal evidence suggests that SB 750 had a substantial initial impact on Arkansas’ prison population. The bill was signed into law in March, 2011.⁸⁵ Between 2011 and 2012, Arkansas’ prison population fell by 8.9%, which is a slightly greater decline than predicted by PSPP.⁸⁶ However, this downtrend was reversed as the prison population rapidly increased by about 1,400 inmates in the three months from June to September 2013.⁸⁷ While we have not investigated this issue in depth, journalistic accounts and conversations with PSPP both suggested that the increased prison population was caused by policy changes made by the Board of Corrections in response to a murder allegedly committed by a parolee who was arrested and charged with felonies repeatedly without having his parole revoked.⁸⁸ These events somewhat reduce our confidence that gains achieved by PSPP will be sustained in the long run (although it is also possible that increased incarceration would have been even greater absent the reforms PSPP worked on). Some of the reforms PSPP assists rely heavily on buy-in from implementers and may face a heavy risk of reversal when a high profile crime occurs.

Post-reform data from additional states In this section, we compare the forecasts of post-reform prison populations to actual post-reform data to further hone our sense of the general reasonableness of PSPP’s forecasts. It is important to emphasize that actual data in line with the forecasts would only imply that PSPP’s impact estimates are accurate if the organization’s baseline forecasts of continued growth in the absence of reform were accurate, something we are not sure of, as discussed in a later section. We also interpret this data cautiously because some states’ reforms were quite recent, which means that their full effects may not yet be visible but also means it is too early to tell if those reforms will be sustained. PSPP provided us with both a) forecasts of post-reform prison populations and b) actual post-reform data for fifteen reforms. In addition, PSPP pointed us towards the Urban Institute’s publication of data from South Carolina.⁸⁹

Graphs of actual prison populations and projected prison populations (with and without reform)
Vertical lines represent year of reform.

For these fifteen reforms, the projections seem to have tracked well with later outcomes, albeit somewhat more likely to underestimate than overestimate the post-reform prison population.⁹⁰

Comparing pre-reform trends to post-reform data

Graphs of pre- and post-reform prison population data could reveal breaks in trend that would be evidence of impact. Again, vertical lines represent year of reform.⁹¹

We view this as a very back of the envelope test for impact because we do not have strong priors about the dynamics of reform (i.e. how quickly a successful bill would lead to a decrease in prison populations), nor do we know the exact effective date at which policy changes are phased in. We only have annual data for most states, and often have few years of post-reform data. With those caveats in mind, based on eyeballing these graphs, there appears to be a reasonable case that reform was followed by a meaningful change in trend in five or six states (Hawaii, Connecticut, North Carolina, Rhode Island, South Carolina, and perhaps Nevada).⁹² In four states (Georgia, Delaware, Oklahoma, and Missouri) it appears to be too soon to tell whether there was a break in trend post-reform. In the remaining fourteen states, we do not see a noticeable change after reform took place. The below table summarizes this data:

Taking a mean across all states with justice reinvestment bills, prison populations declined, on average, by 1.9% between December 31 of the year the bill passed and December 31, 2012. This result is difficult to interpret because it combines data from states less than a year out from reform with data from states reformed five years ago. Therefore, for each justice reinvestment state, we also report changes in prison population between December 31 of the year before reform, and December 31 of each following year. Taking a mean across states, we see that prison populations fell between 0.7% and 1.8% between the year before reform and 1-4 years afterwards.⁹³ Overall, the average size of absolute declines in prison populations following the passage of justice reinvestment bills are consistent with the approximately 1.7% average absolute decline forecast by PSPP. Our beliefs about the overall impact of these bills therefore depend heavily on the counterfactual of what potential growth was actually averted and what would have happened in the bills’ absence.

Comparing post-reform prison growth to data from states where PSPP did not provide assistance

In this section, we examine states PSPP did not assist to provide some context on whether prison population growth in those states PSPP did assist might have stagnated or reversed even if PSPP had not provided assistance. Five factors cause us to exercise particular caution during this analysis:

1. States that receive assistance from PSPP are selected, in part, based on their high need for reform (in terms of rising corrections populations and costs).₉₄ Therefore, even if states that did not receive assistance saw correctional populations decline, states that did receive assistance may have been particularly likely to experience continued growth in the absence of reform. We are somewhat skeptical of this factor’s importance, however, because prison populations seem to have been rising almost as quickly in states that did not receive PSPP’s assistance as in those states that did.⁹⁵
2. States that receive assistance from PSPP may differ from other states because they are required to demonstrate their commitment to the justice reinvestment process.⁹⁶ We would guess that this cuts in the opposite direction from the previous factor and suggests that the states PSPP worked in were more likely to pursue reform in PSPP’s absence than the typical state. On the other hand, PSPP often develops relationships with state leaders, aiming to raise awareness of research and other state reforms, and build the political support necessary to adopt comprehensive reforms.
3. States’ justice systems, criminal penalty laws, and political environments vary greatly and many of the factors that best explain variations in state prison population growth may be idiosyncratic shocks, so population declines in some states may not be highly predictive of population declines in others. For example, PSPP stated to us that California experienced a large, idiosyncratic decline in its prison population due to Brown v. Plata, a Supreme Court case ordering the state to address prison overcrowding, and New York experienced a large idiosyncratic decline due to an NYPD policy of pursuing fewer felony arrests in New York City. PSPP also has observed to us that counts of sentenced prisoners still were rising in 24 states as of 2012, and stated that with so much variety and so many idiosyncrasies, comparisons of trends can raise questions but not ultimately provide answers as to why prison populations have followed the patterns they have.⁹⁷
4. PSPP’s technical assistance in the states where it worked intensively –– and its other work, including organizing convenings, publishing research papers, general media outreach, and promoting nontraditional allies of reform –– may have affected the national dialogue on criminal justice issues and been responsible for some of the reduced prison growth in other states.⁹⁸ If PSPP’s efforts in states where it provided intensive assistance had positive spillover effects on other states by, for example, setting a positive example of reform, our comparison between states where PSPP provided intensive assistance and other states would yield a misleading picture of the impact of PSPP’s assistance. Specifically, spillover effects that were in fact magnifying PSPP’s impact could be misleadingly framed as evidence against PSPP’s impact. PSPP states: “There may be some national trends in policy making by governors and legislators, and in case-level decision making by individual judges, prosecutors, parole board members, probation officers and other system actors, that are influenced by an overall climate of reform and general attitudes toward crime and punishment. While Pew, JRI partners and others working in the field cannot claim anything near exclusive credit for creating an environment for reform, changing the national conversation from “tough vs. soft on crime” to “public safety return on investment” is a key goal of our broader work. This work has included hundreds of research publications, extensive media outreach, and scores of convenings with leading policy makers across the country. These national efforts “contaminate” the group of non-JRI states by exposing policy makers in those states to justice reinvestment data and strategies. Comparing them to the intensive assistance states effectively counts these broader national efforts against the impact of work in intensive JRI states. This is unfortunate since any policy change effort should seek to maximize change with available resources.”⁹⁹
5. PSPP provided limited assistance in some states that did not pass justice reinvestment bills, including California, Florida and Colorado, which may have contributed to declines in the prison population in those comparison states. PSPP stated to us that “Beyond these broad national impacts, Pew and its partners have provided limited assistance in some states, including California, Florida and Colorado. Counting these states among the non-JRI group further complicates comparison of JRI vs. non-JRI states. California, for instance, picked up the Performance Incentive Funding (PIF) concept from 2008 JRI work in Arizona and fashioned it with very light assistance from PSPP into SB 678 of 2009. In its first year of implementation, the California PIF program cut the number of probation revocations to state prison by nearly 6,200, nearly the identical number by which the state’s prison population dropped from 2009 to 2010.”¹⁰⁰

Keeping the above caveats in mind, below we compare the aggregate prison population in states PSPP intensively assisted to the aggregate prison population among all other states.

¹⁰¹

¹⁰²

¹⁰³

The reversal of rapid prison population growth in states unassisted by PSPP initially suggests to us that growth may have subsided absent PSPP’s intensive assistance. When growth in all states is weighted equally (to reduce the potential dominance of idiosyncratic factors in California and New York), the decline in prison population is less pronounced, but the pattern of population growth subsiding in 2007-2008 remains.104

¹⁰⁵

¹⁰⁶

As an additional way to compare post-reform changes in prison population to changes in other states, for each state, we report below a) the percent change in prison population each state experienced between December 31 of the year before that state was reformed and December 31, 2012; b) the change in prison population among states that had not received assistance from PSPP by the end of 2012 over the same time period; and c) the difference between (a) and (b). We do this exercise including and also excluding New York and California from our measure of states unassisted by PSPP. Thus, we look at whether, after reform, states’ prison populations decreased relative to the rest of the country.

Comparison between prison growth in JRI states and non-JRI states over similar time frames

On average, prison populations grow post-reform relative to states that did not receive assistance. When New York and California are excluded, prison populations decline by a mean of 1.1 percentage points relative to populations in the remaining states that did not receive assistance.

What might explain the decline in prison population growth in states that did not receive PSPP’s assistance?

We take the reversal of prison growth in states that did not go through the justice reinvestment process to suggest the possibility that in the states PSPP assisted a) reform might have occurred in PSPP’s absence or b) prison population growth might have reversed without explicit policy change. We have not investigated the possibility that there are causes of declining populations in non-justice reinvestment states that would have left behind the states PSPP assisted. We did, however, ask Mr. Gelb for alternative explanations for changing trends in non-justice-reinvestment states. He stated that: ¹⁰⁷

• As discussed above, idiosyncratic factors such as the Plata decision in California and NYPD policy in New York, systematic differences between states PSPP assisted and other states (i.e. selection effects), positive spillover effects from PSPP’s intensive assistance, and PSPP’s national research reports, conferences, partnerships, and other work all could have made contributions.
• Colorado passed a series of legislative reforms similar to the ones PSPP has assisted with but no other state passed a comprehensive criminal justice reform package without assistance from PSPP or the larger Justice Reinvestment Initiative co-funded by the Justice Department.¹⁰⁸
• Changes in non-justice reinvestment states might be attributable to executive, agency level, judicial, or prosecutorial policy changes. These changes may be more easily reversible than the legislative changes achieved by PSPP.
• Depending on how prison populations are counted, increasing jail populations may substitute for declining prison populations, so incarceration overall may not have decreased.
• PSPP’s technical assistance may have had positive spillover effects or PSPP’s research, public education, and support for nontraditional allies of reform may have led to population decreases in states where PSPP did not provide intensive assistance.
• A temporary response to the Great Recession. We would guess that this last possibility would not apply differentially to states that did not engage PSPP, so, were it to be true, it would probably not affect our overall conclusion.

Our bottom line on comparing states that engaged PSPP with states that did not do so

Comparing states that received intensive assistance from PSPP to states that did not lowered our confidence in the likely magnitude of the bills PSPP worked on. This comparison particularly called into question the idea that the “cost of doing nothing” would be as high as projected by PSPP and raised the possibility that potentially rising prison populations may have been averted by underlying dynamics (such as state fiscal problems, declining crime rates, increasing conservative support for reform, and policy diffusion among states) independent of PSPP. Overall, while this analysis raised questions, the data is compatible with the possibility that the bills PSPP worked on had a large impact on prison populations as well as the possibility that they averted little, if any, growth. If we pursue criminal justice reform in considerable depth, we plan to look into this issue further.

What was PSPP’s role in the bills it assisted?

In addition to vetting the effects of the bills PSPP assisted, we also investigated PSPP’s effect on those bills. In particular, we sought to learn a) whether PSPP was responsible for the passage of legislation that would otherwise not have occurred and b) whether PSPP improved the quality of legislation that otherwise would have passed in some form. To efficiently investigate these questions, we asked PSPP to refer us to policymakers who had worked closely with PSPP on justice reinvestment bills and could speak to that experience. They referred us to Grant Tennille, the Deputy Chief of Staff to Arkansas Governor Mike Beebe during the time of Arkansas’ reform, and Jay Neal, vice-chair of the Georgia State House of Representatives’ Public Safety Committee and co-sponsor of Georgia’s reform.¹⁰⁹ We recognize that PSPP would be unlikely to refer us to policymakers who felt the organization did not play a major role, so our decision to seek referrals in this way means we must interpret these interviews cautiously. Both interviewees did appear to have been key players in their respective bill’s passage and were valuable referrals. Mr. Tennille disclosed that PSPP retained Cranford Johnson Robinson Woods (CJRW) to support the state’s criminal justice reform effort through government and public relations work and that Mr. Tennille’s wife served as CJRW’s lead staff person^.110

PSPP’s role in Georgia according to Representative Neal

This section cites the views of Representative Neal according to our conversation notes.

Georgia’s Governor Deal was a strong proponent of criminal justice reform prior to engaging PSPP.¹¹¹ On the other hand, a conference sponsored by PSPP was directly responsible for Representative Neal’s interest in the issue.¹¹² Representative Neal went on to propose Georgia’s engagement of PSPP and to sponsor the reform bill.¹¹³ PSPP’s assistance and data analysis helped the state’s Special Council on Criminal Justice Reform identify drivers of the prison population, the crimes prisoners had been convicted of, and the lengths of prisoners’ sentences.¹¹⁴ For example, with PSPP’s assistance, the Council found that “the majority of people in prison in Georgia had committed drug or property crimes and were not violent or criminally minded.”¹¹⁵ This type of data helped to reverse Georgians’ attitude toward criminal justice.¹¹⁶ PSPP also helped to design and draft Georgia’s reform bill and conducted stakeholder outreach and education, for example by bringing organizations such as Right on Crime into the state.¹¹⁷ In addition, PSPP helped to address law enforcement’s concerns about the bill.¹¹⁸ It is not clear what would have happened to Georgia’s criminal justice system without PSPP’s involvement.¹¹⁹ Georgia’s Governor, Speaker, and Chief Justice probably would have championed reform in PSPP’s absence, but prior to Representative Neal’s attendance at PSPP’s conference, there was no champion for reform in the General Assembly.¹²⁰ Representative Neal had not heard anyone in the legislature mention the need for reform prior to PSPP’s involvement.¹²¹ Moreover, when the state later tried to reform its reentry system without PSPP’s assistance, the Criminal Justice Reform Council struggled to design a proposal and had to engage a consultant referred by PSPP.¹²² Overall, some sort of criminal justice reform would likely have passed in Georgia without PSPP’s involvement, but it would have relied less on data and experience from other states and may have been less strong.¹²³

PSPP’s role in Arkansas according to Grant Tennille

This section cites the views of Mr. Tennille according to our conversation notes.

Mr. Tennille first became involved in Arkansas’ criminal justice reform process after the bill was introduced, so he is most familiar with PSPP’s role from that point on.¹²⁴ Governor Beebe believed criminal justice reform was necessary in Arkansas before interacting directly with PSPP.¹²⁵ PSPP helped Arkansas’ working group prioritize among potential reforms by analyzing data to help the state understand what crimes most Arkansas prisoners had been convicted of and how different proposals would affect Arkansas’ prison population.¹²⁶ Once an initial set of reforms was chosen, PSPP wrote an outline of a bill to implement those reforms.¹²⁷ Over the course of legislative negotiations, PSPP helped to inform stakeholders about the bill, answered technical questions and provided advice about alternative proposals, and spent time in Arkansas meeting with legislators.¹²⁸ Prior to PSPP’s involvement, the main advocates for criminal justice reform in Arkansas were the state’s Black Caucus and allies in the Democratic Party, who did not have a concrete proposal on the table and were unlikely to pass anything stronger than the eventual legislation.¹²⁹ If PSPP had not been involved, a criminal justice reform bill probably would have been proposed but it would not have been based on data or an analysis of Arkansas’ criminal code; instead policy changes would have been made “fairly haphazardly without regard for how it would actually affect prison populations.”¹³⁰

Our bottom line on our conversations with policymakers and on PSPP’s role

While we are cautious not to place too much weight on the views of policymakers who worked closely with PSPP and were referred to us by the organization, we came away from these conversations believing that PSPP played a major role in these reforms.¹³¹ Both policymakers we spoke with strongly believed that their states would have struggled to pass high quality legislation without PSPP’s assistance. We do note, however, that the governor of each state was interested in reform prior to PSPP’s involvement. PSPP’s engagement may have been more important for improving the bills’ quality than for getting them on the agenda.

Bottom line on PSPP’s track record

Overall, the data are consistent with PSPP’s claim that by averting growth and causing slight declines in prison populations, the bills they worked on reduced prison populations (relative to the counterfactual) by an average of 11%. However, the data are also consistent with the possibility that prison growth would have leveled off even without these reforms and that the bills themselves saved a much smaller number of beds. After vetting PSPP’s overall forecasting process and many of the assumptions behind the forecasts in Arkansas, we were persuaded that PSPP makes a good faith and reasonable, if perhaps slightly optimistic, attempt to accurately forecast the effects of the policy changes made by these bills. Some of the important policy changes in Arkansas had fairly mechanical effects on the prison population. For example, much would have to go wrong for shortening the length of parole to fail to result in fewer revocations. Where the forecasts required more judgment, we felt that PSPP’s judgments were reasonable and had tried to incorporate most issues we, as non-experts, could think of. We also heavily weighed our interviews with two policymakers, both of whom strongly believed that an equally strong and data-driven bill could not have passed without PSPP’s assistance.

Cost effectiveness

Estimating the cost effectiveness of PSPP’s work in the states is challenging due to uncertainties in calculating both costs and benefits. We have explored PSPP’s track record (and discussed the difficulties of precisely estimating its impact) above. On the cost side of the equation, PSPP’s budget includes funding for national research, conferences and many other activities beyond direct assistance to states that likely contribute to the impact of PSPP’s state assistance. In addition, many of the JRI states received assistance from other providers, which receive funding from the federal government.

Below, we attempt an extremely rough calculation of the cost effectiveness of PSPP’s assistance to states as a back of the envelope check on whether funding PSPP would be reasonably cost effective if the bills PSPP assisted with had the effects PSPP claims. As above, our analysis attempts to focus on the benefits and costs of PSPP’s technical assistance to states and we do not attempt to quantify the cost effectiveness of its other work. We acknowledge that our ability to separately estimate the cost-effectiveness of PSPP’s state assistance is limited, because PSPP’s spending on other activities likely contributes to the success of this program.

PSPP’s budget is not public but Pew and the Bureau of Justice Assistance together have spent no more than $25 million on state assistance across all states that have participated in the policy development phase of the justice reinvestment process going back to Texas and Kansas in 2006.¹³² To be conservative, we use this $25 million figure as though it were our estimate of combined Pew/federal spending on state assistance.

PSPP assisted with 29 reforms that passed by the end of 2013 in 27 states. At its peak in 2010, the aggregate prison population among these states was 780,000. Assuming that the impact forecasts accurately predicted each bill’s impact on prison populations, we would expect the bills to lead to about an 11% decrease (the average forecasted reduction among states for which we received forecasts from PSPP was 11%). An 11% reduction in the prison population of these 27 states would lead to about 85,800 fewer prisoners at a given point in time.¹³³ If we assume that the gains from reforms last about ten years and we assume that these gains would not have occurred in PSPP’s absence, then PSPP’s state assistance would have led to about 858,000 fewer person-years in prison. ¹³⁴

If (as noted above) we assumed for the sake of the calculation that total spending to achieve these gains was $25 million, we would then calculate that Pew and BJA spent about $29 per “prison year averted,” again assuming none of the reforms would have happened without PSPP’s involvement. In addition, these averted prison years would have led to billions of dollars in savings to state budgets.

We strongly emphasize the difficulty of quantifying the impact of policy work and advise our readers to treat these calculations as a back of the envelope check on whether PSPP’s cost effectiveness is in the same ballpark as other opportunities and not as a literal estimate of PSPP’s impact. Based in part on these calculations, however, we do believe that PSPP’s past work is plausibly competitive with donations to our top charities, depending largely on how much of the impact of the reforms is (a) accurately estimated relative to the counterfactual and (b) attributable to PSPP.

PSPP’s plans for the next two years

PSPP has informed us that its primary objectives for the next two years include:

• Providing intensive technical assistance to additional states seeking to improve their adult or juvenile justice systems. In 2014, these states include Mississippi, Idaho, and Michigan.
• Cementing gains in states that have undergone reform. PSPP will do this by helping states establish oversight councils to hold agencies accountable for implementation and by helping states construct performance measures to track progress. All new states will receive this assistance as well as certain states PSPP has already worked in.
  • The oversight councils serve two key functions: 1) to receive the performance measurements and implementation updates from the agencies/entities most responsible for carrying out the new sentencing/corrections policies; vet that information; and hold the agencies accountable, and 2) to educate other public officials, the media and public regarding the progress on implementation and performance; suggest modifications to the policies; and advocate on behalf of resources needed to sustain improvements. The councils can be established formally through legislation or executive order, or less formally through Memoranda of Understanding (MOUs). PSPP intends to help states establish these councils and provide support in their initial meetings. Since many of these states are part of the Justice Reinvestment Initiative, PSPP will also facilitate the work of the “Phase II” (or implementation) assistance.
• Documenting the impact of reforms generated by PSPP. PSPP seeks to complete evaluations of at least four more policies the project has recommended in order to expand the evidentiary base for effective criminal justice policies.
• Producing additional research and publications. Studies planned over the coming years include a national report on the costs and consequences of allowing inmates to serve their full sentences and be released without community supervision and another report presenting multi-state juvenile recidivism rates and discussing best practices.
• Engaging and educating influential constituencies and nontraditional allies of reform. This includes supporting the expansion of Right on Crime and working with the National Sheriffs Association to highlight how the reform process includes sheriffs and local government representatives, and how local jails have benefitted from reform.
• Exploring potential reforms at the federal level. PSPP plans to explore whether there is an opening to make progress in the federal criminal justice system and also plans to advocate for an increase in appropriations to the Justice Reinvestment Initiative.
• Monitoring and evaluation. PSPP’s partners plan to ramp up policy evaluations of the Project’s work.¹³⁵

We do not know which, if any, of these activities would be unlikely to happen without GiveWell’s support.

The organization

We find it quite difficult to holistically asses an organization and the people that run it and we have limited observations to assess PSPP. PSPP’s Director, Adam Gelb, has communicated extremely well and directly with us, has had thoughtful answers to most of our questions, and has been quite open about sharing substantive information about PSPP. We first spoke with Mr. Gelb as part of our investigation of the cause of criminal justice reform and found that to be one of the most helpful conversations during that investigation. Both policymakers who worked on legislation with PSPP had extremely strong impressions of the organization, although we would (to some degree) expect referrals from PSPP to speak highly of them. PSPP is the organization we have seen whose approach to criminal justice issues – focusing prison beds on high level offenders, aiming to minimize the fiscal and humanitarian costs of the corrections system while maintaining or improving public safety, taking advantage of the unique opportunity to unite liberals and conservatives – matches the vision of the issues that we have identified as most promising. As we enter new fields in which we have little expertise, we believe it is valuable to collaborate with organizations that share our approach and seem strategic. PSPP’s self-evaluation is quite strong relative to that of other organizations we have seen, but leaves some questions unanswered, as discussed below. We hope to see PSPP’s self-evaluation improve over time and are eager to monitor the results of its development of new performance measures and its focus on the establishment of oversight councils in the states in which it works.

Open questions

Major unresolved issues include:

• The general difficulty of assessing the impact of an organization whose role is technical assistance and facilitating policy change rather than direct aid. As we previously wrote in our review of Deworm the World, “there are substantial potential advantages to supporting such an organization, as it may be able to have more impact per dollar by influencing government policy than by simply carrying out programs on its own, but this situation also complicates impact assessment.”
• The appropriate counterfactual to PSPP’s assistance given the decline in aggregate prison growth even among states where PSPP did not work. This is partially separable into four questions:
  1. Would a bill have been introduced without PSPP’s assistance?
  2. If so, would that bill have passed without PSPP’s assistance?
  3. If a reform bill would have passed without PSPP’s assistance, would the bill have been of lower quality?
  4. If a similar bill would not have passed without PSPP’s help, would some other policy change or non-policy change (such as a decrease in crime) have led to a decline in prison growth?
• The risk that PSPP’s work in future states will be less impactful than the work with which it established its track record because PSPP initially prioritized states in which policymakers were particularly committed to the reform process or in which forecasted prison growth was particularly high.¹³⁶
• The risk that policy gains achieved by PSPP won’t be sustained. In one state, Arkansas, we did see evidence that policy gains from a bill that PSPP worked on were reversed after a parolee whose parole was not revoked despite multiple felony arrests allegedly killed a man in Little Rock (more in the section on Arkansas above). PSPP has told us that over the next two years they plan to significantly expand their efforts to promote the sustainability of policy reforms (see above). It is possible that this is an inherent risk of PSPP’s model, which focuses on working directly with policymakers to achieve reforms in the near-term, but we accept this risk and believe it is outweighed by the model’s apparent effectiveness in achieving policy wins.
• The risk of unintentionally increasing the crime rate. We have not seen evidence of an adverse impact on the crime rate and would guess that PSPP’s work would not increase crime because of PSPP’s emphasis on improving community corrections, which may reduce recidivism, and because PSPP promotes policies that focus prison beds on high level offenders.¹³⁷ However, we have not investigated this issue in-depth.
• The marginal effect of additional funds to PSPP.
• PSPP’s capacity for monitoring and self-evaluation is still improving.¹³⁸ We would have more confidence in PSPP’s track record if such measures already existed and could be examined. PSPP only tracks the accuracy of some of the intermediate-level predictions that are inputs into the forecasts it uses as evidence of impact and, as far as we know, does not always evaluate the accuracy of these forecasts.¹³⁹ However, PSPP’s self-evaluation strikes us as reasonable given the large number of policy changes they have been involved with in a relatively short period of time. Also, PSPP’s plans going forward include helping states establish more rigorous measurement of outcomes, including population, costs, and recidivism.
• We have not closely looked into whether some of the reduction in prison population (either in states PSPP assisted or in other states) might have merely been reallocated into county jails. However, we would be surprised if this were a problem in the states where PSPP worked because of PSPP’s attention to reducing jail backlogs.¹⁴⁰
• We have not looked into whether PSPP’s reforms and the Project’s broader efforts to address and reshape the crime/incarceration debate (through national research and polling, partnership with other organizations, promoting nontraditional voices for reform, and other means) have had an impact on states where PSPP did not provide intensive technical assistance. Much of this work strikes us as impactful, but we find it difficult to rigorously evaluate.

Details of our collaboration with PSPP

Good Ventures will be making a grant of $3,000,000 over two years to PSPP. This grant would constitute less than 25% of PSPP’s biennial budget. This collaboration represents new territory for GiveWell. It is our first close collaboration with an organization whose mission is to facilitate policy change in the U.S. Because of the political nature of PSPP’s mission, communications are often a substantive and sensitive element of their work. For example, framing and communications are crucial parts of the policymaking process, so we believe that writing about legislative negotiations while they are in progress could influence those negotiations. We therefore may have to be somewhat more flexible with respect to our requests for transparency than we have in the past.¹⁴¹ In this context, we have chosen to make our agreement with respect to information sharing, transparency, and other aspects of our collaboration more formal than our agreements with past grantees and top charities. Our overarching goal is to publicly publish our overall opinions about PSPP’s progress and the impact of our grant to them, while respecting the political nature of PSPP’s work and, whenever possible, transparently telling our followers about any limitations on our ability to share confidential information or our opinions. A summary of the agreement’s main points follows:
GiveWell and Good Ventures entered into a grant agreement with The Pew Charitable Trusts in mid-2014 to support Pew’s Public Safety Performance Project. In connection with this relationship:

• GiveWell publishes commentary and updates evaluating Pew’s progress, all of which is shared with Pew in advance of posting. Substantive updates are limited to one in each of two grant periods, and one update after the end of the final grant period.
• If GiveWell or Good Ventures wishes to publish comments about a federal or state policy initiative where Pew is actively working toward legislation that has not passed or regulatory changes that are not finalized, they will only publish content that has been approved in advance by Pew.
• GiveWell and Good Ventures agree to respect the confidentiality of private information shared by Pew.
• GiveWell and Good Ventures provide updates to Pew on all forthcoming publications, commentary, and pending grants related to criminal justice and drug policy.
• Before the finalization of any pending grant whose purpose is to fund work (specifically advocacy work, a state policy initiative, or a ballot initiative) where Pew is actively working toward legislation that has not passed or regulatory changes that are not finalized, Pew will have an opportunity to provide comments to GiveWell and Good Ventures.
• The parties communicate regularly via phone, email, and, when possible, in person to discuss progress.
• The Agreement can be terminated prior to its expiration, unilaterally by any of the parties to it, in which case any uncommitted grant funds will be returned, and GiveWell and Good Ventures will refrain from further public commentary on the project.

This agreement was in place at the time of this page’s publication and – in accordance with our usual procedures – PSPP was able to review and privately comment on an outline of the page and on the page itself before it was published.

Sources