Note: Before the launch of the Open Philanthropy Project Blog, this post appeared on the GiveWell Blog. Uses of “we” and “our” in the below post may refer to the Open Philanthropy Project or to GiveWell as an organization. Additional comments may be available at the original post.
Earlier this year, we announced Chloe Cockburn as our incoming Program Officer for criminal justice reform. Chloe started her new role at the end of August.
This hire was the top priority we set in our March update on U.S. policy. It represents the first time we’ve hired someone for a senior, cause-specific role. Chloe will be the primary person responsible for recommending $5+ million a year of grants in this space. As such, hiring Chloe is one of the highest-stakes decisions we’ve made yet for the Open Philanthropy Project, certainly higher-stakes than any particular grant to date. As such, we are writing up a summary of our thinking (including reservations), and the process we ran for this job search.
We also see this blog post as a major part of the case for future grants we make in criminal justice reform. Part of the goal of this process was to hire a person with context, experience, and relationships that go well beyond what it would be realistic to put in a writeup. We expect that future criminal justice reform grants will be subject to a good deal of critical discussion, and accompanied by writeups; at the same time, for readers who want to fully understand the thinking behind our grants, it is important to note that our bigger-picture bet on Chloe’s judgment will be a major input into each grant recommendation in this area.
Note that Chloe reviewed this post.
Our hiring process
General principles of senior hiring
We’ve had little previous experience with senior hiring, and one of our first steps in the search was talking to people we know about how we should approach it. The general advice we got reflected similar principles to what we’ve picked up from our experiences with hiring more generally:
- The best way to evaluate someone is to work with them. The ideal way to make a senior hire would be to have a longstanding relationship with someone, perhaps as a part-time consulting arrangement.
- Interviews are highly unreliable. The evaluation process should be designed as much as possible to mimic working together.
- References are extremely important, and it can be useful to talk to many people about the candidate (including people who weren’t specifically offered as references).
We saw ourselves as being in a difficult position, given the high stakes of the hire and the fact that we would likely have to hire someone based mostly on an interview process rather than based on trial work and referrals (since we didn’t have very strong pre-existing relationships in the criminal justice reform field). We tried to design our process to hone in early on the qualities we were most confident would be important, and to generally mimic a trial of working together as closely as we could under the circumstances. This meant putting a lot of time up front into defining the role, the associated duties, and the most important qualities for it.
Preparatory work
One of the most important parts of our process was coming up with the job posting, which lays out our goals in the space, example activities, and job duties and desired qualities for the role. We tried to identify the qualities that would be most important for both (a) strong grantmaking, informed by good connections, good knowledge of the field and good judgment; (b) a strong working relationship, in which we could communicate effectively with each other, understand each others’ thinking, and build trust over time. We solicited input on the posting from our contacts in the criminal justice reform space before disseminating it widely. (Also see our previous commentary on what general qualities make a good fit for this type of role, and on what others have written on the matter.)
We had previously done an in-depth “landscape” of the field of criminal justice reform, involving dozens of conversations aiming to get a broad sense of the major players, major approaches, and major opportunities in the field. (The public version of this landscape is available as pages 2-11 of this document (DOCX).) We ended up finding this landscape extremely helpful throughout our search, as discussed below. We had not done it specifically in order to prepare for hiring, but having been through this process once, we now hope to have a similar landscape for any cause in which we are aiming to make a similar hire.
Finding and interviewing candidates
Based on our landscape, we had a sense of who the major funders and some other well-connected people/organizations in the field were, and we sent them our job posting (and in some cases asked for phone calls to discuss and get referrals). We put special weight on the recommendations of people we perceived as aligned with our goals and as generally well-regarded. We initially expected that our strongest candidates would overwhelmingly come via referral, but Chloe submitted her resume via email (though she was referred later, after she had progressed fairly far in our process).
By necessity, the early parts of our process revolved around interviews. Our other ideas for evaluating candidates (below) seemed too time-consuming and high-stakes (for both us and the candidate) to move forward with before we’d met the candidate and had in-depth conversations. We tried to make the interviews as similar as possible to the conversations we might have early in the candidate’s employment. Most of our interviews consisted of discussing the questions, “What do you think of the major organizations and funders in this space?” and “What is your best guess as to what you will end up funding, and what are your major uncertainties?” For stronger candidates, we spent several hours (spread out over multiple sessions) on these questions, trying to get as concrete as possible about what the candidate would do and fund as our Program Officer, and how they would answer unresolved questions.
We were primarily looking for someone who would come in with a lot of pre-existing knowledge and relationships and would be productive quickly, so we felt it was reasonable to ask for a fair amount of detail on their best-guess ideas for grantmaking, while realizing that their vision would likely change significantly on the job.
In these conversations, a common pattern we saw was that a candidate would have a concrete plan for funding one broad kind of work (for example, ballot initiatives, alternative metrics for prosecutors, or research on alternatives to incarceration) but would have relatively little to say about other broad kinds of work. This was where we found the work that had gone into our landscape document particularly useful. When a candidate didn’t mention a major aspect of the criminal justice reform field, we would ask about it and see whether they were omitting it because they (a) had strong knowledge of it and were making a considered decision to de-prioritize this aspect of the field; (b) didn’t have much experience or knowledge of this area of the field.
We sought candidates with breadth: knowledge and familiarity with multiple different approaches to criminal justice reform, such that they could make well-informed and deeply considered tradeoffs regarding what aspects to invest most heavily in. Ultimately, we felt reasonably confident in our reads on different candidates’ breadth, and felt this was probably the most useful thing we learned from interviews. Chloe was one of the strongest candidates on this dimension.
Other things we looked for in interviews:
- We looked for candidates who seemed aligned with our goals regarding criminal justice reform. We place high importance on both reducing incarceration and on maintaining or improving public safety. We asked candidates what their ideal criminal justice system would look like, and what reforms they thought were most worth pushing for.
- We looked for candidates we could communicate effectively with. We generally preferred people who seemed direct and who were consistently able to give reasons for their views that we found logical.
In addition to interviews, we asked candidates for examples of their past work product. In particular, we asked for examples of something they had written laying out the case for a challenging decision. We found these submissions useful, and perhaps more reflective of candidates’ likely professional output than interviews could be. We particularly found them useful for giving us a sense of whether a candidate had experience producing the kind of arguments we would find easy to follow and evaluate. Chloe was a clear standout on this dimension: she sent us memos that were analytical, information-dense, and clearly laid out the reasoning and confidence behind each major claim. We found her work product educational about topics we had already had many discussions about and were interested in. We felt optimistic about our ability to work productively and communicate effectively with her based on this.
Final stages of our process
Chloe was a clear standout based on the early stages of our process, particularly with respect to breadth and communication style. Her application was time-sensitive due to other job offers. Accordingly, she was the only candidate to go through the final stages of our process.
References. We asked Chloe for references: we asked for the people who had worked with her most closely, including both supervising her and being supervised by her, and were best positioned to evaluate her work. Many of the people she referred us to were major figures in the field of criminal justice reform, people we might have reached out to separately had Chloe not cited them as references. We talked to eight people in all, including people who had managed, reported to, and worked alongside her as well as people who represented particularly important funders or other organizations in the field. References were very important to us, partly because we feel that strong interpersonal skills and a strong network are directly relevant to a program officer’s ability to evaluate giving opportunities. We also used this part of the process to get a better sense of Chloe’s background and how well it might prepare her for this role.
Trial work. We set up a “miniature trial” in which we worked with Chloe for ten total hours over the course of about a week (we paid her by the hour for this work). We struggled to come up with a project representative of work we might do together, since the primary role of a Program Officer is representing us to potential grantees and evaluating giving opportunities, which would be inappropriate for this circumstance. We settled on the following:
- We asked Chloe to think about what she could do, subject to the time and logistical constraints of the “mini trial,” to update her working views on funding priorities as much as possible (in the sense of high likelihood of learning important new information and changing her mind per hour spent). She picked a couple of possible funding priorities that she knew relatively little about but had strong contacts who would know more about; spoke to her contacts; and discussed (via email and phone) what she was learning with me.
- We sent Chloe a paper attempting to estimate the costs and benefits of different incarceration policies in a rigorous way, and coming to conclusions that differed fairly significantly from Chloe’s views (in particular, the paper argues that lengthening sentences for a fairly broad class of crimes could be net beneficial). We asked her to review the paper and discuss her thoughts on it with us. This was less of a pure “work trial” than the other assignment, and ultimately we found it less informative. The goal was to get a feel for how Chloe might engage with the quantitative arguments we often make here, and to probe the degree to which our policy goals were aligned; we were not expecting her to engage with the paper’s econometrics in detail, as we do not see this skill as core to the Program Officer role. Much of what we were looking for, instead, was Chloe’s ability to explain the basis for her opinions and key points of uncertainty.
We came away from these final stages feeling that:
- We had had a substantial amount of interaction with Chloe, between the interviews, “mini-trial” and logistical discussions. We felt that we were able to work together well and we were impressed by the work she accomplished in that short time. We felt that we were able to communicate easily with each other, and that we tended to be clear about the reasoning behind her thinking.
- Chloe is extremely well-connected in the field of criminal justice reform, and respected by many people who have very different perspectives on criminal justice reform.
Our bet on Chloe
Note: this section was edited for clarity on June 14, 2016.
Chloe’s role involves a great deal of autonomy. She has context, experience and relationships that greatly exceed ours, and we expect to largely defer to her on many decisions about which giving opportunities to investigate and prioritize. As a result, any grant recommendation in this space is at least partly (and often largely) a bet on Chloe, and can’t be fully understood without that context.
Because of Chloe’s central role in determining our recommendations in this space, and because this is one of the highest-stakes decisions we’ve made to date, we chose to write up a summary of both the major strengths she brings to this role and our reservations. We will not necessarily repeat this for future hires, but having an example will hopefully be helpful to people seeking to understand our approach to this important part of grantmaking. Chloe reviewed this before we published it.
We think the three most important strengths Chloe brings to this role are:
- Breadth: as noted above, Chloe stood out in our process for her pre-existing knowledge and thoughtfulness about many different facets of the criminal justice reform space. We hope that this will position her to see many different types of good giving opportunities, and thoughtfully compare them with each other.
- Relationships: we believe that Chloe has a large number of strong relationships in the field. These will hopefully help her to get thoughtful ideas for how to give well and get honest input on the strengths and weaknesses of different organizations. It also, we believe, reflects that she has generally strong interpersonal skills, which will be important for reasons outlined previously.
- Compatibility as a co-worker and communicator: we expect that we will find it easy to work with Chloe and to understand and evaluate her arguments, which will allow us to build trust over time.
We are fairly confident that all three of these are major strengths of Chloe’s, and we feel that each is among the most important qualities for this role. Between them, we feel that there are good odds that Chloe will have the experience, context, and information to make good judgments and convince us of the merits of her judgments. We also feel that Chloe’s goals in the space are fairly well aligned with ours.
Our biggest reservations are, in order of importance:
- We don’t have much evidence about Chloe’s political savvy and strategic judgment. Her previous role was serving as the Advocacy and Policy Counsel for the ACLU’s Campaign to End Mass Incarceration, heading up the ACLU’s national office support to state-level ACLU affiliates, and she was an attorney prior to that. We have little sense of how effective the ACLU in general, or Chloe in particular, has been at working toward reducing incarceration. We did try to get at this question while taking references, but overall we had the sense that the impact of ACLU’s work was very hard to assess. More broadly, based on the many informal conversations we’ve had in landscaping the field, the ACLU seems to have a mixed (rather than strong) reputation regarding the effectiveness of its advocacy on criminal justice reform.
- We believe our goals are fairly well aligned with Chloe’s, but we aren’t highly confident in the degree of alignment. David Roodman is currently conducting an in-depth literature review on the societal costs and benefits of incarceration. We think it’s possible that, after both we and Chloe review his work, we will believe in a stronger relationship between incarceration and public safety than Chloe believes in.
That said, we think it’s very unlikely that we will come out very far from Chloe; we are fairly confident that the status quo is much too far in the direction of excessive incarceration. - Chloe has a generally different profile from the sorts of people GiveWell has hired in the past. In particular, she is probably less quantitatively inclined than most employees at GiveWell. This isn’t surprising or concerning – most GiveWell employees are Research Analysts, and we see the Program Officer role as calling for a different set of abilities. That said, it’s possible that different reasoning styles will lead to disagreement at times. We think of this as only a minor concern.
More broadly, this kind of role is very new for us, and we believe our ability to define it in advance and evaluate candidates based on our interview process is limited.
Finally, it’s worth noting a couple of things that were not major criteria for us:
- We didn’t look to hire someone with a particular strategy and priorities. We are less knowledgeable about criminal justice reform than most of the people we interviewed, so we think there would have been limited value in our assessments of how promising each person’s strategy is. Rather, we looked primarily for someone who we feel is well-positioned to develop a good strategy – hence the importance of connections and breadth. In other words, we looked for someone who can thoughtfully and knowledgeably compare different approaches to each other, rather than someone whose preferred approaches match a pre-defined strategy on our end.
- Background was a relatively minor consideration. We wouldn’t have hired someone who didn’t have significant experience with the field of criminal justice reform, but most people we interviewed fit that description. The main reason we value experience is because we feel it is necessary to have the key strengths we list above (particularly connections and breadth).
Our plan from here
Chloe started working with us at the end of August. Her role is, broadly, to find and make the case for the strongest possible giving opportunities in the space, and she will have autonomy with respect to the process for finding and prioritizing giving opportunities.
Early on, Alexander Berger and I expect to work closely with her, asking many questions and having many critical discussions about the funding areas and grants she proposes prioritizing. That said, we don’t expect to understand the full case for her proposals, and we will see our role more as “spot-checking reasoning” than as “verifying every aspect of the case.” Over time, we hope to build trust and reduce the intensity of (while never completely eliminating) our critical questions. Ultimately, we hope that our grants in the space of criminal justice reform will be less and less about our view of the details, and more and more about the bet we’re making on Chloe.