US Cochrane Center (USCC) Gets Our First “Quick Grant” Recommendation

Note: Before the launch of the Open Philanthropy Project Blog, this post appeared on the GiveWell Blog. Uses of “we” and “our” in the below post may refer to the Open Philanthropy Project or to GiveWell as an organization. Additional comments may be available at the original post.

Recently, we did something that may strike many GiveWell followers as out of character. We recommended a $100,000 grant to the US Cochrane Center, despite the fact that we have done relatively little investigation of it so far (compared with our investigations of current top charities)—and have many unanswered questions. Good Ventures, which helped with our investigation and therefore followed it closely, was a part of the conversation in which we came to the conclusion that this grant represented a good giving opportunity, and it committed the funds shortly afterward (before we had finalized our writeup; we considered this appropriate since, as we discuss below, speed was desirable in this situation.*)

This post covers two topics:

• Why we believe it is important to be able to make quick grants (i.e., grants with far less than our usual level of investigation) when warranted, and we are working on principles for doing so.
• Why we believe that the grant discussed in this post meets our working criteria for a quick grant.

In brief:

• We believe that in certain cases, speed is valuable in grantmaking: sometimes because circumstances demand action by a certain date (for example, some projects involve close coordination between multiple entities, including governments, and may need to happen on timelines that work for these entities), and sometimes because speed in giving can help organizations make better planning decisions.
• We have developed working principles for when a “quick grant” is called for, which are discussed in this post. We intend to experiment with “quick grants” while ensuring that they represent only a small portion of our money moved in the near future.
• We see the USCC as an ideal recipient for our first “quick grant” partly because we are continuing to investigate the USCC as a potential top charity (in which case we would recommend it to individual donors). The grant discussed in this post meets our working criteria for “quick grants,” and in addition, we are confident that we will learn more over time about the extent to which this “quick grant” was warranted.

Note that the bulk of this post was written in July, following the grant recommendation and grant. We held publication of the post, after drafting it, for a variety of reasons including (a) the desire to get and respond to feedback from all parties discussed in this post (as we generally do); (b) the desire to publish more public communications about our evolution as an organization in order to give more context on how this recommendation fits in.

The importance of making “quick grant” recommendations

To date, our usual approach to making recommendations has been:

• Survey an entire field, looking for the organizations that perform best according to various heuristics.
• Deeply investigate the top-contender organizations, making a major effort to answer all major questions to the degree that we reasonably can.
• Set a deadline (usually giving season) by which we must make or refresh our recommendations; this ensures that we avoid perfectionism and make the best recommendations we can with the information we have.

We’ve put an increasing amount of effort into investigating contenders for our #1 ranking; we’ve found it very important that we be able to fully stand behind any such recommendation, with reasonable answers to any question that one might raise.

There are a lot of positive things about this approach, and we intend to retain it for the recommendations that comprise the bulk of our money moved. We believe that being systematic, deliberate and thorough is likely to lead to much better giving, in general, than relying on happenstance and intuition alone; at the same time, having regular deadlines leads to regular giving (which we favor).

That said, we also believe that in certain cases, speed is valuable. Sometimes speed is valuable because circumstances demand action by a certain date (for example, some projects involve close coordination between multiple entities, including governments, and may need to happen on timelines that work for these entities). Sometimes the argument for speed is more subtle – for example, giving quickly can help organizations make better decisions about how to plan their budgets as well as how much time to invest in fundraising from different sources.

We’ve long been intrigued by the ideas of people like Bill Somerville, and wondered whether the approach of taking calculated risks after a shorter period of research than has been our norm has something to recommend it at times. Since connecting with Good Ventures, this issue has become more salient to us for a couple of reasons:

• We were the recipient of one such “quick grant.”After our first interaction with Cari and Dustin (a 90-minute meeting between them and myself in February 2011), they expressed an intention to contribute $100,000 over two years in operating support to GiveWell. At that time, we were facing a substantial projected deficit and were investing more time than usual in fundraising. This “quick grant” was extremely helpful for our planning and efficiency, in a way that it couldn’t have been if it had required the kind of investigation that GiveWell normally puts in. Noticing this further highlighted the advantages of “quick grants.”
• In addition, our relationship with Good Ventures means that we are able to make “quick grant” recommendations that can be taken up quickly; while we have some other connections who provide similar opportunities, the bulk of our other money moved tends to be concentrated in the month of December.

On May 8, two GiveWell representatives (Stephanie Wykstra and I) visited the US Cochrane Center (the USCC) (note that we have posted notes from the visit). For reasons discussed below, we came away with the impression that (a) the USCC plays an important role in meta-research; (b) the USCC faced a drastic shortage of operating funding, even to the point where it might not be able to continue uninterrupted, minimal staff support. Over the next six weeks (as discussed below), we tried to find persuasive counterarguments to this viewpoint and failed to do so. While I still had many unanswered questions, it struck me that a grant to the USCC, made sooner rather than later, could potentially do an enormous amount of good in terms of helping the USCC plan intelligently; the mere fact that I wasn’t confident in the value of such a grant didn’t seem to change the fact that the expected value of making such a grant quickly seemed high.

While principles such as thoroughness and transparency are important, the principle of doing as much good as possible with one’s giving is more core to GiveWell than any other. We strongly believe that we should never find ourselves passing up what we see as an opportunity to do maximal good, just because the opportunity is an awkward fit with our existing habits and processes. We are constantly watching the actions of other funders and asking whether they are finding great opportunities to do good, in a way that our existing approach would fail to (and if so, what we can do about it).

Thus, once we saw the potential value of a “quick grant” to the USCC, we started actively considering the idea of modifying our approach to facilitate such a grant (rather than passing on the grant because it didn’t fit well with our existing approach). At the same time, we wanted to make sure that we were adopting and discussing a principled modification to our general approach, not simply recommending a “quick grant” on a whim. As such, we worked to develop the best set of general principles we could for when a “quick grant” is called for. These principles were inspired by the case of the USCC, but we took our best shot at making them reasonable for general application.

Principles and process for making “quick grant” recommendations

This section discusses (a) the key questions we feel are appropriate for a potential “quick grant”; (b) the process we intend to follow for answering these questions quickly and efficiently and making “quick grants.”

Key questions for a potential “quick grant”

1. Is there a reason that speed, in and of itself, is valuable for this grant? Some possible reasons that speed may be valuable:
   • There may be a specific reason that funds are needed by a particular date in order to go forward with a particular project.
   • There is always a potential argument that (as outlined above) an early commitment can help an organization plan better and use its time more efficiently. This argument will generally be stronger in the case when (a) a relatively small grant can make a big difference to an organization’s planning (usually because the organization has little in the way of unrestricted support), and (b) we have a positive view of the organization and its people overall (as opposed to a small subset of the work it does).
   • An early grant can also be helpful for learning purposes. If an organization is highly promising from our perspective, but hesitant to engage in our process because of how it perceives the costs and benefits of doing so, an early grant may be valuable as a way to improve our access and ability to learn.
2. Is the organization/project in question focused on work that seems valuable, reasonably cost-effective, suited to philanthropy (as opposed to other approaches) and thus “worth doing” overall? For the purposes of a “quick grant,” the approach to this question will be highly intuitive and perhaps unsatisfying compared to the cost-effectiveness analysis we often do. Explicit cost-effectiveness analysis is very difficult to do with reasonable speed, without sacrificing reasonable robustness, and when comparing radically different approaches to doing good it can be nearly impossible to say with much confidence how they compare in terms of “bang for the buck.” This question acts only as a basic screen: is the organization/project in question filling a valuable role in an important ecosystem, leveraging the work of others, and overall taking on an approach to helping people that is at least defensible from a cost-effectiveness standpoint?
3. Do we see a convincing reason that this organization would not be able to raise the funding it needs in the relevant time frame, even if it made a good case for such funding? As discussed below, one of the things that excites us about the USCC as a funding opportunity is the sense that the USCC is “structurally underfunded”: it struggles to raise funds not because major funders have thoughtfully considered and rejected its case, but because major funders largely have program areas and issue focuses that don’t leave room for the kind of work the USCC does.This is not often the case. We often find ourselves saying, “It seems that if this work were as valuable as the organization claims, funder X would support it.” When such a statement can be convincingly refuted, the case for a “quick grant” (as for a recommendation in general) becomes much stronger.
4. Are the people involved impressive, competent, and capable of making good on-the-fly decisions, such that we’re comfortable with grants that we have fairly little visibility into the specific intended use of? We’ll be writing more in the future about how we evaluate people. This question is important for all recommendations but is particularly important for “quick grants,” since it generally takes us a long time to feel confident about the specifics of how funds will be used; we are much more likely to recommend “quick grants” when these specifics aren’t necessary in order to feel the funds will be used well.
5. How much will we learn in the future about the organization/project and the extent to which the “quick grant” was a good idea? Since “quick grants” are likely to be smaller than the money we moved to our top charities, it isn’t a given that we will engage in the same sort of followup on them that we do for our top charities. In addition, we’ve found that it’s much easier to learn about an organization when we invest up front in defining goals, metrics, etc., which is more difficult to do in the case of a “quick grant.” So by default, there is a risk that we won’t learn much from or about a “quick grant”; we need to be attentive to this and have a strong preference for grants with more learning potential.We expect that there will often be cases in which we consider “quick grants” to organizations that we also plan to evaluate more thoroughly (with the possibility of moving significantly more money to them). In these situations, the situation looks much better in terms of learning about whether our grant was a good one and what ultimately came of it. The USCC is one of these cases.

Our process for making “quick grants”

We’ve provisionally agreed to the following process for making quick grants:

• The process starts when we have some unusually strong signs that the answers to the above key questions are positive. We do not need a definitive case; the case may be largely intuitive and suggestive, but it should be unusually strong in the scope of opportunities we come across.
• We then pick any “low-hanging fruit” in terms of further investigating the answers to our questions – any investigative work that can be done quickly and is likely to lead to a substantially better understanding of the situation.While all of the above key questions are important, we are likely to have a fairly quick and intuitive read on questions #2, #4, and #5, and the questions that we are most likely to focus on investigating are #1 (is there a reason that speed is likely to be helpful?) and particularly #3 (are there other funders who are a logical fit to fund this organization/project or is it underfunded for structural reasons?)
• We actively seek out counterarguments to our views on key questions, as effectively and efficiently as we can. The main approach we used for the USCC – an approach we are likely to use for future “quick grants” as well – is to seek out conversations with funders who seem like the closest logical fit for the funding opportunity, and try to understand whether they are (a) planning to fund the project/organization; (b) planning not to fund the project/organization, for reasons we find compelling; (c) planning not to fund the project/organization, for reasons we don’t find compelling. (A “quick grant” should be made when (c) holds, not when (a) or (b) holds.)
• Before any “quick grant,” we hold a meeting or conference call that combines (a) all staff who have been highly involved with the investigation; (b) some staff who haven’t; (c) funders who are particularly likely to follow the recommendation for a “quick grant.” The staff recommending the “quick grant” summarize the answers to key questions as well as what investigations have been done to learn more about these questions and to identify counterarguments. Others on the call focus on (a) evaluating the strength of the arguments (answers to key questions) given the information already available; (b) determining whether there is other information that would be likely to quickly and substantially shift answers to the key questions (i.e., whether there is “low-hanging fruit” in investigative terms).
• If the basic case for a “quick grant” is accepted, the next step is to determine the size of the grant recommendation (i.e., the dollar amount past which we would stop recommending a “quick grant”). When the “quick grant” is to meet a specific need or fund a specific project, we should understand the nature of the time sensitivity and the size of the need, and the current funding status, before recommending the award. When the “quick grant” is more along the lines of general support for purposes of helping the organization plan and/or improving our access to the organization, we should pay more attention to the size and variance of the organization’s budget (as well as any available room for more funding analysis) and try to aim for something that provides substantial benefit (in terms of planning and/or access) but doesn’t come close to meeting all the organization’s needs.
• When we do recommend a “quick grant,” we publicly write up the recommended grant amount, recipient, and a summary of our process and reasoning in making the recommendation.

How we decided to recommend a “quick grant” to the USCC

We’ve long been familiar with the work of the Cochrane Collaboration, having used it in our research. We’ve noted before that

We have found that its reports generally review a large number of studies and are very clear about the findings, strengths and weaknesses of these studies. For health programs, when there are often many high-quality studies available, we therefore use Cochrane as our main source of information on “micro” evidence when possible.

In April of this year, I attended a meeting on preregistration in development economics and encountered Kay Dickersin of the USCC, who stated to me that (a) the USCC is struggling to attract unrestricted support; (b) if the USCC had sufficient funds, it would provide general support to US-based Cochrane entities, including direct financial grants in cases where these entities appeared underfunded. We scheduled a full-day visit to the USCC in Baltimore to learn more, because (a) we have long respected the Cochrane Collaboration’s work and were surprised to hear that the USCC was struggling to attract unrestricted funding; (b) this was the first concrete giving opportunity we’d encountered in the area of meta-research, which is a new high-priority focus area for us and which we’re seeking to learn more about; (c) the USCC showed a high level of interest in engaging with us and offered to put together a full-day meeting with multiple representatives, which both raised our expectations about what how much we could learn and served as an additional signal that the USCC was struggling to attract sufficient operating funding.

Our notes from the full-day meeting are published online (DOC). We came away from the meeting feeling there was a strong preliminary case for the USCC based on the five key questions above (though we had not yet formalized these questions as the key ones for “quick grants”):

• Is there a reason that speed, in and of itself, is valuable for this grant? The USCC appeared to have a concrete and time-sensitive need for unrestricted funding, including the immediate need for funds to continue uninterrupted, minimal, staff support. In our view, this situation is notable not just because of the specific consequences that a grant might have (allowing the USCC to retain core staff), but also because it more broadly illustrates that the USCC does not have a stable situation in terms of unrestricted funds, and thus that support could help it to plan and set priorities more effectively.
• Is the organization/project in question focused on work that seems valuable, reasonably cost-effective, suited to philanthropy and thus “worth doing” overall? We are positive on the quality of the Cochrane Collaboration’s work, as discussed above; at the meeting we also came away with preliminary reasons to believe the work is influential as well (though we plan to investigate this more). As for the USCC’s role in the Cochrane Collaboration, we saw fairly strong arguments on this point. The Cochrane Collaboration relies on training and supporting volunteers, many of whom are academics. The U.S. has many potential volunteers, including those based within the country’s large university system. But in the U.S. there is far less funding for Cochrane infrastructure (i.e., to train and support volunteers) than in other English-speaking countries such as the UK, Canada and Australia. Jeremy Grimshaw, co-chair of the Cochrane Collaboration’s International Steering Group and Director of Canada’s Cochrane Center, was present by phone at the meeting and supported the message being sent that the USCC is a point of particularly high leverage and importance for the Cochrane Collaboration as a whole. For the reasons discussed above, we don’t feel that formal cost-effectiveness analysis is likely to be helpful in this case.
• Do we see a convincing reason that this organization would not be able to raise the funding it needs in the relevant time frame, even if it made a good case for such funding? We questioned the USCC about many potential sources of funding and were told that the lack of funding was largely for structural, not substantive reasons. That is, the funding needed would support the infrastructure required for the Cochrane Collaboration’s work, such as staff for training, and methodological support for reviews, and not hypothesis-testing research. As such, the Collaboration’s needs do not fit into the pre-defined categories and issue areas of major funders. Thus, potential funders have considered and declined Cochrane requests and applications for general operating support based on “not a good fit” rather than the quality or importance of the work, overall. This was the point we felt we most needed to examine further after the meeting, and we did so, as discussed below.
• Are the people involved impressive, competent, and capable of making good on-the-fly decisions, such that we’re comfortable with grants that we have fairly little visibility into the specific intended use of? Multiple representatives were present, and overall we felt they answered our questions reasonably clearly and well; the USCC also appears comfortable with transparency, having signed off quickly and permissively on our notes from the meeting. Our general positive impression of the Cochrane Collaboration’s work is also relevant here. We currently have moderate confidence on this point; we anticipate learning more as we investigate Cochrane further.
• How much will we learn in the future about the organization/project and the extent to which the “quick grant” was a good idea? We are currently performing an in-depth investigation of the USCC, considering recommending it for more funding than the initial “quick grant,” so we believe that we will learn a great deal about the extent to which this “quick grant” was warranted.

After the meeting, we agreed that the USCC was a promising organization, and our top priority became looking efficiently for counterarguments to the case for funding it. With Good Ventures’s help, we sought out conversations with the major funders that seemed to us like potential fits for the USCC, based both on our prior knowledge and from conversations with the USCC, hoping that we would gain more context on (a) whether it’s true that the USCC doesn’t fit into the issue areas of existing major funders; (b) whether there were general counterarguments to our preliminary views on the USCC’s value and need for more funds.

Feedback was solicited from:

• Representatives of the Gates and Hewlett Foundations (we had no reason to believe they were a fit for the USCC, but thought they might know who would be, and see them as relatively impact-oriented funders in general; we are not cleared to share notes about these interactions).
• A representative from the Wellcome Trust, a large medical research funder (notes available as DOC)
• Representatives from U.S. government agencies: the National Institute of Child Health and Human Development (which contracts with one of the U.S.-based Cochrane review groups but does not provide unrestricted support to the USCC), the NIH Office of Medical Applications of Research in the Office of the Director (which we were pointed to in order to explore whether the USCC might be a fit for funding from the Office of the Director), and the Agency for Healthcare Research and Quality (which has funded the USCC in the past and, like the Cochrane Collaboration, commissions systematic reviews). We are not cleared to share our notes from these interactions.
• A representative from the HIV/AIDS department of the World Health Organization. We are not cleared to share our notes from this interaction.

In these interactions, we asked for general impressions of the Cochrane Collaboration, thoughts on what sorts of funders might be structurally able to support the USCC, thoughts on what other groups do the sort of work that the Cochrane Collaboration does, and (when relevant) reasoning behind an entity’s support (or lack thereof) for USCC. We came away with the impression that the Cochrane Collaboration’s work is widely respected and seen as high-quality and important, that we can’t easily identify any major funders that are a structural fit for the USCC, and that the main other group focused on systematic reviews is AHRQ, which we plan to investigate further. (AHRQ’s role relative to Cochrane’s is discussed in the notes from our visit to the USCC; our takeaways from other conversations were broadly consistent with these notes.)

We also spoke to

• Jeremy Grimshaw, co-chair of the Cochrane Collaboration’s International Steering Group and Director of Canada’s Cochrane Center. We sought to press the question of whether the USCC, specifically, is the best entity to fund in order to supporting the overall mission of the Cochrane Collaboration. Dr. Grimshaw conferred with other international Cochrane Collaboration representatives, including the other Steering Group co-chair, the interim Executive Director and the Editor-in-Chief of the Cochrane Library, and informed us that they endorsed supporting the USCC and would seek the formal endorsement of The Cochrane Collaboration Steering Group. Since then (following the grant), we have further pressed the issue of whether there might be other opportunities to support the Cochrane Collaboration that are higher-leverage than the USCC, and we are continuing to speak with Dr. Grimshaw about how best to work with international representatives to investigate this question. We therefore intend to investigate other Cochrane entities as well, and believe that doing so will take a significant amount of time, so given that the USCC had been endorsed as a strong opportunity in terms of potential leverage for a donation at the time of the grant recommendation (though no single opportunity within the Cochrane Collaboration was put forth as the “best”), we feel it was the right decision to move forward.
• John Ioannidis, whom we see as a leading figure in the field of meta-research generally (and meta-research for medicine in particular). We have published extensive notes from this conversation in transcript form (DOC); Dr. Ioannidis has been involved in the Cochrane Collaboration in the past and believes the USCC to be a strong funding opportunity.
• Professor Steven Goodman of the Stanford School of Medicine, a referral from one of the funders we spoke with (summary forthcoming).

Finally, we obtained detailed room for more funding analysis from the USCC; this analysis is now available online (DOCX).

Having done the above investigations, we felt that

• We had strong – though far from conclusive – reasons to believe that the USCC has strong answers to our key questions. In particular, we believed that it had an urgent need for more unrestricted funding to assist with its planning; that its struggle to attract unrestricted funding could largely be attributed to structural issues (the fact that major funders often focus on particular diseases and do not prioritize meta-research) rather than to substantive objections to the USCC’s work; and that it was a strong candidate for “best leverage point for supporting the overall mission of the Cochrane Collaboration.”
• We had many remaining questions about the USCC, and planned a thorough investigation to answer them. However, we didn’t see any “low-hanging fruit” remaining on the investigative end, and believed it would take a lot of work to obtain more satisfying answers to our key questions.

We discussed a $100,000 grant – enough to replace a specific source of support the USCC expects to lose, and enough to ensure that it can continue uninterrupted, minimal staff support. We came to the conclusion that $100,000 was enough to make a significant difference to the USCC without coming anywhere near meeting its funding needs (as expressed in the “room for more funding” analysis, linked above), and thus that such a grant could be justified not only based on its specific effect (allowing the USCC to retain uninterrupted, minimal staff support) but also on the more general principle (discussed above) of helping a highly underfunded organization with its ability to plan and prioritize. We checked in one more time with the USCC to make sure its funding situation had not changed materially, and recommended the grant.

*There is usually a substantial lag between our coming to a conclusion about a giving opportunity and our writing up & publishing our reasoning. In this case, for reasons discussed below, we did not want to accept the lag of writing up & publishing our reasoning before driving donations, so we made our recommendation via a discussion with Good Ventures and are publishing our reasoning now. In the future, we will generally publish our reasoning publicly before making a recommendation to any particular donor in cases where the funding gap is large and we are seeking to drive donations from many donors, and/or where a lag between the recommendation and the funding commitment is acceptable, but we may act as we have in this case when these conditions do not hold.