SDI Update – March 2015

Table of Contents

Published: March 01, 2015

This page gives an update on the Service Delivery Indicators Program, which is being carried out by the World Bank in partnership with the African Economic Research Consortium and the African Development Bank. Good Ventures contributed to this project based on a joint assessment with GiveWell.

1 Summary

In March 2014, Good Ventures made a $500,000 grant to the Service Delivery Indicators (SDI) program. Since then, SDI has continued to make progress on its surveys. Key updates are:

  • SDI survey results have received some attention and use. Its data has been used by blogs, news agencies, governments, and the World Bank (more).

  • SDI’s expenses are greater than it originally predicted, with surveys costing $700,000 instead of $500,000. SDI needs approximately $4 million in order to reach its goal of surveying 10-15 countries by the end of 2015 (more).

  • SDI continues to seek funding to reach its target number of countries, and is considering options for sustainable funding (more).

2 SDI’s progress

2.1 Data collection, analysis, and publication

As of 2013, SDI aimed to complete assessments for 10-15 countries by the end of 2015.1 SDI still believes it has the capacity to reach that goal with sufficient funding.2 (More on funding needs below.)

Eventually, SDI hopes to be working with 20-25 countries in Africa.3 Once it is surveying this many countries, it will have covered all of the countries in Africa where it believes the political environment is sufficiently open to use its information.4 However, there is a growing interest in SDI’s surveys from other continents.5

SDI’s pilot programs began in 2010.6 As of July 2014, SDI had:

  • Released six complete sets of survey results (Tanzania and Senegal in April 2012, Kenya in July 2013, Uganda in October 2013, Nigeria in June 2014, and Togo in July 2014).7
  • Completed the fieldwork of gathering data in two additional countries (Tanzania’s second round and Mozambique’s first round).8
  • Made preparations to collect a first round of data in four further countries (Niger, Mali, and the Democratic Republic of the Congo).9
  • Started discussions with two new countries about future rounds of data collection (Senegal and Cote d’Ivoire). The latter would be SDI’s first collaborative survey with the African Development Bank.10

2.2 Public use of the indicators

As we wrote in our January 2014 overview of the SDI program, SDI monitors its intermediate success using two indicators:

  1. “Public debate on education and health service delivery [is] initiated and/or informed.”
  2. “Stakeholders (policymakers, media, NGOs, CSO) reporting use of SDI analysis within 6 months after any of the SDI dissemination events.”11

2.2.1 Usage in public debate

SDI has told us that there has been public debate involving SDI’s data in Uganda and Kenya, where both of these countries were in the midst of teachers’ strikes when SDI’s survey results were released.12 SDI claims that its data fueled debates on education, particularly in Uganda, and that its findings are now discussed at many of Uganda’s and Kenya’s education policy forums.13

2.2.2 Usage by stakeholders

Tanzania
On the policy side, SDI believes its greatest success so far has been in Tanzania. SDI has told us that in 2011 SDI results were presented in Tanzania to a closed cabinet meeting.14 Afterwards, the Minister of Education began to visit schools unannounced in order to confirm SDI’s data.15 Then, in April 2013, SDI claims that its data was used during planning for the “Big Results Now” initiative.16 The “Big Results Now” initiative is part of Tanzania’s effort to become a middle-income country; it focuses on improving six areas of the economy, one of which is education.17 Several blogs began to reference SDI’s Tanzania data, especially its data on teacher absenteeism.18 In 2013, SDI’s data appeared in a report on Tanzania’s education system.19 Finally, SDI tells us that in 2014 its data was used to further track the progress of the “Big Results Now” initiative.20

The World Bank
Another policy accomplishment was the World Bank’s decision to use SDI’s data to help appraise programs that are requesting funding.21 The World Bank will agree to fund a country’s program if the program is shown to be achieving certain performance targets, and in both Tanzania and Uganda, the World Bank is now using SDI’s data as a tool to measure whether or not programs are reaching their targets.22 This could end up making SDI’s task more difficult, because if $10-15 million were dependent on the outcomes of SDI’s surveys, that could incentivize people to inflate the countries’ performance metrics, meaning SDI would have to increase its quality-assurance efforts.23

In addition, the World Bank and the Gates Foundation are looking to start a global initiative focused on primary health care. SDI surveys may be used in the future as part of a “Primary Health Care Performance Initiative.”24

Media and other
Articles in high-profile news sources (e.g., the Economist and BBC) are beginning to discuss the data that SDI is collecting.25 SDI is not mentioned in these articles explicitly, but it claims to be the only possible source of the information discussed.26 SDI’s results have also been presented at fora such as:27

  • The 10th Anniversary of the 2004 World Bank Development Report (February 2014)
  • The Oxford University Conference on African Studies (March 2014)
  • The World Health Organization and Spain Colloquium on Water and Sanitation in Health Facilities (April 2014)
  • The third Uganda Economic Update on Decentralization and Service Delivery (June 2014)
  • The World Health Organization Consultative Meeting on Health Facility Surveys (June 2014)

SDI has also developed its own website and is working to improve data visualizations for visitors.28

2.3 Building capacity

SDI aims to build up countries’ capacities to gather and analyze data by having local researchers work on its surveys.29 Therefore, SDI and its partners strongly prefer to use national research institutes to implement the surveys.30 Pursuing this goal of building capacity, a grant in the amount of $840,270 was successfully mobilized to support capacity development activities implemented by the African Economic Research Consortium (AERC), one of SDI’s partners.31 With funds from the grant, AERC successfully launched first training workshop in Kenya on the use of SDI data by national researchers.32

3 Funding updates

3.1 Program Costs

Each round of surveys costs SDI $700k (this includes both the health and the education surveys).33 This is higher than SDI’s original predictions of $400k-$500k per round of surveys.34 SDI underestimated the costs of contracts with survey implementers, the costs of additional oversight for quality assurance, and the costs of disseminating its findings.35

3.2 Room for more funding

SDI does not have enough funding for the last 5-6 countries it would like to survey to reach its 2015 goal of 10-15 countries surveyed; thus, its immediate funding gap was roughly $4 million in July 2014.36 It was hoping to have this funding by July 2015 in order to start the processes needed to finish the surveys by the end of 2015.37 If SDI is unable to find the funds it needs, it will simply survey fewer countries.38

3.3 Sustainability

SDI is also considering plans for its long-term sustainability. So far, SDI has been opportunistic about leveraging sources of funding within the countries it has surveyed.39 For example, it has used funds from the Department for International Development (DFID) and the Gates Foundation on a country by country basis.40 This allows SDI’s trust funds to last longer.41 However, SDI would prefer to have a more sustainable funding model; its current model is costly because of the many transactions and different sources of funding that SDI must keep track of.42

In thinking about how to create a new model, it is important to SDI that it remain neutral and transparent.43 Given the public good nature of the data generated through SDI, the preferred model is where most of its funding is split between governments and development partners, with some additional funding coming from the World Bank.44

4 Sources

DOCUMENT SOURCE
Big Results Now, The United Republic of Tanzania Prime Minister’s Office Source (archive)
DFID blog Source (archive)
Education: NKRA Lab Report Source
Gayle Martin, World Bank Senior Economist, email to GiveWell, June 1, 2014 Unpublished
GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014 Source
Project Appraisal Document on a Proposed Grant to the Government of Uganda Source
Program Appraisal Document on a Proposed IPA Credit to the Republic of Tanzania Source
SDI PowerPoint slide sent to GiveWell on August 8, 2014 Source
SDI Program Document 2011 Source
SDI Senegal Source (archive)
SDI Tanzania Source (archive)
SDI Timeline Source (archive)
World Bank, Service Delivery Indicators, email to GiveWell, March 25, 2015 Unpublished
Expand Footnotes Collapse Footnotes

1.

  • “SDI will assess 10 – 15 countries in Africa on each of the indicators based on a random sample of health and primary school facilities.” GiveWell’s Service Delivery Indicators Program page.
  • In 2011, SDI was aiming to survey 15-20 countries and reassess these countries every two-three years, staggering the countries so that a similar number would be assessed each year:
    • “The vision is to implement the Service Indicator Surveys initially in 15-20 countries in Africa to be repeated with predictable frequency (about every 3 years).” SDI Program Document 2011, page v.
    • “Assuming that a new survey is produced in each country every third year, there will be three survey waves each wave encompassing a third of the countries in the Program. As a point of departure, we suggest five countries in each wave, i.e., 15 countries in total. The number of countries can be expanded as additional resources become available.” SDI Program Document 2011, page 17.

2.“In line with its original objectives, SDI hopes to complete 10-15 countries by the end of 2015.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

3.“In Africa, once SDI is working in 20-25 countries, it will have covered all of the countries where the political environment and public domain is sufficiently open to use its information.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

4.“In Africa, once SDI is working in 20-25 countries, it will have covered all of the countries where the political environment and public domain is sufficiently open to use its information.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

5.“Beyond 2015, there is interest from other regions, which SDI will have to consider.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

6.SDI Timeline

7.

8.“The fieldwork of gathering data has recently been completed in two more countries:

  • Tanzania (the second round of data was collected)
  • Mozambique (the first round of data was collected)”

GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

9.“SDI is almost ready to collect a first round of data in several more countries:

  • Niger
  • Mali
  • Democratic Republic of the Congo
  • Zambia (SDI does not expect to be ready with Zambia until late 2014 or early 2015)”

GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

10.This would be Senegal’s second round of data collection and Côte d’Ivoire’s first round. SDI has postponed implementing its surveys in South Sudan because of conflicts there.
Gayle Martin, World Bank Senior Economist, email to GiveWell, June 1, 2014.

11.SDI Program Document 2011, Pg 11.

12.“Another way in which SDI is having an impact is by generating public discussion. For example, when the information from SDI’s surveys was released publicly in Uganda, it sparked a heated debate about why the education results were very poor. Additionally, teachers were on strike in both Uganda and Kenya when SDI’s data was released in those countries, and when the survey results were released they impacted the discussions around the strikes.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

13.

14.“2011: SDI results were presented at a closed Cabinet meeting: Minister of Education starts doing unannounced visits to schools and confirms SDI’s findings.” SDI PowerPoint slide sent to GiveWell on August 8, 2014.

15.“2011: SDI results were presented at a closed Cabinet meeting: Minister of Education starts doing unannounced visits to schools and confirms SDI’s findings.” SDI PowerPoint slide sent to GiveWell on August 8, 2014.

16.“April 2013: Presidential initiatives use SDI data as diagnostics during planning for major reforms: Reform Compact and Big Results Now!” SDI PowerPoint slide sent to GiveWell on August 8, 2014.

17.Big Results Now, The United Republic of Tanzania Prime Minister’s Office

18.

19.See pages 21 and 50 of Education: NKRA Lab Report.

20.

  • “June 2014: SDI indicators feed into and use to track performance of Government of Tanzania and World Bank/ project values $122 million.” SDI PowerPoint slide sent to GiveWell on August 8, 2014
  • “12. Low rates of teacher time-on-task are major problems in education service delivery in Tanzania. The Service Delivery Indicator survey (SDI, 2011) shows that about one in four teachers is absent from school on any given school day. Even when in school, the teachers are absent from the classroom more than half of the time. On average, students in primary schools in Tanzania are taught for two hours a day, and students in urban areas are taught for just one and a half hours a day (instead of the 5 hours and 20 minutes required of primary school teachers.” Program Appraisal Document on a Proposed IPA Credit to the Republic of Tanzania, pages 8-9. See pages 44-45 for an example of how SDI’s data is used to track Tanzania’s performance.

21.“Finally, the World Bank has begun to use SDI’s indicators as some of its disbursement-linked indicators.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

22.

23.“There is also a risk: this could end up making SDI’s task more difficult, because if $10-15 million is dependent on the outcomes of SDI’s surveys, that could incentivize people to inflate the countries’ performance metrics. This risk will have to be carefully tracked to avoid unintended consequences, meaning SDI may have to increase its quality-assurance efforts.”

GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

24.Gayle Martin, World Bank Senior Economist, email to GiveWell, June 1, 2014.

25.“Furthermore, articles in high-profile news sources (e.g., the Economist and BBC) are beginning to discuss the data that SDI is collecting.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

26.“Furthermore, articles in high-profile news sources (e.g., the Economist and BBC) are beginning to discuss the data that SDI is collecting. SDI is not mentioned in these articles explicitly, but it is the only possible source of the information discussed in the articles.”GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

27.Gayle Martin, World Bank Senior Economist, email to GiveWell, June 1, 2014.

28.Gayle Martin, World Bank Senior Economist, email to GiveWell, June 1, 2014.

29.“A key principle is that the SDI implementation model relies on is the use of national research institutes as implementing partners. Continuing this model will be critical since countries benefit from the capacity built through implementing the SDI surveys.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

30.“A key principle is that the SDI implementation model relies on is the use of national research institutes as implementing partners.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

31.World Bank, Service Delivery Indicators, email to GiveWell, March 25, 2015

32.World Bank, Service Delivery Indicators, email to GiveWell, March 25, 2015

33.“Although it originally predicted $400k-$500k for 2 surveys (health and education) per country, actual costs have been closer to $700k per country (this covers both the health and the education surveys).” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

34.“Although it originally predicted $400k-$500k for 2 surveys (health and education) per country, actual costs have been closer to $700k per country (this covers both the health and the education surveys).” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

35.“SDI has also experienced some funding challenges. Although it originally predicted $400k-$500k for 2 surveys (health and education) per per country, actual costs have been closer to $700k per country (this covers both the health and the education surveys). In part this is because contracts with survey implementers alone cost about $500k. SDI also underestimated the cost of additional oversight for quality assurance and the costs of dissemination of findings.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

36.“SDI is currently not fully funded for all 15 countries that it hopes to reach by the end of 2015. It is underfunded in about 5-6 countries, so its immediate funding gap is roughly $4 million.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

37.World Bank, Service Delivery Indicators, email to GiveWell, March 25, 2015

38.“If SDI is unable to get the funds it is hoping for, it will survey fewer countries, rather than decreasing the budget per country. Scaling down the per-country budget would require SDI to decrease activities like quality assurance and dissemination, which are essential to the project’s success. SDI believes that it would be easier to scale up the number of countries later than to increase the quality of its surveys later.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

39.“So far, SDI has been opportunistic about leveraging sources of funding within the countries it is surveying to complement the resources in the multi-donor trust fund resources.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

40.“For example, it has used funds from the Department for International Development (DFID) and the Gates Foundation (in Nigeria) on a country by country basis.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

41.“Given the funding constraints SDI has preferred to leverage resources at the country level, including World Bank resources, so that its multi-donor trust fund resources last longer. However, the trust money is still being spent to supplement the in-country funds.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

42.“This is not SDI’s preferred model of funding because it elevates transaction costs—costs would be fewer if the funds were all in one pool.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

Also: Gayle Martin, World Bank Senior Economist, email to GiveWell, June 1, 2014.

43.“The future funding model has implications for transparency. A model where funding is exclusively coming from on particular party – whether it be governments or the World Bank – has risks for transparency.” GiveWell’s non-verbatim summary of a conversation with Gayle Martin, World Bank Senior Economist, on July 30, 2014.

44.