Steps for deploying RapidPro for programme monitoring - the Pakistan experience

Launched: May 2016

End of 2015, UNICEF Pakistan used RapidPro to get feedback on hygiene kits that were distributed in rural Pakistan. The collected feedback over SMS was analysed and evaluated to assist in the tailoring and adaptation of hygiene kits design for future UNICEF assistance packages. The report on how RapidPro was used, findings, lessons learned and recommendations can be found here.

The report also contains a useful guide with steps on how to use RapidPro for programme monitoring. These steps are: 

  1. Identify your central question: what are you trying to achieve
  2. Develop a work plan: which activity will be carried out and when
  3. Develop an outreach plan: how will beneficiaries be reached and how will RapidPro interact with them
  4. Develop your survey: define indicators and related questions
  5. Test your system and survey: test RapidPro and your survery in a controlled environment
  6. Launch your survey: “your flows go live”
  7. Monitor and evaluate: monitoring the deployment and evaluating what went well and what did not go well for improvements

Note that the steps above have been modified and added to in this blog post. Also, it is assumed here that you have already secured the resources needed for the implementation (funds, capacity, partnerships, a live and connected RapidPro account, etc.). Furthermore, keep in mind that these steps are based on the Pakistan experience. For a more systematic approach to help you prepare for deploying RapidPro refer to the RapidPro Deployment Toolkit.

1. Identify your central question

What is it that we want our survey to discover? The question should be able to be put into a single sentence such as: “We wish to determine the use and access of men and women for WASH services in Sindh.” “We want to measure the satisfaction of beneficiaries in Punjab regarding their school safety”.

Determine the central questions you want answered by beneficiaries, for example:

  • What is the satisfaction of ‘x’?
  • How is your access to ‘x’?
  • How often do you ‘x’?

You must also carefully consider which comparisons you want to make later in the data analysis:

  • Male vs Female (is there a difference between gender for our measurement?)
  • District/ provincial (is there a difference between geographical locations for our measurement?)
  • Age (is there a difference between life stages for our measurement?)

If these comparisons are relevant to your analysis, there must be registration questions included in your survey. However, any surplus questions should be omitted as the longer any survey is the higher the dropout rate before the survey has been completed. Any survey should not be longer than 7 questions including registration and follow up. Alternatively, you could decide to spread the questions over multiple flows which you within a particular timeframe.

2. Develop a work plan

Now you need to start thinking about when each activity has to be implemented, for example:

  • When is your survey developed, tested and delivered?
  • How long will the survey run for?
  • If communications material is needed, when does it have to be in place and released?
  • If you are working with an implementing partner, when are they expected to carry out their activities?
  • What is the literacy rate of your targeted population?
  • When are the results going to be analysed and shared?

3. Develop an outreach plan

We need to determine how we will reach out to the beneficiaries and how they will interact with RapidPro. There are 2 options:

1. Users register by texting the shortcode from their handsets

If the users must register themselves, the shortcode delivery to the beneficiaries becomes the priority. You need to also to inform the beneficiaries on the relevance of the initiative (particularly to them). This requires a partner that has communication access to target groups. It can be done via letters, billboards, and posters or through communication agencies such as mosques, churches, schools, workplaces etc.

2. Phone numbers are entered into the system by UNICEF staff

This requires contact with an agent or agency that has the beneficiary details and their phone numbers. This method, as it is not voluntary requires consent to be secured with an opt-out function. i.e. “this is a voluntary survey that you do not need to complete”.

4. Develop your survey

Here we need to think about the questions that we will include in the survey. We need to make sure that we are asking questions that can inform our programme, but at the same time we need to be very selective. A survey with more than 6-7 questions over SMS may be too lengthy and can lead to people dropping out. You could also consider spreading the questions over a period of time to not burden respondents.

Here we see the response rates as a percentage using the Pakistan case. The below questions were asked after registration (location, name, gender). As we can see, the percentage of respondents drops from over 70% at the 4th question to 50% at question 7.

Ideally, the programme specialists should be responsible for question development as it requires in depth technical knowledge about the programme topic. This can be then given to the innovations coordinator for building flows.

5. Test your system and survey

1. Translation

Initially, the survey is likely to be developed in English as it is a common language between the technical officers i.e. WASH specialists and the innovations section. Later according to the region, specific translations are to be entered into the RapidPro system by a native speaker of the relevant language.

2. Field testing

Prior to implementation it can be beneficial to conduct a pre-test of survey questions and concepts in an information session or focus group discussion. This requires finding an implementing partner that has an established ground presence and access to a population of willing participants.

Adaptations and changes can be made during and after – this is also a useful tool for advocacy of the RapidPro system.

3. Technical testing

All users of the system should run through the survey on their phones and record any errors/ final adaptations to be made before actual delivery.

6. Launch your survey

Once survey is tested, double check everything again! The data collection process can begin.

7. Monitor and evaluate

Make sure you monitor the connection and the functioning of the system to ensure the interaction with beneficiaries is running smoothly.It is also good to have the contact details of your hosting provider and SMS aggregator at hand in case of failure.

Keeping track of what went well and what did not go so well is important. It will help you reflect and improve future similar initiatives.