This story was shared By Elliot McBride, Innovation Lead at UNICEF Pakistan Innovation Lead. Below is just the executive summary. The full report can be downloaded here.

Executive Summary

Title: Evaluation of WASHkit distribution in rural Punjab using the RapidPro sms feedback system

Author: McBride, E

Date: November 2015

Region: UNICEF ROSA

Type: Evaluation

Partners: PartnerAid

Purpose/Objective

This report provides analysis and insight of the feedback on UNICEF distributed WASHkits to Rajanpur district, Multan; using the sms technology, RapidPro. UNICEF Pakistan’s Water, Sanitation and Hygiene section (WASH) has developed a kit designed at providing equipment which assists in the daily household activities needed to save lives. Activities such as water collection, daily handwashing, clean storage of water and female hygiene have all been targeted by the products within the kit. Initially, distribution was delivered to vulnerable targets throughout Rajanpur district, Multan.

The RapidPro system is a SMS based information highway that sends and receives text messages. Controlled by officers within UNICEF, a series of questions are sent over SMS to beneficiaries who have registered to the service by sending a trigger message to a predefined number. All mobile network operators within Pakistan have registered - as a result, all Pakistani mobile phones are eligible for this service. A sticker with registration instructions was applied to the WASHkits prior to their distribution. Immediately respondents were registered with the RapidPro system and the surveys were completed. In one month over 1200 users had registered with the survey.

This evaluation will assist in the tailoring and adaptation of WASHkits design in future UNICEF assistance packages. Furthermore it will provide direction for follow up surveys targeting unanswered/partially answered queries within the programs survey.

The survey asked respondents 4 questions:

1.       What was the most useful item?

2.       What was the least useful item?

3.       Did anything arrive damaged?

4.       What will improve this kit?

The most useful item and least useful item questions were multiple choice of the four items included in the kit, namely soap, bucket, sanitary cloth and 2 collapsible jerry cans. The question asking for damaged items was also a multiple choice with options for the four items, and an additional option to indicate nothing had been damaged. The final question was an open response, whereby any text sent in reply was received as a valid answer and accepted for analysis. As beneficiaries were able to be identified for gender, results were analyzed using gender as an independent variable for the analysis.

Key Findings/Conclusions

Results for most and least useful were converted into a decimal indicator (1 being very useful, -1 being the least useful). This calculation takes into account votes for most useful, least useful and the total number of votes. For male beneficiaries, the bucket was the most useful item reported with a score of 0.3. The soap was also considered useful by male beneficiaries with a score of 0.22. Beneficiaries were indifferent to the sanitary cloths usefulness (-0.02) and the jerry cans were considered as the least useful item by male beneficiaries (-0.31). Female beneficiaries found soap extremely useful with an overall usefulness score of 0.5. Next was the bucket with a strong positive score (0.3) followed by the sanitary cloth which was found less useful (-0.23). The two jerry cans were scored as the least useful overall items (-0.25).

As the bucket is a multipurpose item its high usefulness score can be interpreted in many ways quite easily for both male and female beneficiaries. Soap as a dispensable item is able to save households money and therefore has direct financial value, adding to its utility. Women, who are especially responsible for child care and cooking are found to put the most emphasis on the usefulness of these items. The score that is the least explainable is the unpopularity of the two collapsible jerry cans. Water collection is a vital and lifesaving household activity that is pivotal to the family routine, yet the two jerry cans which have been designed to make this task easier is the outstanding item in terms of least utility to a household.

For the monitoring of the damaged goods, 4.9% of all responses reported a damaged cloth. 4.2% of all responses reported a damaged bucket. 1.6% reported damaged soap, and 0.2% damaged Jerry Can. 88% of all beneficiaries reported nothing was damaged on arrival. While reports of damaged goods are higher than what would be considered ideal, it can be assumed that over-reporting of damaged goods is more likely than under-reporting. Conditions and frequency of goods that arrived un-damaged is indicative of storage and distribution logistics as effective, and therefore not in need of revision.

The take away from the open response was that nail cutters were the most requested item to improve the kit. As this is quite a specific response, results were analyzed using a scatter graph in real time and discovered that there is a high chance that these results were coordinated to increase frequency and potentially influence results. However, as nail cutters are not lifesaving, they are therefore not within the UNICEF remit for addition to the WASHkit.

Lessons Learned

As there was some duplication in the answers for most useful and least useful, more care should be taken in the future when translating questions. Each question should be easily identifiable and unique to the point where the most basic literacy is required to understand it. Furthermore, as over 90% of the respondents choose to answer in Romanized Urdu, questions should be first developed in Urdu, before being translated into English.

Recommendations

Four main recommendations have been made as a result of this evaluation:

Recommendation 1. Follow-up survey – Jerry Can.
As the jerry can’s unpopularity amongst beneficiaries was so defined, a follow up survey can be used to identify the origin of the problems with the jerry cans and understand what is needed to be adapted to increase their usefulness.

Recommendation 2. SMS Campaign of non-action of nail cutters.
The most popular request made to improve the WASHkit was a set of nail cutters trailed by toothpaste/toothbrush. These requests eclipsed all other responses, however while these items could be included, non-essential, non-lifesaving items are not within the UNICEF remit and will not be incorporated into WASH assistance. However, part of this program is encouraging the use and effectiveness of the RapidPro feedback system. Isolating the respondents who requested nail cutters or toothbrushes/toothpaste, thanking them for their feedback and explaining that non-essential items won’t be included, provides these beneficiaries with confirmation that their responses have been received and accurately recorded. This will lead to a higher percentage of responses for RapidPro SMS feedback, providing beneficiaries with the assurance that their responses are being heard. 

Recommendation 3. Revise and redesign WASHkit for next year.
A consolidated narrative with the findings and conclusions for this report should be shared with the design team for the WASHkit. Regardless of the action taken, doing so will provide perspective for future kit design as well as confirming or answering questions that the design team might have after kit distribution.

Recommendation 4. Finalize data from other provinces.
As this report is only from the findings of one province out of 3. Analysis between different districts will add another level of insight to this report. This distribution is expected to be completed in Sindh province before the end of January.