Evaluation of WASHkits and distribution in rural Punjab using RapidPro
Launched: December 2015
This story was shared By Elliot McBride, Innovation Lead at UNICEF Pakistan Innovation Lead. Below is just the executive summary. The full report can be downloaded here.
Executive Summary
Title: Evaluation of WASHkit distribution in rural Punjab using the RapidPro sms feedback system
Author: McBride, E
Date: November 2015
Region: UNICEF ROSA
Type: Evaluation
Partners: PartnerAid
Purpose/Objective
This report provides analysis and insight of the feedback on UNICEF distributed WASHkits to Rajanpur district, Multan; using the sms technology, RapidPro. UNICEF Pakistan’s Water, Sanitation and Hygiene section (WASH) has developed a kit designed at providing equipment which assists in the daily household activities needed to save lives. Activities such as water collection, daily handwashing, clean storage of water and female hygiene have all been targeted by the products within the kit. Initially, distribution was delivered to vulnerable targets throughout Rajanpur district, Multan.
The RapidPro system is a SMS based information highway that sends and receives text messages. Controlled by officers within UNICEF, a series of questions are sent over SMS to beneficiaries who have registered to the service by sending a trigger message to a predefined number. All mobile network operators within Pakistan have registered - as a result, all Pakistani mobile phones are eligible for this service. A sticker with registration instructions was applied to the WASHkits prior to their distribution. Immediately respondents were registered with the RapidPro system and the surveys were completed. In one month over 1200 users had registered with the survey.
This evaluation will assist in the tailoring and adaptation of WASHkits design in future UNICEF assistance packages. Furthermore it will provide direction for follow up surveys targeting unanswered/partially answered queries within the programs survey.
The survey asked respondents 4 questions:
1. What was the most useful item?
2. What was the least useful item?
3. Did anything arrive damaged?
4. What will improve this kit?
The most
useful item and least useful item questions were multiple choice of the four
items included in the kit, namely soap, bucket, sanitary cloth and 2
collapsible jerry cans. The question asking for damaged items was also a
multiple choice with options for the four items, and an additional option to
indicate nothing had been damaged. The final question was an open response,
whereby any text sent in reply was received as a valid answer and accepted for
analysis. As beneficiaries were able to be identified for gender, results were
analyzed using gender as an independent variable for the analysis.
Key Findings/Conclusions
Results for most and least useful were converted into a decimal indicator (1 being very useful, -1 being the least useful). This calculation takes into account votes for most useful, least useful and the total number of votes. For male beneficiaries, the bucket was the most useful item reported with a score of 0.3. The soap was also considered useful by male beneficiaries with a score of 0.22. Beneficiaries were indifferent to the sanitary cloths usefulness (-0.02) and the jerry cans were considered as the least useful item by male beneficiaries (-0.31). Female beneficiaries found soap extremely useful with an overall usefulness score of 0.5. Next was the bucket with a strong positive score (0.3) followed by the sanitary cloth which was found less useful (-0.23). The two jerry cans were scored as the least useful overall items (-0.25).
As the bucket is a multipurpose item its high usefulness score can be interpreted in many ways quite easily for both male and female beneficiaries. Soap as a dispensable item is able to save households money and therefore has direct financial value, adding to its utility. Women, who are especially responsible for child care and cooking are found to put the most emphasis on the usefulness of these items. The score that is the least explainable is the unpopularity of the two collapsible jerry cans. Water collection is a vital and lifesaving household activity that is pivotal to the family routine, yet the two jerry cans which have been designed to make this task easier is the outstanding item in terms of least utility to a household.
For the monitoring of the damaged goods, 4.9% of all responses reported a damaged cloth. 4.2% of all responses reported a damaged bucket. 1.6% reported damaged soap, and 0.2% damaged Jerry Can. 88% of all beneficiaries reported nothing was damaged on arrival. While reports of damaged goods are higher than what would be considered ideal, it can be assumed that over-reporting of damaged goods is more likely than under-reporting. Conditions and frequency of goods that arrived un-damaged is indicative of storage and distribution logistics as effective, and therefore not in need of revision.
The take away
from the open response was that nail cutters were the most requested item to
improve the kit. As this is quite a specific response, results were analyzed
using a scatter graph in real time and discovered that there is a high chance
that these results were coordinated to increase frequency and potentially
influence results. However, as nail cutters are not lifesaving, they are
therefore not within the UNICEF remit for addition to the WASHkit.
Lessons Learned
As there was
some duplication in the answers for most useful and least useful, more care
should be taken in the future when translating questions. Each question should
be easily identifiable and unique to the point where the most basic literacy is
required to understand it. Furthermore, as over 90% of the respondents choose
to answer in Romanized Urdu, questions should be first developed in Urdu,
before being translated into English.
Recommendations
Four main recommendations have been made as a result of this evaluation:
Recommendation
1. Follow-up survey – Jerry Can.
As the jerry can’s unpopularity amongst beneficiaries was so defined, a follow
up survey can be used to identify the origin of the problems with the jerry
cans and understand what is needed to be adapted to increase their usefulness.
Recommendation
2. SMS Campaign of non-action of nail cutters.
The most popular request made to improve the WASHkit was a set of nail cutters
trailed by toothpaste/toothbrush. These requests eclipsed all other responses,
however while these items could be included, non-essential, non-lifesaving
items are not within the UNICEF remit and will not be incorporated into WASH
assistance. However, part of this program is encouraging the use and
effectiveness of the RapidPro feedback system. Isolating the respondents who
requested nail cutters or toothbrushes/toothpaste, thanking them for their
feedback and explaining that non-essential items won’t be included, provides
these beneficiaries with confirmation that their responses have been received
and accurately recorded. This will lead to a higher percentage of responses for
RapidPro SMS feedback, providing beneficiaries with the assurance that their
responses are being heard.
Recommendation
3. Revise and redesign WASHkit for next year.
A consolidated narrative with the findings and conclusions for this report
should be shared with the design team for the WASHkit. Regardless of the action
taken, doing so will provide perspective for future kit design as well as
confirming or answering questions that the design team might have after kit
distribution.
Recommendation
4. Finalize data from other provinces.
As this report is only from the findings of one province out of 3. Analysis
between different districts will add another level of insight to this report.
This distribution is expected to be completed in Sindh province before the end
of January.