How we assess our impact
Social norms interventions are notoriously hard to evaluate. Change happens incrementally over a long period of time, and not all social norms interventions will lead to immediate service uptake increases. We draw from sector best practices as well as MSI’s strong data systems to find ways to monitor and evaluate the impact of social norms activities, with the strategy often bespoke to a given intervention.
The SNAP approach
All of our approaches are rooted in the SNAP analysis framework and accompanied by a robust theory of change (ToC). This allows us to capture small changes that according to the ToC will lead to bigger future impact. We use an adapted version of CARE’s SNAP analysis framework. There are 4 steps to a SNAP approach informing evaluations:
SNAP framework: created to break down the social norms the project is trying to address.
Mixed method evaluation design: identifying relevant data sources; often including a simple vignette to be used during qualitative interviews describing a scenario influenced by the social norms
Semi-structured discussions: using a discussion guide, we capture participant reactions to the vignette. The discussion explores the different aspects of the SNAP framework for these social norms, including any changes participants have observed and how the project may have contributed to these.
Data analysis: qualitative data is analysed thematically, complemented by any relevant quantitative data.
Given how complex the issue of social norms change is, the study can provide only an indication of norm change. The qualitative approach also means that results are not generalisable to the wider population. Sometimes, given the intervention’s particular design and context, it is necessary to deviate slightly from these 4 steps.
The data we use
The data we collect to support the evaluation depends on the design of the intervention and where possible includes a mix of qualitative and quantitative insights. Examples include:
Analysis of routine service and monitoring data: used to understand participation in the different activities, the number and profile of referrals resulting from activities, redemption of the referrals and uptake of services in the areas where activities took place.
Qualitative discussions with participants: In-depth qualitative discussions with a range of participant types involved in the activities to explore perceptions of the social norms being addressed.
Observation of activities: structured observations of the activities to look at the content and tone of discussions; themes and information discussed (topics, accuracy of information); engagement of participants.
For examples of analyses completed of social norms interventions, please refer to the Social Norms Library.