You’ve probably seen—or even been directly involved—a campaign of online petitions to garner hundreds of thousands of signatures. What do you think is the measure of the success of these campaigns? Number of “likes” and “shares”? Media coverage? Considered “viral”? Number of signatures supporting the online petition? These are only minor aspects and even less relevant to assessing whether a campaign to promote social change is effective or not.

All civil society organizations (abbreviated as CSOs) aspire to improve the lives of the people who are their constituents, but there are many obstacles that hinder this dream. One of the obstacles is the existence of policies or practices that interfere with the common good (e.g. policies that discriminate or encourage environmental damage). One of the other obstacles is the behavior of some people who are considered to be detrimental to themselves or others.

To overcome this obstacle, many CSOs end up conducting public campaigns. If the obstacle they want to overcome is a policy, then the campaign will aim to mobilize citizens to demand that the policy be changed. If the obstacle is the behavior of some people, then the campaign aims to change it. Campaigns aimed at changing policies or changing behavior fall into the category of “social change campaigns”.

Since Indonesians are passionate about using social media, it is easy for us to come across examples of social change campaigns. Various groups launched campaigns to change people’s behavior at the beginning of the COVID-19 pandemic. These campaigns generally invite people to adopt new behaviors such as wearing masks, washing hands more frequently in the right way, maintaining physical distance from other people, and avoiding crowds. After the COVID vaccine is available, a campaign appears inviting people to be fully vaccinated.

An example of a campaign to push for policy change, the latest in Indonesia is demanding the ratification of the Draft Law on the Crime of Sexual Violence (RUU TPKS) into law. Efforts to encourage the emergence of a law have actually started 10 years ago, but public campaigns have started to emerge at least since the second half of 2021. The materials for this public campaign start from media coverage, explanations of what sexual violence is and why there is a need for a special law in Indonesia. Indonesia, until it finally spread into organic talk (not triggered by campaigners) on social media. Of course, don’t miss the support-raising for the online petition which has garnered 349,743 signatures. This policy advocacy and public campaign has shown quite encouraging results: the TPKS Bill was finally passed by the DPR into Law on April 12, 2022.

Can we judge that because the TKPS Law was finally passed, we can see that the public campaign to garner public support for this bill is effective or successful? By looking at the number of cases, the number of deaths, and the reproducibility rate (R_o) of COVID19 in Indonesia, can we judge whether the existing behavior change campaigns are effective or not?

So what about other social change campaigns? If the policies that are being pushed have not been passed after years of campaigning for social change, will we be able to convict that the campaign has failed? Then what if the policy that is passed turns out to be very different from the CSO’s proposal? Can the social change campaign that accompanies policy advocacy efforts still be successful?

The same questions can be asked of behavior change campaigns. When can a behavior change campaign be considered effective? We have often seen the campaign “raise public awareness about mental health issues” on social media. If more and more people are using mental health terms in everyday conversation but the number of people receiving psychological or psychiatric services is still not increasing, can we say that this campaign failed?

When asked about the effectiveness of social change campaigns, those who are managing social change campaigns (especially those that take place on social media channels) usually mention the numbers measured by impressions, reach, likes, shares, engagement rates, responses, or the number of signatures support online petitions. Sometimes they even mention how much media has covered their campaign. If the scores are considered pretty good (though it’s often unclear what the benchmarks are), they usually feel quite proud that they have accomplished an important or noble task. But is it true that the response to the uploaded material equals effectiveness?

So how should the effectiveness of a social change campaign be measured? The short answer is: a social change campaign is considered effective if it succeeds in bringing about the desired changes. So how do we determine what changes a social change campaign can realistically bring about, and not other factors? The trick is to develop a theory of change as the first step in making a campaign strategy.

A campaign must have a theory of change

The theory of change sounds complicated or even mystical, but it’s actually quite simple. All CSOs want to bring about change that leads to the good of society. Most of these changes will only occur if there is one or more problems that need to be solved, so the current situation is not ideal. The theory of change is a set of hypotheses about how a program run by a CSO can solve the problem(s), so that the desired change can finally occur. The form of the theory of change can be a simple diagram, or a short paragraph with sentences in the format “if…, then…”.

Many CSOs have made theoretical changes before starting each program. Unfortunately, most of the CSO campaign activities do not start with making a theory of change. Based on questions and answers with CSO staff, we suspect that this is because social change campaigns are usually only seen as one of the tactical activities in the program, and are considered to be in line with events such as seminars or workshops.

How to develop a theory of change for a social change campaign? Follow these steps:

1. Describe what you would see if the key change your organization was pushing for was finally achieved (“what success looks like?”). What did you see in that situation that was different from the current state? Write in a box.
2. Step back: before that “final victory” is achieved, what must be there first? If necessary you should go back several steps (and fill in a few boxes) until you meet conditions such as “X policy changed (revoked, revised, or passed)” or “X group did (specific behavior”).
3. After writing the boxes and arriving at the box about policy change or behavior change, then pause for a moment. You have to jump into the current situation. Describe what is happening now (baseline situation).
4. Write a new box connecting the boxes you filled in in steps 2 and 3. This box should be the goal of the social change campaign.

Here’s an example of a theory of change for a campaign to encourage policy change:

And here’s an example of a theory of change for a campaign to encourage behavior change:

By developing a theory of change, campaigners will understand two things better: what is the purpose of the campaign, and what changes must occur first in the campaign objectives, so that the desired impact is more likely to occur. The effectiveness of social change campaigns is measured by the extent to which the goals that the campaign has successfully reached have actually changed their behavior. Changes that occur in the campaign objectives are referred to as the outcomes of the social change campaign.

For example, the effectiveness of a policy change campaign on forest conservation is measured by how many of them are exposed to campaign materials who end up voicing their demands. However, we know that the number of people who are demanding does not guarantee that policy makers will do something (eg because there are other interests). Similarly, the behavior change campaign related to TB control. The measure of its success is the number of people who end up coming to the TB screening clinic after seeing the campaign. The number of people who are tested and then diagnosed with TB depends on other factors, such as the availability of tools and materials, so it is not entirely the “responsibility” of the campaign.

If the theory of change has made us clearer about campaign objectives and expected outcomes, then we can now explore how to monitor performance and evaluate social change campaigns.

• Reach in the digital realm is calculated from how many Internet users are exposed to a campaign material. If impressions count duplication (the same person exposed to a material 5 times counts as 5 impressions), reach counts only per individual (in this example, 5 impressions count as 1 reach because there is only one Internet user). It’s really difficult to directly measure reach in non-digital media (except the number of visitors to an event, maybe), and usually it can only be estimated based on a mathematical model.

To answer the second question (“Are the campaign goals interested in the content?’), we measure engagement metrics. Engagement in the digital realm is calculated from the number of interactions from the audience aimed at a campaign material. In social media, engagement can be in the form of likes, shares, and the number of responses. On websites, it is measured using bounce rate, time per session, pages per session.

To answer the third question, the only way to measure it is to conduct a short survey with people who have viewed the content. We’ll ask if they can identify the essential attributes of the campaign, such as the core message, key visuals, or organization of the campaign.

From the explanation above, it is clear that there are two factors that influence the size of the output. The biggest factor is the budget for distributing campaign materials, including through paid advertising. The second factor is how campaign materials are created: whether they are eye-catching enough, make it easy for people to understand key messages, remember important visuals, or identify who is campaigning.

How do you measure the outcome of a social change campaign? In this case, the question to be answered is “Does the campaign target change its behavior as expected after being exposed to the campaign material?”. The best way to measure results is to compare the baseline to the endline. In other words, we should do two surveys about what campaign targets think, feel, and behave: once before the campaign starts, and once again after the campaign ends. Unfortunately, this ideal measurement is rare in Indonesia because many organizations skip audience research when determining the baseline.

A more likely but less valid option is to use a proxy. However, this modeling approach also requires comparison of the baseline with the endline (or midline). The factor being compared is the interest or sentiment of people towards the issues discussed. This comparison can use Google Trends to find out how far this topic is being searched before vs. after the campaign. This comparison can also use network analysis on social media, or see the volume of mentions and distribution of anyone who mentions key terms from the topic on social media before vs after the campaign.

Another less valid option is to measure only the endline (opinions, feelings, or behaviors) of two groups with the same characteristics, and differ only in their exposure to the campaign material. The challenge is to ensure that the groups differ only in the level of exposure to the content, and not in other things such as socio-economic level, education, culture, etc.

Want to know more about how to strategize a social change campaign? Or do you want us to assist your organization in running the campaign? Join the training from C4C by calling our WhatsApp number (+62 8966-6666-727)