A new approach to measuring the impact of open data
Strong evidence on the long-term impact of open data initiatives is incredibly scarce. The lack of compelling proof is partly due to the relative novelty of the open government field, but also to the inherent difficulties in measuring good governance and social change. We know that much of the impact of policy advocacy, for instance, occurs even before a new law or policy is introduced, and is thus incredibly difficult to evaluate. At the same time, it is also very hard to detect the causality between a direct change in the legal environment and the specific activities of a policy advocacy group. Attribution is equally challenging when it comes to assessing behavioral changes – who gets to take credit for increased political engagement and greater participation in democratic processes?
Open government projects tend to operate in an environment where the contribution of other stakeholders and initiatives is essential to achieving sustainable change, making it even more difficult to show the causality between a project’s activities and the impact it strives to achieve. Therefore, these initiatives cannot be described through simple “cause and effect” relationships, as they mostly achieve changes through their contribution to outcomes produced by a complex ecosystem of stakeholders — including journalists, think tanks, civil society organizations, public officials and many more — making it even more challenging to measure their direct impact.
We at the Sunlight Foundation wanted to tackle some of the methodological challenges of the field through building an evidence base that can empower further generalizations and advocacy efforts, as well as developing a methodological framework to unpack theories of change and to evaluate the impact of open data and digital transparency initiatives. A few weeks ago, we presented our research at the Cartagena Data Festival, and today we are happy to launch the first edition of our paper, which you can read below or on Scribd.
The outputs of this research include:
- A searchable repository of more than 100 examples on the outputs, outcomes and impacts of open data and digital technology projects;
- Three distinctive theories of change for open data and digital transparency initiatives from the Global South;
- A methodological framework to help develop more robust indicators of social and political change for the ecosystem of open data initiatives, by applying and revising the Outcome Mapping approach of IDRC to the field.
The repository
The case stories we collected aim to illustrate the social impact of open data and digital transparency initiatives in different countries, cities and communities. We gathered examples on how users are empowered by open government initiatives as well as how open data and digital transparency projects are changing the behavior, relationships, activities and actions of users. Instead of basic project descriptions and mere output, we wanted to seek evidence on how open data and technology help to influence governance and improve lives, both directly and indirectly. After reviewing the examples, our research team compiled the data to form a database with basic facts, short descriptions and links, then categorizing the repository into a couple of significant fields/sectors. The database is a living document and available here.
For the sake of our research, we defined open data initiatives as projects using data that can be freely used, reused and redistributed by anyone. After filtering the 136 cases, we had a total of 110 open data initiatives in our repository, with 64 cases from the Global North, 37 from the Global South and nine with impact on both regions. The most common type of open data initiatives focused on legislatures, elections and elected officials; budget and spending information (including procurements); open data portals; business; environment; the judiciary; extractive industries; and health and lobbying.
Not surprisingly, the submitted case stories to our repository approached "impact" in various ways, ranging from mere output (such as publishing a certain amount of raw data), through mid-term outcomes (through, for instance, increasing participation in a democratic process), to long-lasting impact (such as increasing social equity). In some cases, the descriptions focused on short-term outputs only, even when the project does seem to achieve long-lasting change, while other cases claimed impact without credible proof on attribution.
While acknowledging that the open government and open data agenda is much broader than the scope of our repository, we identified four potential benefit categories for project outcomes in the open government space:
- Educate or inform citizens so that they can make more informed choices;
- Promote direct civic engagement and increase citizen participation in democratic processes;
- Gather feedback for policy-makers and/or the private sector; and
- Monitor and hold officials and/or the private sector accountable.
In the second phase of our research, we selected three projects from the Global South: Vota Inteligente from Chile, Open Duka from Kenya and A Tu Servicio from Uruguay to work more closely with. We developed a methodological framework and built distinctive theories of change for all three projects as well as a set of social change indicators based on the identified progress markers and strategy maps, through testing and revising the approach of outcome mapping.
Next steps
Assessing impact means we should be able to prove if there has been some kind of change in the ecosystem. Whether that change is “good” or “bad” will always depend on a normative position, while attribution is incredibly challenging — even impossible — for most open government projects. The outcome mapping approach, as well as other robust evaluation methods, has a strong potential for the long-term evaluation of complex projects through detecting and documenting the desired change model in the behaviors, relationships and activities of people and organizations an open data initiative interacts with. It might also provide a good framework for identifying other research methods (such as randomized control trials or quantitative surveys) to prove causality.
In the next few months, we will need to refine the assessment frameworks designed for the three projects we examined. After the launch of our paper, Sunlight’s international policy team intends to conduct another round of group interviews with the three implementing teams. We plan to make some adjustments to the social change indicators and identify the data collection methods for documenting. The project implementing groups will need to decide what they will monitor on an ongoing basis and what they will evaluate in depth in the next few years. In each case, they will have to revisit the indicators on a regular basis for the best impact.
Sunlight’s aim with this research has been to boost the conversation around impact assessment in the open government field, as well as to provide a base for advocacy purposes and perhaps further research. We believe that focusing on mid-term outcomes, as opposed to long-term impact or short-term output, might be the next step for the open data community in order to create a solid base for evaluation. We also think that further work is needed to develop clear change models for most open data and open government projects.
You can read the study below:
