Principles and tools
Case study analysis
In order to reflect the complexity of the innovations and their environments, the method proceeds by case study analysis. This analysis tool on the one hand enables a common inquiry across cases following a single framework and on the other hand remains open to diversity among cases (Yin, 2009). The unit of analysis is the innovation process from the beginning of the research work to ownership by users.
Thirteen case studies reflecting the diversity of CIRAD's activities and geographical areas were selected from a hundred proposals made by CIRAD scientific management. Nine of them are ex-post whereas 4 are in itinere (ongoing) evaluation cases. This makes it possible to consider the emerging outcomes from another perspective and to find the way to conduct the impact hypotheses.
Contribution analysis of impact
The impact process of agricultural research is a complex multi-causal phenomenon. Attribution analysis is not appropriate for assessing this impact since a counterfactual scenario (situation without the intervention) cannot be elaborated given that the nature of the intervention evolves throughout the innovation process and multiple factors, both internal and external, contribute to the final impact. Contribution analysis is therefore more appropriate. It makes it possible to address causality by elaborating an impact pathway, verify its assumptions, identify internal and external factors of success or failure and lastly understand the part played by rival hypotheses in producing impacts (Mayne, 2001).
Iterative and participatory building of evaluation tools
The guiding principle of the ImpresS method is to associate, as far as possible, stakeholders of the innovation (both those involved in the innovation process and those impacted by it) with the evaluation process. Therefore, if the evaluation team first sketches out hypothesis for potential impacts, indicators, impact pathways or innovation stories, these products are systematically confronted with stakeholders’ perceptions of the innovation. The aim of the study is to use these tools in order to reach a consensus on the innovation process, or at least to provide evidence for divergent assessments.
Focusing on the outcomes black-box: the capacity building process
All along the impact pathway, the research stakeholders often participate in implementing the innovation in order to foster its appropriation by the target audience. This interaction of actors within the innovation process to produce outcomes involves many types of learning processes (formal, informal, through networks, for individuals, groups, etc), it often builds the capacity of stakeholders and even generates new capacity to innovate. The mapping and analysis of these processes by ImpresS serve to trace the intermediary role of research in achieving the impacts.
The innovation story
The innovations under study and their outcomes and impacts are not the result of a linear process. CIRAD’s agricultural research is conducted in close partnership with stakeholders, academics and non-academics whose diversity depends on the cases and the context of intervention. Hence, it is necessary to account for the chronology of this often lengthy iterative process and the co-production of the innovation. In particular, this tool sheds light on the evolution of the innovation’s stakeholder network.
The impact pathway
The impact pathway is a tool grounded on the theory-driven evaluation literature (Douthwaite, 2003). This diagram represents the causal process of the intervention under study and includes:
- inputs: the resources used by the research team to produce scientific results and products
- outputs: the results produced by the research team (publications, technical innovation, etc)
- outcomes: appropriation of those results by the beneficiaries or intermediate stakeholders that often lead to adaptation.
- primary impacts: impacts of the use of the innovation on the first target audience
- secondary impacts: scaling out or scaling up of this innovation to other territories and audiences and its impact.
The impact pathway also represents the causal links between the various steps in the innovation process. See below the ImpresS impact pathway diagram.
The ImpresS methodology plans to include a cross-sectional analysis of several aspects (impacts, mechanisms, etc). The first dimension already implemented refers to the capacity of research to influence policy-making.
Research may impact the political agendas, arguments and policy horizons of policy-makers as well as their policy objectives and strategies, including laws and regulations, public spending patterns, policy implementation activities and policy capacities at different levels. For the innovations under study, we outline research-policy linkages between research-innovation processes and policy-making processes, relying on the impact pathway analysis.