Principles and tools
Case study analysis
The ImpresS ex post method relies on case study analysis, in order to be able to address the complexity of agricultural innovations. This approach enables the use of a common inquiry framework across cases, while being able to allow for diversity among cases (Yin, 2009). The unit of analysis is the innovation process from the beginning of the research work to appropriation of research outputs by their intended users.
Thirteen case studies reflecting the diversity of CIRAD's research actions and geographical scope were selected from a hundred proposals made by the institutions’ scientific management. Nine of them were ex-post case studies, whereas four were in itinere (ongoing) evaluation cases.
The process by which agricultural research contributes to developmental impacts is a complex multi-causal phenomenon. The ImpresS method stems from the recognition that attribution analysis should not appropriate for assessing this impact since a counterfactual scenario (situation without the intervention) is extremely difficult to elaborate, given that the nature of the agricultural interventions and the actors involved evolve throughout the innovation process. Multiple factors, both internal and external, contribute to the final impact. Contribution analysis is therefore more appropriate. Contribution analysis addresses causality by elaborating an impact pathway, verifying its assumptions, identify internal and external factors of success or failure and lastly understanding the part played by rival hypotheses in producing impacts (Mayne, 2001).
Iterative and participatory building of evaluation tools
The guiding principle of the ImpresS method is to associate, as far as possible, stakeholders of the innovation (both those involved in the innovation process and those impacted by it) in the evaluation process. Therefore, even if the evaluation team initially sketches out potential impact hypotheses, impact indicators, impact pathways and innovation stories, these products are systematically confronted and re-elaborated with stakeholders’. This allows to include their perceptions of the innovation process and reach a consensus on this process, or at least substantiate divergent assessments.
Focusing on the outcomes black-box: the capacity building process
All along the impact pathway, different stakeholders participate in the innovation process, in order to foster the appropriation of the innovation by its intended users (the outcomes). This interaction of actors within the innovation process to produce outcomes involves many types of learning processes (formal, informal, through networks, for individuals, groups, etc). It can also build different capacities of stakeholders and even generates new capacity to innovate. The mapping and analysis of these processes serve to trace the intermediary role of research in achieving impacts.
The innovation story
The innovations studied in ImpresS ex post and their outcomes and impacts are not the result of a linear process. CIRAD’s agricultural research is conducted in close partnership with stakeholders, academics and non-academics in different contexts. Hence, ImpresS ex post details the timeline of this often lengthy and iterative process of co-production of an innovation. Linked with the mapping of actors involved at different phases of an innovation, the timeline also sheds light on the evolution of the innovation’s stakeholder network.
The impact pathway
The impact pathway is a tool grounded in the theory-driven evaluation literature (Douthwaite, 2003). As a diagram, it represents the causal logic of an intervention (or an innovation). In the context of agricultural research, the impact pathway includes four principal elements, namely:
- inputs: the resources used by the research team to produce scientific results and products
- outputs: the output is generated by research activities or by the interactions of research activities with stakeholders in the projects. The output can be knowledge, scientific or not (publications, reports, database, methods …), trainings (academic or technical), expertise, technology, network, or other forms of production. The outputs can contribute to initiate the innovation process when they are appropriated by actors. In the ImpresS approach, research outputs (knowledge, prototypes …) elaborated before the intervention will be considered as inputs, whereas those elaborated during the intervention will be considered as outputs. .
- outcomes: the appropriation of results (outputs) by by actors interacting directly or indirectly with research, which leads to new practices (agricultural or managerial), new organizations, new rules
- primary impacts: primary impacts are measured on actors interacting directly or indirectly with research and/or major actors of the innovation process, and can be evaluatedwith those actors.
- secondary impacts: scaling out or scaling up of this innovation to other territories and audiences and its impact.