How does evidence-based development policy change happen?
(Note: This paper was written in the first quarter of 2018 as part of a Research Design course)
This paper discusses the criteria for the selection of 6 cases to explore the causal mechanism of how evidence-based recommendation lead to policy change. The mechanism through which this works was described as “translation and mobilization”. While focusing intensively on each case, selecting case studies across time and space would reveal any variations in the mechanism, be instrumental in identifying entry points and areas where the causal mechanism can be strengthened and move us closer towards generalization.
Policy and the Importance of Evidence Based Insights
A public policy is an action which employs governmental authority to commit resources in support of a preferred value.[1] It is what governments and the public sector choose to do or choose not to do.[2] This preferred value forms the basis for government objectives and proposed policy actions. However, multiplicities of values necessitate that rationales and the cost and benefits of governments’ preferred value is compared to that for other groups or sectors. Without quantitative measures of these values, there is no common yardstick for evaluating costs and benefits and policy decisions end up being political decisions.[3] Evidence-based recommendations/insights not only helps to identify groups or sectors that should be targeted for intervention, it also fosters criteria necessary for good public policy. These are that the policy solution achieves identified objectives (efficient), produces the greatest benefit at the least possible cost (effective), delivers changes in the real world (outcome focused), inclusive by taking full account of the needs and experience of all those likely to be affected by policy, involve a careful appraisal of costs and benefits, and based on learning from experience through continuous learning and make use of evidence and research about the problems being addressed.[4] Definitely, a public policy is a deliberate and careful decision that provides guidance for addressing selected public concerns. Objectives are set, methods to achieve them are identified and assessed based on their strengths and weaknesses. However, without evidence as the starting point providing justification for their selection and the amount of intervention that is appropriate per time, policy would just be without direction or focus.[5] Thus, evidence-based recommendation and insights prove to be an enabler for objective/rational policy making that is in the best interest of the public.
Causal Mechanism linking Evidence based insight to Policy
Evidence-based research can play this role of an enabler for objective policy making because it provides resources for translation and mobilization; the causal mechanism through which evidence-based research becomes policy outcomes. These resources are the arguments or insights that the research reveals.
A clear and succinct definition of causal mechanism (mechanism) defines it as, “intervening processes through which causes exert their effects”.[7] They can also be described as producing an “outcome by the interaction of a number of parts”.[8] Although, mechanisms have a consistent pattern, the presence of an exception does not invalidate the mechanism. They are robust, non-accidental and hold in virtue of the fact that they describe the behavior of the mechanism.[9] Therefore, they have also been defined as “mechanistically fragile generalizations”[10] Explaining a social event involves describing the various causal chains linking all the elements involved in constituting a social fact, identifying the relevant elements between which causal relationships exist, and determining their nature. Viewed in this way, a mechanism is the set of elements and their causal links that regularly lead from an initial social state to a subsequent one.[11] The requirements of this definition are exemplified in the Evidence Based Recommendation (X) to Policy (Y) causal mechanism diagram above.
The context in which this causal mechanism operates is the presence of a sub-optimal condition that causes questions to be raised about what should be done, how, when and why. This necessitates a research which serves as a “trigger”, setting the causal mechanism in motion.[12] The strength of the evidence to inform policy is influenced considerably by adherence to the principles of both scientific good practice and democratic representation, which are also contributory factors to the success of agenda setting and framing activities.[13] Agenda setting and framing serve as information based instruments that present Evidence based recommendation in formats that specific target groups in society (individuals, Civil Society, Interest Groups, Legislators) can relate with such that coalitions that support the recommendation begin to build up.[14] Policy options considered are constrained by availability of resources and competing needs. Assuming prior stages are successful, and evidence-based recommendation is now at the table of legislators. Researcher, think tanks, NGO, academic institution whose research the evidence-based recommendation originated from at this point provide cost-benefit analysis of their recommendation in comparison to maintaining the status quo or other options available, paying attention to value for money components, how recommendation meet the needs of a considerable portion of the population and the overall objectives of society.[15] Finally, at the output end of the mechanism is the adoption of selected portions of the recommendation as policy.
These three statements aptly summarize the underlying factors that fuels this causal mechanism of translation and mobilization;
· visibility and how results are framed generates attention,
· facts that are trusted expedites consent,
· catering for collective interests and the presence of consent sets the causal mechanism on its final stage towards policy adoption/change.[16]
Case selection process and rationale for selection
To explore this mechanism, I propose a theory testing case study analysis of this causal pathway. Since X is known, focusing on the causal pathway allows exploration of the causal mechanisms (M) and/or verification of other aspects of chosen cases to confirm that they align with expectations from theory. Furthermore, theory testing case studies help “strengthen or reduce support for a theory, narrow or extend the scope conditions of a theory, or determine which of two or more theories best explains a case, type, or general phenomenon”.[17] A causal case study is organized around a central argument about a change in X that generates a change in Y. While X usually refers to a single factor, it may also include several related factors or even all the causes of Y.[18] In the causal mechanism being explored in this paper X refers to a single factor “Evidence based recommendations” and the causal inference tested is imprecise (e.g., ‘‘An increase in X causes an increase in Y’’).[19] A good case (s) for causal analysis are selected such that they mimic quasi-experimental properties; replicating the virtues of a true experiment by ensuring that variation in X is not correlated with other factors that are also causes of Y (confounders).[20] This ensures that a spurious (non-causal) relationship between X and Y is not generated.[21] Therefore, to explore the causal mechanism in this paper, cases where other causes of Y are present and could serve as confounders would not be selected. For example, an “Evidence based recommendations” for climate change policy that coincides with a period of drought or other climate change disasters or “Evidence based recommendations” for Violence against women policy happening while a Supranational organization the country is a member signs an International Treaty for Violence against women.
The importance of testing the theory behind a causal mechanism in various contexts has been highlighted in the literature. The importance of inclusive as against narrow testing ( which focuses on a specified and near-conclusive procedures for falsification or verification) has also been emphasized.[22] It is argued that theory testing should focus on external validity, on replication of case studies with the purpose of identifying whether previous results extend to new cases and by so doing evaluate the theory’s (mechanism’s) explanatory power and boundaries.[23] When a researcher tests a theory, then propositions, logical conclusions or predictions, are derived from the theory and are compared to observations, or data, in the case.[24] The more often and the more conclusively the theory is confirmed, the more faith in that the theory reflects reality.[25] It is, however, not merely a function of the number of cases observed, but rather the range of characteristics of the units and the range of conditions under observation[26] and selecting cases strategically or purposefully is essential for generalization.[27] Cases are also are selected on the expectations about their information content[28], i.e., their potential for learning[29]. A total of 6 case studies would be selected across time and space to control for temporal or spatial variations in the causal mechanism.
Since our interest is testing the underlying theory behind our causal mechanism, 1,1 cases would be selected to explore this mechanism; cases that have both the cause (X) and outcome of interest (Y) with the goal of determining whether the causal process works as expected within individual observations. This is necessary, so we are not guilty of informed speculation like the doctors who prescribes medicine and when symptoms go away assumes that the medicine worked without analyzing the causal pathway through which the medicine, affected the symptom.[30]
Countries have differing characteristics based on their stage of development. These characteristics have pervading influences that carry on into the evidence to policy space. For example, institutions in richer countries are generally better and stronger than those in poorer ones. With weaker institutions, the incentives to find and use the evidence to improve policies are weaker, if not absent.[31] Studying case studies across countries with different economic characteristics would lend more weight to our case study by making our findings more generalizable.
The Overseas Development Institute, an independent think tank on international development and humanitarian issues, monitors the progress of various evidence to policy initiatives and one of our cases in the LDCs, Rainwater Harvesting in Tanzania,[32] is selected from their database. The search for appropriate 1,1 cases is on-going and would be finalized if this case study is to be implemented.
Conclusion
Governments has a responsibility to utilize resources to serve the common good and meet the most pressing needs. Producers of Evidence based research provide a tool that makes government accountable to this responsibility such that it becomes a collective responsibility if failure or success results. Various organizations are working with Civil Society Organizations (CSO) and Think tanks and other agencies that occupy the forefront of this task to make them credible providers of valid insights, provide the tools they need to properly set agendas and frame issues and be strategic to navigate non-responsive governments so that positive policy change is a reality. This case study research would be instrumental to adding and developing the body of knowledge regarding what works or not to identify strategies that strengthen this causal mechanism.
References
[1] Considine, M., 1994, Public Policy: a critical approach, Melbourne: Macmillan
[2] Colebatch H., 1998, Policy, Buckingham: Open University Press
[3] Matheson, C. 1997. ‘The Premises of Decision-Making within the Australian Public Service’, Australian Journal of Public Administration, 56, 1, pp. 13–24. https://onlinelibrary.wiley.com/doi/epdf/10.1111/j.1467-8500.1997.tb01237.x
[4] Curtain, R. 2001. Good Public Policy Making: How Australia Fares. Agenda: A Journal of Policy Analysis and Reform. Vol. 8. №1. Pp 33–36. ANU Press.
[5] Torjman, S. 2005. What is policy? The Caledon Institute of Social Policy.
[6] Green, D. 2005. The Policy Funnel — a way to sharpen up our advocacy? http://oxfamblogs.org/fp2p/the-policy-funnel-a-way-to-sharpen-up-our-advocacy/, (accessed 18 December 2014).
[7] Goertz, G. and Mahoney, J. 2012. A tale of two cultures: qualitative and quantitative research in the
social sciences. Princeton: Princeton University Press. pg 100.
[8] Glennan, S. 1996. Mechanisms and the nature of causation. Erkenntnis, 44:49–71.
[9] 2010. Ephemeral mechanisms and historical explanation. Erkenntnis 72:251–66.
[10] Craver, C. F. (2007). Explaining the brain: what a science of the mind-brain could be. New York: Oxford
[11] Demeulenaere, P. 2011. Introduction. In P. Demeulenaere (ed.), Analytical sociology and social mechanisms. Cambridge:
Cambridge University Press.
[12] http://www.compasss.org/files/goertz2016-pupv6.pdf
[13] Parkhust, J. 2017. The politics of evidence: from evidence-based policy to the good governance of evidence. Routledge Studies in Governance and Public Policy.
[14] Jones, D. 2018. Getting evidence and policy to speak the same language. https://www.bond.org.uk/news/2018/01/getting-evidence-and-policy-to-speak-the-same-language
[15] Ibid
[16] Mabey, N. 2014. Driving Change and Opportunity through Strategic Influencing. E3G, UK. https://www.e3g.org/docs/Driving_Strategic_Change_-_Westminster_Hub_March_2014.pdf
[17] George, A. L. and Bennett, A. (2005) Case Studies and Theory Development in the Social Sciences, MIT Press, Cambridge, MA.
[18] Gerring, J., and L. Cojocaru. 2016. Selecting cases for intensive analysis: a diversity of goals and
methods. Sociological Methods & Research 45:392–423.
[19] Ibid
[20] Gerring, J. & McDermott, R. 2007. “An Experimental Template for Case-Study Research.” American Journal of Political Science 51:3 (July) 688–701.
[21] Gerring, J. 2016. Case Study Research: Principles and Practices, 2d ed. Cambridge University Press. pg 134
[22] Crabtree, B. and Miller, W. (1999) Doing Qualitative Research, Sage, London.
[23] Yin, R. K. (2014) Case Study Research Design and Methods, Sage, Thousand Oaks, CA.
[24] Cavaye, A. L. M. (1996) “Case study research: a multi-faceted research approach for IS”, Information Systems Journal, 6, pp227–242.
[25] Ibid
[26] Kennedy, M. M. (1979) “Generalizing from single case studies”, Evaluation Quarterly, 3, pp661–678.
[27] Flyvbjerg, B. (2004) “Five misunderstandings about case-study research”, in Clive Seale, G. G., Gubrium, J. F., and Silverman, D. (eds.) Qualitative Research Practice, Sage, London
[28] Ibid
[29] Stake, R. E. (2000) “Case studies”, in Denzin, N. K. and Lincoln, Y. S. (eds.) Handbook of Qualitative Research, Sage, Thousand Oaks, CA.
[30] Goertz, G. and Mahoney, J. 2012. A tale of two cultures: qualitative and quantitative research in the
social sciences. Princeton: Princeton University Press.
[31] Howes, S. Betteridge, A & Sause, L. 2017. Evidence-based policy making in the tropics http://www.devpolicy.org/evidence-based-policy-making-tropics-20170728/
[32] Court, J & Young, J. 2003 Bridging Research and Policy: Insights from 50 Case Studies. Working Paper 213. Overseas Development Institute. pp 6 — Smallholders who farm in the tropical drylands of Tanzania must cope with two realities of their natural environment: they live in dry places and their rainfall is unreliable. Both have negative impacts on human well-being, livestock condition, the productivity of crops, and livelihoods. While policymakers recognize this, and few doubt the critical importance of rainfall, they have not previously considered the importance of using run-off for increasing water supply. Indeed, the overriding view was that run-off is a hazard rather than a resource and leads to soil erosion. This perception has driven government policy for many years. The research and communication activities of an inter-disciplinary research team over a 10-year period, led by the Sokoine University of Agriculture, has informed policymakers at national and district levels on the benefits of managing rainfall by harvesting run-off. Interestingly, the adoption of Rain Water Harvesting (RWH) by smallholder farmers in Tanzania is significant and began well before an enabling policy for its promotion by planners and development workers was in place.