Why is public policy implementation so difficult?

This is the first of my posts solely on public policy, and it focuses on the end stage of public policy formation, the implementation stage. The implementation stage of the policymaking process is arguably the most important, this paper highlights why and also lists the core issues that make policy implementation so difficult.

(This essay formed part of my Masters of Science in International Public Policy)


The importance and complexities of policy implementation are well known in the 21st century, however, this has not always been the case. Policy analysis in the early 1960’s focused primarily on the ‘front end’ stages of the policy process, with implementation largely a neglected, background issue. By drawing attention to policy failures at the implementation stage, it was hope that policymakers could learn from them. This essay will answer the question as to why policy implementation is so difficult, using the Economic Development Administration (EDA) project in Oakland as a central case study. This essay will also draw on additional cases in Nigeria and the UK. First, this essay will outline three definitions, beginning with Pressman and Wildavsky’s, before moving on to Mazmanian and Sabatier’s top-down perspective, finishing with Van Meter and Van Horn’s bottom-up definition. This essay will seek to stress the importance that definition plays in the implementation process, particularly at ‘street-level’ where discretion or principal-agency is often a feature of implementation. The varied characteristics of top-down and bottom-up implementation will be explored and the roles they have in the early and late stages of the policy and implementation process will be analysed, assessing the advantages and disadvantages of both perspectives. An important central question of this essay is; to what extent are formal institutions and public policies also affected by path-dependence and what does this mean for implementation? The inherent problems of measuring progress, and factors that are outside a decision makers control will be highlighted, along with the solutions that New Public Management (NPM) can offer. Finally, this essay will conclude with some core issues that make implementation so difficult and some concise lessons for effective implementation on a broad range of policies. Whilst highlighting this is not an exhaustive list of reasons, and some cases fail at the implementation stage for varying reasons, highlighting the fact that not all cases are the same.

Definitions are critical to understanding the attributes of each perspective. Implementation is most commonly described as the process of turning policy in practice. How a bureaucrat or decision maker begins and carries out the process of implementation largely depends on how they define implementation, or to which perspective they ascribe to. The first definition this essay considers is that of, Pressman and Wildavsky defined implementation as ‘to carry out, accomplish, fulfil, produce, complete’ (Pressman and Wildavsky, 1984, p.xxi). They also point to an important question; how do we distinguish between a policy and its implementation? (Pressman and Wildavsky, 1984, p.xxi). This definition expresses the role of public servants in the government system to provide efficient and just services to citizens. A second definition is that of Mazmanian and Sabatier who offer a top-down perspective by stating that ‘Implementation is carrying out of a basic policy decision, usually incorporated in a statute but which can also take the form of important executive orders or court decisions’ (Mazmanian and Sabatier, 1983, p.21). Finally, Van Meter and Van Horn from a bottom-up perspective describe implementation as ‘Those actions by public or private individuals (or groups) that are directed at the achievement of objectives set forth in prior policy decisions’ (Meter and Horn, 1975, p.447-8).

Policy analysis neglected the ‘rear end’ of policy-making (the implementation stage) with an over focus at the top end, which brought many problems to the fore such as; methodological problems, and problems about the extent to which the very practical concerns of implementation studies may involve, explicitly or implicitly, including identification with actor’s views of what should happen (Hill and Varone, 2017, p.228). For deLeon & deLeon (2001), this ‘first wave’ of implementation focused solely on addressing the gaps between policy formulation and execution (deLeon and deLeon, 2002). Which were only partially successful. Sarbaugh-Thompson and Zald (1995) indeed argued that these first-generation studies which saw implementation success or failure, as a function of flawed, or imperfect primary legislation and a failure of bureaucratic compliance (Schofield, 2001, p.249). The top down approach itself is built on two related assumptions: Descriptive, where decisions are made at the ‘top’ (by central government or the legislature) then are carried out at the ‘bottom’ (by implementing organisations (Cairney, 2012, p.37), and prescriptive, which holds that decisions should be made at the top and carried out at the bottom (Cairney, 2012, p.37). There was a presence of significant implementation gaps in both perspectives, implementation gaps can be identified by the differences between the expectations of policymakers and the actual policy outcomes (Cairney, 2012, p.34). The presence of these gaps has led to the deepening of implementation studies, and a combination of the two perspectives later in the 1990s.

In their formative publication in the early 1970s, Pressman and Wildavsky accepted that the policy process was unidirectional, and they drew conclusion that policy was formulated by policymakers at the top and then carried out by bureaucrats at implementation stage. Pressman and Wildavsky identified that implementation of the EDA would be difficult right from the beginning. Key issues involved the emergence of unexpected decisions, and the necessity of two decision paths in order to achieve the EDA program goals; and within each path, the agency found that the number of decisions and clearances required was constantly growing (Pressman and Wildavsky, 1973, p.112). These additional issues required decisions had not initially been a part of the EDAs plans, policymakers had only one goal, to reduce unemployment. The two decision paths required; financing the construction of the works, and developing a hiring plan that involved African-American workers. Even if these two decision paths had been included in the original policy, it is still unclear as to whether the failure of the project could have been avoided. In this case, there was no study or inclusion into the policy implementation process of bottom-up actors, even the factoring of unintended consequences as Pressman and Wildavsky have discussed was limited in the initial policy. In addition to these concerns, the workers consulted in this process would become active actors themselves, and the implementers of policy would be using their discretion. Whilst Browne and Wildavsky (1983) later recognise multi-causal interactive patterns (Browne and Wildavsky, 1983, p.233). Discretion is not fully studied in their work at the time.

The second case considered is that of the National Policy on Education which was formed in Nigeria in 1977, this case set in a developing country is in stark contrast to other cases put forward in this essay. Dr.Ikechukwu and Chukwuemeka identify the obstacles to effective implementation in Nigeria in their paper titled, the obstacles to effective policy implementation by the public bureaucracy in developing nations: The case of Nigeria. The National Policy on Education was formulated to achieve the objective of acquiring appropriate skill and competence, both mental and physical, as equipment for an individual to live in and contribute to the development of Nigerian society (Ikechukwu Ugwuanyi and Chukwuemeka, 2013). Whilst Nigeria has indeed made vast progress in terms of the number of children in primary education UNESCO concluded that more remains to be done in both quantity and quality (Irigoyen, 2017). The initial formulation process in 1977 was largely focused on the top end of the implementation process, which was determined by previous educational policies and culture, this path dependency has continued into the present. UNESCO found that the quality of the national school curriculum is undermined by the generally low quality of teachers who implement it, which translates into low levels of learning achievement (Irigoyen, 2017). Similar to the Oakland case study, stakeholder and was found to be weak. There was no evidence of communication or consultations between institutions at the federal and state level in the design of the Universal Basic Education programme, which led to misalignment later in the implementation phase (Irigoyen, 2017).

Inferior communication and consultation between the federal and state institutions, creates problems with funding allocations internally from state to state. As is the case in Oakland there is a sizeable implementation gap between what was formulated, and what was implemented. Such a case differs markedly with the two others depicted in this essay. The locality, culture, political, economic, and social situations in any country must be taken into account in the policy process, particularly at the implementation stage. For this reason, comparing developing countries with those developed is not appropriate.

Hindering implementation was the National Policy on Education is over focus on state agencies, which do not possess the requisite manpower and financial resources to effectively implement them. Despite the teachers being implementers of the policy, there was no initial consideration to include them in the implementation process. However, finances could play an influential role here. As Ikelegbe (2006) and Dick (2003) state that the national government, sometimes, do not budget adequately, not enabling the public bureaucracy to properly implement formulated policies (Ikechukwu Ugwuanyi and Chukwuemeka, 2013). The reason for this lack of bottom-up consideration could in part be down to how the Nigerian government defines public policy implementation, and to the extent Nigerian institutions are path dependent. Nigeria having only gained independence from Britain in 1960, took time to develop its own educational system and faces a number of political and economic challenges. Prior to 1977, Nigeria operated an educational policy inherited from Britain at independence (Irigoyen, 2017). Under colonial rule the educational policy of Nigeria was fixed, with very limited interaction from those implementing the policy. Path dependence holds that when a commitment has been established and resources devoted to it, over time it produces ‘increasing returns’ and it becomes increasingly costly to choose a different path (Cairney, 2012, p.127 citing Pierson, 2000). This is clearly demonstrable in this case, with policymakers often unwilling or unable to diverge.

By focusing on implementing policy from the top-down, the Nigerian government has neglected to factor in discretion and ambiguity that street-level bureaucrats such as teachers, and head teachers implement. The National Policy on Education is exceptionally ambiguous with regards to how bureaucrats at the base level should implement policy. Irigoyen, argues that there is no clarity on the management structures and guidelines for the programme’s delivery (Irigoyen, 2017). This is a considerable problem when implementing policy in such a location, although additionally the challenge of keeping away personal interest, prejudice and the influence of early values in the conduct of official business by bureaucrats is equally very critical in Nigeria (Ikechukwu Ugwuanyi and Chukwuemeka, 2013). Self-interest can also affect policy at any stage, as Dr.Ikechukwu and Chukwuemeka note it is observable that each new political leadership in Nigeria is usually and primarily concerned with making its own impression on public programmes and projects (Ikechukwu Ugwuanyi and Chukwuemeka, 2013). Nigeria also operates under ineffective and corrupt political leadership (Ikechukwu Ugwuanyi and Chukwuemeka, 2013). Leadership corruption, and ineptitude, for instance, affects the content and quality of policy at formulation stage (Ikechukwu Ugwuanyi and Chukwuemeka, 2013). If policy is botched at the formulation stage a successful implementation is even less likely, regardless of how involved bottom up actors are. All such considerations make this particular policy hard to implement in Nigeria.

If a policy passes through the formulation stage effectively. Many bottom up theorists would then argue that if local level implementers are not given the freedom to adapt to local conditions, then it is likely that the policy will fail at the implementation stage (Matland, 1995). Matland’s statement is case dependent. This freedom to adapt would only come from implementers at the base being fully included into the implementation process with low-ambiguity. Implementer’s face two important issues that they may have to use their discretion to tackle in this case or may have been using; social and cultural barriers are hindering both female participation and the lack of enforcement of the UBE Act 2004 on enrolment and retention (Irigoyen, 2017). Elements such as the lack of basic training, the lack of communication, coordination and consultation with these base level implementers led to low levels of learning and achievement, and high levels of discretion as implementers have tried to offset these problems with little governmental oversight. Low-ambiguity is essential to effective implementation in this case. It is worth noting that implementers may also suffer cognitive limitations that decision makers do. When they become the implementers they become decision maker, the implementer may suffer bounded rationality.

In the 1998 edition of the National Policy on Education it gave specific provision for policy makers and implementers, recommending the establishment of the State Agency for Mass Education (SAME) (Irigoyen, 2017). This specific provision was expected to provide some solution to these issues and lower ambiguity at street level. However, the culture and path dependency within schools and other Nigerian institutions, mean the increasing governmental oversight and lower policy-means ambiguity are likely the only options to correct the government’s initial failure in implementing this particular policy at its initial implementation. This is an example of path dependence hampering future policy amendments, history matters in implementation, this is especially true of such inceptive highly ambiguous policies. Irigoyen finds the structural alignment of institutions at the national level is not strong, with several organisations overlapping, or with unclear management objectives (Irigoyen, 2017). With these unclear management objectives and increasingly overlapping organisations ambiguity will be high.

Self-interest is present at every stage of the policy process. Politicians may want to be seen to be in favour of certain ideals or goals while actually doing nothing about them (Hill and Varone, 2017, p.232). As highlighted by both the Oakland case study and the National Policy of Education case of 1977 in Nigeria. The self-interests of the principal (elected officials and institutions) and the agent (the bureaucrat, who is tasked with implementation) are potentially divergent. This is the case in Oakland and even the case of rail privatisation in the UK. Where actors (British Rail managers) opportunity networked in the implementation process in order to secure additional resources. As Grantham states these managers, after early objections to the form of the privatization, then collaborated fully in the process with some of them becoming wealthy as a result (Grantham, 2001). Where the resource is financial, actors manoeuvre to secure those additional resources for themselves subject to an evaluation of transaction costs (Grantham, 2001). As implementers of passenger transport authority policy, the managers were low-discretion actors in administrative networks, however their own financial knowledge and informational resources were substantial (Grantham, 2001).

Later academic studies have avoided taking either extreme positions. Saetren (2014) points to the ‘the gradual silencing of the protracted and unproductive debate between top-down and bottom-up scholars’ (Saetren, 2014, p.95). It is this later position that the UK case of rail privatisation takes up, in many ways it is more of a balance between top-down and bottom-up perspectives. New Public Management (NPM) sought to tackle the implementation ‘gaps’ in the public sector, which the Oakland and Nigeria case are examples of. NPM rose to the fore in the early 1980s under UK Prime Minister Margaret Thatcher and US President Ronald Regan. The central state was perceived as ‘overloaded’, monitoring, oversight and performance management (through the use of targets) needed to be increased, and the discretion of public sector professionals had to be reduced. The promotion of internal markets and term contracts are a feature of this perspective, which are justified as having ‘rivalry as the key to lower costs and better standards; contracts as the key to explicating performance standards’ (Hood, 1995, p.96). Whereas traditional top-down and bottom up frameworks tend to give only a partial explanation of outcomes, NPM sought to measure and monitor outcomes (Grantham, 2001). NPM provides solutions to inherent problems of measuring progress that earlier perspectives tended to neglect.

For Grantham, the successful delivery of this policy depended largely on activity located in a series of distinct implementation networks, they were only a feature of the transition to privatisation (Grantham, 2001). The NPM framework was evident through the implementation period, with grants and direct handouts from the Treasury. The establishment of competitive contracts for private rail operators (a public service), was expected to lower costs simultaneously and was the core perception of improved policy. For NPM supporters, the use of performance indicators, targets and new technology allow for more even more effective implementation. However, NPM does have significant disadvantages, for which the paradigm faces significant criticism for at present. One is that public servants are wasteful, to which opponents of NPM blame accuse it of demoralising the public sector. Arguments are made by supporters that stress the ‘need to cut direct costs, raise labour discipline, and to do more with less’ (Hood, 1995, p.96). Whilst offering an alternative at present it is a deeply contested paradigm, for varying reasons that are beyond the scope of this essay.

One key theme present in all three cases presented is that of unintended outcomes. This is true of the implementation process as a whole. As former US Secretary of Defence stated ‘we also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones’ (Federal News Service Inc., 2002). The task for policymakers is to anticipate and prepare for unintended outcomes in the early stages of the policymaking process. Implementation failure from unintended outcomes can be mitigated by learning from similar past cases, and anticipation by having funds, and manpower aside for any unintended consequence. For example, an unintended consequence of the later 2004 over focus on secondary education and skills for the workplace led to the decrease by 4% for primary level enrolment education in Nigeria (Irigoyen, 2017).

It is often very difficult to know if a policy failure is due to poor policy formulation, to an imperfect policy implementation or to a subtle mix of both elements (Hill and Varone, 2017, p.229). Hogwood and Gunn (1984) attribute implementation failure to three main factors: bad execution, bad policy, and bad luck (Hogwood and Gunn, 1984, p.197). Cairney (2012) suggests that the implementation gap is wide because participants have unrealistic expectations (Cairney, 2012, p.34). This essay has so far shown that most of the statements are correct, however, there are many variables that cause implementation to fail and these are largely case dependent.

This essay has shown that the distance between, implementers and the policy formulators is critical to successful implementation. The deeply complex and contested policy arena makes it harder to design effective policy, this bears directly into the implementation process. Politics, culture, economics, location are huge factors that influence the implementation process even in relatively stable developed countries, output is unpredictable.

It is important to note that this essay is only outlining three case studies. Each case involves different political incentives, goals, and expectations, amongst a wide array of other factors. Each of the different variables affect implementation cases to a greater or lesser degree. Most implementation studies are Western focused, this essay has tried to buck that trend with the inclusion of Nigeria. As the Nigeria case study proves path dependence, locality and culture play a significant part in determining whether the policy goals are met. Implementers may also suffer the issue of bounded rationality.

Self-interest and discretion affects implementation in the three cases in this essay. It is reasonably safe to assume these two variable have to be taken into account in all cases. NPM provides an opportunity for performance measurement that other perspectives do not, however, it does have its own disadvantages. This essay has come to the conclusion that learning from all perspectives is the best solution, as is finding a balance. Whilst it is clear that implementation contains many pitfalls, they can be mitigated against.

References

Cairney, P. (2012) Understanding Public Policy, First. Hampshire, Palgrave Macmillian [Online]. (Accessed 25 October 2017).

deLeon, P. and deLeon, L. (2002) ‘What Ever Happened to Policy Implementation? An Alternative Approach’, Journal of Public Administration Research and Theory: J-PART, vol. 12, no. 4, pp. 467–492.

Federal News Service Inc. (2002) Defense.gov Transcript: DoD News Briefing – Secretary Rumsfeld and Gen. Myers [Online]. Available at http://archive.defense.gov/Transcripts/Transcript.aspx?TranscriptID=2636 (Accessed 27 February 2018).

Grantham, A. (2001) ‘How Networks Explain Unintended Policy Implementation Outcomes: The Case of UK Rail Privatization’, Public Administration, vol. 79, no. 4, pp. 851–870 [Online]. DOI: 10.1111/1467-9299.00283.

Hill, M. J. and Varone, F. (2017) The Public Policy Process, Seventh. London, New York, NY, Routledge, Taylor & Francis Group.

Hood, C. (1995) ‘Contemporary public management: a new global paradigm?’, Public Policy and Administration, vol. 10, no. 2, pp. 104–117 [Online]. DOI: 10.1177/095207679501000208.

Hogwood, B. and Gunn, L. (1984) Policy Analysis for the Real World (Oxford: Oxford University Press).

Ikechukwu Ugwuanyi, B. and Chukwuemeka (2013) ‘The Obstacles to Effective Policy Implementation by the Public Bureaucracy in Developing Nations : The Case of Nigeria’, Kuwait Chapter of Arabian Journal of Business and Management Review, vol. 2, pp. 59–68 [Online]. DOI: 10.12816/0001218.

Irigoyen, C. (2017) ‘Universal Basic Education in Nigeria’, Centre for Public Impact [Online]. Available at https://www.centreforpublicimpact.org/case-study/universal-basic-education-nigeria/ (Accessed 27 February 2018).

Matland, R. E. (1995) ‘Synthesizing the Implementation Literature: The Ambiguity-Conflict Model of Policy Implementation’, Journal of Public Administration Research and Theory: J-PART, vol. 5, no. 2, pp. 145–174.

Mazmanian, D.A. and Sabatier, P. (eds.) (1981) Effective Policy Implementation (Lexington: Lexington Books).

Meter, D. S. V. and Horn, C. E. V. (1975) ‘The Policy Implementation Process: A Conceptual Framework’, Administration & Society, vol. 6, no. 4, pp. 445–488 [Online]. DOI: 10.1177/009539977500600404.

Pressman, J. L. and Wildavsky, A. (2018) Implementation – How Great Expectations in Washington Are Dashed in Oakland; Or, Why It’s Amazing that Federal Programs Work at All, This Being a Saga of the Economic Development Administration as Told by Two Sympathetic Observers Who Seek to Build Morals on a Foundation of Ruined Hopes – The Oakland Project,.

Saetren, H. (2014) ‘Implementing the third generation research paradigm in policy implementation research: An empirical assessment’, Public Policy and Administration, vol. 29, no. 2, pp. 84–105 [Online]. DOI: 10.1177/0952076713513487.

Schofield, J. (2001) ‘Time for a revival? Public policy implementation: a review of the literature and an agenda for future research’, International Journal of Management Reviews, Vol. 3 (3), pp. 245-263.

====

(This essay formed part of my Masters of Science in International Public Policy)

Evidence based policy making – A discussion

This is an edited article, I originally authored this as part of an MSc in International Public Policy.

Title: Policy making is all too often more about policy-based evidence making rather than evidence-based policy making
————-

An Introduction
Many imperatives can lead to the utilisation of policy-based evidence (PBE), rather than evidence-based policy making (EBPM) of which this article will seek to explore. Numerous studies have been produced on whether EBPM exists as just figure of speech, or as part of a rational process.

Whilst covering significant ground, this is by no account a fully comprehensive analysis, which would go beyond the scope of this article.

Split into four sections, this article will first examine fact interpretation, and the role this plays in the discourse. Second, the use of policy learning through evaluations, and randomised controlled trials (RCTs) as an instrument for better policy outcomes will be examined. Third, the way policymakers and decision makers understand problems and establish evidence for solutions with the influence of ideology, along with the always present bounded rationality will be discussed. The role ambiguity plays in the EBPM discourse will be highlighted. Finally, an introduction to behavioural public policy will be presented, specifically that of nudges including that of an innovative policy instrument namely payments by results (PbR) and the contribution they create to the discussion.

Many variables can influence a policymaker’s decision making, the main variables will be noted and discussed were possible. However, this article also recognises that there are ever more variables yet to be researched in an increasingly complex world. The principal case study utilised through this article is that of RCTs in the English school system. Consequently, this article will conclude by making an argument that policy-based evidence making is all too prevalent.

Defining and exploring ‘Evidence-based policy making’
Evidence-based policy making is a deeply ambiguous and contested metaphor, differing significantly to the term policy-based evidence. For proponents of the former, the metaphor “evidence-based” suggests that scientific evidence comes first and acts as the primary reference point for a decision (Cairney, 2017). Whilst the underlying definition may be reasonably clear for proponents of EBPM, for its many detractors this provides a lens of “rational” policy making that does not exist in the real world.

For the critics of EBPM, policy-based evidence provides an descriptive and prescriptive lens allowing for the description of anything from a range of mistakes by policy makers – including ignoring evidence, using the wrong kinds, “cherry picking” evidence to suit their ideologies, agendas, and/or producing a disproportionate response to evidence – without describing a realistic standard to which to hold them (Cairney, 2017). For instance, is policy more evidence-informed then based? The definitional problem for EBPM is never really solved, ambiguity always remains over what it is, and each actor defines it differently.

Fact or opinion

Fact Interpretation: Seperating fact from opinion

Fact Interpretation:
Fact or opinion, clear or muddy?

Muddying the waters to any policy that aims to be more evidence-based is that facts can sometimes be misinterpreted, either intentionally or unintentionally.

As Kisby puts it facts are framed as evidence to support or contradict a particular theoretical claim or hypothesis (Kisby, 2011). Evidence itself can be politicised. Trends do not all go in one direction with governments willing to politicise in line with ideological and political agenda (Kisby, 2011).

Meaning is inscribed through, but also derived from use of evidence, which may be open to a wide range of plausible interpretations (Kisby, 2011). Assumptions based on models and politicisation determine what constitutes as evidence, and in addition with politicisation and assumptions comes hierarchy. The selection of a preferred piece of evidence over another significantly alters the course of a policy at various stages in the policy making process.

Credits to Bilal et al., https://bit.ly/3kbgh33

The role of New Public Management
The rise of New Public Management, ushered in policy learning through evaluations which became commonplace throughout 1980s and 1990s, primarily focusing on the realist method-based evaluation approach. Government ministers have increasingly made use of the term ‘what matters is what works’; with evidence of what works provided through substantially increased research and evaluation programmes and the use of pilot projects to test out new approaches (Sanderson, 2001).

Policy learning through evaluations, and randomised controlled trials (RCTs)
Programmes such as the single work-focused gateway, Health Action Zones (HAZs), and Sure Start were all subject to initial piloting (Sanderson, 2001). Sanderson states that evaluations have been set up to answer two key questions: first, ‘does it work’; and second, ‘how can we best make it work?’ (Sanderson, 2001). The constraints on the ability of evaluations to provide a suitable answer to the first question are that of time, and imperfect information. For example, it can take considerable time for pilot projects to become fully established (Sanderson, 2001).

The political constraints: Being politically bound
Political considerations can constrain the length of time for pilot programmes to operate effectively (Sanderson, 2001). Policymakers may be eager to implement policies for this constraint. When initiatives arise from political manifesto commitments policymakers are impatient to receive results that will provide evidential support for decisions to proceed with full implementation (Sanderson, 2001). Such issues can lead to evidence being misinterpreted, or undue pressure applied to departments working under government producing the evidence.

The importance of neutral (ideologically free), independent institutions comes into its own here as the opportunity for evidence-based policy can be degraded. The case for increased methodological pluralism in policy analysis is strengthened with these considerations.

Understanding problems and establishing evidence
Rationalist approaches suggest that the quality of the evaluation itself should lend itself to evaluation use (Stewart and Jarvie, 2015). However, other more critical approaches stress that policy making is a process based on argumentation, and that evaluation use is best viewed interactively (Stewart and Jarvie, 2015). There is some evidence to support this argument, two-way policy transfer occurs in most policy processes.

Networks are key vehicles for interactive evaluation, while the state is key in transfers it by no means the exclusive actor. International organisations, bureaucrats, non-state actors, and in particular knowledge-based actors are all involved in the export of ideas (Stone, 2001). However, as Stone (2001) argues, the inclusion of think tanks, consultants or other expert groups into a global policy network, ‘doesn’t depend so much on its innovative ideas, but rather upon whether or not it shares a common value system with the government or international organisation that desires the policy transfer to occur’ (Stone, 2001). Key constraints based outside a policymaker’s control may come from outside the state, thus the rationalist approach is inadequate and ignores the realities of politics.

Quantitative data has increasingly been used to inform policy decisions, most notably in the UK education system. Its use has increased significantly since the Conservative-Liberal Democrat coalition came to power.

Reducing ambiguity: A role for Randomised controlled trials (RCT’s)
RCTs are one policy instrument that can be used to reduce ambiguity at a critical phase in the policy process and has been done so with some success. Two notable examples of this success are that of the Education Endowment Foundation (EEF) which has made significant use of RCT’s, commissioning 10% of all randomised controlled trials ever carried out in education research (The Economist, 2017). The other being the UK cabinet Office’s Behavioural Insights Team (BIT). John (2014) highlights a key factor in the BIT’s success is its willingness to work in a non-hierarchical way, its skills at forming alliances, and the BIT’s ability to form good relationships with expert audiences (John, 2014). The two organisations present themselves as evidence-based. Making high use of RCTs and networking effectively can be seen as an example of their organisational confidence in obtaining knowledge that can alter behaviour, certainly, knowing evidence produced will be at the top of any hierarchy is a motivating factor.

Results from RCTs are often referred to as scientific, non-biased and rational by proponents of the method (Wesselink et al., 2014). Although the BIT works in a flat, non-hierarchical way, evidence produced was naturally at the top of the government’s hierarchy up to 2015. Such a hierarchies is created for evidence producing institutions to solve what many studies predominantly focus on, which is; identifying the problem of uncertainty and incomplete information (Cairney et al., 2016).

However, this hierarchy is partly artificially created, the EEF and BIT have had significant government involvement. Raising the question of how much they benefit over other forms of evidence, which may be more reliable but further down the hierarchy, thus widening the divide further between evidence and policy.

Issues around the implementation of RCTs exist, such as randomisation and poor study design. Terry Wrigley, a visiting professor at Northumbria University and Professor Biesta of Brunel University London accuse the EEF’s toolkit of “bundling together” very different studies, carried out in a range of differing contexts, to come up with the average number of months of progress gained, when some of the studies contradict the headline statistic (Smith, 2017). In addition to these issues of randomisation, ideological agendas also feature.

Libertarian paternalism holds that governments ‘nudge’ citizens into making more rational choices (Thaler and Sunstein, 2008, p.47). Nudges and behavioural policies are known as “soft paternalism” with the aim to not upholding the “nanny state”, which is of appeal to both libertarians and conservatives (The Economist, 2014). The BIT which was established under the Conservative-Liberal Democrat coalition, was initially created in June 2010 as a unit within the Cabinet Office (John, 2014). The institutional norms created by the establishment of the BIT, and the large amount of work it does with central government departments means it can hardly be termed as neutral ideologically (John, 2014). Furthermore, Policymakers may be resistant to innovation and new policies because they may be associated with younger post holders whose careers might benefit (John, 2014).

Nevertheless, it would appear the UK is pushing and adopting evidence from RCTs and behavioural studies more so then other countries. Van Deun et al (2018) confirm this explaining comparatively wider attention within academia and the government for the subject, and the establishment of ‘nudge units’ in both countries (Van Deun et al., 2018). They also state that worldwide 40% of the articles are linked to health policies and almost 20% relate to environmental policies, with around 2% on education (Van Deun et al., 2018). For instance, EEF claims it has commissioned 10% of all randomised controlled trials ever carried out in education research, with a third of English state schools involved in at least one of its trials (The Economist, 2018). The UK is taking the lead in certain sectors of behavioural policy research.

Interestingly, Van Deun et al point to the small size of field experiments, which seems to suggest researchers are only limitedly involved in applying nudges in public policies and in testing the effects of interventions (Van Deun et al., 2018). Naturally, a question as to who is applying nudges in public policies, and who is testing the effects of interventions comes about? However, it is noted that the application of nudges in public policy is limited, and nudge policies do still constitute a new field of study. The UK is also pioneering research in education; however, it lacks any comparison. Lack of comparison or choice in this sector is a significant argument against EBPM.

An image explaining bounded rationality

A explanation of Bounded Rationality courtesy of tutor2.net

Bounded rationality: Are policymakers bound?
The way policymakers understand problems and establish evidence for solutions is a key factor in determining whether policy is evidence informed or driven by policy itself. In formulating policy there are significant limits to a policymaker’s attention, ‘solutions’ often exist before ‘problems’ and human knowledge is necessarily incomplete and fragmented (Parsons, 1995, p.278). Often policymakers are satisficying rather than maximising, fallible, in an open setting, with ambiguous scope, imperfect information, and with limited time (Simon, 1965, p.45). A policy maker often is operating in a world of bounded rationality which means they are bounded due to social differentiation. Problem interpretations are differentiated, for example, there are situations where clients can perceive a problem differently to staff, and the decision maker can appreciate aspects of a situation that either clients or staff do (Forester, 1984).

Information is never perfect…
In this situation, information is not only imperfect, but it is of highly varying quality, in equally varying locations, with less than simple and direct accessibility (Forester, 1984). As time is a socially variable resource, different actors have different amounts of it to devote to decision or problems at hand (Forester, 1984). Satisficying is no longer adequate in this situation, it needs to be supplemented by the use of social intelligence networks where decision-making, and evidence is produced in a socially differentiated environment. Satisficying, which is selecting evidence that is suboptimal to form a policy over the optimal solution poses a challenge to EBPM. This may have occurred in the trial of signature placement by a local authority that did not confirm Dan Arieley’s argument that signatures at the top of documents increased compliance (John, 2013).

The vast amounts of data collected which is required to nudge an individual in a chosen direction should mean evidence-informed or evidence-based policy should be a given. As what is learned about individuals is fed into other policy amendments, however, as depicted policy learning is complicated, with bounded rationality being just one of the many complications. Thaler and Sunstein explain that as alternatives increase and/or vary on more dimensions, policymakers are likely to adopt simplifying strategies (Thaler and Sunstein, 2008, p.232).

Increased data means more choices. Thus, choice architecture matters to policymakers when formulating a policy, as good choice architecture will provide structure and structure affects policy outcomes (Thaler and Sunstein, 2008, p.92). Whilst it may be true that evidence-based/informed policy will increase with more choice, critically, we know in the real world a policymaker is bounded. So, the question comes to what extent is the policy evidence-based? And as more evidence is presented to the policymaker, might this in fact degrade real evidence-based policy? These are the important pressing questions.

An introduction to behavioural public policy: Payment by results (PbR)
PbR is another innovative policy instrument, with PbR payments are dependent on independent verification of results. Payment by results allows the government to pay a provider of services on the basis of the outcomes their service achieves rather than the inputs or outputs the provider delivers (Puddicombe et al., 2012). The definition of what a result consists of is a key issue, as it defines what type of evidence constitutes as an acceptable result.

In the follow up of their seminal paper titled ‘Payment by results and social impact bonds in the criminal justice sector: New challenges for the concept of evidence-based policy?’ (Fox and Albertson, 2011). Fox and Albertson state that the Ministry of Justice notes there are several measures of re-offending which might be taken as indicators of success of a PbR scheme (Puddicombe et al., 2012). Fox and Albertson go on to claim that to date a simple, binary measure of proven re-offending seems to be their preferred option (Puddicombe et al., 2012). There is no reference as to how the Ministry of Justice chose this measure from other measures that were available to them. Nevertheless, as discussed previously, an adequate choice architecture provides efficient structure.

It can also be argued that such a definition ignores the complexity of individual’s journey to desistance, which might involve initially moving to less serious or less frequent offending (Puddicombe et al., 2012). Such definitions that are headline targets can be counterproductive, such as in education and the health sector for example, and other sectors where success is not easily measured (Puddicombe et al., 2012). These drawbacks suggest simply shifting emphasis from inputs to outcomes is not the solution to this problem. In addition, choice and hierarchy issues are still present and still a factor for a decision maker in pursuit of a more evidence-based policy.

Considerable research has been undertaken on efficiency and savings. Albertson and Fox point out that the formal evidence base and the personal experience of practitioners suggest modest reductions in re-offending are more likely (Puddicombe et al., 2012). Savings may be too small to identify with such an instrument being implemented. As much as the economics may be called into question, so too the social effects, and the unintended consequences. Corry points to the strengths of PbR, that a provider of services will go the extra mile to try and hit outcome targets (Puddicombe et al., 2012). However, this can also lead to undesirable and/or unexpected behaviours (Puddicombe et al., 2012). Pointing to the temptation to ‘park and cream’, providers may work hardest with those that they think they can get over the line and so achieve outcome targets and associated payments (Puddicombe et al., 2012). This behaviour can be desirable and comes naturally but it can mean for example that those offenders who need help are ignored. The real setback is that prime contractors (often for-profit organisations) will pass the toughest cases on to charity sub-contractors (Puddicombe et al., 2012).

The temptation to alter figures is also present. If top-down pressure is applied in an organisation to hit specified targets, then actors in the process may be tempted to make sure data (evidence) says what those at the top of the hierachy want to hear. The answer to such a problem, is openly accessible data. Corry states the more publicly available the less room for this to happen, as people can monitor and pick up on fraud more easily (Puddicombe et al., 2012). Overall, the focus on improving efficiency and cost cutting could push governments to use PbR, especially a government driven more by ideology.

If a government is driven significantly by ideology and has tough manifesto commitments it intends on following through, then it is doubtful that this would lead to evidence-based/evidence-informed policy outcomes. A present example is that of the UK’s withdrawal from the EU, where the UK government is pursuing a policy that most sources state will leave the UK worse off economically (Crerar, 2018). However, the present government is following through on the 2016 June Referendum with government ideology and interpretation, playing a significant role in what sort of result the UK is likely to have. It can certainly be said that examples of policy-based evidence making exist at the local, national, and international sphere.

What this discussion presents
This article has shown that evidence-based policy making is an extremely contested arena, with no consensus on any definition. This leaves a tricky starting point on which to discuss what evidence-based policy making really is. Evidence-based policy making as a metaphor has many detractors. Arguing that it provides a lens “rational” policy making that does not exist in the real world, to the question of whether policy is more evidence-informed then based.

Framed, or unintentionally misinterpreted further adds to the contestation. Governments politicise, individuals and groups have agendas, and ideologies in the evidence “soup”. Assumptions based on models and politicisation determine what constitutes as evidence, and with such politicisation and assumption comes hierarchy.

This article has shown that policy learning, through evaluations is increasingly complex. By focusing on the realist method-based evaluation approach, and the phrase ‘what matters is what works’ the use of RCTs as a means of generating evidence has become more common. However, this article has highlighted that political considerations can constrain the length of time pilot programmes operate considerably. The importance of independent institutions cannot be underestimated in such instances, as is its effect on evidence produced.

The increased use of RCTs in most nudging and payment by results implementations are all heavily dependent on structured evidence. The increased use of RCTs can in some instances allow for greater evidence-informed policy and delivery, however, this article has argued that hierarchies can influence a decision maker negatively. RCT evidence is generally at the top of any government hierarchy and is often selected for use first because it is seen as reliable, without taking into account how the evidence was produced or whether the organisation benefited from significant government involvement. RCTs are also open to poor design and issues with randomisation are present, how “random” are they for instance.

Nudging and PbR present opportunities for more evidence-informed policy. As the push for evidence to hit specified targets is necessary, however, ideology and interpretation can influence PbR implementation and outcomes. However, this article has shown the push for comprehensive rationality is not achievable. In addition, it has been demonstrated a decision maker is always bound in the real world.

References
Cairney, P., Oliver, K. and Wellstead, A. (2016) ‘To Bridge the Divide between Evidence and Policy: Reduce Ambiguity as Much as Uncertainty’, Public Administration Review, vol. 76, no. 3, pp. 399–402 [Online]. DOI: 10.1111/puar.12555.

Crerar, P. (2018) Each Brexit scenario will leave Britain worse off, study finds | Politics | The Guardian [Online]. Available at https://www.theguardian.com/politics/2018/apr/18/each-brexit-scenario-will-leave-britain-worse-off-study-finds?CMP=share_btn_link (Accessed 23 April 2018).

Fox, C. and Albertson, K. (2011) ‘Payment by results and social impact bonds in the criminal justice sector: New challenges for the concept of evidence-based policy?’, Criminology & Criminal Justice, vol. 11, no. 5, pp. 395–413 [Online]. DOI: 10.1177/1748895811415580.

John, P. (2013) ‘Policy entrepreneurship in UK central government: The behavioural insights team and the use of randomized controlled trials’: Public Policy and Administration [Online]. DOI: 10.1177/0952076713509297 (Accessed 17 April 2018).

Kisby, B. (2010) Interpreting Facts, Verifying Interpretations: Public Policy, Truth and Evidence – Ben Kisby, 2011 [Online]. Available at http://journals.sagepub.com/doi/abs/10.1177/0952076710375784 (Accessed 11 April 2018).

Parsons, W. D. (1995) Public policy, Brookfield, Vt, Edward Elgar Pub.

Puddicombe, B., Corry, D., Fox, C. and Albertson, K. (2012) ‘Payment by results: Bill Puddicombe, Dan Corry, Chris Fox and Kevin Albertson debate the merits and disadvantages of payment by results’, Criminal Justice Matters, vol. 89, no. 1, pp. 46–48 [Online]. DOI: 10.1080/09627251.2012.721982.

Sanderson, I. (2002) ‘Evaluation, Policy Learning and Evidence‐Based Policy Making’, Public Administration, vol. 80, no. 1, pp. 1–22 [Online]. DOI: 10.1111/1467-9299.00292.

Smith, E. (2017) Exclusive: EEF toolkit ‘more akin to pig farming than science’ | News [Online]. Available at https://www.tes.com/news/school-news/breaking-news/exclusive-eef-toolkit-more-akin-pig-farming-science (Accessed 17 April 2018).

Stewart, J. and Jarvie, W. (2015) ‘Haven’t We Been This Way Before? Evaluation and the Impediments to Policy Learning’, Australian Journal of Public Administration, vol. 74, no. 2, pp. 114–127 [Online]. DOI: 10.1111/1467-8500.12140.

Thaler, R. H. and Sunstein, C. R. (2008) Nudge: Improving Decisions about Health, Wealth, and Happiness, New Haven, USA, UNITED STATES, Yale University Press [Online]. Available at http://ebookcentral.proquest.com/lib/gmul-ebooks/detail.action?docID=4654043 (Accessed 20 April 2018).

The Economist (2014) Nudge unit leaves kludge unit – The market for paternalism [Online]. Available at https://www.economist.com/blogs/freeexchange/2014/02/market-paternalism (Accessed 17 April 2018).

The Economist (2018) England has become one of the world’s biggest education laboratories – The big classroom experiment [Online]. Available at https://www.economist.com/news/britain/21739671-third-its-schools-have-taken-part-randomised-controlled-trials-struggle-getting (Accessed 20 April 2018).

Van Deun, H., van Acker, W., Fobé, E. and Brans, M. (2018) Nudging in Public Policy and Public Administration: A Scoping Review of the Literature | The Political Studies Association (PSA) [Online]. Available at https://www.psa.ac.uk/conference/psa-annual-international-conference-2018/paper/nudging-public-policy-and-public-0 (Accessed 21 April 2018).

Wesselink, A., Colebatch, H. and Pearce, W. (2014) ‘Evidence and policy: discourses, meanings and practices’, Policy Sciences, vol. 47, no. 4, pp. 339–344 [Online]. DOI: 10.1007/s11077-014-9209-2.