1.9 The Three Es of Policy Analysis

Prof Jarman introduces a simple framework for more balanced policy analysis and evaluation. 

• HMP 615 Canvas site
• Find work from the HMP Governance Lab at www.hmpgovernancelab.organd on Twitter @HMPgovlab
• Music: 'Blippy Trance' by Kevin MacLeod

Transcript

Holly Jarman: 0:13

Welcome to the HMP governance podcast. My name is Holly Jarman. I'm a professor of health management and policy at the University of Michigan. And I'm going to be talking to you today about the three E's, which is a framework for trying to understand policy and conduct policy analysis and evaluation. And it's a framework I actually created for this class, because I was not happy, let's say with some of the other options out there for policy analysis. And so feel free to use or not this framework, feel free to debate with it or critique it, it would be actually great to hear some of your comments and questions around this. So a lot of the time when we look at policies, and when we discuss them in the public discourse, we focus on, for example, the cost of a policy. So we look at the cost of health care. And a lot of the public debate when we have health reform discussions is around, well, how much is this going to cost? What is the bottom line? What will taxpayers be on the hook for in terms of this policy? And so this was one reason why I felt like we needed to have some balance between this focus on cost, which can kind of take over health reform debate sometimes. And some other words or other concepts that actually help us understand more about the effects of the policy, and overall, not just its cost, because sometimes, as I contend, you know, good policies cost money. And it's a matter of how much money but certainly also, what do you get out of that spending. And so I've been trying to think about ways to express this. And I came up with the three E's framework. So the three E's are efficiency, efficacy, and equality. And I think especially the last one, equality does not get the attention that it deserves. In policy analysis. We have offices in the federal government whose job it is to take a look at, quote unquote, red tape and regulation and policies and review them. And so we and but they do that, from the perspective of budget or regulatory burden on businesses, we don't have a prominent office whose job it is to look at health equity, which I think is a big issue. So the way that our federal government and then also, state governments and legislatures look at policy is quite often skewed in this way towards thinking about costs. And sometimes it's thinking about cost effectiveness. But there are some big areas of US policy where cost effectiveness is not allowed to be considered by federal law as an integral concept. And so I think it's a little bit easier if we think in terms of these three things, and trying to use some common sense around the balance between them. So let me talk about each in turn. First of all, efficiency. Please note that this is not economics, it is not just looking at the cost of something. But it's actually looking at the cost against what is produced by the policy, what do we actually get from it. And so in that sense, it's closer to the the concept of some kind of value based policy, we have the acronym vbid, for about value based insurance design, which is a little closer to this concept. We need to think about what we value as a society, in terms of our health policy, what are we actually trying to achieve? What are those goals? And then what is the minimum amount of resources that we have to commit in order to obtain those goals? That's a better way of thinking about health and just thinking about costs. If we look across the world, actually, the cost of health care is increasing in pretty much every industrialized country. And so we face some of those same trends in the United States. To the to some extent, we do a little better because our population is not aging as rapidly, but there are other factors pushing up our healthcare costs. So we are right to be concerned about costs, but by saying we should be thinking about efficiency. I'm talking about more of a engineering style concept where we look at inputs and we look at outputs, how efficient is our policy machine in that sense? How much money are we putting Putting in to obtain what outcomes. And I think that's very important, because when you're trying to make that argument you're incorporating then the fact that we do want health as a goal. And there are many policy positions where we think it's fair and right to support people with government money in order to obtain a benefit, which is not just necessarily a benefit to that individual, but a benefit to society as well. So when we provide people with health care coverage, you could argue, one of the things we are doing is improving productivity in the workforce overall. And so while it might be more costly to provide people with health care coverage, there are benefits to society that we may value. And we might consider that to be more efficient than say, not giving people coverage. And then having the costs fall on emergency rooms when people show up in the emergency room, because they've had no preventative care, and they have a condition that's out of control. So that's the first e efficiency. The second E is efficacy. So we very often in American politics, get the idea into our head, a policy will work. And it doesn't really in practice. So one of the things we absolutely should be looking at for all health policies is, is this an actually an effective answer to the problem that we see. Now, if you think back to previous podcasts, we've talked a lot about how the definition of a problem is quite fluid in public policy. So there's some some of that plays into our discussion of efficacy, right? So how did we define the problem? What do we see as the the issue and then what policies will fit that solution and actually be efficacious in solving the problem or alleviating somewhat the problem? And so you can see how it's important to understand the problem when you're as perceived in the legislation? And what is the intent of this in the legislation or the regulations? In order to then say, well, let's assess this policy for efficacy does it do what it said it would do? And so there are a lot of examples here of policies that don't necessarily do what we think they will do their policies that instead signal to an audience that we did something. Or there might be unexpected problems with implementation, because we're trying to implement them in a complicated system that's already cluttered with with policies and laws. And so when we make a tweak to the health system that we think will incentivize providers to provide better preventative care, for example, maybe that will work, or maybe it will only work in some circumstances, and there might be some unanticipated effects of the policy. So understanding whether a policy is efficacious means understanding the problem definition, but also what's in the law, and how it's implemented. So where did the problems arrive in this process? Was the problem in fact framed incorrectly leading to the policy to not really be efficacious in solving the problem, because the problem wasn't really characterized correctly. Was there some issue with the way that the law or the regulations were constructed that meant there were exceptions for certain businesses or groups or individuals, or there wasn't enough money put behind the policy? That meant that it actually wasn't effective in practice, even though it might have been effective on paper as originally conceived? Or was there something wrong at the implementation stage that meant? Well, providers didn't, for example, behave the way we expected them to. And it turned out there was a big loophole in practice, or the regulations were just poorly constructed, and they didn't actually fit onto the way that individuals behave or the way that healthcare organizations work in practice. And then you have a serious problem of efficacy. So that's the second e efficacy. The third E is equity. So this is the part where I feel a number of existing frameworks for doing policy evaluation really fall short, is they certainly focus on efficacy. So there's a lot of evaluation in particular retrospective evaluation. The recommendations from federal government in the CDC are to evaluate continuously, but there's almost certainly an evaluation that takes place near the end of the project. We call that a retrospective evaluation.

Unknown: 10:02

And

Holly Jarman: 10:03

those evaluations quite often focus on efficacy because you're looking at the goals of the law or policy, or the regulations, and you're comparing them to outcomes. Fair enough. And you may well be looking at efficiency, because you might be looking at the cost of the program relative to outcomes. But quite often, we're not really thinking so much about equity. And I think that's been a blind spot in American politics for a long time. So when we're thinking about policy analysis, I think equity is important to include here, because we need to know not just the effects of the policy overall, potentially, and the impact on the population. But we need to know how are those impacts distributed? That's really important. So when we look at something like the cares act, and we say, Well, you know, how did that actually play out in practice? What were the differential effects of the cares Act funding on foot, let's say families at different levels of income. And we see that families with different levels of income behave differently, people who can afford to do so may well put some of that money away for future expenditures, whereas people who are receiving the money and have bills to pay or are in a more precarious financial situation where they immediately have to spend any money that comes in on food or necessities, they're going to have a different spending pattern. So we want to think about the differential effects of a policy in every context of equity, not just the economic context that I've just mentioned. So does this policy discriminate against some groups rather than others? And I think if we did that, for every federal law, we might see some very surprising and well, to some people surprising to some people not so surprising results. So that's equity. So I would say, I want you to carry this framework in your head as a cognitive tool. So when you are looking at cost benefit analysis, cost effectiveness analysis, or some other framework, like the Eightfold Path, which is very popular in policy analysis. Or you're doing a policy evaluation, and you might be following some guidelines, which come from the US federal government or the scene, like the CDC, for example. Or another popular framework is the magenta book, which comes from the UK. And you're thinking about how to evaluate the successful policy as a whole as a whole. Please do think about these three things, efficacy, efficiency, and equity, and try to think about whether different policy solutions might fulfill some of those criteria more readily than others. I think if we started to think like this, and incorporate equity, especially into all of these policy analysis frameworks, we might get a more a fairer system. And we might actually look at policies in a slightly different way. And hopefully, it would widen the range of things that we consider when we're thinking about what would be the best solution to a problem. All right, that was a very brief guide for me. I'm looking forward to hearing your feedback about how you feel about these kinds of frameworks. So bye for now, and I'm looking forward to hearing from you. This has been a podcast from the government snap. If you're interested in our research, come and visit us at HMP governance lab.org or on Twitter at HMP. gov.