Thoughts on Public Policy, Law & Economics

Eight (Partly Incompatible) Uses for Performance Measures

Added on by William D. Walker.

In the early 2000’s, I was briefly swept up in the recurrent fashion of performance measurement. Only now do I find this useful 2003 article, Why Measure Performance? Different Purposes Require Different Measures.1 I’ll be ready the next time.

Here is the abstract:

Performance measurement is not an end in itself. So why should public managers measure performance? Because they may find such measures helpful in achieving eight specific managerial purposes. As part of their overall management strategy, public managers can use performance measures to evaluate, control, budget, motivate, promote, celebrate, learn, and improve.

Unfortunately, no single performance measure is appropriate for all eight purposes. Consequently, public managers should not seek the one magic performance measure. Instead, they need to think seriously about the managerial purposes to which performance measurement might contribute and how they might deploy these measures. Only then can they select measures with the characteristics necessary to help achieve each purpose.

Without at least a tentative theory about how performance measures can be employed to foster improvement (which is the core purpose behind the other seven), public managers will be unable to decide what should be measured.

Did you catch the list?

  1. Evaluate
  2. Control
  3. Budget
  4. Motivate
  5. Promote
  6. Celebrate
  7. Learn
  8. Improve

  1. Behn, R. D., September/October 2003. Why measure performance? Different purposes require different measures. Public Administration Review 63 (5), 586–606. ↩

An Illusion to Conflate

Added on by William D. Walker.

From the politics-and-policy-are-different category:

[I]t is a mistake to confuse rhetoric with reality, or feelings with actions, and it is an illusion to conflate political aspirations with actual policymaking.1


  1. P. 311 in Jonathan B. Wiener, Brendon Swedlow, James K. Hammitt, Michael D. Rogers, and Peter H. Sand (2013): Better Ways to Study Regulatory Elephants, European Journal of Risk Regulation, 2013, 311–319. ↩

Making the Precautionary Principle Narrower but More Powerful

Added on by William D. Walker.

In a working paper on the precautionary principle,1 Taleb and co-authors write:

Traditional decision-making strategies focus on the case where harm is localized and risk is easy to calculate from past data. Under these circumstances, cost-benefit analyses and mitigation techniques are appropriate. The potential harm from miscalculation is bounded.

On the other hand, the possibility of irreversible and widespread damage raises different questions about the nature of decision making and what risks can be reasonably taken. This is the domain of the PP [precautionary principle].

Before, when people said “we should follow the precautionary principle”, I heard “don’t do anything I’m afraid of”. In that interpretation, the precautionary principle is just rhetoric.

What I like about the Taleb et al. paper is that narrows the precautionary principle to situations with irreversibility and widespread damage. That makes it less broadly applicable, but more powerful when applied.


  1. Taleb, N. N., Bar-Yam, Y., Douady, R., Norman, J., Read, R., 2014. The precautionary principle: Fragility and black swans from policy actions. Working paper. ↩

Fighting over the Future, Today

Added on by William D. Walker.

Last June, the Washington Post reported on a conflict over flood maps in North Carolina. These are not your standard, everyday flood maps. They do not limit themselves to definite-but-boring historical evidence of flooding. They go a step beyond, to include computer models of how future flooding may change as the climate changes. As was the case with Armstrong’s hop off the ladder, that is one small step for a flood mapper, but a giant leap for the law and politics of flood mapping. Too giant a leap, as we shall see.

The story starts with a state agency project to estimate how much the sea would rise by the year 2100, under the influence of a warming climate. The agency projected a 39-inch rise by century’s end and made maps showing precisely which parts of the North Carolina coast would be under water under that assumption. The agency was in the process of making a public website to share those maps, when the legislature intervened. The project is now limited to using historical trends extrapolated to a 30-year forecast. Modelers are projecting an 8-inch rise with those assumptions.

The Post story linked to Steven Colbert’s satirical take on the conflict. He joked that the legislature had “addressed the crisis predicted by these climate models, by outlawing the climate models” (minute 2:40). “If your science gives you a result you don’t like, pass a law saying the result is illegal” (minute 4:04).

Of course the legislature did not outlaw a result, it prohibited a state agency from using a particular model. But the puzzle is still there. Isn’t it odd to legislate against a model?

Colbert and others saw the legislature’s action as evidence that legislators are anti-science. They may be, but I think the restriction comes from a different social conflict, one that will challenge climate adaptation far more than scientific debates will. Namely, should the state be able to make authoritative statements about the distant future without considering the present-day effects of those statements?

The legislature was acting on behalf of people whose property is in the areas that were projected to be under water by 2100. The owners were objecting to that specific fact: that the state asserted that their property would be under water by 2100.

The assertion is specific as to place — the maps were at a fine resolution. It is also specific as to time — by 2100. Those two things are necessary to mobilize landowners as an interest group. Without them, no individual’s interest is sufficiently threatened by the map.

But why is it the map that is seen as the threat instead of sea level rise itself? Answer: the sea threatens future landowners, who may not yet be born. The map threatens today’s landowners.

The threat comes from the first italicized word above: “asserted”. The maps were an assertion, a claim, an allegation. Statements like that, once launched into the public debate, can be used, reinterpreted, and modified by anyone. Their original creators no longer control them.

The threat to the landowners was that the state’s maps would be used by others in a way that hurts the landowner’s property values. Specifically: that potential buyers would not pay as much for the property, due to a misunderstanding of the map’s significance or some other reason.

If the maps were known to be 100% true projections of the year 2100, the landowners would still have a reason to object to the presentation of the maps today. But the objection is strengthened by the uncertainty of the maps.

The uncertainty of the models is technical, hard to understand, and hard to find in the underlying analysis. Anyone who wanted to disagree on the merits would have to do a comparable level of work. And unlike the form of science heralded by Colbert, projections of the distant future are non-disprovable. There is no way to test them.

Of course there are many reasons to used modeled projections anyway. They can help people consider long-term risks to present-day investment decisions. This reason is especially salient for municipalities who are investing in very long-term infrastructure. Climate projections can also be seen as a public good, under-provided by firms acting independently. Maybe there are scale economies of production or learning-by-doing that support state work on projections. Perhaps they play a signaling role, like a bargaining position or a pre-commitment device by the state.

Those reasons may be sufficient from the state’s point of view. They may even be sufficient from a social welfare point of view. But they do not negate the landowners’ objections.

Should the state be able to make authoritative statements about the distant future without considering the present-day effects of those statements? Those harmed by the statements will say “no”. Before we judge them too harshly, let’s remember that, at some point, we may also be the ones harmed by a state action.

A Systematically Inquisitive World View Makes Life Easier

Added on by William D. Walker.

Peter Welch, wrapping up a delightful story about figuring out why his office’s door lock is hard to open:

I won’t lie and say there are no emotional or personal components to my skeptical world view, but situations like this are the reason I stick with it. Being skeptical and applying the scientific method isn’t something you only do with beakers and goggles: it’s an approach to the getting through the day that consistently makes life easier.

“Skeptical” and “scientific” are his words. I prefer “inquisitive” and systematic”, but the approach is the same.

Improvement Needs the Right Incentives

Added on by William D. Walker.

Andrew Means asks whether program evaluation is “dying”, and notes that evaluation can prevent improvement:

Program evaluation actually undermines efforts to improve. Once I have my evaluation report I have less of an incentive to innovate or improve because I now have a piece of paper saying that what I do is working. If I were to change my intervention, that piece of paper would become less valid and thus less valuable.

Such complacency may be common, but it is not a property of evaluation itself. They key point is the word “incentive”. If a program’s only test is the artificial one of passing an evaluation, then yes, evaluation will discourage innovation.

The fix is not replacing evaluation, but putting better incentives on a program.

Here is an idea: make the evaluation competitive. If a competing program can do better, your program will no longer pass the evaluation. Improve or be left behind.

(He also makes what to me is an unrelated point: program evaluation makes too-little use of data. See also his follow-up post on data.)

Why Do Some Government Offices Perform Well and Others Poorly?

Added on by William D. Walker.

Speaking of government performance, here is the central question in Fisman and Sullivan’s 2013 article about the Hudson Street passport office:

But as the praise for the Hudson Street office shows, not all government customer service is terrible. Which raises another question: Why do some government offices perform well and others poorly, even when they’re providing the same services and working with comparable resources?

The article reports on research with a few suggestions. Spoiler: They involve the hard work of good management.

Treating the Consumer "Like an Unwelcome Intruder"

Added on by William D. Walker.

A colleague recently shared with me an unsolicited but enjoyable rant about how it is folly to expect government to “run like a business”. It reminded me of Robert Murphy’s 2013 essay explaining why that is the case.

He covers several reasons and gives several examples. The following point is an important illustration of the theme:

Murray Rothbard pointed out that by separating payment from service, government enterprises end up treating the consumer ‘like an unwelcome intruder, an interference in the quiet enjoyment by the bureaucrat of his steady income.’

Government employees experience this first hand even within an agency. If one of us needs service from another unit — such as IT or facilities maintenance — we are the recipient of the service, but we don’t pay for it. And yes, we are often treated “like an unwelcome intruder”.

Murphy’s main point is that the limits on government effectiveness imply that there should be fewer government enterprises. I’m making a narrower point: there are ways to induce higher performance from government, but they are different ways than those that work for business, because of the reasons Murphy reviews. Agency managers — especially those recently from the private sector — should learn about them.

Negotiating Requires Both Parties to Have "Rational Underlying Interests"

Added on by William D. Walker.

Judith White:

Remember, 99 times out of 100, your counterpart has rational underlying interests that you will eventually discover with patience and the right strategies. The secret to negotiating, after all, is to find out what the other party wants and how much it’s worth to him. In those rare cases when your counterparty wants to use the negotiation to control or punish you, however, it doesn’t matter how much it’s worth to him. It’s worth more to you to be free of him and able to get on with your business.

Strategic Thinking is Always Hard and Dangerous

Added on by William D. Walker.

Roger Martin disagrees that the world is becoming more volatile, uncertain, complex, and ambiguous (VUCA):

There is really no actual evidence that ours is a more VUCA world than previous ones. In fact, every generation claims that its times are the most turbulent ever. Partly this is because the past is known and understood, so we forget how uncertain everything seemed at the time.

If the world is more VUCA, we might be tempted to stop thinking strategically about the future. Martin warns us this would be a mistake:

I believe the fad for adaptive strategy models is an excuse for not making hard, dangerous choices.

Tim Cook Is Improving Apple's Internal Operational Efficiency

Added on by William D. Walker.

Speaking of getting processes right, here is a point from John Gruber’s recent essay about Tim Cook’s progress as Apple’s CEO:

As the Cook era as Apple’s CEO unfolds, what we’re seeing is something we didn’t know, and I think few expected. Something I never even considered: Tim Cook is improving Apple’s internal operational efficiency.

It’s hard to overstate how good Apple is at process. Their manufacturing has excelled under Cook. Now he is applying the same improve-the-processes idea to Apple’s internal collaboration, design, software tools, etc.

Let's Fix the Process First

Added on by William D. Walker.

Seth Godin:

If the process we’ve used in the past is broken, let’s fix it, because, in fact, getting that process right is actually more urgent than the problem we’ve got right now.

Government processes are terrible. People constantly chase the new and novel project instead of getting the organization working well in the first place.

Create a Problem in the Mind

Added on by William D. Walker.

Seth Godin, on the importance of problems:

Or consider the case of a non-profit seeking to raise funds or gain government support. Without a doubt, they have to create a problem in the mind of the donor, or there will be no funds or no support to solve that problem.

Tell us What Problem You're Solving

Added on by William D. Walker.

Here is a marketing idea that some policy people have trouble with. When you are promoting a solution to a problem, make sure your audience knows about the problem and knows that it is a problem. If you are at the same time downplaying the problem, stop.

I have seen the last guideline violated in a few cases recently. I’ll blend them into the following example.

Many groups are now working on the problem of adapting to climate change. Consulting firms and research groups are springing up to help.

The problem with climate change is twofold. First, as climate changes in a place, human activities that were suited to the old climate become less well suited. They may even become impossible, as will happen with at-sea-level housing in poor countries.

Second, climate change is uncertain. We do not know the details of change for a given place. How much warmer, or cooler, or wetter, or dryer, or stormier, or calmer will this place become?

If climate change were certain — if we knew precisely how change would affect a given place — there would be no need for advice. We would not need people to model climate change, assess its impacts, estimate risks, or experiment with adaptation approaches.

The fact that climate change is uncertain is precisely why the adaptation community has so many climatologist, impacts modelers, and experimenters. They are selling their services; offering to help others with uncertainty. They should therefore talk about uncertainty and how they can help.

Yet, some in the the climate adaptation community also downplay uncertainty and exaggerate what is known. I think they emphasize certainty to convince people that climate change is important. But that’s missing the trees for the forest.

Climate adaptation is about coping with uncertainty — reducing it if possible and building resilience against it otherwise. You can’t sell your services as an uncertainty manager without talking about uncertainty in the first place.

Just Give us CSV Files

Added on by William D. Walker.

At Flowing Data, Nathan Yau gives ideas for improving government data sharing, starting with this:

Just give us CSV files. Everybody wins.

…and ending with this:

Decide what’s important, archive the rest.

He finds so many things to fix that he includes an intermission photo halfway through for a break.

Consider it a checklist for government data site designers.

What Is "Efficient" Depends on Our Level in the Project Stack

Added on by William D. Walker.

Charles Marohn, in a post on transit-oriented development, reminds us that what is “efficient” depends on our level in the project stack:

Yes, [gradual transit expansion] is less ‘efficient’ than getting out there and building the light rail to the parking lot now, but only if we judge efficiency on the project level. … If we judge efficiency on a macro-financial level, however, it is more efficient to allocate scarce resources to endeavors that are proving their success than to those that may be successful but, thus far, are unproven.

If we are sure that a complete rail system will be needed, then total cost will (probably) be lower if we build the whole system now.

But we are never sure how development or other major social changes will play out. Overconfidence about the future can lead to huge waste on failed bets.

A Label or a Claim?

Added on by William D. Walker.

I’ve had several professional labels over the years: “lawyer”, “budget analyst”, “policy advisor”, and “economist”. Now I’ve got a new one — “social scientist” — and I don’t like it. It’s more than a label — it’s used also as a claim, in ways I am not comfortable with.

To label an object is to attach a piece of fabric, paper, metal or other material that gives information about the object. These are Mary’s shoes. This pump dispenses 87 octane unleaded. This is the final resting place of Abraham Lincoln. This hook is where the pruning shears belong; that hook is where the shovel belongs.

Labels tell observers things that are not obvious, although they are true. The label on “Room 15” is there to help visitors. No one bothers to label the lamps, though we might label the switches if we can’t remember which lights they control.1

You can hedge your bets in a label: Some people believe this was the birthplace of Famous Person. A hedge may be implicit, as with the quotes around advertising copy: “America’s Finest Beer”. But a statement that is not generally known to be true is not really serving as a label. It is instead a claim.

A claim is about persuasion. It starts with something that is in dispute or not widely known, and proceeds from there. The claimant may know — or believe — the claim is true, but they realize that the audience is not with them. So they make an effort. They state their case.

The point of a claim is to overcome people’s resistance to it. Labels don’t have that problem. We accept that the label Room 15 is accurate. Otherwise, labelers would have to add evidence. See, look! Here is the building’s floor plan showing room numbers, initialed by the architect; and here is an affidavit from someone who agrees this is really room 15; and here is a photograph of this being room 15 back in 1976; ….

Claims put us on guard, because we know from experience that many claimers care more about convincing us than they do about truth. The harder they push, the more on guard we become. Even people who want to be honest will tailor their claims to their audience.

“Science” can of course be just a label. What does she do for a living? — she’s a scientist. Or … what are you reading? — a book on the science of cooking. The word “science” gives us useful information in those contexts. She’s not a salesperson, she’s a scientist. It’s not a recipe book, it’s a science-of-cooking book. We have no reason to doubt those labels.

“Science” can also be used to label a group, like “the physical sciences” or “the biological science”. The meaningful words are “physical” and “biological” but “science” is useful filler. I don’t object when “social science” is used a grouping label in the same way.2

What bothers me is the use of “social science” to claim seriousness, objectivity, trustworthiness, or validity; calling something social “science” to convince us it is true. That is an appeal to authority, which is the antithesis of the scientific method. A scientific result stands or falls on its merits, regardless of label. The stronger the result, the less its proponents need to resort to persuasion.

In other words, insisting that something is science is prima facie evidence that it is wrong. The claim undermines itself.

On the flip side, if something is true, the fact that it doesn’t fit into a supposedly scientific discipline is not a mark against it. The scientific method is a sieve to separate truth from untruth, but not all propositions fit in its machinery.

I have noticed an even more dangerous use of the word “science” — as a shield against review or verification. In other words, I have heard people argue that they should not have to defend or explain their work, because it is “science”. There is a truth lurking here: review can be used as a weapon to delay or obfuscate work. People cannot be expected to answer every question or defend every point in detail. But they should share data and methods so reviewers can do the checking on their own.

You may happily use the term “social science” without intending to claim anything. But I think that the claim is made so often by others that it has become part of the term, whether any individual intends it or not. Just as we cannot excuse our use of words the culture has deemed derogatory, we are stuck with the culture’s use of “social science”. In short, think twice before using that term.


  1. If you have designed a lamp that people cannot recognize, labeling it “lamp” is an admission of failure. You should have designed a better lamp.  ↩

  2. See, e.g., RSeek.org’s list of Statistics for the Social Sciences. It’s a convenient grouping of methods that are commonly used in various social sciences. On the other hand, I’ve seen a lot of disagreement about what should count as a social science. Some people specifically exclude economics. Some include law and business. Others seem to equate it with survey methods. Still others use it to mean the psychology of persuasive communication. Wikipedia has the broadest list I’ve seen. ↩

They Rarely Answer the Practical Questions

Added on by William D. Walker.

Daniel Duane on exercise science:

This raised a question: If all the latest cutting-edge scientific research says that outdated barbell movements have to be updated with core stability tricks and then integrated into super-short high-intensity muscle-confusion routines, how come none of that did much for me, while the same five lifts repeated for a year caused profound structural changes to my body?

The answer, it turns out, is that there are no cutting-edge scientific studies.

I don’t mean that exercise physiologists don’t conduct brilliant research. They do. I mean that they rarely research the practical questions you and I want answered, like which workout routine is best.

Similarly, economists and others rarely answer the practical questions that public managers want answered.

Apt to Be Conflicting Factors

Added on by William D. Walker.

B. H. Liddell Hart, referring to German defensive maneuvers in the eastern front, in the summer of 1943:

[T]ime and force are apt to be conflicting factors, and the outcome was a lesson in the way that increased force may really prove less weighty than early timing that carries a greater measure of surprise.1

Weakness early may be better than strength late.


  1. P. 295. Strategy. B. H. Liddell Hart. Meridian, second revised edition, Mar. 1991. ISBN 0452010713. Originally published in 1941 — without the WWII analysis — as “The Strategy of Indirect Approach”. Emphasis in original. ↩

We Are Falling Behind!

Added on by William D. Walker.

When I started my career in public policy, one of my jobs was to evaluate budget proposals and make recommendations to the higher-ups. One of their favorite questions was “what are other states doing?”

That question has always puzzled me.

Of course it can be helpful to have some context for an issue. What are other states doing? can indicate how common an issue is and what the range of responses are. It can also be a check on error. If many states face the same problem but none use our proposed solution, maybe our solution is no good. It can also be a substitute for evidence. There may be no time to evaluate a proposal before deciding on it, so a glance at other states’ experiences may be all we have to go on.

But what are other states doing? is used in other situations where it seems inappropriate.

Here is one variant: “we are falling behind other states!”. We are falling behind is a curious justification for a policy choice. It requires an ordering, but doesn’t say which one: total spending? Per-capita spending? Output per unit input? Some outcome measure?

Worse, it makes the standard of success relative to others. That makes sense in a zero-sum game, like sports or law school ranking.1 But it rarely makes sense in policy. For instance, a state should hope for “zero road deaths”, not “fewer road deaths than other states”. Sorry Mrs. Smith. Your son died at a known-dangerous intersection, but we have fewer road deaths than Nevada, so it’s okay.

Policy should be aimed at solving important problems efficiently. We are falling behind can lead us away from that model. For instance, law students have been known to sabotage their competing students by hiding library books required for an assignment. They care more about their class rank than their own learning.

It’s a shame when policy-makers care more about their state’s rank than about effective problem solving.


  1. Law schools traditionally report student achievement strictly by rank. Letter grades don’t matter; they may not even be awarded. What matters is the percent of other students you are better than. “Top 1%” is the goal. It is unattainable by 99% of students. ↩