Tyler Cowen, proposing that anthropology is more interesting than it appears to be in writing:
We so often confuse 'what can be translated into print well' with 'what is important and interesting.'
Tyler Cowen, proposing that anthropology is more interesting than it appears to be in writing:
We so often confuse 'what can be translated into print well' with 'what is important and interesting.'
It is employee evaluation time in my office; another chance to learn from flaws in the process. This year, I am interested in skills development. In our process skills are an afterthought, but I think they should come first, as in sports. I will explore that idea here. Spoiler: skills come in bundles, which balance the tradeoffs in skill development and the synergies in skill use.
But first, allow me to complain about our employee evaluation process.
The process is built around annual goals. The form (there is always a form) gives space for around 5–10 goals. The supervisor and employee can put nearly anything as a “goal”. A goal can be small or large; ambitious or timid. It could be a mere input (i.e., effort by the employee), an output (e.g., a written product), or an outcome (e.g., an improvement in customers’ lives).
The form has training section at the end. What goes there? Ideas for classes, readings, or other ways for the employee to learn things. (This year, I’m going to learn something about Python.) The form does not suggest a connection between goals and training. In my experience, most people gloss over training, often citing — truthfully — the lack of money for training expenses.
The form gives no help in choosing skills, a real missed opportunity. For instance, the form could — but does not — include a list of the employee’s ongoing duties and derive relevant skills from that list. Similarly, the form could — but does not — include a list of the employee’s current skills and note options for developing them.
The current skills idea is a good place to bring sports in to the discussion. It is common to talk about players in the form of a scouting report, which covers skills the player has; skills she ought to have but doesn’t; and habits or tendencies. Something like this:
She’s a stay-at-home defenseman with good vision and puck sense, a quick but not powerful shot, and solid skating skills. She needs to work on her physical presence in the neutral zone and on distributing the puck. She tends to allow more of a gap than needed.
This way of talking about players seems so common that it goes unremarked upon by athletes. My kids, for instance, talk this way naturally about all of their teammates, their opponents, and players they just watch. It is noteworthy that scouting talk is always evaluative, but need not be disparaging. It seems most valuable when it is accurate and supportive of improvement.
A scouting report mostly describes the player, but it also describes the situation around the player, namely how her skills fit into the rest of the team. “She is a defenseman” describes both her skill set and her role.
Here we get to the idea of tradeoffs in skill development. To be a defenseman requires a different set of skills than to be a forward or a goaltender. True, they all skate, they all handle the puck, they all see the flow of the game. But defensemen have to be able turn, skate backwards, and maintain a proper gap, things that are not essential for forwards.
At a beginner level, the skating skills of forwards and defensemen may be indistinguishable. But at an elite level, they are very distinguishable. A forward who is out of position and having to act as a defensemen, may be just weak enough in defensive skating skills as to allow a goal.
As players in any sport develop, they work on the skills most relevant to their role. Skating backwards is a skill. Defenseman is a role. Taking face offs is a skill. Centerman is a role.
Although any hockey player can skate, when we assign players to roles we are choosy. We want the best backwards-skaters on defense. And once they are in that role, we are choosy about what skills they continue to develop. We want defenseman to be competent at face offs, but not at the expense of their defensive skating ability.
Put another way, if a defenseman has an extra hour for practice, most of that should go toward skills appropriate to her role; not to, say, goaltending. Meanwhile, the goaltenders are using their extra hour for rebound control, glove work, or stickhandling behind the net.
We might think of a role as a bundle of skills that together yield high value. To fit well in a role together, skills either need to be compatible enough that they can be developed in tandem with little conflict, or they need to yield high value when paired. Of course, some skills are general purpose, like hand-eye coordination in sports.
In my experience, governmental performance evaluations neglect this idea. They do not think consciously about which skills to bundle in roles. Nor do they consider the value yielded by different bundles. They treat skills as an unlimited grab bag: choose as many as you like; work on what ever skills interest you. Indeed if an employee doesn’t work on a lot of skills, she is seen as a slacker or a failure.
But ignoring tradeoffs and synergies among skills doesn’t make them disappear. No, I can’t learn just any skills that interest me. Or more accurately, if I do that, I’ll be a poor contributor to the team.1 What people need from a performance evaluation is a thoughtful characterization of their role, of the skills that comprise that role, and a plan for improving those skills.
In short, we need scouting reports on our own bundle of skills.
I’m not saying that people should never work on skills outside their role. Rather, doing so should be a conscious choice with some value and an understanding of what’s being given up. A mid-career player with some aptitude for teaching may want to shift some of her practice time to teaching younger players, as an investment in her future teaching career, for instance. ↩
(From April 2013) Sheila Jasanoff’s notes something that I am often amused by, scientists are slow to hold the mirror to themselves:
Ironically, there is one kind of science that science advisers rarely turn to for insights into how best to define their role in the policy process. That is the body of scholarship which has emerged from some three decades of research in the field of science and technology studies (STS).
STS starts from the proposition that scientific knowledge is one of the basic means with which modern societies make sense of who they are, how they relate to the world around them, and what they can or should do in order to improve their conditions. It follows that societies also need to study the processes through which science and technology are made — in order to understand how knowledge advances, what makes innovation happen, and why things do not always progress as hoped or planned.
Her essay includes four lessons on science advice that are worth studying and reviewing periodically.
Jean Tirole, recent Nobel Laureate, writing in 2007:
Every action of the State must be subject to a double independent evaluation. The first should be before the action: Is public intervention necessary? What are the costs and benefits? The second is after. Did it work? Was it cost effective? On this point, it would be necessary to require that the audit recommendations (for example, those of the Audit Court) be either followed according to a strict schedule, or rejected with a convincing justification.
The passive-voice is appropriate and effective in the last sentence.
Russ Floate, in a mildly vulgar post:
The internet security world has for years had white-hat hackers—people whose job it is to test code for security flaws. It’s time for designers to adopt the idea. Next time you’re working on a long-term project, appoint a designated white-hat jerk; someone whose job it is to keep thinking about how a person or group with a bit of time on their hands might try to bend and twist your system for a few laughs. This isn’t simply asking someone to be a tedious Devil’s Advocate—it’s ensuring that someone is always thinking ‘How could someone fool around with this, and what would that mean for our end product?’
In the early 2000’s, I was briefly swept up in the recurrent fashion of performance measurement. Only now do I find this useful 2003 article, Why Measure Performance? Different Purposes Require Different Measures.1 I’ll be ready the next time.
Here is the abstract:
Performance measurement is not an end in itself. So why should public managers measure performance? Because they may find such measures helpful in achieving eight specific managerial purposes. As part of their overall management strategy, public managers can use performance measures to evaluate, control, budget, motivate, promote, celebrate, learn, and improve.
Unfortunately, no single performance measure is appropriate for all eight purposes. Consequently, public managers should not seek the one magic performance measure. Instead, they need to think seriously about the managerial purposes to which performance measurement might contribute and how they might deploy these measures. Only then can they select measures with the characteristics necessary to help achieve each purpose.
Without at least a tentative theory about how performance measures can be employed to foster improvement (which is the core purpose behind the other seven), public managers will be unable to decide what should be measured.
Did you catch the list?
Behn, R. D., September/October 2003. Why measure performance? Different purposes require different measures. Public Administration Review 63 (5), 586–606. ↩
From the politics-and-policy-are-different category:
[I]t is a mistake to confuse rhetoric with reality, or feelings with actions, and it is an illusion to conflate political aspirations with actual policymaking.1
In a working paper on the precautionary principle,1 Taleb and co-authors write:
Traditional decision-making strategies focus on the case where harm is localized and risk is easy to calculate from past data. Under these circumstances, cost-benefit analyses and mitigation techniques are appropriate. The potential harm from miscalculation is bounded.
On the other hand, the possibility of irreversible and widespread damage raises different questions about the nature of decision making and what risks can be reasonably taken. This is the domain of the PP [precautionary principle].
Before, when people said “we should follow the precautionary principle”, I heard “don’t do anything I’m afraid of”. In that interpretation, the precautionary principle is just rhetoric.
What I like about the Taleb et al. paper is that narrows the precautionary principle to situations with irreversibility and widespread damage. That makes it less broadly applicable, but more powerful when applied.
Taleb, N. N., Bar-Yam, Y., Douady, R., Norman, J., Read, R., 2014. The precautionary principle: Fragility and black swans from policy actions. Working paper. ↩
Last June, the Washington Post reported on a conflict over flood maps in North Carolina. These are not your standard, everyday flood maps. They do not limit themselves to definite-but-boring historical evidence of flooding. They go a step beyond, to include computer models of how future flooding may change as the climate changes. As was the case with Armstrong’s hop off the ladder, that is one small step for a flood mapper, but a giant leap for the law and politics of flood mapping. Too giant a leap, as we shall see.
The story starts with a state agency project to estimate how much the sea would rise by the year 2100, under the influence of a warming climate. The agency projected a 39-inch rise by century’s end and made maps showing precisely which parts of the North Carolina coast would be under water under that assumption. The agency was in the process of making a public website to share those maps, when the legislature intervened. The project is now limited to using historical trends extrapolated to a 30-year forecast. Modelers are projecting an 8-inch rise with those assumptions.
The Post story linked to Steven Colbert’s satirical take on the conflict. He joked that the legislature had “addressed the crisis predicted by these climate models, by outlawing the climate models” (minute 2:40). “If your science gives you a result you don’t like, pass a law saying the result is illegal” (minute 4:04).
Of course the legislature did not outlaw a result, it prohibited a state agency from using a particular model. But the puzzle is still there. Isn’t it odd to legislate against a model?
Colbert and others saw the legislature’s action as evidence that legislators are anti-science. They may be, but I think the restriction comes from a different social conflict, one that will challenge climate adaptation far more than scientific debates will. Namely, should the state be able to make authoritative statements about the distant future without considering the present-day effects of those statements?
The legislature was acting on behalf of people whose property is in the areas that were projected to be under water by 2100. The owners were objecting to that specific fact: that the state asserted that their property would be under water by 2100.
The assertion is specific as to place — the maps were at a fine resolution. It is also specific as to time — by 2100. Those two things are necessary to mobilize landowners as an interest group. Without them, no individual’s interest is sufficiently threatened by the map.
But why is it the map that is seen as the threat instead of sea level rise itself? Answer: the sea threatens future landowners, who may not yet be born. The map threatens today’s landowners.
The threat comes from the first italicized word above: “asserted”. The maps were an assertion, a claim, an allegation. Statements like that, once launched into the public debate, can be used, reinterpreted, and modified by anyone. Their original creators no longer control them.
The threat to the landowners was that the state’s maps would be used by others in a way that hurts the landowner’s property values. Specifically: that potential buyers would not pay as much for the property, due to a misunderstanding of the map’s significance or some other reason.
If the maps were known to be 100% true projections of the year 2100, the landowners would still have a reason to object to the presentation of the maps today. But the objection is strengthened by the uncertainty of the maps.
The uncertainty of the models is technical, hard to understand, and hard to find in the underlying analysis. Anyone who wanted to disagree on the merits would have to do a comparable level of work. And unlike the form of science heralded by Colbert, projections of the distant future are non-disprovable. There is no way to test them.
Of course there are many reasons to used modeled projections anyway. They can help people consider long-term risks to present-day investment decisions. This reason is especially salient for municipalities who are investing in very long-term infrastructure. Climate projections can also be seen as a public good, under-provided by firms acting independently. Maybe there are scale economies of production or learning-by-doing that support state work on projections. Perhaps they play a signaling role, like a bargaining position or a pre-commitment device by the state.
Those reasons may be sufficient from the state’s point of view. They may even be sufficient from a social welfare point of view. But they do not negate the landowners’ objections.
Should the state be able to make authoritative statements about the distant future without considering the present-day effects of those statements? Those harmed by the statements will say “no”. Before we judge them too harshly, let’s remember that, at some point, we may also be the ones harmed by a state action.
Peter Welch, wrapping up a delightful story about figuring out why his office’s door lock is hard to open:
I won’t lie and say there are no emotional or personal components to my skeptical world view, but situations like this are the reason I stick with it. Being skeptical and applying the scientific method isn’t something you only do with beakers and goggles: it’s an approach to the getting through the day that consistently makes life easier.
“Skeptical” and “scientific” are his words. I prefer “inquisitive” and systematic”, but the approach is the same.
Andrew Means asks whether program evaluation is “dying”, and notes that evaluation can prevent improvement:
Program evaluation actually undermines efforts to improve. Once I have my evaluation report I have less of an incentive to innovate or improve because I now have a piece of paper saying that what I do is working. If I were to change my intervention, that piece of paper would become less valid and thus less valuable.
Such complacency may be common, but it is not a property of evaluation itself. They key point is the word “incentive”. If a program’s only test is the artificial one of passing an evaluation, then yes, evaluation will discourage innovation.
The fix is not replacing evaluation, but putting better incentives on a program.
Here is an idea: make the evaluation competitive. If a competing program can do better, your program will no longer pass the evaluation. Improve or be left behind.
(He also makes what to me is an unrelated point: program evaluation makes too-little use of data. See also his follow-up post on data.)
Speaking of government performance, here is the central question in Fisman and Sullivan’s 2013 article about the Hudson Street passport office:
But as the praise for the Hudson Street office shows, not all government customer service is terrible. Which raises another question: Why do some government offices perform well and others poorly, even when they’re providing the same services and working with comparable resources?
The article reports on research with a few suggestions. Spoiler: They involve the hard work of good management.
A colleague recently shared with me an unsolicited but enjoyable rant about how it is folly to expect government to “run like a business”. It reminded me of Robert Murphy’s 2013 essay explaining why that is the case.
He covers several reasons and gives several examples. The following point is an important illustration of the theme:
Murray Rothbard pointed out that by separating payment from service, government enterprises end up treating the consumer ‘like an unwelcome intruder, an interference in the quiet enjoyment by the bureaucrat of his steady income.’
Government employees experience this first hand even within an agency. If one of us needs service from another unit — such as IT or facilities maintenance — we are the recipient of the service, but we don’t pay for it. And yes, we are often treated “like an unwelcome intruder”.
Murphy’s main point is that the limits on government effectiveness imply that there should be fewer government enterprises. I’m making a narrower point: there are ways to induce higher performance from government, but they are different ways than those that work for business, because of the reasons Murphy reviews. Agency managers — especially those recently from the private sector — should learn about them.
Remember, 99 times out of 100, your counterpart has rational underlying interests that you will eventually discover with patience and the right strategies. The secret to negotiating, after all, is to find out what the other party wants and how much it’s worth to him. In those rare cases when your counterparty wants to use the negotiation to control or punish you, however, it doesn’t matter how much it’s worth to him. It’s worth more to you to be free of him and able to get on with your business.
Roger Martin disagrees that the world is becoming more volatile, uncertain, complex, and ambiguous (VUCA):
There is really no actual evidence that ours is a more VUCA world than previous ones. In fact, every generation claims that its times are the most turbulent ever. Partly this is because the past is known and understood, so we forget how uncertain everything seemed at the time.
If the world is more VUCA, we might be tempted to stop thinking strategically about the future. Martin warns us this would be a mistake:
I believe the fad for adaptive strategy models is an excuse for not making hard, dangerous choices.
Speaking of getting processes right, here is a point from John Gruber’s recent essay about Tim Cook’s progress as Apple’s CEO:
As the Cook era as Apple’s CEO unfolds, what we’re seeing is something we didn’t know, and I think few expected. Something I never even considered: Tim Cook is improving Apple’s internal operational efficiency.
It’s hard to overstate how good Apple is at process. Their manufacturing has excelled under Cook. Now he is applying the same improve-the-processes idea to Apple’s internal collaboration, design, software tools, etc.
If the process we’ve used in the past is broken, let’s fix it, because, in fact, getting that process right is actually more urgent than the problem we’ve got right now.
Government processes are terrible. People constantly chase the new and novel project instead of getting the organization working well in the first place.
Seth Godin, on the importance of problems:
Or consider the case of a non-profit seeking to raise funds or gain government support. Without a doubt, they have to create a problem in the mind of the donor, or there will be no funds or no support to solve that problem.
Here is a marketing idea that some policy people have trouble with. When you are promoting a solution to a problem, make sure your audience knows about the problem and knows that it is a problem. If you are at the same time downplaying the problem, stop.
I have seen the last guideline violated in a few cases recently. I’ll blend them into the following example.
Many groups are now working on the problem of adapting to climate change. Consulting firms and research groups are springing up to help.
The problem with climate change is twofold. First, as climate changes in a place, human activities that were suited to the old climate become less well suited. They may even become impossible, as will happen with at-sea-level housing in poor countries.
Second, climate change is uncertain. We do not know the details of change for a given place. How much warmer, or cooler, or wetter, or dryer, or stormier, or calmer will this place become?
If climate change were certain — if we knew precisely how change would affect a given place — there would be no need for advice. We would not need people to model climate change, assess its impacts, estimate risks, or experiment with adaptation approaches.
The fact that climate change is uncertain is precisely why the adaptation community has so many climatologist, impacts modelers, and experimenters. They are selling their services; offering to help others with uncertainty. They should therefore talk about uncertainty and how they can help.
Yet, some in the the climate adaptation community also downplay uncertainty and exaggerate what is known. I think they emphasize certainty to convince people that climate change is important. But that’s missing the trees for the forest.
Climate adaptation is about coping with uncertainty — reducing it if possible and building resilience against it otherwise. You can’t sell your services as an uncertainty manager without talking about uncertainty in the first place.
At Flowing Data, Nathan Yau gives ideas for improving government data sharing, starting with this:
Just give us CSV files. Everybody wins.
…and ending with this:
Decide what’s important, archive the rest.
He finds so many things to fix that he includes an intermission photo halfway through for a break.
Consider it a checklist for government data site designers.