Measurement of intangibles at Harvard Business School

I read this article, published in the New York Times on September 7, about the administration's efforts to ensure gender equity at Harvard Business School with a great deal of interest. It's partly for the detailed glimpse into another world, and it's partly due to some of the great comments people made.

But during [graduation] week’s festivities, the Class Day speaker, a standout female student, alluded to “the frustrations of a group of people who feel ignored.” Others grumbled that another speechmaker, a former chief executive of a company in steep decline, was invited only because she was a woman. At a reception, a male student in tennis whites blurted out, as his friends laughed, that much of what had occurred at the school had “been a painful experience.”

The article is relevant here because of the focus of the administrators on measurement. Here's what they did:

* Turned what was subjective into an objective measure:  Women in the B School lagged in class participation - but participation grades are both subjective and dependent on memory. The business school administrators posted stenographers in every class so faculty no longer had to rely on their memories.

* Provided information quickly: As the reporter describes it, "[n]ew grading software tools let professors instantly check their calling and marking patterns by gender." Information was used to identify and change behavior, not to punish. In fact, it appears that the administrators pushed responsibility for change down to the front lines, making it a team effort. One professor stated that the message he got from the administration was: “We’re going to solve it at the school level, but each of you is responsible to identify what you are doing that gets you to this point.” Management trusted workers to get the message, and to change.

* Developed a theory and tested it: the article reports that an additional factor, one the school really couldn't control, was contributing to the women's minimal participation. Social success was as important as academic success, possibly more important, and class participation could hurt social capital.

As you have probably surmised, all this was expensive and time consuming. At least so far, the B School administrators haven't been able to try other approaches. And it's too soon to tell whether salaries for men and women will be comparable 10 years or more after graduation. The atmosphere, as reported, sounds a lot better, but, as the article points out, the experiment brought with it unintended consequences and helped other issues - like class differences - surface. The story's not done yet, but the tale so far makes for fascinating reading. (The graphics online are much better than they were in the physical newspaper.) And so do the comments.

Update: This post has been updated to correct a typo.

Peter Singer has an interesting Op-Ed "Good Charity, Bad Charity" in yesterday's New York Times, in which he argues that there are clear answers to the question 'to which charity should I donate?' He argues that there is a stark choice between donating to organizations that provide medical and social services, and to cultural organizations like museums.

I tend to agree, though I question his assumption that donating to a cause overseas is more important than donating to an organization that serves people in the US. (I can also see an argument that funding a museum is important - future artists need a place to go and view art, and without public museums most art would be in private collections. The rest of us sometimes need to see art too.)

But what's most interesting about Singer's piece, and the reason I'm linking to it here, is his final point: there is now objective evidence of a charity's effectiveness available, and donors can use it as part of their decision-making. Singer links to both the sites of GiveWell and GiveDirectly. The former ranks charities by their effectiveness. It's still fairly small, and focuses on finding outstanding charities rather than ranking all (or many) charities but to its credit GiveWell is open about its processes and its mistakes.

GiveDirectly, which is highly rated by GiveWell, transfers donations directly (and electronically) to recipients' cell phones. Recipients then use the money for whatever is important to them. GiveDirectly reports:

• The most frequent self-reported use of funds is purchasing a metal roof. We estimate the annual rate of return on on metal as opposed to thatch roofing to be 15%-20%, suggesting this is an attractive investment.
• 1% of recipients report regrets about the way they used their transfer. For example, one woman chose not to pursue a business opportunity but later wished that she had.
• 1% of recipients report having had some of their transfer stolen.
• On net, 100% report being better-off as a result of the transfer.

Helping organizations understand the impact of the work they perform is one of the most important things I do so it's heartening to see the progress. Organizations don't have to wait for a a GiveWell to tell them how they're doing - with a little effort, they can do it themselves. It's well worth the investment.

Harvard Business Review on measuring impact

Here's a a link to a thoughtful piece by Alnoor Ebrahim in the Harvard Business Review called "Let's Be Realistic about Measuring Impact." It compares the measurement processes of three different organizations, each of which uses a different approach. One measures outputs, another measures short-term outcomes and estimates long-term outcomes, while the third thinks hard - and waits for - long-term outcomes. Each uses some form of proxy measures when necessary. Each organization puts the resources necessary into making its measurements meaningful. And they've figured out a way to get around, where necessary, their research limitations. Says Ebrahim:

Notice that none of these three organizations typically measures impact directly. They hypothesize what the outcomes and impacts might be but only in some instances are they able to follow through by commissioning their own research or multi-year evaluations. And these are sophisticated funders and investors who are much better positioned to measure long-term results than the front line organizations that contend with funding shortages and operational challenges every day.

I've written before about the importance of using outcome measures. It's hard to do well, but well worth the effort. Has your organization tried? What were the problem areas? What worked well?

"The Non Nonprofit" by Steve Rothschild

A friend recommended "The Non Nonprofit" by Steve Rothschild and I'm glad he did, because the book takes a completely different approach to the problem of paying for social services from anything else I have read. The book is organized around seven principles, with an additional chapter about how important it is to put the principles into practice together.

Rothschild is a former executive at General Mills who went on to use his business experience, acumen, and connections to form a nonprofit, Twin Cities RISE!, whose purpose is to reduce concentrated poverty. Notice the use of the word purpose where you might expect to see mission. Organizations need both, Rothschild explains. Purpose guides programming, reinforces a long-term perspective, and encourages flexibility and innovation. Mission, on the other hand, is concrete. It sets out what the organization does (and does not) do; it puts the purpose into operation. Rothschild's first principle? "Have a clear and appropriate purpose."

The other principles are worth bearing in mind also - measure what counts, be market driven, create mutual accountability, support personal empowerment, and be learning driven. All good things, and Rothschild's take on them, particularly the mutual accountability and personal empowerment sections, are worth reading. Alert readers will have notice that so far I have mentioned only six principles.

The seventh is "Create Economic Value from Social Benefit." In this chapter, Rothschild argues that since every improvement in social good creates social value it is up to nonprofits to quantify who is receiving the value and what it is worth. As he puts it, our current approach to financing social service work "doesn't make sense." Why not? We have big problems, but pay small organizations to deal with them. The problems are systemic, but we spend money in one to two year cycles. Nonprofits focused on these issues don't have access to the long-term capital that for-profits do.

Rothschild outlines two alternative approaches to what he calls the "no financial return expected" approach - the one that government and philanthropic funders use now. These approaches go beyond performance based contracting as it is practiced now. Both depend on calculating a firm and credible return on investment.

I have discussed one of the alternatives, social impact bonds, here (as well as its cousin health impact bonds, here.) Rothschild's alternative, and he appears to have invented the concept (and founded an organization that worked for passage of a Minnesota law to try it out), is the Human Capital performance bond. One explanation of Human Capital Bonds is:

Human Capital Performance Bonds . . . are State AA "annual appropriation" bonds that fund high-performing human services.

The bonds pay market rates, and might be bought by private investors, financial institutions, or social investors. As Rothschild puts it:

The basic mechanism of the [human capital bond] is simple: if the social outcomes that a nonprofit generates create economic value that is greater than the state's cost of borrowing the funds, then the state will have both a social and economic incentive to sell the bonds.

Here's a diagram of how it works:

Human Capital bonds have yet to come into existence, even in Minnesota, and there are lots of questions remaining to be answered. For one interesting take, see this article.

Using Outcome Measures

While some sectors of the not-for-profit world have been using outcome measures for many years, others providers are still doubtful about the benefits and costs of using them. What are outcome measures? Outcome measures are simply standards that a program's staff sets to determine whether the program is accomplishing what it exists to accomplish.

So what is an outcome? It's a benefit or change for participating individuals or populations during or after the program's activities. For a foster care program, an outcome might be reducing length of stay in placement. For an HIV prevention program, it might be reducing the number of new infections. (Note: While these are simple statements, developing and calculating the outcomes should be a considered process. That will be the subject of a later post.)

The advantages of using carefully thought through outcome measures are many. For example, you can use them with outsiders in some of the follow ways.

• Once you have outcome measures, you can state clearly what you have accomplished. For example, Twin Cities RISE, a program designed to provide employers with skilled workers - mostly men from communities of color -  states on its website that people who complete its programs increase their annual income by over 355%, and 70% remain employed in the second year after placement.
• Using outcome measures lets all stakeholders know what your expectations for service providers are. For example, in New York City the Administration for Children's Services Scorecard sets out a detailed process of review, rating and rebuttal for providers of foster care and preventive services. You can download a copy of the Scorecard methodology here.
• More and more, funders like government and foundations are requiring outcome measures,  are basing funding decisions on prior outcomes, or letting performance-based contracts. In addition, accrediting agencies often require outcome measures. Having a system in place means access to more funding opportunities.

Sometimes directors or managers believe that they don't have time, or staff, or money to develop or use outcome measures, or that their clients or services are too specialized to make generalizations possible. They're wrong. There are also many internal reasons to develop and use outcome measures:

• Your staff can see whether their efforts really make a difference in people's lives. 
• If not, then staff are in a position to suggest changes or improvements to programs.
• You can target effective services for expansion or replication.
• You can compare program performance from year to year.

This post is part of a series about developing and using high quality management indicators.You can see related posts here and here.

Data managers and proxy data

I've been thinking about measurement lately (see also here for a slightly different view). So I read this post in the HBR Blog Network with some interest. In it, two consultants recommend that corporations need a Chief Data Officer - and I think it's a concept that has an analogy in the not-for-profit world as well. The CDO's purpose, say the authors, Anthony Goldbloom and Merav Block, is "oversight and evangelism at the highest levels of management." Specifically, the person:

• Figures out how data can be used to support the organization's most important priorities - it's easy, after you've gone to the trouble of setting up outcome measures and other data management systems, to think that the functions can now be handed off to mid-level staff. But there's always something pressing that new or different data might help resolve.
• Keep checking to make sure the organization is collecting the right data - what was once an outcome may now be an output, or you may have obtained funding to do further followup. Having a senior manager responsible for thinking about what is collected allows organizations to collect the data they need. It also allows you the possibility of experimenting with different service models because you can compare data. 
• Ensuring the organization is able to collect the needed data - a top level view combined with the clout of a senior manager means that the organization will probably make better decisions about allocating limited resources for developing or enhancing data systems.

Update: Just after I posted this I came across an article that nicely illustrates the second point. There is a great deal of uncertainty whether hurricanes are increasing in number along with global warming - not all hurricanes strike land, and while we have good records from the 1970s forward (when we started using satellites to track hurricanes) that's not enough time to tell whether we are seeing natural fluctuations are a change.

Now a scientist at the University of Copenhagen, Alex Grinsted has published a report looking not at hurricanes but at storm surges, which have been measured reliably by tide gauges since the 1920s. It's a nice use of a different type of data that helps you find the information you are looking for. Grinsted's conclusion?

“Using surges as an indicator  . . . we see an increase in all magnitudes of storms when ocean temperatures are warmer.” As ocean temperatures have risen inexorably higher in the general warming of the planet due to human greenhouse-gas emissions, the scientists concluded, hurricane numbers have moved upward as well. The implication: they’ll keep increasing along with global temperatures unless emissions are cut significantly.

"God's Hotel" by Victoria Sweet

Update, August 20, 2012: For an insight into which direction Big Medicine may be moving, see Atul Gawande's article "Big Med" in last week's New Yorker. (As of this writing it's not behind a pay wall but that may change.)

I am a person who believes in using numbers to understand and manage programs, and who regularly asserts with some confidence that anything can be measured. And that those measurements can be put to work constructively. So it was with an increasing sense of discomfiture that I read Victoria Sweet's wonderful memoir, "God's Hotel." Sweet, a doctor and medical historian, worked at Laguna Honda Hospital as it made a difficult transition from hospital to modern health care facility. When Sweet first started working there Laguna Honda was one of the very few descendants remaining in the United States of 18th and 19th century almshouses that provided care for the chronically disabled. It had open wards and lots of nursing care, almost no computers, and very few departments - doctors, not techs or specialists, read the x-rays they had ordered. By the time she ends her book, several investigations, lawsuits, and one Ph.D. later, it had become a modern facility with private rooms and many administrative staff including utilization managers, but fewer nurses. What happened at Laguna Honda is only one small ripple of a much larger wave social changes playing out in the country, but seeing the jolting as doctors and patients moved from backwater to modern medical center is instructive.

Sweet pairs the story of the hospital's move forward into her own explorations back in time, into the world of premodern medicine, or medicine as it had been practiced for most of history through the end of the 19th century. Her particular focus was Hildegard of Bingen, a 12th century mystic and author of various medical texts. Working at Laguna Honda, Sweet learns the virtues of caring for the whole person, the body, and also what she thinks of as the anima, the force that animates the body (in the premodern sense). "Even when there is nothing to do for a patient, there is something to do," she says. Her example is a new pair of glasses. Along the way, Sweet explores the history and development of hospitals, from medieval monastery hospices for the poor through de-institutionalization to our modern tertiary care centers. Sweet argues that the old system had some virtues that have been lost in the transition.

And what are those virtues? Things that we all think we need more of: time, peace and quiet in which to heal, good food, and, well, sex. Medieval thinkers tried to keep the humors in the body in balance, and as she learns more, Sweet expands the metaphor. If you think of the body as a plant, the immediate environment is the garden, and the garden and plant contribute to each other's equilibrium. After finishing her doctoral work, Sweet celebrated by following the medieval pilgrimage route from Le Puy to Santiago de Compostela. Along the way, Sweet thought about her often difficult patients. Her lessons are several: hospitals are open to those in need. As a doctor, she provided care. Following the pilgrimage route, Sweet became the stranger experiencing hospitality. This will come in useful, as eventually everyone, including doctors and hospital visitors, will become a patient.

Sweet brings her garden metaphor into work with her. Laguna Honda provided care for the indigent, the homeless, people with complex triple diagnoses of mental illness, drug addiction, and the complex medical problem that had landed them in the hospital. Often, patients had no home to go to. Sometimes they had partners or street friends who took advantage of them; they smoked and drank illicitly even at Laguna Honda. These were not model patients; they were not easy ones. But the hospital allowed doctors lots of time, time to sit and reflect, to spend time with each patient, which Sweet explains often gave her the time to find a diagnosis lurking among all the information in the chart. Time also gave patients time to heal. But time is the one thing that modern medicine doesn't allow, at least time in the hospital: it's too expensive. Unfortunately, and this is where things become uncomfortable for the analyst, time was one of the first things to be eliminated. Face-to-face contact was another victim, replaced by paperwork: forms, procedures, policies.

And forms, procedures, and policies all have their place, as does aggregating data about individuals. The changes were hard on everyone; as Sweet puts it, "the practice of medicine had become the delivery of health care." It's hard to argue with that conclusion. But I think there is some place for worrying about outcomes. There's a difference between counting units of service for their own sake, and counting something  -- the output -- as it contributes toward an outcome you are seeking. It's possible that at least some of the work Sweet describes was about outputs, without outcomes. Second, there's no use in developing outcome measures without thinking about the context in which the work is taking place. Some of the patients discharged under the new managed utilization regime returned, sicker than before. Others were never heard of again.

Everyone who has ever been a patient, or known one, should read this book. At the moment, administrators are in control of hospitals, but perhaps the pendulum will swing again. Taking the long view, I will give Sweet the last word:

It was true that medicine and nursing and administration had always been at odds . . . but then I remembered what Florence Nightingale had written about the struggle between medicine and nursing and administration. That struggle was irresolvable and should not be resolved, she said, because it was in the patients' best interest. If medicine ever won control of the hospital, too much would be practiced on the patient; if administration, too little; if nursing, medical progress would be curtailed in the interest of the spiritual and emotional care of the patient.

There's a nice slideshow of images from the old Laguna Honda hospital here.

Image above via http://jsri.jp/English/ojo/2008/sanfran/zenhospice.html

How looking at outcomes can give you useful information

Here's another example showing how thinking about outcome measures, and using information available, can spark a larger discussion about priorities and solutions. In today's print and on-line New York Times, reporter Eduardo Porter writes, in an article titled "Numbers Tell of Failure in Drug War,"

If there is one number that embodies the seemingly intractable challenge imposed by the illegal drug trade on the relationship between the United States and Mexico, it is $177.26. That is the retail price, according to Drug Enforcement Administration data, of one gram of pure cocaine from your typical local pusher. That is 74 percent cheaper than it was 30 years ago. [Link in original.]

Porter goes on from there to discuss US and Mexican policies and identifies other numbers, such as the large numbers of people in US prisons, and how many are there for carrying small amounts of marijuana, and how much it costs. And there are uncountable numbers of people dead in Mexico. Porter uses the numbers to raise a number of possible policies the US could adopt. There are a lot of possible arguments among the alternatives, which Porter discusses. But one thing is clear: the decline in the street price of most drugs is a pretty good indicator - outcome measure - that the policies we have adopted are may not be working.