subscribe by email
In 2007 Booz Allen Hamilton set out to help small nonprofits in the Washington, D.C. area gain access to expert advice and guidance on management and fundraising that was otherwise unattainable for them due to cost. In the four years since, the Booz Allen Hamilton Nonprofit Development Conference Series has grown exponentially and now successfully serves more than 400 unique nonprofit organizations with top tier guest speakers from around the metropolitan area who provide the sorely needed advice and guidance.
Measurement is more than a good thing to do – it’s the right thing
By Bea Boccalandro, Boston College Center educator
Is it ethical to refuse a child tutoring services in order to produce a report on the effectiveness of such services? Should you divert resources from programming to measurement when there are more children to serve, families to help, problems to solve and an otherwise overwhelming number of unmet needs?
If actions speak for themselves, then corporate citizenship professionals have answered a resounding “NO!” We have long refused to invest in social sector impact measurement, which measures whether a program generates the ultimate community change it purports to generate, because it means fewer resources for service delivery.
|
“We can’t measure whether our program truly makes a difference without cutting services,” corporate citizenship professionals often say, “so we just can’t afford to prove impact.” Faced with zero-sum funding decisions, we overwhelmingly choose more people served over more knowledge gained. We consider it heartless to favor an evaluation report over a child. Virtually every corporate citizenship program supports services. Precious few know whether such services make an impact.
Yet, how ethical is it to enroll an underprivileged child in a program we do not know to be effective? How respectful is it to not even bother to determine if the program is helping them? Services extended to victims of domestic violence for decades proved to be ineffective when they were eventually assessed. How can that delay in assessing those services – while thousands of women and children (mostly) suffered – be considered acceptable?
We expect our doctors to give us the probability of success of treatment options but we don’t give this information to those we serve. Paradoxically, by reacting compassionately to the needs around us we might be responding unethically.
The cost-effectiveness of our decision is also suspect. Based on the body of impact evaluations done in the past five years, it appears no more than half of nonprofit services generate the purported change – be it reducing high school dropouts, improving mental health or reducing crime – to a meaningful degree. By eschewing impact evaluation, we tacitly accept that half of our social sector investments are unproductive and that it’s acceptable to remain in the dark as to which half. Forgoing impact evaluation in favor of delivering services is shortsighted. It’s a commitment to activity, not to change.
We don’t pit activity against efficacy in other endeavors. We don’t build power plants before being assured that they will produce power. We don’t put MRI machines in every hospital before being reasonably certain that they will work. We don’t mass produce brooms that haven’t been shown to sweep dust. R&D departments, pilot studies and beta tests are testaments to our willingness to invest the bulk of our resources in testing of effectiveness.
Yet, when it comes to corporate citizenship, we are hesitant to spend a meaningful proportion of a project’s budget on evaluation – even if it’s the only way to obtain evidence of its effectiveness. As a result, we support programs, toolkits, services and other “solutions” without knowing if they solve anything. Ensuring that our corporate citizenship actually works should be our first order of business even if – for a period of time, at least – it swallows a considerable amount of our corporate citizenship resources.
I do not advocate that we allow the homeless man to go hungry and unsheltered on a cold night while we perfect a job-training curriculum. It is a human instinct and a moral obligation to respond to suffering, independent of long-term impact. However, we should be clear about the purpose of our corporate citizenship and hold ourselves accountable accordingly. If we aim to bring someone in from the cold, there is no need to measure whether they are employed a year later. We only need to know that they are sheltered and warm. If, on the other hand, we design a program to employ the homeless, it is our managerial and moral obligation to know that it increases the chances of employment before we offer it to thousands of individuals.
If we are generous with impact evaluation investments, most technical constraints melt away. Longitudinal studies are possible. Control groups to determine causality can often be arranged. Measuring long-term meaningful change is within reach.
Once impact evaluation identifies what works, the need for continued impact evaluation is minimal, allowing us to invest heavily in those services that are known to work. Society has invested resources to prove that interventions such as vaccinations, reading to children and exercise work. We no longer need to track whether an immunized child will be disease-free, whether a student who is read to early in life will perform better in school, or whether a person who exercises will improve their health. A corporate citizenship effort that focuses on any of these areas need not spend resources on impact evaluation. It can dedicate itself to serving as many people as possible, knowing that the bulk of them will truly be helped. Unfortunately, these are exceptions. Most corporate citizenship programs have little to no assurance of their effectiveness.
Some pioneering companies have already made evaluation of their social sector impact an integral part of their corporate citizenship program. The Best Buy Children’s Foundation, for example, produced a research report, “Teen Voice 2009: The Untapped Strengths of 15-Year-Olds” that is used to help ensure Best Buy’s @15 TM corporate citizenship program makes a difference in the lives of teenagers. Furthermore, IBM’s Corporate Service Corps program, an international community service assignment for high-potential IBM employees, commissioned an independent evaluation in its launch year that showed, among other things, that most recipient nonprofit and government organizations reported improved leadership and strategic direction due to Corporate Service Corps.
Other corporate citizenship programs might want to follow the lead of Best Buy and IBM. For in failing to properly invest in impact evaluation, we are likely squandering our corporate citizenship investments, the hopes of those we mean to help and the opportunity to maximize social sector benefits.
Impact evaluation is not a luxury. It is a neglected necessity that can elevate corporate citizenship to unprecedented efficacy and higher moral grounding.
February 10th, 2010 at 8:44 AM
An outstanding article Bea. This is a pervasive issue that must be addressed, and the strong compelling argument of your article is exactly what needs to be said.
Non-profits have be required to produce, in some cases, exacting data and complicated measurement processes in order to qualify for foundation grants and public money. While I think that sometimes these requests are onerous and beyond any kind of usefulness, on the whole, they’ve been a helpful development within the world of philanthropy.
But here’s the thing. Non-profits almost always have fewer resources and capacities for measurement related processes. While many of the businesses who posit this dichotomy of ‘doing good’ or ‘measuring good’ have tremendous resources and capacity for measurement.
To my way of thinking, it is a bit suspect to spend the appropriate amount of money on evaluating all areas of the business while relegating ‘doing good’ to the category of ‘good enough’.
True Corporate Citizenship and CSR treats community engagement as an integral and necessary part of the overall business. Let’s take a moment and see if we’re doing as much good as we believe we are.
I just don’t think I could agree more with you on this one Bea, thanks for saying it so well!
Chris Jarvis
Co-Founder, Senior Consultant: Realized Worth
chrisjarvis@realizedworth.com
c: 317-840-8955
Toronto, Canada | Indianapolis, Indiana
February 10th, 2010 at 7:26 PM
Agree – very well said, Bea! I know that for us, as corporate citizenship and CSR continue to become a more integral part of the business, it will start to necessitate better measurement of programs.
Another thought is as corporations realize the importance and value of measurement (and put fund behind it) they can partner with the aforementioned nonprofits to conduct evaluations.
February 11th, 2010 at 9:41 AM
the article is so good, that reading it mulitple times is a good recommendation – thx Bea!
The comments are intriguing, too, though I think we have to take into account the role of public sector and civil socitey (also with the respective role in different countries).
Leaving the funding of measurement effort entirely to businesses is overdoing it. By the way, the immediate issue would be that NPOs then depend on businesses for getting the ‘well done’-mark (from a measurement point of view). Not sure we want this.
Taking as example volunteering as one potential target for better measurement I would think we ought to consider all sectors being part of it: civil society, business and public sector. So a common approach ( a ‘standard’, as those usually tend to develop a life of their own) appears to be key.
February 12th, 2010 at 11:49 AM
Hello everybody. I’m from Guayaquil – Ecuador (South America). I found this article very very intersting. I have worked for many years in different social projects.
I worked, for instance, in a literacy program for adults. I worked there for about 4 years and, because it was a educational program, we could meassure our success according to the number of people who finished their elementary school.
However we had no idea of the real impact that our program had over society. For instance, we graduated one hundred adults each year but what happened with them? Did they found a better job? Did they continue stuying their high-school? Did any of them ever finish the university? Are they still more proud of themselves that when they were unliterated? To be honest I don’t know.
I also worked in a program for children who worked in streets. We graduated 30 students from that program but haven’t meassure our real impact. Some girls became pregnant just some months after they leaved the program… some boys are in gangs… of course many kids (now young adults) are excellent citizens and they work very hard to keep reaching more goals. But what happened to the rest?
What were the factor that make possible for some kids to change and other dont’t? Family influence? Neighborhood influence? As in the case before, I don’t now. I can guess but I don’t have any scientif investigation that can give real conclusions.
I still believe that if just one kid or just one adult improve their life-stile, everything worth the effort. But I wish I have a evaluation that can give me some clues about how I can warranty some kids or adults that they will be successful.
February 12th, 2010 at 11:59 AM
Bea, great article. In my view measurement is a double edged sword. As a Corporate Responsibility Practitioner and passionate about sustainability, I embrace the quote from Einstein ‘Not everything that can be counted counts, and not everything that counts can be counted’. But I also strongly advocate being able to quantify and demonstrate benefit otherwise you don’t spend limited resources most cost effectively.
I wrote a provocative post on my own blog recently suggesting that the money we have spent in Haiti might have been more cost effective, in terms of suffering alleviated, had we spent it creating a better infrastructure before the earthquake than rescuing people after.
I think we have to find a way for the two approaches to coexist. In fact, I see that as a key characteristic of sustainability. We also need to make sure that we measure the right thing for the right reasons and don’t allow meeting of quantified targets to distract us from taking a holistic perspective.
April 18th, 2010 at 8:34 PM
I whole heartedly agree that outcome evaluation is critical to proving and improving the efficacy of any program whether it is being implemented by a private or non-profit entity. I currently work for a pharmaceutical company who spends a tremendous amount of money on each and every “investment opportunity.” This scrutiny starts when a product is in the research pipeline (controlled clinical trials), through a product’s pre-launch (research using physicians as key opinion leaders/advisors), through the post marketing studies to prove a product’s longer term efficacy and safety in treating a targeted serious disease. In the U.S. Pharma group, we would not even consider investing in an advertising campaign without conducting focus groups to determine a proposed campaign’s effectiveness at delivering the intended message regarding the product in question.
I find the concept of an independent program evaluator especially intriguing. It would be really cool if there was an independent nonprofit program evaluation review board to lend credibility to program evaluation results and allow a:
· volunteer to compare nonprofit programs to determine where best to spend his/her limited volunteer time.
· foundation or individual donator to compare among “nonprofit investments.”
· private corporation to determine the risk that a nonprofit program may fail and thereby taint the corporation’s reputation in the community.