Evaluation: Why? What Kind? For Whom?
Evaluation has gotten a bad rap. Although viewed by many as an accountability tool that will tell the board which heads should roll, evaluation is really a tool for organizational learning.
In fact, at the April 15 Governance Matters Round Table, Evaluation: Why? What Kind? For Whom? it was suggested that the internal evaluators be titled "Chief Learning Officers" or "Directors of Organizational Excellence."
The titles summarize the consensus of panelists and participants: Evaluation is a tool that helps a nonprofit find out what impact it is having on its client community, what makes programs effective, and how programs can be improved to have greater and better impact.
That's not the heads-will-roll approach many fear when the word "evaluation" comes up or that boards may envision when they decide to implement organizational or program evaluation.
Overcoming this view of evaluation at every level is the first task of consultants and internal evaluators. While evaluation is often driven by the need for accountability and the demands of funders for outcome data the real value is in organizational learning. If the organization is not learning something about itself, why invest time and money in evaluation?
Why Evaluate?
Evaluation provides information that can and should:
- Inform program design;
- Make sure programs are having the desired impact on clients;
- If not having the desired impact, why not?
- Guide corrections needed to improve impacts.
Nonprofit staff want to have a positive impact on the communities they serve. By framing evaluation as a way to quantify and improve that impact, staff will be more willing to undertake the extra work of evaluation.
For funders, evaluation clarifies the organization's goals. Most important, however, is what the organization does with the data from the evaluation. How is the data used to improve programs and client services? It is important for funders to realize that evaluation costs time and money. If they want data from their grantees, they should provide the money or training needed to develop an evaluation program.
For the board, evaluation can show which programs are most effective and why. This information can guide budget decisions. For example, an after-school sports program may find that the relationship between client and coach is the most important factor in program effectiveness. With this information, the board may decide that money should be allocated to hiring more coaches rather than to buying new equipment.
The board should understand that an evaluation is not a guide to who should be fired and who retained. It is a guide to help staff correct problems and allocate resources most effectively. It's about using information to improve, not punish.
Evaluation is all about learning and using that learning to build better programs with better outcomes.
When Not to Evaluate
If a program is new still in the development phase outcomes cannot be realistically measured. One can evaluate whether the program is going in the direction it should and use the data to make mid-course corrections. Outcome measurement should wait until the program is established.
Also, do not undertake a long-term research project in the guise of evaluation. Longitudinal studies are generally the stuff of academic research and not useful to the evaluation of controllable impacts of particular programs.
As an example, it is unrealistic for a program that serves 3rd graders to collect data on the high school graduation rates of its clients. Too many factors intervene between 3rd grade and high school. It is realistic to collect data on characteristics that affect high school graduation rate, such as self-confidence and hope for the future. These can be measured and improved at the 3rd grade level.
It is a waste of money to measure outcomes over which your organization has little control.
Preparing for Evaluation
First, make sure that everyone knows evaluation is about learning, not about retribution. Then be sure the data reflects what the organization and staff want to know and need to know in order to improve.
Staff will give their time if they will find out something they care about, and they care about how they are doing and whether they are making things better for their clients.
Define what you can measure that shows if you are making a change. Evaluation is not one measurement. It involves multiple instruments at different points in time and with different people clients, staff, parents.
Participation by board and staff even clients is essential in defining what will be evaluated and how the data will be used. Not only are staff viewpoints valuable in defining what outcomes should be and how they might be measured, but including staff in the design will help allay fears that evaluation is about firing people. Effective evaluation relies on staff buy-in because staff time will be taken up by the process.
Client input may reveal factors that affect performance but may have been unrecognized by staff or require rethinking of program design. Such factors might include, for examples the realities of the community in which the client lives, peer pressure, and the economic situation in the family.
Check out the Good Governance Guide for more information about Program Evaluation and Organizational Evaluation.
Moderator
Judy Katz, president On Target Strategies, works with organizations and small businesses to design a more effective future through program assessment and performance metrics.
Panelists
Amy Barger is managing director of Tiger Foundation, which provides financial support to nonprofit organizations and training to philanthropists.
Kim Sabo Flores is director of evaluation and organizational learning at ActKnowledge, which connects social change practice with the study of how and why initiatives work.
Kira Krenichyn, Ph.D., director of program evaluation at Ramapo for Children, teaches graduate courses in assessment and educational psychology at Hunter College.
|
