Evaluating Projects of a NORC Program
Guiding Principle # 2
Each project evaluation is sharply focused and designed to be practical.
Actions and Considerations
Decide what data you need to collect and how you will collect it. Generally, you should first document the problem you are addressing—for example, before beginning a project to prevent falls, you need baseline data on the number of senior residents at risk or the percentage of seniors that have received flu shots. Then, you should measure again after a project has been in place for a period of time, typically six months, to make comparisons. (See Deepdale CARES NORC Program Profile.)
Keep your measures simple and streamlined. Sometimes, you can answer a critical question just by counting heads—for example, you can get information about social engagement by counting how many seniors participate in a friendly visiting initiative. The measures you choose depend on what you are evaluating, the audience who will review the findings, and the resources available for the undertaking. Where possible, build on data your NORC program already collects regularly.
Consider both quantitative and qualitative approaches. Combining both approaches helps to confirm your findings and highlights key insights.
- Quantitative methods are more structured, usually involving the objective analysis of surveys and questionnaires.
- Qualitative approaches, which include interviews and focus groups, help you collect participant opinions, perceptions, and experiences. These tend to be more open-ended and descriptive, and will generally yield more anecdotal insights.
Count your outputs. The next step in completing your logic model is to list outputs. Outputs are linked directly to your activities and show the quantifiable results of what you did. Usually, they are measured in terms of volume—the number of classes taught, the number of people served, the amount of material distributed. Outputs alone do not offer insight about the effect of a project, but they are necessary for measuring outcomes.
Identify your outcomes. The final step in completing your logic model is to record outcomes, which are changes in conditions, behavior, attitudes, skills, or knowledge that occur as a result of participating in a project. There are three types of outcomes:
- Immediate outcomes typically focus on near-term changes and generally demonstrate changes in participation rates, awareness, knowledge, perception, or beliefs – for example, a comparison of the number of providers using standardized fall assessment tools before and after attending a training session on preventing falls among seniors.
- Intermediate outcomes demonstrate changes in practice and behavior. An example may be an increase in shared care planning among providers of seniors at risk of falling. Or an increased client self-advocacy or confidence around falls. Intermediate outcomes are often the most practical way to determine whether a NORC project has made a positive difference in the lives of the people it serves.
- Long-term outcomes are generally consequences of the project, including demonstrated changes in use of services, health status, and quality of life, such as a reduction in risks of falls among senior residents.
Focus on tracking outputs and immediate outcomes. These are your building blocks. Showing success here suggests a project is making progress towards longer-range outcomes. Long-term outcomes themselves are often difficult to measure because they are subject to many influences and can take a long time to become evident.
Intermediate-term outcomes are often the most practical way to feel confident that a project has made a positive difference in the lives of the people it serves. For example, if ten providers attended a falls prevention training session (that’s the output), began using a standardized falls assessment tool (that’s the immediate outcome), and increased shared care planning among providers (intermediate outcome), you can feel more confident that your project will ultimately reduce the risk of falls themselves (long-term outcome).
Keep in Mind
A streamlined evaluation can still be rigorous. Recognizing the limits of staff and resources at most NORC programs, measures should be no more complicated or broad than necessary.
Getting early agreement from all partners about what you are measuring, and how, lessens the chance of conflict over time.