Assessing a Healthcare Program/Policy Evaluation
- Review the healthcare program or policy evaluation and reflect on the criteria used to measure the effectiveness of the program or policy described.
The Assignment: (2–3 pages)
Based on the program or policy evaluation you selected, complete the Healthcare Program/Policy Evaluation Analysis Template. Be sure to address the following:
- Describe the healthcare program or policy outcomes.
- How was the success of the program or policy measured?
- How many people were reached by the program or policy selected?
- How much of an impact was realized with the program or policy selected? Assessing a Healthcare Program/Policy Evaluation
- At what point in program implementation was the program or policy evaluation conducted?
- What data was used to conduct the program or policy evaluation?
- What specific information on unintended consequences was identified?
- What stakeholders were identified in the evaluation of the program or policy? Who would benefit most from the results and reporting of the program or policy evaluation? Be specific and provide examples.
- Did the program or policy meet the original intent and objectives? Why or why not?
- Would you recommend implementing this program or policy in your place of work? Why or why not?
- Identify at least two ways that you, as a nurse advocate, could become involved in evaluating a program or policy after 1 year of implementation. Assessing a Healthcare Program/Policy Evaluation
Ongoing program evaluation is just as important for self-management support programs as it is for other service delivery programs. Purchasers and builders will want to routinely receive information that allows them to assess the program’s operation and performance, especially whether it ultimately benefits patients with chronic illnesses. Yet there is at present no standard format for such information, and a purchaser or builder of a self-management support program will find a broad array of possible evaluation measures. To some extent, the choice depends on the program’s main goals, but a selection of endpoints may lead to the best understanding of how the program is working. In the following section, “Evaluation Measures,” we describe the range of evaluation measures revealed by our literature review and expert interviews and discuss the key issues that were raised. Additional methodological issues are discussed in the next section, “Evaluation Methodology.” Assessing a Healthcare Program/Policy Evaluation
Program success may be assessed at many points along the chain of effects presented in Figure 1. One can examine whether:
- Program structure matches what was called for in the contract.
- Coaches are engaging eligible patients and performing the self-management support activities.
- Patients’ knowledge and self-efficacy have increased.
- Patients’ health-related behaviors have changed.
- Rates of provider adherence to guidelines have increased.
- Disease control has improved.
- Patient health outcomes have improved. Assessing a Healthcare Program/Policy Evaluation
- Patient satisfaction has improved.
- Utilization has declined, and patient productivity has improved.
- Health care costs have declined.
Each of these interim and long-term goals is important and provides a possible endpoint for evaluation. They are discussed in more detail below.
Tables 3a, 3b, and 3c provide an overview of the endpoints that are examined in the recent literature on existing self-management support programs. The columns in the table parallel the boxes in Figure 1. It can be seen that endpoints all along the chain of effect (following program structure) have been utilized by researchers. Among our interviewees, some use of each category was reported. Neither our literature review nor our interviews identified research on programs that measured endpoints in all categories. Utilization, costs, provider behavior, and disease control endpoints received the greatest emphasis, most likely reflecting considerations of data availability, reliability, and other tradeoffs associated with different data sources Assessing a Healthcare Program/Policy Evaluation
Measures of Program Structure
Commonly, purchasers of external services look for accreditation of a program as a measure of the program’s structural soundness. While few studies in the literature used structural measures for program evaluation, a purchaser or developer might want to monitor whether the program has the components and features that were called for in the original plan or contract. One might also question if the features are plausibly capable of supporting the kinds and extent of self-management support activities desired. Are the staff and caseload as expected? Do staff members have the qualifications and training to perform their duties? Can they reasonably be expected to support the kinds of activities envisioned for the intended number of patients? Are procedures and protocols in place to ensure that coaching tasks, as well as education, are performed? Structural measures also may be used to assess whether claims about the program’s success based on other measures are plausible. Assessing a Healthcare Program/Policy Evaluation
Most structural measures rely on information from the program’s management. Patient self-report is another data source to consider. The Disease Management Association of America (DMAA) Participant Satisfaction Survey, for example, asks patients if their program has a toll-free number they can call.49
Self-management Support Process Measures
Another approach to evaluation is to monitor the performance of the program staff by examining the extent to which they perform the tasks and activities intended; i.e., how well the process of providing support to chronically ill patients actually works. Not only are program process measures critical for program supervision and management, they also can tell purchasers and developers if the program is being implemented according to plan. Measures of reach and implementation can help reveal factors that contribute to success or failure and be useful for monitoring of staff performance and program improvement. As shown in Table 3, process measures in the program literature have focused on the reach of the program, processes for assessing patients’ self-management skills and needs, education processes, and coaching processes Assessing a Healthcare Program/Policy Evaluation