Toolkit: POMP To Go Part 1

Resource Topic Guide

The POMP website contains all the information and tools necessary to conduct performance-related surveys of Older Americans Act service recipients on the state and local level. These tools may also be useful to other social service and support programs.

POMP Tutorial for the POMP Toolkit (WMV, 7 minutes) to familiarize you with the POMP web pages.

AoA Performance Measurement Toolkit

The Administration on Aging (AoA), U.S. Department of Health and Human Services, supported the Performance Outcome Measurement Project (POMP) to develop tools and procedures to measure the impact of programs funded under the Older Americans Act (OAA). POMP was a collaborative effort between AoA and grantees that represented State Units on Aging (SUAs) and Area Agencies on Aging (AAAs), with their providers and university research partners across the country.

Complete Toolkit

Contents

  • Acknowledgements
  • Chapter 1. Introduction
  • Chapter 2. Measure Program Performance Through Survey Data
  • Chapter 3. Select the Survey Instrument
  • Chapter 4. Determine the Data Collection Method
  • Chapter 5. Develop the Work Plan and Budget
  • Chapter 6. Select the Client Sample
  • Chapter 7. Administer the Survey
  • Chapter 8. Enter, Review, and Analyze the Data
  • Chapter 9. Prepare the Report and Disseminate the Results
  • Chapter 10. Compare the Results to Other National Surveys
  • Appendix A: Survey Instruments (Mail and Telephone)
  • Appendix B: Supplemental Materials

AoA Program Evaluations and Related Reports

AoA program evaluations insure that: the most relevant data are available to policy makers; programs demonstrate value to the taxpayer; and programs have a track record of results. AoA strives to evaluate programs in an integrated manner combining process, outcome, impact and cost-benefit analysis of evaluation activities. This site provides links to reports and results from these evaluation efforts.

Completed Evaluations and Studies

  • Process evaluation of the Chronic Disease Self-Management Education Program (CDSME):
    In September 2011, AoA awarded Contract HHSP233201100492G to IMPAQ International and Altarum Institute. This process evaluation examined state CDSMP programs funded through Communities Putting Prevention to Work: Chronic Disease Self-Management Program, an initiative of the Administration for Community Living/Administration on Aging (AoA) in collaboration with the Centers for Disease Control and Prevention (CDC) and the Centers for Medicare & Medicaid Services (CMS). The research team employed a multi-method approach using multiple qualitative and quantitative data sources. The program data were also used to conduct regression analyses to examine the influence of various factors on participant completion rates. The report provides information about: characteristics of State grantees, CDSMP participants, CDSMP implementation, participant completion rates, site-level data collection, program sustainability, and recommendations for program improvements. The final process evaluation report with Appendices is now available. In addition to the process evaluation, ACL is coordinating with CMS’ efforts in evaluating evidence-based wellness programs. For additional information about this grant program please also see the reports created by ACL’s technical assistance grantee the National Council on Aging at https://www.ncoa.org/resources/
  • Serving Elders at Risk: The Older Americans Act Nutrition Programs – National Evaluation of the Elderly Nutrition Program, 1993-1995 (Institute of Medicine of the National Academy of Sciences/National Research Council)
  • Real People Real Problems: An Evaluation of the Ombudsman Programs of the Older Americans Act (The National Academies Press)
  • Assessment of Title III-D of the Older Americans Act: Disease Prevention and Health Promotion (Research Triangle Institute)
  • 2004 Survey of Adult Protective Services: Abuse of Adults 60 Years of Age and Older (National Center on Elder Abuse)
  • Information Systems Management Study (NASUA/Westat) – Final Report
  • Title III-B Supportive Services Evaluation

Reports on Caregiver Programs

Ongoing Studies

Evaluation of the Aging and Disability Resource Centers (ADRCs)

As of 2012 there are almost 500 ADRC sites across the nation. Since the inception of this initiative, ADRCs have had over 4.8 million contacts with consumers, caregivers, providers and professionals. The focus of the evaluation is to determine the extent to which Aging and Disability Resource Centers (ADRCs) are fulfilling the goal of improving awareness of and access to long term supports and services for older adults and individuals with disabilities and also how well ADRCs are contributing to the overall AoA mission. The process data collection will include all State and local ADRC grantees and the outcome study sample will include approximately 3,000 consumers from 43 sites (23 ADRCs and 20 AAAs). AoA received OMB Paperwork Reduction Act clearance and the process data collection started April 1, 2013 and will be completed during the summer of 2013. The outcome data collection started in May of 2013 and is expected to run for approximately 6 months. You can view the training provided to sites participating in the outcome study (YouTube).

The final report is expected to be completed in late FY 2014. The results of this evaluation will influence future performance measures and indicators. This work is being completed by IMPAQ International with Abt Associates under contract HHSP233201000692G.

Evaluation of Title III-C Elderly Nutrition Services Program (ENSP)

SUA AOA Flyer

Under contract HHSP233201200606G, Mathematica Policy Research, Inc. is conducting a process evaluation and cost study of the ENP; the third since the program was created in 1972. The primary research question that the evaluation is designed to answer is, how effective and efficient is the OAA Title III-C Program at addressing the Program’s legislative intent and program goals of helping to keep older Americans healthy and active in their homes and communities and prevent the need for more costly interventions through the provision of healthy meals, social interaction, health promotion, and linking older adults to other appropriate services. This overarching research question addresses two primary and two secondary research domains: Primary Domains:

    • Program Efficiency: streamlining costs, increasing service cost-effectiveness, leveraging resources and eliminating redundancy;
    • Program Effectiveness and Client Outcomes: targeting services to older adults with greatest social or economic need; extent of integration with other home- and community-based long-term care services; client service outcomes such as those associated with a comprehensive and coordinated food and nutrition service system, including health promotion and disease prevention, and maintenance in the community rather than placement in a nursing home; and client satisfaction with services.

Secondary Domains:

      • System Infrastructure: policies, resources, use of technology, staff skills, methods of program administration and service delivery.
      • Interagency Partnerships: Program coordination and the outcomes of collaborative efforts with other agencies, organizations and stakeholders.

Evaluation of Title III-E National Family Caregiver Support Program (NFCSP)

This national, comprehensive evaluation of the NFCSP will be the first for this relatively new program. Several reports on early implementation activities have been produced but until now the programs have not been mature enough for an evaluation of client outcomes. The Lewin Group, Inc, under award HHSP23337031T, is conducting the evaluation with the overall purpose understanding and documenting the extent to which and how NFCSP goals are being met and if the investment is producing a high quality, cost-effective program. The evaluation seeks to assess the impact of the program at the individual (family caregivers and the persons for whom they care), program, and LTC policy/HCBS system levels.

The evaluation is organized around three broad research areas:

      • How does the program meet its goals (legislative, state, local, and provider level)? Do caregivers have easy access to a high quality, multi-faceted system of support and services that meets caregivers’ diverse and changing needs and preferences? What systems need to be in place in order to achieve this?
      • What are the outcomes and impacts on caregivers and care recipients? Does the caregiver support provided through the NFCSP have an effect on quality of life and health outcomes for caregivers and care recipients? To what extent has the NFCSP enhanced caregiving? Has it enabled care recipients to remain in their homes and communities when that is the family’s desired outcome?
      • Has the program contributed to LTC system efficiency? How is the NFCSP integrated or coordinated with other long-term care programs and what is the effect? Does the NFCSP contribute to any cost savings in the long-term care system by helping families avoid or delay costly institutionalization?

Evaluation Design Projects

Evaluation design for the Long Term Care Ombudsman Program (LTCOP)
In September 2011 the Administration for Community Living (ACL) awarded contract HHSP233201100500G to NORC at the University of Chicago (NORC) to develop an evaluation study design to better understand and assess the effectiveness of Long-Term Care Ombudsman Programs (LTCOPs). The task included building the evidence base on LTCOPs in order to develop recommendations for a rigorous and comprehensive study design that investigates program efficiency and program effectiveness at multiple levels, including the resident/family, facility, local/state/program, and federal levels. Key tasks of the design process involved the development of a family of four logic models and a set of overarching research questions to guide the evaluation, as well as the identification of data collection tools and data sources that inform those questions. The diversity of proposed activities reflects the ACL’s goals for this evaluation, the commitment to a population health frame of reference, and seven critical LTCOP characteristics that influence design options. The final evaluation design report (PDF, 2MB) was completed in January 2013.

Evaluation Design for a Global Evaluation of Older American Act (OAA) Services

Under this project, the Administration on Aging (AoA) seeks to study the impact of OAA programs and services, specifically services provided under Title III, on key outcomes including HCBS use, health care use, community tenure, and long-term services and supports (LTSS) expenditures. OAA services and programs are diverse, often integrated and/or provided in combination with other services, funded through multiple funding streams, and administered and delivered by different state and local-level agencies with varying data collection capacity. This design focuses on the impact of services provided through OAA funded HCBS programs authorized under Titles III-B, C, D, and E of the OAA. These programs include a range of supportive services, nutrition services, health promotion and disease prevention programs, as well as services for family caregivers. While the mix and type of services offered differ by state and locality, the vast majority of OAA funding is used for the provision of nutritional services. Proposed research questions include:

      • What is the impact of OAA funded HCBS programs and services on:
        • Community tenure
        • Health care utilization
        • Costs of care for older adults (e.g., LTSS, health care costs)
        • Physical, mental, and emotional health and wellness (e.g., preventive measures) of care recipients and caregivers
        • Unmet needs among older adults
        • Caregivers (e.g., strain, burden, depression, health, etc.)
        • Coordination of services (e.g., care management)?
      • What is the impact of OAA services alone or in combination with services paid for by other sources?
      • What is the impact of service mix and intensity on outcomes of interest? [if possible, we will isolate OAA services]
      • What subgroups had the most favorable outcomes? (e.g., health conditions, demographics, functional status)

A primary recommendation from the final evaluation design report is that the Administration for Community Living (ACL) first facilitate an exploratory study to determine the feasibility and limitations of conducting this evaluation with state OAA data. Such a study will assist ACL in identifying actions needed to better position the states for participation in a nationwide evaluation.

Performance Measurement and Evaluation Definitions

http://www.gao.gov/assets/80/77277.pdf

This publication supersedes GAO-05-739SP, Performance Measurement and Evaluation: Definitions and Relationships, June 2005. Both the executive branch and congressional committees need evaluative information to help them make decisions about the programs they oversee–information that tells them whether, and why, a program is working well or not. In enacting the Government Performance and Results Act of 1993 (GPRA), Congress expressed frustration that executive and congressional decision making was often hampered by the lack of good information on the results of federal program efforts. To promote improved federal management and greater efficiency and effectiveness, GPRA instituted a government-wide requirement that agencies set goals and report annually on performance. Many analytic approaches have been employed over the years by the agencies and others to assess the operations and results of federal programs, policies, activities, and organizations. Most federal agencies now use performance measures to track progress towards goals, but few seem to regularly conduct in-depth program evaluations to assess their programs’ impact or learn how to improve results. Individual evaluation studies are designed to answer specific questions about how well a program is working, and GPRA explicitly encourages a complementary role for these types of program assessment. The GPRA Modernization Act of 2010 aims to improve program performance by requiring agencies to identify priority goals, assign officials responsibility for achieving them, and review progress quarterly. Complete and accurate information on how well programs are working and why will be key to its success. This glossary describes and explains the relationship between two common types of systematic program assessment: performance measures and program evaluation. Based on GAO publications and program evaluation literature, it was first prepared in 1998.