Evaluation of Healthy Families Massachusetts

Five-tiered Approach

Our evaluation of HFM (both MHFE-1 and MHFE-2) was and is rooted in Jacobs’s Five-tiered Approach to evaluation,¹ a developmental model that moves evaluation activities from a primary focus on descriptive and process-oriented information to an emphasis on program effects.² Tier One activities produce needs and demand assessments, and usually are conducted prior to the program’s implementation. Evaluation activities at Tiers Two and Three are directed at program processes: they describe program staff, services, clients, and costs; examine program implementation compared to model standards; and provide feedback to programs for improvement. Tiers Four and Five focus on outcome evaluation activities, assessing the extent to which a program is meeting its short-term and long-term goals. The primary difference between Tier Four and Tier Five is the use of an experimental design in Tier Five; when such scientific rigor is possible, researchers are more confident that changes they observe in participants are the result of the intervention being studied.

Description of Evaluation Activities for the Five-tiered Approach:
 

Tier	Purposes of Evaluation	Types of Evaluation Activities
Tier 1: Needs Assessment	Document the size and nature of a public problem Determine unmet need for services in a community Propose program and policy options to meet needs Set a data baseline from which later progress can be measured Broaden the base of support for a proposed program	Review existing community, county, and state data Determine additional data needed to describe problem and potential service users Conduct "environmental scan" of available resources Identify resource gaps and unmet need Set goals and objectives for intervention Recommend program model
Tier 2: Monitoring and Accountability	Monitor program performance Meet demands for accountability Build a constituency Aid in program planning and decision making Provide a groundwork for later evaluation activities	Determine needs and capacities for data collection and management Develop clear and consistent procedures for collecting essential data elements Gather and analyze data to describe program along dimensions of clients, services, staff, and costs
Tier 3: Quality Review and Program Clarification	Develop a more detailed picture of the program as it is being implemented Assess the quality and consistency of the intervention Provide information to staff for program improvement	Review monitoring data Expand on program description using information about participants’ views Compare program with standards and expectations Examine participants’ perceptions about effects of program Clarify program goals and design
Tier 4: Achieving Outcomes	Determine what changes, if any, have occurred among beneficiaries Attribute changes to the program Provide information to staff for program improvement	Choose short term objectives to be examined Choose appropriate research design, given constraints and capacities Determine measurable indicators of success for outcome objectives Collect and analyze information about effects on beneficiaries
Tier 5: Establishing Impact	Contribute to knowledge development in the field Produce evidence of differential effectiveness of treatments Identify models worthy of replication	Decide on impact objectives based on results of Tier 4 evaluation efforts Choose appropriately rigorous research design and comparison groups Identify techniques and tools to measure effects in treatment and comparison groups Analyze information to identify program impacts

Adapted from Jacobs (1988) for use in Jacobs, Easterbrooks, Brady, & Mistry (2005)

[1] Jacobs. F. (2003).  Child and family program evaluation:  Learning to enjoy complexity.  Applied Developmental Science, 7(2), 62-75; Jacobs, F. H. (1988). The Five-tiered Approach to evaluation: Context and implementation. In H. B. Weiss & F. H. Jacobs (Eds.), Evaluating Family Programs, New York: Aldine DeGruyter; Jacobs, F.,&  Kapuscik, J. (2000). Evaluating family preservation services: A guide for state administrators. Medford, MA: Tufts University.
[2] Since the first MHFE was initiated, the Five-tiered Approach has been used in other home visiting evaluations, including a recent evaluation of Healthy Families Alaska (see Duggan, A., Rodriguez, K., Burell, L., Shea, S., & Rohde, C. (2005). Evaluation of Healthy Families Alaska: Final report: January 21, 2005. 
Downloaded June 19, 2006 from http://www.hss.state.ak.us/ocs/Publications/JohnsHopkins_HealthyFamilies.pdf.