2010-18013. Maternal, Infant, and Early Childhood Home Visiting Program

AGENCY:

Health Resources and Services Administration and Administration for Children and Families, HHS.

ACTION:

Request for public comment on criteria for evidence of effectiveness of home visiting program models for pregnant women, expectant fathers, and caregivers of children birth through kindergarten entry.

SUMMARY:

The Health Resources and Services Administration and Administration for Children and Families, HHS, solicit comments by August 17, 2010 on proposed criteria for evidence of effectiveness of home visiting program models for pregnant women, expectant fathers, and primary caregivers of children birth through kindergarten entry. Final criteria for evidence of effectiveness will be included in the program announcement inviting eligible entities to apply for funding under the Affordable Care Act Maternal, Infant, and Early Childhood Home Visiting Program.

SUPPLEMENTARY INFORMATION:

Invitation to Comment: HHS invites comments regarding this notice, both on the proposed criteria and proposed methodology for HHS's systematic review of the evidence. To ensure that your comments have maximum effect, please identify clearly the specific criterion or other section of this notice that your comment addresses.

1.0 Purpose of Program

The Affordable Care Act (ACA) Maternal, Infant, and Early Childhood Home Visiting program is designed to strengthen and improve home visiting programs, improve service coordination for at risk communities, and identify and provide comprehensive evidence-based home visiting services to families who reside in at risk communities. The legislation specifies that most program funds must be used for “evidence-based” home visiting program models. This Notice (1) proposes criteria to be considered in assessing whether home visiting models have evidence of effectiveness, and (2) describes the methodology for a systematic review of evidence, applying the criteria proposed in this Notice, which HHS is currently conducting. The Notice solicits comments on both items.

2.0 Background

2.1 Legislative Context

On March 23, 2010, the President signed into law the Patient Protection and Affordable Care Act of 2010 (Pub. L. 111-148) (also known as the Affordable Care Act or ACA); historic legislation designed to make quality, affordable health care available to all Americans, reduce costs, improve health care quality, enhance disease prevention, and strengthen the health care workforce. Through a provision adding Section 511 to Title V of the Social Security Act to create the Maternal, Infant, and Early Childhood Home Visiting program, the Act responds to the diverse needs of children and families in at risk communities and provides an historic and unique opportunity for collaboration at the Federal, State, and local level to assure coordination and delivery of critical health, development, early learning, and child abuse and neglect prevention services to most effectively serve these children and families. By supporting evidence-based home visiting program models, the ACA Maternal, Infant, and Early Childhood Home Visiting program plays a crucial role in national efforts to build quality, comprehensive statewide early childhood systems for pregnant women, parents, and caregivers, and children from birth to 8 years of age.

The ACA Maternal, Infant, and Early Childhood Home Visiting Program is designed: (1) To strengthen and improve the programs and activities carried out under Title V; (2) to improve coordination of services for at risk communities; and (3) to identify and provide comprehensive services to improve outcomes for families who reside in at risk communities. At risk communities will be identified through a statewide assessment of needs and of existing resources to meet those needs. HHS intends that the home visiting program will result in a coordinated system of early childhood home visiting in every State that has the capacity to provide infrastructure and supports to assure high-quality, evidence-based practice.

The program enables eligible entities to utilize what is known about effective home visiting services to provide evidence-based programs to promote: improvements in prenatal, maternal and newborn health; child health and development including prevention of child injuries and maltreatment and improvements in cognitive, language, social-emotional, and physical development; parenting skills; school readiness; reductions in crime or domestic violence; improvements in family economic self-sufficiency; and improvements in the coordination and referrals for other community resources and supports.

2.2 Use of Funds for “Evidence-Based” Programs

Section 511(d)(3)(A) of Title V, as amended by the Affordable Care Act, reserves the majority of grant funds for home visiting program models with evidence of effectiveness based on rigorous evaluation research. The legislation specifies that models must meet the following requirements in order to be considered “evidence-based”:

(I) The model conforms to a clear consistent home visitation model that has been existence for at least 3 years and is research-based, grounded in relevant empirically-based knowledge, linked to program determined outcomes, associated with a national organization or institution of higher education that has comprehensive home visitation program standards that ensure high quality services delivery and continuous program improvement, and has demonstrated significant, (and in the case of the service delivery model described in item (aa), sustained) positive outcomes, as described in the benchmark areas specified in paragraph (1)(A) and the participant outcomes described in paragraph (2)(B), when evaluated using a well-designed and rigorous—

(aa) randomized controlled research designs, and the evaluation results have been published in a peer-reviewed journal; or

(bb) quasi-experimental research designs.

The legislation charges the Secretary of Health and Human Services with establishing criteria for evidence of effectiveness of the home visiting program models and ensuring the process for establishing the criteria is transparent and provides the opportunity for public comment.

This Notice (1) proposes criteria to be considered in assessing whether home visiting models have evidence of effectiveness and (2) describes the methodology for a systematic review of evidence, applying the criteria proposed in this Notice, which HHS is currently conducting. The Notice solicits comments on both items. After comments are received, HHS will finalize the criteria and methodology and complete the systematic review of the available evidence of effectiveness of selected home visiting program models.

It is expected that eligible entities will also have an opportunity to present documentation in their applications for the ACA Maternal, Infant, and Early Childhood Home Visiting program to demonstrate that additional home visiting models meet the final criteria. Such documentation will be reviewed by HHS using the same procedures applied in HHS' systematic review and described below.

The criteria proposed in this notice apply only to the home visiting program for States and territories authorized by Section 511(c) of Title V. Criteria for the ACA Tribal Maternal, Infant, and Early Childhood Home Visiting Program authorized by Section 511(h)(2)(A) of Title V will be issued separately. Based on a careful review of available research evidence on home visiting interventions with Tribal populations, the Secretary will develop alternative evidence-based criteria for identifying home visiting models likely to improve outcomes for families in Tribal communities.

3.0 Proposed Criteria for Evidence of Effectiveness

A home visiting model must have been (1) evaluated using rigorous methodology and (2) shown to have a positive impact on outcomes in order to meet criteria for evidence of effectiveness. The following two types of criteria (3.1 and 3.2) must be met in order for a home visiting model to be considered evidence-based for the purposes of the Maternal, Infant, and Early Childhood Home Visiting Program:

3.1 Criteria for Well-Designed, Rigorous Impact Research

In order to ensure the highest probability of producing unbiased estimates of program impacts, there are a number of variables that should be considered. These variables include study design (i.e. randomized controlled trial [RCT] or quasi-experimental design [QED]), level of attrition, baseline equivalence, reassignment of participants from one condition to another in the trial, the reliability and validity of outcome measures studied, and confounding factors.

Two types of impact study designs have the potential to be both well-designed and rigorous: Randomized controlled trials and quasi-experimental designs. HHS proposes to define randomized controlled trials as a study design in which sample members are assigned to the program and comparison groups by chance. Randomized control designs are often considered the “gold standard” of research design because personal characteristics (before the program begins) do not affect whether someone is assigned to the program or control group. HHS proposes to define a quasi-experimental design as a study design in which sample members are selected for the program and comparison groups in a nonrandom way. For example, families may self-select into groups (deciding whether they want services or not) or an administrator may assign families to groups based on family risk factors. Quasi-experimental designs are considered weaker than randomized controlled trials because characteristics that may be related to outcomes, such as motivation or need, may also influence whether someone is in the program or comparison group.

HHS proposes that an impact study will be considered high, moderate or low quality depending on the study's capacity to provide unbiased estimates of program impact. Studies that are rated “high” and “moderate” (see Table 1 below), therefore, would meet requirements to be considered “well-designed, rigorous impact research.” In brief, the high rating would be reserved for random assignment studies with low attrition of sample members and no reassignment of sample members after the original random assignment. The moderate rating would apply to studies that use a quasi-experimental design and to random assignment studies that, due to flaws in the study design or execution (for example, high sample attrition), do not meet all the criteria for the high rating. To receive the moderate rating, studies would have to demonstrate that at the study's onset, the intervention and comparison groups were well matched on specified measures (i.e. baseline equivalence), such as a pretest measure of targeted outcomes or race and maternal education. Studies that do not meet all of the criteria for either high or moderate quality would be considered low quality.

As summarized in Table 1, the rating scheme would consider five dimensions: (1) Study design, (2) attrition, (3) baseline equivalence, (4) reassignment, and (5) confounding factors.

Table 1—Criteria for Well-Designed, Rigorous Impact Research

Rating Criteria	Rating
	High	Moderate	Low
Study design	Random assignment	Quasi-experimental design with a comparison group; random assignment design with high attrition or any reassignment	Studies that do not meet the requirements for a high or moderate rating.
Attrition	Meets “What Works Clearinghouse” (Dept. of Education) standards for acceptable rates of overall and differential attrition	No requirement
Baseline equivalence	No requirement other than random assignment; Statistically significant differences must be controlled	Must establish baseline equivalence of study arms on selected measures (see Table 1, Note 2 below)
Reassignment (see Table 1, Note 1 below)	Analysis must be based on original assignment to study arms	No requirement
Confounding factors	Must have at least two participants in each study arm and no systematic differences in data collection methods	Must have at least two participants in each study arm and no systematic differences in data collection methods

Table 1, Note 1: In random assignment studies, deviation from the original random assignment (for example, moving families from the treatment to the control group) can also bias the impact estimates. Therefore, in order for a RCT to meet our criteria for the high rating, the analysis must be performed on the sample as originally assigned. Subjects may not be reassigned for reasons such as contamination, noncompliance, or level of exposure. RCTs that somehow alter the original random assignment but otherwise meet the criteria for the high rating are considered for a moderate study rating, provided they meet the other criteria for that rating. Our criteria are similar to those developed by the WWC, which allows a study to be downgraded as a result of reassignment.

Table 1, Note 2: When possible, baseline equivalence should be established on outcomes of interest. For some studies it is not feasible to collect baseline measures on the outcome of interest, for example, children's outcomes when baseline is collected prenatally. For all studies, baseline equivalence must be established on two demographic factors: (1) The parent or child's race and ethnicity and (2) socioeconomic status.

3.2 HHS Proposed Criteria for Evidence of Effectiveness of a Home Visiting Service Delivery Model

In order to have confidence in the impact estimates created from a high or moderate quality study design, a number of variables should be considered. These variables include statistical significance, whether impacts are sustained, and whether the impacts were found for the full sample or only for non-replicated subgroups.

3.2.1 The ACA Maternal, Infant and Early Childhood Home Visitation Program legislation includes a number of participant outcome and benchmark areas. In determining program effectiveness HHS proposes to examine programs for impacts in the following eight program domains:

(1) Maternal health

(2) Child health

(3) Child development and school readiness, including improvements in cognitive, language, social-emotional or physical development

(4) Prevention of child injuries and maltreatment

(5) Parenting skills

(6) Reductions in crime or domestic violence

(7) Improvements in family economic self-sufficiency

(8) Improvements in the coordination and referrals for other community resources and supports.

3.2.2 Taking into account the legislative language and the two types of criteria discussed in 3.1 and 3.2 above, HHS proposes to consider a program model eligible for evidence-based funding for the purposes of the ACA Maternal, Infant, and Early Childhood Home Visiting Program if it meets the following minimum criteria:

• At least one high- or moderate-quality impact study (see 3.1) of the program model finds favorable, statistically significant impacts in two or more of the eight outcome domains (see 3.2.1); or
• At least two high- or moderate-quality impact studies using different samples (see 3.1) of the program model find one or more favorable, statistically significant impacts in the same domain (see 3.2.1).

In both cases, the impacts considered must be found either for the full sample or, if found for subgroups but not for the full sample, impacts must be replicated in the same domain in two or more studies using different samples.

Additionally, if the program model meets the above criteria based on findings from randomized control trial(s) only, then one or more impacts in an outcome domain must be sustained for at least one year after program enrollment, and one or more impacts in an outcome domain must be reported in a peer-reviewed journal (consistent with section 511(d)(3)(A)(i)(I)).

Isolated positive findings, and impacts found only for a subgroup, but not the full sample in a study, raise concerns about false positives that may be artifacts of multiple statistical tests rather than reflecting true impacts. The requirements for replication of positive findings across samples or for findings in two or more outcome domains are meant to guard against this problem. HHS recognizes the importance of subgroup findings for determining impacts on subgroups of the population of interest, including specific racial or ethnic groups, and plans to report information on subgroup findings, whether replicated or not.

4.0 Proposed Methods for HHS's Systematic Review of Evidence of Effectiveness

HHS is conducting a comprehensive and detailed program model-by-model review of the available evidence of effectiveness of home visiting programs that support the following legislatively specified benchmarks and outcomes: Maternal health; child health; child development and school readiness including improvements in cognitive, language, social-emotional, and physical development; prevention of child injuries and maltreatment; parenting skills; reductions in crime or domestic violence; improvements in family economic self-sufficiency; and improvements in the coordination and referrals for other community resources and supports.

The review is being carried out through a contract to Mathematica Policy Research, Inc. and led by the Administration for Children and Families in collaboration with the Health Resources and Services Administration, the Office of the Assistant Secretary for Planning and Evaluation, and the Centers for Disease Control and Prevention. The review will apply the HHS criteria proposed above to determine which of the program models reviewed meet the criteria for evidence of effectiveness. The review will be completed after comments on this notice are received and considered.

4.1 Review Process

To conduct a through and transparent review of the home visiting program model research literature, the systematic review project is following five main steps, the first three of which have been provisionally completed. Comments on steps 4 and 5 are especially encouraged.

1. Conduct a broad literature search;

2. Screen studies for relevance;

3. Prioritize program models for review;

4. Rate the quality of impact studies with eligible designs;

5. Assess the evidence of effectiveness for each program model.

In addition, the project plans to review and make available implementation information for each program. Steps taken to address potential conflicts of interest are also described below.

4.1.1 Step 1: Conduct a Broad Literature Search

The literature search included four main activities:

1. Database Searches. The project team searched on relevant key words in a range of research databases. Key words included terms related to the service delivery approach, target population, and outcome domains emphasized in the Patient Protection and Affordable Care Act. The initial search was limited to studies published since 1989; a more focused search on prioritized program models included studies published since 1979 (see Prioritizing Programs below).

2. Web Site Searches. The project team used a custom Google search engine to search more than 50 relevant government, university, research, and nonprofit Web sites for unpublished reports and papers.

3. Call for Studies. In November 2009, Mathematica issued a call for studies and sent it to approximately 40 relevant listservs for dissemination.

4. Review of Existing Literature Reviews and Meta-Analyses. The project team checked search results against the bibliographies of recent literature reviews and meta-analyses of home visiting programs and added relevant missing citations to the search results.

The literature search yielded approximately 8,200 unduplicated citations, including 150 articles submitted through the call for studies.

4.1.2 Step 2: Screen Studies for Relevance

The project team then screened all citations identified through the literature search for relevance. Studies were screened out for the following reasons:

• The model under study did not use home visitation as a primary service delivery strategy. Programs that are primarily center-based with infrequent or supplemental home visiting were excluded. In order to be considered a home visiting model, a program must offer home visiting services to most or all participants and these services must be integral to programmatic goals. Visits should occur solely or primarily where participating families reside but occasionally may occur elsewhere if the families are homeless or uncomfortable conducting visits in the home. The services could be voluntary or mandated (for example, court ordered).
• The study did not use an eligible design as described in 3.1 above (randomized controlled trial, quasi-experimental design). The project team also included any studies on the implementation of specific home visiting models. These studies were used in the implementation reports described in section 5.0 of this Notice.
• The program did not include pregnant women and families with children from birth to kindergarten entry.
• The study did not examine any outcomes in the domains of: Maternal health and/or child health; child development and school readiness; reductions in child maltreatment; reductions in juvenile delinquency, family violence, and crime; positive parenting practices; and family economic factors. The legislatively specified domain of improvement in coordination and referrals for community resources and supports was not used in screening because of challenges in specifying discrete measures.
• The study did not examine a clear home visiting program model. For example, the study might focus on a specific home visiting strategy, such as comparing the use of professional and paraprofessional home visiting staff within home visiting program model broadly rather than a specific program model. Without a clearly identified program model, the evidence review could not use the impact study to assess the effectiveness of a specific program.
• The study was not published in English. This limitation reflects practical considerations, given the limited time available for the review.
• The study was published before 1989 for the initial search or 1979 for the focused search on prioritized program models. These limitations balance practical considerations, given limited time available, and were designed to ensure that seminal research was included.

4.1.3 Step 3: Prioritize Program Models for Review

After screening, the initial search yielded studies on more than 250 home visiting program models. Timing and resources do not allow for a detailed review of all of these home visiting program models prior to the implementation of the ACA Maternal, Infant, and Early Childhood Home Visiting Program. For each model the team examined the number and design of impact studies, sample sizes of the impact studies, the availability of implementation information, whether the program was currently in widespread use in the U.S., and whether the program had been implemented only in a developing-world context. The project staff eliminated programs that had no information available about implementation, were implemented only in a developing-world context, or were no longer in operation and provided no support for implementation. This decision was made so that resources could be focused on reviewing program models that States and territories would be readily able to implement and that would be likely to meet other statutory requirements.

4.1.4 Step 4: Rating the Quality of Impact Studies

For the purposes of the systematic review, HHS plans to assign each impact study a rating of high, moderate, or low, per the criteria described in 3.1 above.

4.1.5 Step 5: Assessing Evidence of Effectiveness

After rating the quality of all available impact studies for a program, HHS plans to assess the evidence across all studies of the program models that received a high or moderate rating and measured outcomes in at least one of the legislatively specified participant outcome domains utilizing the HHS proposed criteria for evidence for effectiveness discussed in 3.2 above.

5.0 Implementation Reviews

To assist in implementation of the ACA Maternal, Infant and Early Childhood Home Visiting Program, the project plans to collect and publish information about implementation of the prioritized program models. The project plans to provide two kinds of implementation reports for each program model. One implementation report will focus on the support available to assist interested entities to implement the model (such as program model technical assistance staff or trainings) or infrastructure required to implement the model (such as the purchase of a specific data management system or curricula). The second kind of implementation report will focus on implementation experiences during the impact trials or in implementing the model in the field. These reports will provide information on the study samples in the impact trials, describe the locations where the specific model has been implemented, the average number of visits the participants receive, any available research on adaptations of the program models and lessons learned about implementing the models that have been reported in the available research.

6.0 Addressing Conflicts of Interest

All members of the review team have signed a conflict of interest statement in which they declared any financial or personal connections to developers, studies, or products being reviewed and confirmed their understanding of the process by which they must inform the project director if such conflicts arise. The review team's project director assembled signed conflict of interest forms for all project staff and subcontractors and monitors for possible conflicts over time. If a team member is found to have a potential conflict of interest concerning a particular home visiting program model being reviewed, that team member is excluded from the review process for the studies of that program model. In addition, reviews for two programs evaluated by Mathematica Policy Research are being conducted by contracted reviewers who are not Mathematica^® employees.

7.0 Future Allocations Based on Application Strength

To encourage exemplary programs and direct Federal funds where they can have the greatest impact, HHS plans to allocate the ACA Maternal, Infant and Early Childhood Home Visiting Program funding available in future years that exceeds funding available in FY 2010 competitively based upon States' capacity and commitment to improve child outcomes specified in the statute through improvements in service coordination and the implementation of home visiting programs with fidelity to high-quality, evidence-based models. HHS plans to evaluate applications based on multiple criteria and invites comments on what criteria are appropriate. Among the criteria, HHS proposes to give significant weight to the strength of the available evidence of effectiveness of the model or models employed by the State. In this context, the use of program models satisfying the criteria outlined in section 3.2.2 would be a minimal requirement, but HHS would consider additional criteria that further distinguish models with greater and lesser support in evidence. HHS is committed to ensuring that these criteria are transparent, methodologically sound, and increase the likelihood that federal funds will contribute to improved outcomes for at-risk children and families.

There are a number of different ways that such a system could be structured. We invite comments on the proposal to distinguish among evidence-based models based on a rubric that weighs factors relating to research quality and findings. For example, one relatively simple approach would rate models using an index constructed by weighting several factors equally. Models might be given points for meeting each of the following criteria: Favorable impacts sustained at least one year after program completion, favorable impacts replicated in distinct samples, favorable impacts in studies conducted by independent evaluators, quality and relevance of outcome measures; and balance between favorable and unfavorable and null findings. Additional factors which might be considered could include further indicia of the quality of the research design and implementation (as reflected in randomization, sample size, attrition, and baseline equivalence). We invite comments on HHS' proposal to use evidence for program models as a factor in determining allocations of additional funds, how various factors should be weighed in assessing the evidence of effectiveness, how to define these categories, and any other role distinctions related to the strength of the evidence should play in funding allocation. As noted above, strength of evidence is proposed to be only one factor in the evaluation of the strength of States' applications, and we invite comments on other appropriate factors as well.

8.0 Future Considerations

We invite comment on the following:

• HHS anticipates the criteria for evidence-based models will likely need to be altered over time as the state of the field changes. If HHS believes the criteria need to be changed in future years, it is anticipated the public will have an opportunity to comment on the proposed revisions.
• HHS intends to review the evidence base for home visiting models on an ongoing basis to ensure that new evidence is incorporated.

9.0 Submission of Comments

Comments may be submitted until August 17, 2010 by e-mail to: [e-mail].

Dated: July 19, 2010.

Mary K. Wakefield,

Administrator, Health Resources and Services Administration.

Carmen R. Nazario,

Assistant Secretary, Administration for Children and Families.