2018-13420. Decisions Related to the Development of a Clearinghouse of Evidence-Based Practices in Accordance With the Family First Prevention Services Act of 2018

AGENCY:

Administration for Children and Families, HHS.

ACTION:

Request for public comment.

SUMMARY:

The Administration for Children and Families, HHS, solicits comments by July 22, 2018 on initial criteria and potential candidate programs and services for review in a Clearinghouse of evidence-based practices in accordance with the Family First Prevention Services Act of 2018. The Clearinghouse will identify promising, supported, and well-supported practices for mental health and substance abuse prevention and treatment programs, in-home parent skill-based programs, and kinship navigator programs appropriate for children who are candidates for foster care pregnant or parenting foster youth, and the parents or kin caregivers of those children and youth.

SUPPLEMENTARY INFORMATION:

Invitation to Comment: HHS invites comments regarding this Notice. To ensure that your comments have maximum effect, please identify clearly the section of this Notice that your comment addresses.

1.0 Background and Legislative Context

The Family First Prevention Services Act (FFPSA) was signed into law as part of the Bipartisan Budget Act (H.R. 1892) on February 9, 2018. FFPSA enables States to use Federal funds available under parts B and E of title IV of the Social Security Act to provide enhanced support to children and families and prevent foster care placements through the provision of evidence-based mental health and substance abuse prevention and treatment services, in-home parent skill-based programs, and kinship navigator services. As described in the statutory language, these services and programs are intended “for children who are candidates for foster care or who are pregnant or parenting foster youth and the parents or kin caregivers of the children”.

FFPSA requires an independent systematic review of evidence to designate programs and services as “promising,” “supported,” and “well-supported” practices, defined as follows in section 471(e)(4)(C):

• Promising Practice: “A practice shall be considered to be a `promising practice' if the practice is superior to an appropriate comparison practice using conventional standards of statistical significance (in terms of demonstrated meaningful improvements in validated measures of important child and parent outcomes, such as mental health, substance abuse, and child safety and well-being), as established by the results or outcomes of at least one study that—(1) was rated by an independent systematic review for the quality of the study design and execution and determined to be well-designed and well-executed; and (2) utilized some form of control (such as an untreated group, a placebo group, or a wait list study).”
• Supported Practice: “A practice shall be considered to be a `supported practice' if (I) the practice is superior to an appropriate comparison practice using conventional standards of statistical significance (in terms of demonstrated meaningful improvements in validated measures of important child and parent outcomes, such as mental health, substance abuse, and child safety and well-being), as established by the results or outcomes of at least one study that—(aa) was rated by an independent systematic review for the quality of the study design and execution and determined to be well-designed and well-executed; and (bb) was a rigorous random-controlled trial (or, if not available, a study using a rigorous quasi-experimental research design); and (cc) was carried out in a usual care or practice setting and (II) the study described in sub-clause (I) established that the practice has a sustained effect (when compared to a control group) for at least 6 months beyond the end of treatment.”
• Well-supported Practice: “A practice shall be considered to be a `well-supported practice' if (I) the practice is superior to an appropriate comparison practice using conventional standards of statistical significance (in terms of demonstrated meaningful improvements in validated measures of important child and parent outcomes, such as mental health, substance abuse, and child safety and well-being), as established by the results or outcomes of at least two studies that—(aa) were rated by an independent systematic review for the quality of the study design and execution and determined to be well-designed and well-executed; and (bb) were rigorous random-controlled trials (or, if not available, studies using a rigorous quasi-experimental research design); and (cc) were carried out in a usual care or practice setting and (II) at least one of the studies described in sub-clause (I) established that the practice has a sustained effect (when compared to a control group) for at least 1 year beyond the end of treatment.”

In accordance with FFPSA, practices must also meet the following requirements:

• Book or manual: The practice has a book, manual, or other available writings that specify the components of the practice protocol and describe how to administer the practice.
• No empirical risk of harm: There is no empirical basis suggesting that, compared to its likely benefits, the practice constitutes a risk of harm to those receiving it.
• Weight of evidence supports benefits: If multiple outcome studies have been conducted, the overall weight of evidence supports the benefits of the practice.
• Reliable and valid outcome measures: Outcome measures are reliable and valid, and are administrated consistently and accurately across all those receiving the practice.
• No case data for severe or frequent risk of harm: There is no case data suggesting a risk of harm that was probably caused by the treatment and that was severe or frequent (section 471(e)(4)(C)(ii) of the Act).

In order to meet these requirements, the Administration for Children and Families (ACF) in the Department of Health and Human Services (HHS) intends to establish and maintain a public Clearinghouse of practices, including culturally specific, or location- or population-based adaptations of practices, identified via a systematic review of evidence on relevant programs and services. In accordance with FFPSA and building from other federal evidence reviews, HHS is developing initial criteria that will be used to designate programs and services as promising, supported, and well-supported practices. HHS will also identify a preliminary list of candidate services and programs that will be considered for systematic review.

This Notice (1) identifies and requests comment on potential initial criteria for (a) identifying eligible programs and services for review by the Clearinghouse, (b) prioritizing eligible programs and services for review, (c) identifying eligible studies aligned with prioritized programs and services, (d) prioritizing eligible studies for rating, (e) rating studies, and (f) rating programs and services as promising, supported, and well-supported practices. This Notice (2) requests comment on potential programs and services that may meet the aforementioned criteria and that should be considered as candidates for systematic review. After comments are received, HHS will revise and publish the initial criteria and a preliminary list of candidate programs and services to be considered for review; and begin to conduct reviews. This Notice is one step in ensuring that activities associated with the development of a Clearinghouse are transparent and build from the existing knowledge of States, federal agencies, researchers, evaluators, program and service developers, key stakeholders and experts, and the general public.

2.0 Initial Criteria

2.1 Program or Service Eligibility Criteria. Programs or services may be eligible for inclusion in the Clearinghouse if they meet the following criteria developed in accordance with FFPSA statutory language [sections 471(e)(1)(B) and 471(e)(1)(c)]:

2.1.1 Types of Programs and Services. HHS intends to limit eligibility to mental health and substance abuse prevention and treatment services, in-home parent skill-based programs (including parenting skills training, parent education, and individual and family counseling), and kinship navigator programs.

2.1.2 Book/Manual/Writings Available. HHS intends to limit eligibility to programs or services that have a book, manual, or other available documentation that specifies the components of the practice protocol and describes how to administer the practice.

2.2 Program or Service Prioritization Criteria. Timing and resources may not allow for a detailed review of all programs and services determined to be eligible by the criteria detailed in section 2.1 Program or Service Eligibility Criteria. Programs or services may be prioritized for review on the basis of the following criteria:

2.2.1 Types of Programs and Services. As noted in 2.1.1. Types of Programs and Services, HHS intends to limit eligibility to mental health and substance abuse prevention and treatment services, in-home parent skill-based programs (including parenting skills training, parent education, and individual and family counseling), or kinship navigator programs. This Notice requests comment on the scope of programs and services and topic areas of interest within the aforementioned categories that should be prioritized for inclusion.

2.2.2 Target Population of Interest. HHS intends to prioritize programs or services for review that have been developed or used to target children and families involved in the child welfare system or populations similar to those involved in the child welfare system. This Notice requests comment on populations that may be considered “similar” to those involved in the child welfare system.

2.2.3 Target Outcomes. HHS intends to prioritize programs or services for review that aim to impact target outcomes. Target outcomes should be defined in accordance with FFPSA statutory language [section 471(e)(4)(C)] and include those outcomes that “. . . prevent child abuse and neglect, and reduce the likelihood of foster care placement by supporting birth families and kinship families and improving targeted supports for pregnant and parenting youth and their children.” These may include, but are not limited to, “. . . important child and parent outcomes, such as mental health, substance abuse, and child safety and well-being.” This Notice requests comment on which types of mental health, substance abuse, and child and family outcomes should be considered as `target outcomes' and requests research evidence to support recommendations of `target outcomes'. HHS does not intend to include access to service, satisfaction with programs and services, and referral to programs and services as `target outcomes'.

2.2.4 Number of Impact Studies. HHS intends to prioritize programs or services with at least two studies with non-overlapping analytic samples and distinct implementations examining effectiveness/impact.

2.2.5 In Use/Active. HHS intends to prioritize programs or services currently in use in the U.S. Programs or services that are no longer in operation or have no information available about active implementation will not be prioritized.

2.2.6 Implementation and Fidelity Support. HHS intends to prioritize programs or services that have implementation training and staff support and/or fidelity monitoring tools and resources available to implementers in the United States.

2.2.7 Trauma-Informed. HHS may also prioritize services and programs that have been implemented using a trauma-informed approach. FFPSA statutory language [section 471(e)(4)(B)] states, “The services or programs to be provided to or on behalf of a child are provided under an organizational structure and treatment framework that involves understanding, recognizing, and responding to the effects of all types of trauma and in accordance with recognized principles of a trauma-informed approach and trauma-specific interventions to address trauma's consequences and facilitate healing.” This Notice requests comment on the feasibility of prioritizing programs and services based on past implementation in accordance with trauma-informed principles.

2.2.8 Delivery Setting for In-Home Parent Skill-Based Programs and Services. HHS intends to prioritize in-home parent skill-based programs and services where the primary service delivery strategy takes place in the caregivers' place of residence.

2.3 Study Eligibility Criteria. HHS intends to engage in a broad literature search to identify studies examining prioritized programs and services. This search may include databases, websites, existing literature reviews, and meta-analyses. HHS intends to screen studies for eligibility using the following criteria:

2.3.1 Impact Study. HHS intends to limit eligibility to studies included in government reports and peer-reviewed journal articles that assess effectiveness (i.e., impact) using quantitative methods.

2.3.2 Target Outcomes. HHS intends to limit eligibility to studies that examine the impact of the service or program on at least one `target outcome', as described in section 2.2.3. Target Outcomes. As noted above, this Notice requests comment on specific outcomes in accordance with FFPSA statutory language that should be considered `target outcomes' and requests research evidence to support recommendations of `target outcomes'.

2.3.3 Conducted in the U.S., U.K., Canada, New Zealand, or Australia. HHS intends to limit eligibility to studies conducted with samples in the U.S., U.K., Canada, New Zealand, or Australia to ensure that the evidence base reflects the populations where programs and services will be implemented.

2.3.4 Study Published in English. HHS intends to limit eligibility to studies published in English.

2.3.5 Published or Prepared in or after 1990. HHS intends to limit eligibility to studies published or prepared in or after 1990.

2.3.6 Usual Care or Practice Setting. HHS intends to limit eligibility to studies carried out in a usual care or practice setting in accordance with FFPSA [section 471(e)(4)(C)]. This Notice requests comment on the definition of usual care or practice settings.

2.4 Study Prioritization Criteria. Timing and resources may not allow for a detailed rating of all studies determined to be eligible by the criteria identified in section 2.3 Study Eligibility Criteria. HHS intends to conduct a high-level scan of eligible studies to determine which should be prioritized for rating. This Notice requests comment on criteria that can be used to prioritize eligible studies for rating.

2.4.1 Implementation Period: FFPSA [section 471(e)(1)(A) and (B)] states that the Secretary may make a payment to a State for providing services or programs “for not more than a 12-month period”. This Notice requests comment on whether studies with program or service implementation periods of longer than 12 months should be considered for review and if so, whether any other implementation period cutoff should be included as a study prioritization criterion.

2.4.2 Sample of Interest. HHS intends to prioritize studies that include samples of children and families involved in the child welfare system or populations similar to those involved in the child welfare system. This Notice requests comment on populations that may be considered “similar” to those involved in the child welfare system.

2.5 Study Rating Criteria. HHS intends to rate studies on the following criteria:

2.5.1 Favorable Effects. HHS intends to rate studies based on whether they demonstrate at least one meaningful favorable effect (i.e., positive significant effect) on a `target outcome' as specified in section 2.3.2 Target Outcomes. A meaningful effect will be defined using conventional standards of statistical significance (i.e., two-tailed hypothesis test and a specified alpha level of p < .05). This Notice requests comment on whether and how ratings should consider the number or magnitude of favorable effects.

2.5.2 Unfavorable Effects. HHS intends to rate studies based on the number of unfavorable effects (i.e., negative significant effects) on either `target' or non-target outcomes as specified in section 2.3.2 Target Outcomes. Effects will be defined using conventional standards of statistical significance (i.e., two-tailed hypothesis test and a specified alpha level of p < .05). This Notice requests comment on whether and how studies should also be rated on the number of null effects on `target outcomes', and on whether and how ratings should consider the number or magnitude of unfavorable effects.

2.5.3 Sustained Favorable Effect. HHS intends for studies with at least one favorable effect on a `target outcome', as determined by the criteria in 2.5.1 Favorable Effects, to be rated on whether or not they demonstrate a sustained favorable effect. As noted in section 471(e)(4)(C), a `supported practice' must have at least one study that demonstrates “a sustained effect (when compared to a control group) for at least 6 months beyond the end of treatment” and a `well-supported practice' must have at least one study that demonstrates “a sustained effect (when compared to a control group) for at least 1 year beyond the end of treatment.” HHS intends to classify studies as not demonstrating a sustained favorable effect (i.e., effects are demonstrated for less than 6 months), demonstrating a sustained favorable effect of 6 months or more (but less than 12 months), or demonstrating a sustained favorable effect of 12 months or more.

2.5.4 Rigorous Study Design. HHS intends to rate studies as either high, moderate, or low on the rigor and appropriateness of their study design. Study designs that receive the highest rating will be either Randomized Controlled Trials (RCTs) or rigorous quasi-experimental designs. HHS defines randomized controlled trials as a study design in which sample members are assigned to the program or service and comparison groups by chance. Randomized control designs are often considered the “gold standard” of research design because personal characteristics (before the program or service begins) do not affect whether someone is assigned to the program or service or control group. HHS defines a quasi-experimental design as a study design in which sample members are selected for the program or service and comparison groups in a nonrandom way. Similar to criteria considered in other federal evidence clearinghouses, rigorous study designs will be those that are appropriately powered, include an appropriate control group, maintain original assignment to study arms, and are appropriate to combat threats to internal validity. This Notice requests comment on threats to internal validity that should be considered. This Notice requests comment on appropriate thresholds for evaluating and assigning a rating to a study design.

2.5.5 Rigorous Study Analysis. HHS intends to rate studies as either high, moderate, or low on the rigor and appropriateness of their analysis. Study analyses that receive the highest rating may be those that tested and established baseline equivalence, appropriately accounted for overall and differential sample attrition, appropriately accounted for multiple comparisons, and when necessary accounted for clustering. This Notice requests comment on appropriate thresholds for evaluating and assigning a rating to a study analysis.

2.5.6 Reliability, Validity, and Systematic Administration of Outcome Measures. HHS intends to rate studies as either high, moderate, or low on the extent to which `target outcome' measures are reliable (i.e., the extent to which a measure produces the same results when used repeatedly), valid (i.e., the extent to which a measure captures what it is intended to measure), and were administered consistently and accurately across all those receiving the practice in accordance with FFPSA statutory language [section 471(e)(4)(C)] or receiving the appropriate comparison practice. This Notice requests comment on appropriate thresholds for evaluating and assigning a rating to the reliability, validity, and administration of `target outcome' measures.

2.6 Program or Service Rating Criteria. HHS intends for programs or services to be rated as promising, supported, or well-supported practices if they meet the below criteria that collectively assess the strength of evidence for a practice and build from the individual study criteria described in section 2.5 Study Rating Criteria. These criteria were developed in accordance with FFPSA statutory language [section 471(e)(4)(C)].

2.6.1 Promising Practice: HHS intends to designate a program or service as a `promising practice' if the program or service has at least one study that demonstrates a favorable effect on a target outcome as described by criterion 2.5.1 Favorable Effects and achieves, at a minimum, moderate ratings on criteria 2.5.4 through 2.5.6.

2.6.2 Supported Practice: HHS intends to designate a program or service as a `supported practice' if the program or service has at least one study that demonstrates a favorable effect on a target outcome as described by 2.5.1 Favorable Effects, demonstrates a sustained favorable effect on a target outcome of at least 6 months beyond the end of treatment as described in Section 2.5.3 Sustained Favorable Effect, and achieves the high rating on criteria 2.5.4 through 2.5.6.

2.6.3 Well-Supported Practice: HHS intends to designate a program or service as a `well-supported practice' if the practice has at least two studies with non-overlapping analytic samples and distinct implementations that demonstrate favorable effects as described by 2.5.1 Favorable Effects, demonstrate sustained favorable effects of at least 12 months beyond the end of treatment as described in Section 2.5.3 Sustained Favorable Effect, and achieve the high rating on criteria 2.5.4 through 2.5.6.

HHS does not intend to rate a program or service as a `promising', `supported', or `well-supported practice' if there is an empirical basis, as evidenced by multiple unfavorable effects on target or non-target outcomes across reviewed studies, as described in 2.5.2 Unfavorable Effects, that suggest the overall weight of evidence does not support the benefits of the program or service. This Notice requests comment on approaches for determining that promising, supported, and well-supported practices do not constitute a risk of harm. As described in FFPSA [section 471(e)(4)(C)], “There is no empirical basis suggesting that, compared to its likely benefits, the practice constitutes a risk of harm to those receiving it”, “If multiple outcome studies are conducted, the overall weight of evidences supports the benefits of the practice”, and “There is no case data suggesting a risk of harm that was probably caused by the treatment and that was severe or frequent”.

3.0 Recommendations of Potential Candidate Programs and Services for Review

This Notice requests comment on potential candidate programs and services to consider for the systematic evidence review. Comments should identify how recommended programs and services meet the criteria described in section 2.1 Program or Service Eligibility Criteria. These criteria include: Types of Programs and Services and Book/Manual/Writings Available. Comments should also identify how recommended programs and services meet the criteria described in section 2.2 Program or Service Prioritization Criteria. These criteria include: Types of Programs and Services, Target Population of Interest, Target Outcomes, Number of Impact Studies, In Use/Active, Implementation and Fidelity Support, Trauma-Informed, and Delivery Setting for In-Home Parent Skill-Based Programs and Services. In order to leverage new insights from the field, HHS may put forth additional future Notices requesting recommendations of potential candidate programs and services for review.

4.0 Submission of Comments

Comments may be submitted until July 22, 2018 by email to [e-mail].

Naomi Goldstein,

Deputy Assistant Secretary for Planning, Research, and Evaluation.