2023-23391. Title IV-E Prevention Services Clearinghouse Handbook of Standards and Procedures, Draft Version 2.0

AGENCY:

Administration for Children and Families, Department of Health and Human Services.

ACTION:

Request for public comment.

SUMMARY:

The Administration for Children and Families (ACF), within the U.S. Department of Health and Human Services (HHS), oversees the Title IV–E Prevention Services Clearinghouse. ACF seeks comments on proposed changes and clarifications to existing standards and procedures in the Handbook of Standards and Procedures, Version 2.0.

DATES:

The deadline for comments on this notice is November 24, 2023.

ADDRESSES:

Interested parties may submit written questions, comments, and supplementary documents by email to [e-mail] with “Title IV–E Prevention Services Clearinghouse FRN comment” in the subject line. To ensure that your comments have maximum effect, please identify clearly the section of the draft Handbook of Standards and Procedures, Version 2.0 that your comments address.

Readers are referred to the full version of the draft Handbook of Standards and Procedures, Version 2.0 on the Clearinghouse website ( https://preventionservices.acf.hhs.gov/​resources/​comment-draft-handbook).

SUPPLEMENTARY INFORMATION:

1.0 Background and Legislative Context

The Family First Prevention Services Act (FFPSA) was signed into law as part of the Bipartisan Budget Act (H.R. 1892) on February 9, 2018. FFPSA amended the Social Security Act (the Act) to enable use of Federal funds available under parts B and E of title IV of the Social Security Act to provide enhanced support to children and families and prevent foster care placements through the provision of evidence-based “mental health and substance abuse prevention and treatment services, in-home parent skill-based programs, and kinship navigator services.” As described in the statutory language, these services and programs are intended “for children who are candidates for foster care or who are pregnant or parenting foster youth and the parents or kin caregivers of the children.” The Act requires an independent systematic review of evidence to designate programs and services as “promising,” “supported,” and “well-supported” practices.

In order to meet these requirements, ACF established the Title IV–E Prevention Services Clearinghouse (the Clearinghouse). The Clearinghouse carries out a systematic review process implemented by trained reviewers using consistent, transparent standards and procedures. The Handbook of Standards and Procedures, Version 1.0 ( https://preventionservices.acf.hhs.gov/​review-process) provides a detailed description of the standards used to identify and review programs and services for the Clearinghouse and the procedures followed by the Clearinghouse staff. The Handbook of Standards and Procedures, Version 1.0 was informed by public comments submitted in response to Federal Register Notice 83 FR 29122 ( https://www.federalregister.gov/​documents/​2018/​06/​22/​2018-13420/​decisions-related-to-the-development-of-a-clearinghouse-of-evidence-based-practices-in-accordance), consultations with research and practice experts, and the review processes developed and used by other prominent evidence clearinghouses.

2.0 Overview of 2021 Request for Public Comment on Title IV–E Prevention Services Clearinghouse Handbook of Standards and Procedures, Version 1.0

ACF solicited feedback on the Prevention Services Clearinghouse Handbook of Standards and Procedures, Version 1.0 (subsequently referred to as Handbook Version 1.0) through a Federal Register Notice 86 FR 37332 ( https://www.federalregister.gov/​documents/​2021/​07/​15/​2021-15065/​title-iv-e-prevention-services-clearinghouse-handbook-of-standards-and-procedures) published on July 15, 2021. This comment period was open for 30 days and closed on August 16, 2021. One hundred four unique commenters submitted feedback, including 10 commenters from state and local child welfare agencies. Commenters included state and local government administrators, program and service developers, Federal staff, researchers and evaluators, foundation and non-profit organization staff, and other interested parties. ACF ensured the careful review and consideration of all of the comments in developing the draft Handbook of Standards and Procedures, Version 2.0 (subsequently referred to as Handbook Version 2.0). Comments were considered within the context of the statutory requirements of FFSPA, the necessity to conduct a systematic, objective, and transparent evidence review, and resource considerations. The public comments informed discussions with a large number of experts whose comments were also considered in developing the proposed revisions.

Summary of Comments. Comments highlighted how the standards and procedures specified in Handbook Version 1.0 might be revised to better reflect the goals and requirements of the Executive Order on Advancing Racial Equity and Support for Underserved Communities Through the Federal Government. For example, commenters recommended prioritizing the review of programs and services that have been implemented and/or studied with diverse populations (Section 2.2). Commenters also recommended engaging diverse individuals and those with lived experience to inform the systematic review process and allowing greater flexibility for culturally adapted programs and services. Commenters recommended providing additional detail to clarify the existing standards and procedures. For example, comments requested technical clarification regarding the definition of an available written protocol, manual, or other documentation (Section 2.1.2), determination of the length of time after the end of treatment (Section 6.2.3), determination of whether program or service or study adaptations are substantial (Section 4.1.6), and calculations of effect size and statistical significance (Section 5.1.0). Commenters recommended broadening the definitions of the program or service areas (Section 2.1.2) to be more inclusive regarding the types of programs and services that may be eligible for review. Commenters recommended broadening the definition of eligible comparison conditions (Section 4.1.4) and making the design and execution standards (Chapter 5), particularly those related to baseline equivalence (Section 5.7), more flexible. Finally, commenters provided recommendations to ACF that did not pertain to the Clearinghouse. For example, comments recommended ACF provide further support and investment in building evidence, particularly of programs and services designed to serve communities of color and others disproportionally represented in the child welfare system as well as for kinship navigator programs.

Summary of Proposed Revisions. The draft Handbook Version 2.0 aims to be responsive to the diversity of comments received, to enhance the transparency of the systematic review process, and to support efforts to advance equity in accordance with the Executive Order on Advancing Racial Equity and Support for Underserved Communities Through the Federal Government. For example, revised program or service area definitions (Section 2.1) are inclusive of a broader range of programs and services, new program or service prioritization criteria have been added to consider the child welfare relevance and diversity of populations served (Section 2.2) with similar criteria also added for study prioritization (Section 2.3), and the range of eligible comparison conditions for studies has been expanded to include studies that compare one intervention to another intervention (Section 4.1.7). Additional clarification and guidance are now provided on program or service and study adaptations, including new examples of how standards are applied to culturally adapted programs and services (Sections 2.3.2 and 4.1.9). Clarification is also provided that eligible outcomes and outcome measures may be defined differently across studies to reflect the different ages, backgrounds, cultures, locations, and contexts of the study participants, with examples provided (Section 4.1.8). Formulae used in effect size and statistical significance calculations are now provided directly in the Handbook (Chapter 6) and additional guidance and clarification is provided on design confounds, including clarification that studies with a single provider unit shared across the intervention and comparison conditions are not considered a confound (Section 5.9.3). A broader range of options is provided for establishing baseline equivalence and low attrition randomized group design contrasts are no longer assessed for baseline equivalence (Section 5.7). The Handbook now provides additional information on how the risk of harm assessment is conducted, with additional considerations for cases where the comparison group receives another intervention (Section 7.2.1). Further, additional clarification on how time since the end of treatment is calculated is provided (Section 7.2.3). The Handbook now clearly specifies how any member of the public can submit recommendations of programs or services for review or information about studies of those recommended programs and services to the Clearinghouse at any time (Chapters 1 and 3).

Additional Relevant Activities. The Clearinghouse also intends to conduct additional activities to be responsive to public comments and to support efforts to advance equity in accordance with the Executive Order on Advancing Racial Equity and Support for Underserved Communities Through the Federal Government. First, the Clearinghouse is planning to display study participant characteristics on the program or service page of the Clearinghouse website. Display of participant characteristics is intended to promote transparency on the extent to which diverse populations are represented in research reviewed by the Clearinghouse. Second, the Clearinghouse plans to develop two new reports focused on equity. These two reports are intended to provide additional information about diverse populations included in studies of the programs and services that have been reviewed by the Clearinghouse and identify gaps in evidence. Third, enhanced activities are planned for future public calls for program and service recommendations in order to comprehensively identify culturally adapted and culturally grounded programs and services that may be eligible for review. The Clearinghouse plans to conduct targeted outreach to providers of culturally adapted and culturally grounded programs and services and community-based organizations serving diverse populations to improve engagement. The Clearinghouse also plans to clearly communicate in future public calls how the public, including community-based organizations and providers of culturally adapted and culturally grounded programs and services, can recommend programs and services and submit relevant studies of programs and services to the Clearinghouse. Further, the Clearinghouse plans to make future public call materials available in both English and Spanish. Fourth, the Clearinghouse intends to revise its author Reporting Guide to clarify recommended reporting related to culturally adapted and culturally grounded programs and services and the characteristics of their participants. Fifth, the Clearinghouse intends to revise existing resources for Clearinghouse users, such as its Frequently Asked Questions (FAQ) website section and fact sheet resources, with person-centered design principles to ensure information about the Clearinghouse and its standards and procedures are accessible. Sixth, the Clearinghouse plans to publicly post all programs and services that have been recommended for review and will continue to explore additional ways to improve transparency such as through data sharing.

A comprehensive list of specific revisions and clarifications to the Clearinghouse's Standards and Procedures is provided in the following section. Subsequent chapter and section numbers all refer to the chapter and section numbering for the draft Handbook Version 2.0 unless the text explicitly indicates a reference to Handbook Version 1.0 chapter and section numbering.

3.0 Revisions and Clarifications to the Clearinghouse's Standards and Procedures in the Draft Handbook Version 2.0

3.1 Introduction

The revised introduction includes a description of the Clearinghouse website and resources available on the website. This includes reference to the FAQ section that includes information on how members of the public can submit a program or service recommendation and how to provide information about studies to the Clearinghouse.

3.2 Chapter 1. Identify Programs and Services

Revisions clarify that all program and service recommendations are retained for consideration, including those submitted during public calls and ad hoc recommendations submitted to the Prevention Services Clearinghouse inbox. Revisions also clarify that any member of the public may submit a program or service recommendation at any time to the Clearinghouse via email and that suggested information to include as part of a program or service recommendation can be found on the FAQ section of the Clearinghouse website. Additionally, this section now indicates that all programs and services identified as potential candidates for review will be posted on the Clearinghouse website.

3.3 Chapter 2. Prioritize and Select Programs and Services

3.3.1 Revisions and Clarifications to Program or Service Area Definitions (Section 2.1.1)

Based on FRN feedback and consultation with experts in the fields of mental health, substance use, parenting and parent skill-based programs and services, kinship navigator programs, and child welfare, the draft Handbook Version 2.0 revised and clarified the in-home parent-skill based and substance use prevention and treatment program or service area definitions, as noted below.

• In-home parent skill-based programs and services. The revised definition is more flexible and now indicates that eligible programs and services involve direct intervention with a parent or caregiver and target parenting skills or other skills that can be applied to where the child resides, including in the home. The revised definition also clarifies that delivery of programs and services can occur in the home or other settings and defines necessary content for a program or services to be considered “skill-based.”

Revised examples of eligible and ineligible in-home skill-based programs and services are provided in Exhibit 2.3.

• Substance use prevention and treatment programs and services. The revised definition clarifies that programs or services:

• targeting recovery from substance use (as well as those targeting prevention, treatment, remediation, elimination and/or reduction of substance use or misuse) are eligible; and

• without client-oriented substance use prevention or treatment components, such as mass communications/media campaigns or interventions that solely target broader community-level or policy systems, remain not eligible.

Revised examples of eligible and not eligible programs and services are now provided in Exhibit 2.2. Specifically, one new example clarifies that programs or services targeting parents or caregivers aiming to prevent substance use among children and youth are eligible.

Minor wording changes were made to the kinship navigator program or service area definition for clarification purposes. Experts did not suggest any changes to eligible outcomes for kinship navigator programs and services.

No changes were made to the mental health prevention and treatment programs and services definition. New examples of eligible and ineligible programs and services are provided in Exhibit 2.1.

3.3.2 Clarifications to Available Protocols, Manuals, or Other Documentation (Section 2.1.2)

To be eligible for review by the Prevention Services Clearinghouse, programs and services must be clearly defined and replicable. To meet this criterion, programs and services must have available written or recorded protocols, manuals, or other documentation that describes how to implement or administer the practice (referred to subsequently in this notice as a “manual” for brevity). Revisions to this section clarify that materials to satisfy this requirement may be presented in a web-based format and that “manual” can include recorded videos or online learning systems if these materials describe how to implement or administer the practice. The Clearinghouse notes that, consistent with Handbook Version 1.0, there are no language requirements for manual eligibility.

3.3.3 Revisions and Clarifications to Program or Service Prioritization (Section 2.2)

As of July 2023, the Prevention Services Clearinghouse has reviewed 148 programs and services. Yet there remains a high volume of potentially eligible programs and services identified for review. As a result, the Prevention Services Clearinghouse must continue to prioritize programs and services for review. The draft Handbook Version 2.0 continues to highlight the prioritization of programs and services with available evidence of eligibility and programs and services in active use (Section 2.2). New to this section is further clarification about additional prioritization considerations. These additional prioritization criteria were informed by recommendations from public comments and consultation with experts. Listed below are the additional prioritization criteria included in the draft Handbook Version 2.0.

• Number and source of program or service recommendations received;
• Child welfare relevance;
• Population(s) served;
• Previous evaluations and studies; and
• Implementation supports.

The Clearinghouse continues to prioritize programs and services in a way that ensures representation across the four program and service areas. Additional clarification is provided in draft Handbook Version 2.0 noting that the Clearinghouse assesses prioritization criteria by examining publicly available information, other clearinghouses' websites, and materials submitted with program or service recommendations.

3.3.4 Clarifications on Program or Service Selection (Section 2.3.1)

Given the large volume of programs and services identified, resource considerations mean that not all programs and services can be selected for review at once. To help clarify the distinction between the prioritization and reviewing process, the draft Handbook Version 2.0 adds a new section on selection of a program or service for review (Section 2.3.1). Based on the prioritization process, specific programs and services are selected for review at a given time, as indicated by publication on the working list of programs and services planned for review available on the Prevention Services Clearinghouse website. The final eligibility of a program or service for review by the Clearinghouse is determined after a program or service is selected for the working list.

3.3.5 Revisions to Program or Service Adaptations Criteria (Section 2.3.2)

Multiple public comments requested clarification regarding the program or service adaptation standards specified in Handbook Version 1.0 (found in Section 4.1.6 of this version) and recommended increased inclusivity, particularly with respect to cultural adaptations. The Prevention Services Clearinghouse sought input from a range of experts specifically focusing on program or service adaptations, including those with expertise in cultural adaptations designed to serve historically underserved communities. Underserved communities, as articulated in the Executive Order on Advancing Racial Equity and Support for Underserved Communities Through the Federal Government, include Black, Latino, and Indigenous and Native American persons, Asian Americans and Pacific Islanders and other persons of color; members of religious minorities; lesbian, gay, bisexual, transgender, and queer (LGBTQ+) persons; persons with disabilities; persons who live in rural areas; and persons otherwise adversely affected by persistent poverty or inequality.

To meet the eligibility criteria of being clearly defined and replicable, a program or service must have publicly available written or recorded protocols, manuals, or the documentation (hereafter referred to as “manuals”) that describe how to implement the practice (Section 2.1.2). A new section (2.3.2) clarifies the procedures used to identify and review relevant manuals for a program or service. This includes procedures for identifying a primary manual for review and addressing cases with multiple potential manuals.

Many programs and services have multiple manuals, including manual editions ( e.g., editions of a manual as a program or service evolves over time or expands) and manual variants ( e.g., adaptations of a program or service or a manual to address new issues, different populations, or alternative approaches to delivering the program or service). This section clarifies the standard process by which the Prevention Services Clearinghouse assesses whether alternative manual editions or variants have any substantial adaptations, compared to the primary manual identified. This process consists of the following steps, followed as needed based on the nature of the program or service:

• Step 1: Determining whether the adaptation is explicitly prohibited in the primary program or service manual under review or is the result of adding another separate program or service to the existing program or service ( i.e., “bundling”);

• Step 2: Determining whether the adaptation is explicitly allowed by the primary program or service manual under review;

• Step 3: Determining whether the adaptation substantially changes a program element in the primary program or service manual under review;

• Step 4: Gathering additional information and consulting with senior content experts on the Clearinghouse.

A revised table (Exhibit 2.4) classifies program elements and gives examples of acceptable and substantial adaptations—including expanded examples of adaptations that may be made in the process of culturally adapting a program or service. (These criteria and procedures are aligned with those used to assess any program or service adaptations identified in studies during the study eligibility process, described in Section 4.1.9). Manuals that are substantially adapted from a primary manual may be considered as a separate program or service when reviewing studies. Studies with these substantial adaptations would be ineligible in a review based on the primary manual identified for a particular program or service. Alternatively, manuals without substantial adaptations may be considered the same program or service when reviewing studies. Studies without substantial adaptations would be included in a review based on the primary manual.

3.4 Chapter 3. Literature Search

To help ensure identification of studies conducted with American Indian and Alaska Native populations, the draft Handbook Version 2.0 adds Healthy Native Youth to its list of clearinghouses used to identify relevant research. The list of bibliographic databases has been trimmed for efficiency and resource considerations. Some databases in Handbook Version 1.0 were largely providing duplicative results. This section clarifies that any publicly available research from program or service websites is incorporated into the search. Clarification is also provided on procedures for incorporation of research that is submitted to the Prevention Services Clearinghouse inbox ad hoc or during public calls.

3.5 Chapter 4. Study Eligibility Screening and Prioritization

3.5.1 Revision to Study Definition (Section 4.1)

In alignment with other Federal evidence clearinghouses, the Prevention Services Clearinghouse intends to focus on degree of sample overlap in applying its definition of a study as “one research investigation of a defined subject sample, and the interventions, measures, and statistical analyses applied to that sample.” Additional study definition criteria (based on the What Works Clearinghouse v4.0 study definition) in Handbook Version 1.0 have been dropped in the draft Handbook Version 2.0.

3.5.2 Clarifications on Source of Publication Criteria (Section 4.1.2), Language of Publication (Section 4.1.3) and Location of Study (Section 4.1.4)

The draft Handbook Version 2.0 clarifies the definition of “publicly available” and “published” for the source of publication standard (Section 4.1.2), in response to public comments. Dissertations, theses, and conference papers remain ineligible. Given the priority of reviewing a large number of programs and services, the Prevention Services Clearinghouse intends to continue to exclude such sources in the interests of efficiency.

Some public comments indicated confusion about whether studies conducted outside of the United States or those conducted in non-English-speaking countries are eligible. The draft Handbook Version 2.0 clarifies that the standard from Handbook Version 1.0 that studies must be available in English (Section 4.1.3) is inclusive of studies originally published in another language that have published English language translations available. The draft Handbook Version 2.0 explicitly clarifies that studies conducted in any country are eligible (Section 4.1.4), as they were under Handbook Version 1.0.

3.5.3 Revisions to Study Design and Intervention Condition Criteria (Sections 4.1.5, 4.1.6)

The draft Handbook Version 2.0 provides clarification on definitions for randomized group designs and quasi-experimental group designs with respect to eligible study designs (Section 4.1.5). It clarifies that single-group pretest-posttest designs and interrupted time series designs without comparison groups are not eligible. It also clarifies that group assignment must be exclusive for an outcome measured at a given point in time—that is, participants cannot be counted in both the intervention and comparison condition. The criterion for eligible intervention conditions—that the intervention group is offered an eligible program or service that is essentially the same for all participants in the group—remains the same as in Handbook Version 1.0, with minor clarifications, but is presented as a distinct subsection in the draft Handbook Version 2.0 (Section 4.1.6) for clarity.

3.5.4 Revisions to Eligible Comparison Conditions (Section 4.1.7)

Many public comments requested expansion of eligible study comparison conditions beyond no or minimal treatment and treatment as usual to include more active comparison conditions. Many experts also recommended that the Prevention Services Clearinghouse consider including active comparison conditions. One consideration voiced by multiple experts consulted is that active comparison conditions are increasingly recommended, particularly if there are other available interventions considered to be efficacious. Revision to this standard was considered in the context of the FFSPA legislative criterion that a program or service must be demonstrated as being superior to an appropriate comparison practice.

The draft Handbook Version 2.0 allows for five types of eligible comparison conditions:

• No intervention or wait list —offered no services or services at a later date (clarifying that outcomes measured after a wait list group is offered the intervention are not eligible).

• Minimal intervention —including informational materials or psychoeducation, referrals to available services, or similar nominal services.

• Placebo or attention control —conditions designed to account for nonactive effects of treatment, such as participants' expectations, contact time with an interventionist, or the relationship between interventionist and participants; includes psychological or pharmacological placebos, attention placebos, and nonspecific therapy in which participants receive the same or similar amount of attention or contact as the participants in the intervention condition.

• Treatment as usual —The draft Handbook Version 2.0 clarifies that both “usual or typical services” ( i.e., individuals do not receive anything they would not have been able to receive outside the context of the study) or “services consistent with usual or typical services” ( i.e., services as part of the study that are not offered in the community but are clearly described as consistent with the usual or typical services that would be received by individuals or families similar to those in the study) are considered eligible under treatment as usual. Therapeutic or pharmacological interventions that meet the definition of treatment as usual are eligible.

• Head-to-head comparisons —assigned to another intervention that is not a variant of the program or service under review (may also be referred to as alternative interventions, active interventions, or comparator interventions); excluded are comparisons to pharmacological interventions that do not meet the definition of treatment as usual above.

The draft Handbook Version 2.0 indicates three types of comparison conditions that are explicitly not eligible for review and provides a rationale for each:

• Intervention variants —assigned to an intervention that is a variation of the intervention under review. Examples include dismantling studies ( e.g., full version of intervention compared to one lacking one or more components); bundled intervention studies ( e.g., full version of intervention compared to a version with a second intervention added); studies comparing different delivery modes, providers, dosage, or fidelity levels for the same intervention; sequencing studies ( e.g., both conditions receive the same interventions, but in a different order).

• Population-level data or benchmarks —constructed from population norms or statistics derived from other studies, surveys, censuses, or similar sources.

• Comprised only of intervention refusers or dropouts —composed entirely of individuals who were offered the intervention condition but refused the offer or dropped out of the intervention after being offered the intervention.

3.5.5 Revisions to Outcomes (Section 4.1.8)

Definitions of outcome domain, outcome, and outcome measurement have been provided for clarity. Clarifications have been included regarding eligible outcomes within the child safety and child permanency outcome domains and family functioning outcomes within the adult well-being outcome domain. The clarifications to the child safety and child permanency outcomes were previously described in the FAQ section of the Prevention Services Clearinghouse website. Additionally, eligible educational achievement and attainment outcomes in the child well-being outcome domain have been expanded to include school attendance and absenteeism as eligible outcomes. These outcomes, though not direct measures of educational achievement and attainment, are viewed as closely related and relevant outcomes. Clarification is provided that outcomes that are composites of one or more eligible outcomes within the eligible outcome domains are eligible; those that are composites of eligible and ineligible outcomes are not eligible. Clarification is also provided that eligible outcomes and outcome measures may be defined differently across studies to reflect the different ages, backgrounds, cultures, locations, and contexts of the study participants, with examples provided.

The Prevention Services Clearinghouse currently does not have measurement standards for assessing the validity or reliability of biomarker measures ( i.e., a physiological measure used as an indicator of a physical, psychological or emotional state), such as the use of cortisol as a measure of psychological stress. Expert consultations on biomarkers did not indicate a clear set of standards that could be broadly applied for review of such measures. As a result, the draft Handbook Version 2.0 indicates that biomarker measures are not currently eligible for review as child well-being or adult well-being outcomes.

3.5.6 Revisions to Study Program or Service Adaptations Criteria (Section 4.1.9)

Consistent with Handbook Version 1.0, the draft Handbook Version 2.0 indicates that, to be eligible for review, studies of a program or service must all represent similar implementations of the program or service selected for review. Revisions in the draft Handbook Version 2.0 clarify that the process of assessing program or service adaptations for study eligibility is based on having identified a particular manual (or set of manuals) of the program or service under review (see Sections 2.3.1, 2.3.2).

The standard process used to identify whether program or service adaptations are present in the studies being screened for eligibility is clarified. The procedures and criteria for assessing whether adaptations identified in studies are acceptable or substantial mirror those specified in Section 2.3.2 for adaptations found in manual editions or variants. The end result of these procedures is the determination of study eligibility for the particular program or service under review (in Section 2.3.2, the end determination is whether two manuals are substantively similar or represent different programs or services). Studies with any substantial adaptations are ineligible for review as a study of the program or service under review (such studies may be eligible for review as a study of different program or service and its associated manual). Studies with only minor adaptations may potentially be eligible if all other study eligibility criteria are met.

3.5.7 Revisions to Study Review Prioritization Criteria (Section 4.2)

The Prevention Services Clearinghouse notes that study prioritization criteria are distinct from study eligibility criteria. When a program or service has more than 15 studies eligible for review, study prioritization criteria are applied to order the review of eligible studies. The study prioritization process ensures efficiencies in the reviewing process to review a large number of programs and services.

The Prevention Services Clearinghouse notes that only 12 of the 148 programs and services reviewed as of July 2023 had more than 15 eligible studies identified, requiring the use of study prioritization criteria in these reviews to prioritize the first 15 eligible studies for review using the design and execution standards. Of these 12 programs and services, nine had 16 to 25 eligible studies, with a few having a much larger number of eligible studies ( e.g., 75 or 90). All other programs and services reviewed had 15 or fewer eligible studies, with all eligible studies reviewed using the design and execution standards. Therefore, as in Handbook Version 1.0, the study prioritization criteria continue to apply only when there are 15 or more eligible studies of a program or service in the draft Handbook Version 2.0.

Three modifications have been made to the process of assigning prioritization points for identifying the order in which studies are reviewed in the draft Handbook Version 2.0. First, given that programs or services must demonstrate sustained favorable effects 6 or 12 months beyond the end of treatment (Section 7.2.3) to receive a rating of supported or well-supported, the Prevention Services Clearinghouse intends to increase the prioritization points given to studies that include outcomes measured 6 or 12 months beyond the end of treatment to ensure that these studies are reviewed earlier when present, increasing the prioritization points for such studies to 3 and 6 points, respectively (compared to 1 and 2 points, respectively, in Handbook Version 1.0). Second, some public commenters and experts consulted noted the importance of statistical power for being able to detect intervention effects. The draft Handbook Version 2.0 adds one prioritization score point for studies that report an analysis of statistical power. Third, many public comments recommended that points be awarded to studies based on populations served. The draft Handbook Version 2.0 intends to add one prioritization score point for the child welfare relevance of populations served and two prioritization points for studies with samples from underserved communities. Prioritization points for studies with outcomes in multiple outcome domains have been decreased from a maximum of three to a maximum of one. The draft Handbook Version 2.0 provides procedural details clarifying how ties in prioritization scores are resolved in cases where more than 15 eligible studies are identified.

The draft Handbook Version 2.0 includes efficiency enhancements based on the study prioritization process for programs and services where more than 15 eligible studies are identified. If, after review of the first 15 eligible studies prioritized for review, a program or service has not achieved a rating of well-supported, additional studies are reviewed using the design and execution standards in their prioritized order until either no eligible studies remain that could result in further improvement to the program or service rating or all eligible studies have been reviewed. Determination of potential for program or service ratings to improve upon review of additional eligible studies is based on (1) the program rating from studies already reviewed using the design and execution standards and (2) the duration of effects examined in the remaining studies (as assessed according to study review prioritization criteria). Detailed examples of the application of this policy are described in Section 4.2. The draft Handbook Version 2.0 retains the policy from Handbook Version 1.0 of reviewing all studies against design and execution standards when 15 or fewer eligible studies are identified. All eligible studies are reviewed for risk of harm.

3.6 Chapter 5. Evidence Review Using the Design and Execution Standards

3.6.1 Revisions and Clarifications to Contrasts Rated, Design and Execution Rating Categories, Method of Assignment, and Integrity of Random Assignment (Sections 5.1 to 5.5)

The draft Handbook Version 2.0 indicates that contrasts from all eligible comparison conditions (Section 5.1) will be rated, whereas under Handbook Version 1.0, only contrasts from the least-intensive eligible comparison condition for a particular contrast were rated if multiple comparison conditions were eligible for review ( Handbook Version 1.0, Section 4.1.4). Given the priority of reviewing a large number of programs and services, the draft Handbook Version 2.0 retains the policy from Handbook Version 1.0 of only reviewing full-sample analyses and not reviewing subgroup or sensitivity analyses due to resource considerations. For any studies that receive a moderate or high design and execution rating and report subgroup analyses, the Clearinghouse intends to indicate whether subgroup analyses were conducted for informational purposes only. New and revised examples are provided to clarify integrity of randomization standards for individual and cluster-assignment designs.

3.6.2 Revisions and Clarifications to Attrition, Baseline Equivalence, and Pretest Standards (Sections 5.6 to 5.8)

Based on expert feedback, and in alignment with other Federal clearinghouses (in particular, the What Works Clearinghouse and Home Visiting Evidence of Effectiveness [HomVEE]), the draft Handbook Version 2.0 no longer requires baseline equivalence to be established for a contrast from a low attrition randomized group design to receive a “High” support of causal evidence rating.

Public comments expressed a desire for greater flexibility regarding options for demonstrating baseline equivalence and reconsideration of participant sociodemographic characteristics that could be used to establish baseline equivalence when a pretest alternative is not available. Informed by expert consultations, the draft Handbook Version 2.0 maintains a general preference for using the same (or nearly the same) measure as the outcome ( i.e., a “direct pretest”) for baseline equivalence but now allows any eligible outcome measure demonstrated to be correlated with the outcome at a threshold of 0.60 or higher to be used to establish baseline equivalence (here referred to as a “correlated pretest measure”). Also informed by expert feedback, when a correlated pretest measure or pretest alternative is not available, the draft Handbook Version 2.0 provides greater flexibility in the form of two options for establishing equivalence on sociodemographic characteristics, allowing an expanded set of individual characteristics and the use of a set of neighborhood characteristics if only one individual characteristic is available. Option 1 requires demonstration of baseline equivalence on at least two of the following individual characteristics: race or ethnicity, socioeconomic status, household composition, or age. If only one of the four individual characteristics from Option 1 is available, baseline equivalence can still be established under Option 2 if equivalence is demonstrated on a measure of each of the following neighborhood characteristics: race or ethnicity, socioeconomic status, and household composition. When sociodemographics are used to establish baseline equivalence, a new requirement indicates that study authors must clearly describe all criteria used to create the intervention and comparison groups and affirmatively indicate that the same or similar criteria were used to create each group.

Binary measures have different statistical properties than continuous measures that can potentially reduce their reliability as indicators of baseline equivalence—particularly when events are rare or in smaller samples. To address this, the draft Handbook Version 2.0 indicates a preference for continuous correlated pretests over direct pretests when establishing baseline equivalence for a binary outcome. It also permits use of continuous pretest alternative measures when outcomes are binary, even if it was feasible to measure a direct pretest. Specifically, continuous measures that meet the correlated pretest measure or pretest alternative criteria are preferred over a direct pretest of the binary measure, when available.

3.6.3 Revisions and Clarifications to Statistical Model Standards (Section 5.9)

The statistical model standards (Section 5.9.1) have been revised in the draft Handbook Version 2.0 to clarify procedures used when statistical models do not meet standards and alternative statistical models are not available or do not meet standards. In such cases, the Prevention Services Clearinghouse will seek to review the contrast based on unadjusted means and standard deviations and the statistical significance test procedures specified in Chapter 6.

The measurement reliability standard for inter-rater reliability in Handbook Version 1.0 was revised in the draft Handbook Version 2.0 (Section 5.9.2), with specific thresholds for inter-rater reliability (correlation), inter-rater agreement on the basis of percentage agreement (0.80 or higher), and inter-rater agreement based on kappa (0.60 or higher). These revised standards are aligned with current What Works Clearinghouse standards.

Some public comments expressed concern that confound standards prevent inclusion of studies conducted in rural or underserved areas where only a single service provider is available may not be able to meet standards. The draft Handbook Version 2.0 clarifies that studies with a single person or administrative unit are not automatically confounded, with detailed clarifying examples added to this section. Specifically, if a single provider (or a single administrative unit) provides treatment or services to at least some participants in both the intervention and comparison condition, a design confound is not considered to be present. Expert feedback indicated that the confound standards in Handbook Version 1.0 were appropriate causal evidence standards, informing the retention of these confound standards in the draft Handbook Version 2.0.

3.7 Chapter 6. Record and Characterize Impact Estimates

Public comments requested additional information about the formulae used for computing effect sizes and procedures used for determining statistical significance. The draft Handbook Version 2.0 provides all standard formulae used in computing effect sizes reported and for computing statistical significance. For models that meet statistical model standards in the design and execution requirements (Section 5.9), the draft Handbook Version 2.0 indicates that author-reported statistical significance is preferred in covariate-adjusted models and certain models for which the Prevention Services Clearinghouse does not currently have standards for computing statistical significance ( e.g., time-to-event models). When such models are not available or do not meet statistical model standards, the formulae provided are used to conduct a post-hoc statistical significance test based on the natural metric of the outcome reported ( e.g., continuous, binary, count, or time-to-event).

Clarification is provided on information needed and procedures used to compute effect sizes and statistical significance for repeated measures models ( e.g., growth curve models). In alignment with other Federal clearinghouses (in particular, What Works Clearinghouse, HomVEE), point-in-time estimates for each measurement time period are required. If such information is not reported, unadjusted means and standard deviations for each point in time are used (or requested if not reported), with appropriate post-hoc significance tests performed based on the natural metric of the outcome.

3.8 Chapter 7. Program or Service Ratings

3.8.1 Revisions and Clarifications to Program or Service Ratings (Section 7.1) and Risk of Harm (Section 7.2.1)

No changes were made to the criteria for promising, supported, or well-supported program or service ratings in the draft Handbook Version 2.0 (Section 7.1). This section clarifies that intention of the Prevention Services Clearinghouse is for program or service ratings from reviews conducted under Handbook Version 1.0 to be retained until such time that a program or service is re-reviewed under Handbook Version 2.0 (see Section 8.5.1 below regarding re-review procedures).

A new standard specified in the risk of harm section (Section 7.2.1) of the draft Handbook Version 2.0 is that contrasts in head-to-head comparison conditions or placebo or attention control comparison conditions where the comparison condition has any evidence for risk of harm cannot contribute to a promising, supported, or well-supported rating. If risk of harm is present in these kinds of comparison conditions, impact estimates are not clearly interpretable as evidence of intervention effectiveness—as it is possible that both the intervention and comparison condition could be made worse off than if they had not participated in the study at all. When risk of harm is not present in the comparison condition, favorable impacts can be interpreted as the intervention group being at least better off than they would have been if no treatment had been offered at all and can potentially contribute as evidence of effectiveness. Standard procedures for identifying potential risk of harm in comparison conditions are detailed in this section.

3.8.2 Revisions and Clarifications to Usual Care or Practice Settings Definition (Section 7.2.2)

The definition of usual care or practice settings (Section 7.2.2) in the draft Handbook Version 2.0 has been clarified to indicate that community settings, such as schools, with embedded service providers that may provide eligible programs or services as part of their typical operations ( e.g., school counselors), are also considered usual care or practice settings. It clarifies that clinics that provide services solely for participants in research studies or clinical trials ( i.e., that do not provide any services to persons not participating in research studies as part of their typical operations) do not constitute usual care or practice settings.

3.8.3 Revisions and Clarifications to Beyond the End of Treatment (Section 7.2.3)

Some public comments requested clarification on how the Prevention Services Clearinghouse assesses the duration of sustained effects, particularly in cases where the end of treatment is flexible across participants. Section 7.2.3 of the draft Handbook Version 2.0 includes revisions to clarify the order of preference for information that may be provided in studies about the end of treatment and procedures for computing the duration of sustained effects when the duration of treatment is fixed, when the duration of treatment is defined and varies across participants, and when the duration of treatment is undefined. Treatment of boosters in computing the duration of sustained effects is now explicitly addressed. Detailed procedures and examples can be found in Section 7.2.3 of the draft Handbook Version 2.0.

3.9 Chapter 8. Prevention Services Clearinghouse Procedures

The draft Handbook Version 2.0 represents the first update to the Handbook of Standards and Procedures since the beginning of the Title IV–E Prevention Services Clearinghouse in 2018. The basic procedures for identifying eligible studies (Section 8.3) and reviewing studies against the design and execution standards (Section 8.4) remain essentially the same, with minor clarifications to operational procedures. Author query policies (Section 8.4.2) have been clarified; new content has been added clarifying the reasons that the Prevention Services Clearinghouse may query program and service developers for information about programs or services (Section 8.4.3). New content and more substantive revisions are described below.

3.9.1 Selection of Handbook of Standards and Procedures Version To Use in Reviews (Section 8.2)

The intention of the Prevention Services Clearinghouse is to conduct reviews of any program or service not previously reviewed under Handbook Version 1.0 solely under the standards and procedures specified in Handbook Version 2.0 once it is finalized. Programs or services that are included on the working list prior to when Handbook Version 2.0 is finalized may be reviewed under Handbook Version 1.0 or Handbook Version 2.0. The version of the handbook used to conduct a review (or re-review) of a program or service will be clearly stated on the working list and on the program or service's review page on the Prevention Services Clearinghouse website.

3.9.2 Program and Service Re-Reviews and Study Re-Reviews (Sections 8.5.1, 8.5.2)

The Prevention Services Clearinghouse intends to conduct program and service re-reviews solely under Handbook Version 2.0 after it is finalized (Section 8.5.1). The intention of the Prevention Services Clearinghouse is that all existing program and service ratings determined under Handbook Version 1.0 will remain in effect until such time that a program or service re-review is conducted of a program or service.

Programs and services reviewed by the Prevention Services Clearinghouse under Handbook Version 1.0 may be considered for re-review under Handbook Version 2.0 if a re-review has the potential to change the program or service rating (Section 8.5.1). Program or service ratings could potentially change due to application of Handbook Version 2.0 standards to studies already identified in a prior review ( e.g., studies previously ineligible now being eligible; studies being able to demonstrate baseline equivalence under revised standards) or the emergence of new evidence since the original review. The intention of the Prevention Services Clearinghouse is that the rating of a re-reviewed program or service would be based solely on the standards and procedures in Handbook Version 2.0 ( i.e., the previously assigned rating would no longer be in effect).

The intention of the Prevention Service Clearinghouse is to conduct study re-reviews ( i.e., due to missing information or errors in the currently published review of an individual study) under the version of the handbook used to review the program or service (Section 8.5.2). That is, for a program or service reviewed under Handbook Version 1.0 where the program or service has not been re-reviewed under Handbook Version 2.0, a study re-review would be conducted under Handbook Version 1.0. For a program or service where a program or service rating has been assigned using Handbook Version 2.0, study re-reviews would be conducted using Handbook Version 2.0. This policy is consistent with other Federal evidence clearinghouses with multiple handbook versions ( e.g., HomVEE). The Prevention Services Clearinghouse's intention is that the emergence of substantial new evidence that has the potential to change program or service ratings ( e.g., a newly published study) should be addressed through a program or service re-review. Similarly, cases where application of Handbook Version 2.0 standards to a study reviewed under Handbook Version 1.0 could affect the program or service rating are intended to be addressed through a program or service re-review. Study re-reviews are intended to be limited solely to addressing missing information or errors in studies already reviewed.

3.9.3 Manual Citation Updates (Section 8.5.3)

The Prevention Services Clearinghouse recognizes that program or service manuals may be updated in the course of time after a review of a program or service has been published. Should a new manual edition (as defined in Section 2.3.2) be published, the public may request consideration of an update to the manual citation used for the program or service as outlined in Section 8.5.3 of the draft Handbook Version 2.0. If updated manual editions do not have substantive modifications or adaptations from the manual reviewed (per the criteria specified in Section 2.3), a manual citation may be updated to reflect that a newer manual edition is in active use that is substantively similar to the original primary manual selected for the review of the program or service. In considering whether an update to a manual citation is warranted, the Prevention Services Clearinghouse must have sufficient information available to be able to apply the procedures specified in Section 2.3 for determining whether any substantive adaptations are present in the newer manual edition compared to the original edition reviewed. If the manual citation is updated, the original manual citation used to conduct the review of evidence for the program or service will also be noted for clarity.

4.0 Timeline for the Clearinghouse To Apply New Standards and Procedures

The Prevention Services Clearinghouse proposes to apply the standards and procedures upon publication of a final Handbook Version 2.0. The public will be clearly notified on the Prevention Services Clearinghouse website and via other avenues ( e.g., email to subscribers to the Prevention Services Clearinghouse email list) when the final published Handbook Version 2.0 will go into effect for reviewing programs and services.

Per the procedures in Chapters 7 and 8 of the draft Handbook Version 2.0, all existing program and service ratings established under Handbook Version 1.0 will remain in effect until such time that a program or service re-review is conducted of a program or service under Handbook Version 2.0.

5.0 Request for Information (RFI)

To facilitate the review of submissions, please identify the chapter, section, and/or page number of the draft Handbook of Standards and Procedures, Version 2.0 ( https:// preventionservices.acf.hhs.gov/​resources/​comment-draft-handbook ) that your comments address. This RFI is for information and planning purposes only and should not be construed as a solicitation or as an obligation on the part of ACF or HHS. For more information about the Prevention Services Clearinghouse, visit: https://preventionservices.acf.hhs.gov.

Lauren Supplee,

Deputy Assistant Secretary for Planning, Research, and Evaluation.