2021-05840. Revised Single-Case Design Procedures and Standards: Home Visiting Evidence of Effectiveness (HomVEE) Review

AGENCY:

Administration for Children and Families, U.S. Department of Health and Human Services.

ACTION:

Request for public comment.

SUMMARY:

The Administration for Children and Families (ACF), within the U.S. Department of Health and Human Services (HHS), oversees the Home Visiting Evidence of Effectiveness (HomVEE) review, which is proposing to revise the procedures and standards that guide its work with single-case design (SCD) research. The current Federal Register notice (FRN) seeks comments on proposed changes related to revised procedures and standards for SCD research. Readers are referred to the full version of the HomVEE Version 2 Handbook on the HomVEE website for more details, particularly Appendix D (https://homvee.acf.hhs.gov/​publications/​methods-standards). HomVEE will release an updated Handbook (Version 2.1) after consideration of public comments received in response to this FRN.

DATES:

Send comments on or before April 19, 2021.

ADDRESSES:

Submit questions, comments, and supplementary documents to [e-mail] with “HomVEE SCD procedures and standards FRN comment” in the subject line.

SUPPLEMENTARY INFORMATION:

Invitation to Comment: HHS invites comments regarding this notice. To ensure that your comments are clearly stated, please identify the section of this notice that your comments address.

1.0 Background

To help policymakers, program administrators, model developers, researchers, and the public identify rigorous research and understand which early childhood home visiting models are effective, ACF's Office of Planning, Research, and Evaluation (OPRE) within HHS oversees the HomVEE review, in partnership with the Health Resources and Services Administration (HRSA). HomVEE's mission is to conduct a thorough and transparent review of early childhood home visiting models that serve pregnant women and children from birth to kindergarten entry. The review identifies well-designed, well-executed research, then extracts and summarizes the findings from that research.

One critical use of HomVEE's results is to determine which home visiting models meet the HHS criteria for an “evidence-based early childhood home visiting service delivery model” (see Section 1.1 below), a key requirement of eligibility for implementation of the model with Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program funding.

The MIECHV Program is administered by HRSA in partnership with ACF. Created in 2010, the MIECHV Program provides funding to states, territories, and tribal entities to implement home visiting models. MIECHV awardees have the flexibility to tailor the program to serve the specific needs of their communities. Through a needs assessment, awardees identify at-risk communities and select home visiting service delivery models that best meet state and/or local needs. In accordance with MIECHV's authorizing statute, state and territory awardees must spend the majority of their MIECHV Program grants to implement evidence-based home visiting models with up to 25 percent of funding available to implement promising approaches that will undergo rigorous evaluation.

In December 2020, HomVEE released an updated Version 2 Handbook of Procedures and Evidence Standards (Sama-Miller et al. 2020). The updates incorporated input from methodological experts, other federal evidence reviews, and public comments on two FRNs released on August 5, 2020 (85 FR 47376 and 85 FR 47384). The HomVEE Version 2 Handbook adopted the What Works Clearinghouse (WWC) Version 4.1 Procedures and Standards Handbooks (2020a and 2020b), including revised SCD procedures and standards.

This FRN proposes further changes to the HomVEE Version 2 Handbook to clarify how HomVEE would apply the WWC 4.1 procedures and standards that HomVEE already adopted. The proposed changes focus exclusively on SCD research, including:

(1) Requirements for considering SCD research toward the HHS criteria for evidence-based models (discussed in Section 2 of the FRN).

(2) Flexibilities for applying certain SCD standards (discussed in Section 3 of the FRN).

A work group consisting of federal staff and HomVEE contractor staff, including methodological and home visiting experts, met to discuss and develop the proposed changes outlined in this FRN. Through this FRN, ACF seeks to gather stakeholder input on draft changes and to provide a transparent account of how the review operates. The remainder of Section 1 summarizes recent and current HomVEE procedures and standards with respect to SCD research.

After a period of public comment (and consultation with selected experts outside HomVEE), HomVEE will release an updated Handbook (Version 2.1). Section 5 of this FRN specifies where in the updated Version 2.1 Handbook HomVEE would insert the proposed changes.

1.1 HHS Criteria for an “Evidence-Based Early Childhood Home Visiting Service Delivery Model”

As described in Section 2 of this FRN, HomVEE is proposing changes to requirements for how SCD research would be considered toward the HHS criteria. However, HomVEE is not proposing changes to the HHS criteria themselves. To meet HHS criteria for an “evidence-based early childhood home visiting service delivery model,” models must meet at least one of the following criteria:

(1) At least one high- or moderate-quality impact study of the model finds favorable, statistically significant impacts in two or more of the eight outcome domains.

(2) At least two high- or moderate-quality impact studies of the model using non-overlapping analytic study samples find one or more favorable, statistically significant impacts in the same domain.

In both cases, the impacts must either (1) be found in the full sample for the study, or (2) if found for subgroups but not for the full sample, be replicated in the same domain in two or more studies using non-overlapping analytic study samples.

Additionally, following the MIECHV authorizing statute, if the model meets the above criteria based on findings from randomized controlled trials only, then two additional requirements apply. First, one or more favorable (statistically significant) impacts must be sustained for at least one year after program enrollment. Second, one or more favorable (statistically significant) impacts must be reported in a peer-reviewed journal.

Since HomVEE's inception, research about SCD studies has had to satisfy certain requirements before manuscripts about those studies could be considered toward the HHS criteria. These requirements, known as the “5-3-20 rule,” are as follows:

(1) At least five studies examining the intervention either met the WWC's SCD standards without reservations or met those standards with reservations (corresponding to a “high” or “moderate” rating in HomVEE, respectively).

(2) The SCDs were conducted by at least three research teams, with no overlapping authorship at three institutions.

(3) The combined number of cases was at least 20.

Beyond the 5-3-20 rule, the question of statistical significance is relevant to findings from SCD research because the HHS criteria focus on favorable, statistically significant findings. Authors of SCD manuscripts rarely describe their findings in terms of statistical significance, which HomVEE needs to apply the HHS criteria. However, the WWC Version 4.1 Procedures Handbook, adopted in the HomVEE Version 2 Handbook, introduces the design-comparable effect size (D-CES), from which statistical significance can be determined.

2.0 Proposed Changes to How SCD Research Would Be Considered Toward HHS Criteria for an “Evidence-Based Early Childhood Home Visiting Service Delivery Model”

The HomVEE Version 2 Handbook adopted the WWC Single-Case Design Procedures and Standards (Version 4.1). Therefore, the latest round of updates to each of the WWC and HomVEE procedures and standards included three key complementary changes. The changes, described in more detail in Subsections 2.1 through 2.3, are as follows:

(1) The WWC no longer uses the “5-3-20” rule, which established thresholds for when SCD research could contribute to evidence ratings. Consequently, HomVEE's previous requirements for SCD research about an early childhood home visiting model to be considered toward the HHS criteria (see Section 1.1) no longer align with current best practices for systematic reviews.

(2) Under HomVEE's new procedures as defined in the Version 2 Handbook and adopted from the WWC, it is now possible to calculate effect sizes and determine statistical significance for SCD research using the D-CES. Consequently, HomVEE proposes to align the application of the HHS criteria for SCD research to the procedures already in effect for other eligible research designs.

(3) Reviewers will query authors for numerical data for calculating the D-CES, a step that had not been relevant under prior versions of procedures for HomVEE (or the WWC). Because calculation of the D-CES requires numerical data from each time point, HomVEE proposes to query authors, as needed, for the information required to calculate the D-CES.

2.1 The 5-3-20 Rule No Longer Applies

Additional requirements for considering results from SCD research toward the HHS criteria (the 5-3-20 rule) have been in effect since HomVEE's inception; those requirements were consistent with the WWC's then-current approach to SCD research. The 5-3-20 rule provided a threshold for determining when SCD research could contribute toward a decision about whether a model was evidence-based. Specifically, for SCD research to be considered toward the HHS criteria, the model's SCD research has been required to consist of at least five studies with high or moderate internal validity, to be conducted by three independent research teams, and to include at least 20 cases. To date, no models reviewed by HomVEE have met the requirements of the 5-3-20 rule.

The WWC Version 4.1 Procedures Handbook removed the 5-3-20 rule. Correspondingly, HomVEE now proposes to remove the 5-3-20 requirement for SCD research to be considered toward the HHS criteria.

2.2 Effect Sizes and Statistical Significance Can Now Be Computed

The HHS criteria for evidence-based models require evidence of favorable, statistically significant findings. However, HomVEE's procedures did not previously specify how HomVEE would apply the HHS criteria to SCD research, for which researchers generally express findings in terms of visual patterns rather than statistical significance.

Advances in meta-analysis have made it possible to calculate an effect size (the D-CES) and determine associated statistical significance for findings from most SCD research. Therefore, under its Version 4.1 procedures and standards, the WWC calculates an effect size from SCD studies, if feasible and appropriate, which is then treated as any other effect size when determining intervention ratings. Consistent with the WWC, the HomVEE Version 2 Handbook includes an explanation of how the review will calculate the D-CES for SCD findings.

The D-CES is comparable to a standardized mean-difference effect size and can be interpreted similarly to effect sizes from group design impact studies, such as randomized controlled trials, regression discontinuity designs, and non-experimental group designs. Because HomVEE can calculate the D-CES and then use the D-CES to determine the statistical significance of a finding for most SCDs, it is now possible to align the application of the HHS criteria for SCD research to group design impact studies. HomVEE proposes to align the application of HHS criteria accordingly.

A D-CES can be computed for SCDs using the following designs:

(1) Multiple baseline designs focused on the same outcome across three or more cases.

(2) Multiple probe designs focused on the same outcome across three or more cases.

(3) Reversal/withdrawal designs focused on the same outcome across three or more cases.

A D-CES cannot be computed for SCDs using the following designs:

(1) Multiple baseline designs focused on the same case across three or more settings.

(2) Multiple probe designs focused on the same case across three or more settings.

(3) Several reversal/withdrawal designs focused on the same case.

HomVEE proposes to review all eligible SCD manuscripts. HomVEE also proposes to determine statistical significance based on the D-CES for all manuscripts for which a D-CES can be calculated. For SCD manuscripts for which a D-CES cannot be calculated, for either design or data reasons, HomVEE proposes to report on the rating of the manuscript and the findings reported in it. However, research for which a D-CES cannot be calculated will not be included in the summary of evidence that contributes to the assessment of whether a model meets HHS criteria. Without a D-CES, HomVEE cannot determine statistical significance of the manuscript's findings.

2.3 Proposed Changes to Author Query Procedures

SCD manuscripts frequently include graphical representation of data at each time point. To calculate the D-CES requires numerical data from each time point. HomVEE proposes to request numerical data via an author query, as needed. If the author does not provide the numerical data, then HomVEE proposes to use a software package, such as WebPlotDigitizer (Rohatgi 2020), to extract numerical data from graphical presentations. This approach is consistent with WWC procedures.

2.4 Summary of Proposed Changes to Requirements for SCD Research To Be Considered Toward the HHS Criteria

To account for the elimination of the 5-3-20 rule and to promote consistency in how well-designed, well-executed impact studies are considered toward the HHS criteria, HomVEE proposes the following updates:

(1) Eliminate the 5-3-20 additional requirement that SCD research must meet to be considered toward the HHS criteria and instead consider SCD research toward the HHS criteria in a manner consistent with other designs.

(2) To support the use of SCD findings toward the HHS criteria, implement three procedural steps that align with the Version 4.1 practices of the WWC, as follows:

a. Request numerical data from authors, if necessary.

b. Calculate the D-CES, if possible, for those designs that are rated moderate or high.

c. Calculate statistical significance for the D-CES.

These proposed changes ensure that any impact study with high or moderate quality, as determined by the application of HomVEE Version 2 standards, can contribute to the determination of whether a model meets the HHS criteria for an “evidence-based early childhood home visiting service deliver model.”

3.0 Flexibilities for Applying Certain SCD Standards

HomVEE proposes to clarify the flexibilities related to the application of certain SCD standards consistent with current WWC practice. Such flexibilities are intended to ensure that HomVEE standards are not unnecessarily stringent as more SCD manuscripts are reviewed. HomVEE proposes to update the HomVEE Version 2 Handbook text to clarify how existing flexibilities may be applied in the HomVEE review process.

3.1. Flexibility Related to the Timing of Probe Point Collection When a Case Receives the Intervention

The first flexibility concerns the collection of probe data points in a multiple probe SCD. The current standard is strict requiring that baseline probes for cases without the intervention be collected in the same session when another case starts the intervention. The purpose of the baseline probe collection is to assess whether cases not receiving the intervention have changes in outcomes prior to receiving the intervention.

HomVEE, and the WWC, recognize the requirement that baseline probes for cases with and without the intervention take place in the exact same session may be overly restrictive. The goal of the requirement related to the timing of baseline data collection for cases not yet receiving the intervention is to provide support that any change in outcomes in the cases receiving the intervention is likely due to the intervention and not some external factor (internal validity).

In alignment with WWC procedures, HomVEE currently grants exceptions to this standard for individual manuscripts or interventions in consultation with subject matter experts. HomVEE proposes making the requirement more flexible by clarifying that baseline probe points may be collected when the intervention is introduced or in subsequent baseline sessions.

3.2 Flexibilities Related to the Minimum Number of Effects Demonstrated or Data Points Required

The other flexibilities relate to the demonstration of effects—either the minimum number of effects demonstrated or the data points required to demonstrate an effect. HomVEE currently allows exceptions to each of these standards. These flexibilities are needed to accommodate possible nuances in outcomes that may be examined in SCDs — for example, outcomes that are challenging for researchers to collect without burdening families (such as outcomes based on skills that may be frustrating to be tested on repeatedly if they have not been taught) or outcomes that are dangerous to collect repeatedly without intervening (such as some child maltreatment outcomes).

To facilitate a transparent review, HomVEE proposes to clarify the process for granting and documenting the application of these flexibilities. Specifically, HomVEE proposes to specify that, if warranted, the HomVEE team can grant exceptions in collaboration with subject matter experts, and the exception (and its rationale) will be documented clearly in the review and related dissemination efforts.

4.0 Timeline for HomVEE To Apply New Procedures and Standards

HomVEE proposes to apply the new procedures and standards for SCD research beginning with the 2021 review.

5.0 Summary of the Proposed Changes and Placement in the Version 2 Handbook

Following the 2020 substantial update to HomVEE procedures and standards, HomVEE proposes additional changes focused specifically on SCD research. The table below summarizes the proposed changes to be made to the HomVEE Version 2 Handbook and the relevant handbook section(s) that HomVEE proposes to update.

	Topic	Description of proposed change and relevant FRN section	Relevant HomVEE Version 2 Handbook section
1	Requirements for considering single-case design (SCD) research toward HHS criteria for an “evidence-based early childhood home visiting service delivery model”	SCD studies no longer have additional requirements to meet. The statistical significance of a finding (based on a design-comparable effect size, or D-CESCES) will be considered for SCD research when applying the HHS criteria. Manuscripts for which the D-CESCES cannot be calculated will be reviewed and reported but will not contribute to summaries of evidence. (See Sections 2.1, 2.2, and 2.4 for more details.)	Section B.2.c and Exhibit II.11.
2	Author queries for numerical data for SCD manuscripts	HomVEE will query authors for the numerical data underlying graphical results in manuscripts, as needed. The numerical data are necessary for calculating the D-CESCES, which is used to determine statistical significance. If the author does not provide numerical data, then HomVEE proposes to use a software package to extract numerical data from graphical representations. (See Section 2.3 for more details.)	Section B.1.b.i Appendix D, Section D.1..
3	Additional baseline data point requirements for multiple probe SCDs	HomVEE will require multiple probe designs to have baseline data in the same or subsequent baseline session for cases not receiving the intervention when a case starts the intervention. (See Section 3.1 for more details.)	Appendix D, Section B.5, Footnote 80 and related statement.
4	Flexibility in the requirement for demonstration of three attempts of intervention effects at three different points in time	HomVEE may grant an exception to this requirement in consultation with subject matter experts. If granted, it will be documented in the review and related dissemination efforts, including the rationale for the exception. (See Section 3.2 for more details.)	Appendix D, Section B.6, Footnote 81.
5	Number of data points required	HomVEE may grant an exception to this requirement in consultation with subject matter experts. If granted, it will be documented in the review and related dissemination efforts, including the rationale for the exception. (See Section 3.2 for more details.)	Appendix D, Section B.6, Footnotes 82 and 83.

6.0 Request for Information (RFI)

Through this FRN, ACF is soliciting information from a broad array of stakeholders on the proposed revisions to HomVEE's procedures and standards related to SCD research. Federal, state, and local decision makers rely on HomVEE to know which home visiting models are effective. Applying the HHS criteria to SCDs as they are applied to other quasi-experimental designs or randomized controlled trials means that well-designed, well-executed SCD research will be treated on par with other forms of well-designed, well-executed impact studies.

Responses to this FRN will inform ACF's ongoing discussion about HomVEE's procedures and standards with the aim of publishing a final HomVEE Version 2.1 Handbook of Procedures and Standards in 2021. This RFI is for information and planning purposes only and should not be construed as a solicitation or as an obligation on the part of ACF or HHS.

Authority: Social Security Act Title V § 511 [42 U.S.C. 711], as extended by the Bipartisan Budget Act of 2018 (Pub. L. 115-123) through fiscal year 2022.

Naomi Goldstein,

Deputy Assistant Secretary for Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

References

Rohatgi, A. WebPlotDigitizer, version 4.3. 2020. Available at https://automeris.io/​WebPlotDigitizer. Accessed February 26, 2021.

Sama-Miller, E., J. Lugo-Gil, J. Harding, L. Akers, and R. Coughlin. Home Visiting Evidence of Effectiveness (HomVEE) Systematic Review: Handbook of Procedures and Evidence Standards: Version 2, December 2020. Available at https://homvee.acf.hhs.gov/​publications/​methods-standards. Accessed January 7, 2021.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, What Works Clearinghouse. What Works Clearinghouse Procedures Handbook: Version 4.1., 2020a. Available at https://ies.ed.gov/​ncee/​wwc/​Docs/​referenceresources/​WWC-Procedures-Handbook-v4-1-508.pdf. Accessed June 19, 2020.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, What Works Clearinghouse. What Works Clearinghouse Standards Handbook: Version 4.1., 2020b. Available at https://ies.ed.gov/​ncee/​wwc/​Docs/​referenceresources/​WWC-Standards-Handbook-v4-1-508.pdf. Accessed June 19, 2020.