2020-17001. Revised Procedures and Standards: Home Visiting Evidence of Effectiveness (HomVEE) Review

AGENCY:

Administration for Children and Families, U.S. Department of Health and Human Services.

ACTION:

Request for public comment.

SUMMARY:

The Administration for Children and Families (ACF), within the U.S. Department of Health and Human Services (HHS), oversees the Home Visiting Evidence of Effectiveness (HomVEE) review, which is proposing to revise the procedures and standards that guide its work. The revised procedures and standards will be presented in two separate Federal Register notices. The current Federal Register notice seeks comments on proposed changes and clarifications to several procedural topics and on the standards for assessing the quality of impact study designs. Readers are referred to the full version of the HomVEE Draft Version 2 Handbook on the HomVEE website (https://homvee.acf.hhs.gov) for more details. Another Federal Register notice summarizes updated definitions, rules, and procedures related to handling home visiting model versions (commonly referred to in the home visiting research literature as adaptations) in the review.

DATES:

Send comments on or before September 1, 2020.

ADDRESSES:

Submit questions, comments, and supplementary documents to [e-mail] with “HomVEE procedures and standards FRN comment” in the subject line.

SUPPLEMENTARY INFORMATION:

Invitation to Comment: HHS invites comments regarding this notice. To ensure that your comments are clearly stated, please identify the section of this notice or the chapter and section of the HomVEE Draft Version 2 Handbook that your comments address.

1.0 Background

To help policymakers, program administrators, model developers, researchers, and the public identify rigorous research and understand which early childhood home visiting models are effective, ACF's Office of Planning, Research, and Evaluation within HHS oversees the HomVEE review. HomVEE's mission is to conduct a thorough and transparent review of the research literature on home visiting for families with pregnant women and children from birth to kindergarten entry. The review team identifies well-designed research within that pool and extracts and summarizes the findings from that research.

One critical use of HomVEE's results is to determine which home visiting models meet the HHS criteria for an “evidence-based early childhood home visiting service delivery model” (see Exhibit II.11 in the HomVEE Draft Version 2 Handbook), a key requirement of eligibility for implementation with the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program funding. The MIECHV Program is administered by the Health Resources and Services Administration in partnership with ACF. Created in 2010, the MIECHV Program provides funding to states, territories, and tribal entities to implement home visiting models. MIECHV awardees have the flexibility to tailor the program to serve the specific needs of their communities. Through a needs assessment, awardees identify at-risk communities and select home visiting service delivery models that best meet state and/or local needs. As per MIECHV's authorizing statute, state and territory awardees must spend the majority of their MIECHV Program grants to implement evidence-based home visiting models, with up to 25 percent of funding available to implement promising approaches that will undergo rigorous evaluation.

For the first time since its inception in 2009, HomVEE is proposing to revise the procedures and standards that guide the systematic review. The proposed revisions include (1) clarifying and updating standards and procedures for rating the quality of impact studies that are used to determine which home visiting models meet HHS criteria for an “evidence-based early childhood home visiting service delivery model” and (2) clarifying definitions, rules, and procedures for handling model versions (commonly referred to in the home visiting research literature as adaptations) in the review (presented in a separate Federal Register notice). The current Federal Register Notice focuses on the former set of revisions.

The proposed changes to HomVEE procedures and standards generally bring the review into alignment with procedures and standards for other federally sponsored systematic evidence reviews. The proposed revisions also specify standards for research designs that are becoming increasingly common in home visiting studies. Over the course of the past 2 years, HomVEE consulted with methodological experts and other federal evidence reviews to refine and update the procedures and standards.

Through this Federal Register notice, HomVEE seeks to provide a transparent account of how the review operates and to gather stakeholder input on draft changes. The sections below summarize the main changes to the procedures and standards. A preliminary version of these revisions, the HomVEE Draft Version 2 Handbook, is available during the public comment period on the HomVEE website at (https://homvee.acf.hhs.gov).

After a period of public comment (including close consultation with selected methods experts outside of HomVEE), HomVEE will release a final Version 2 Handbook.

2.0 Changes and Clarifications to HomVEE Procedures

It is natural in the course of a systematic review for issues to arise that cannot be addressed by existing procedures. These issues require the HomVEE team to develop internal guidance to guide the review.

Clarifications to the procedures added to the HomVEE Draft Version 2 Handbook, summarized in Sections 2.1 through 2.7 below, represent HomVEE's attempt to formalize internal guidance generated over the course of conducting reviews so that the procedures are applied systematically.

Similarly, as reviews evolve, it is necessary for ongoing systematic reviews to change their procedures to meet the needs of the field. Changes are new procedures proposed to align with best practices in systematic reviews and keep the HomVEE review current.

2.1 Clarify Definitions of Research Terms

Recognizing the importance of clear communication and consistent terminology when applying systematic review rules, HomVEE clarifies the definitions of important research terms in the HomVEE Draft Version 2 Handbook. Exhibit I.3 of the HomVEE Draft Version 2 Handbook presents these, as listed below.

• A study evaluates a distinct implementation of an intervention (that is, with a distinct sample, enrolled into the research investigation at a defined time and place, by a specific researcher or research team). HomVEE reviews eligible manuscripts about studies that examine the impact of an early childhood home visiting model by comparing an intervention condition (in which study participants are offered the home visiting model under study) and a comparison condition (in which study participants are not offered that model). This includes eligible manuscripts about studies on model replications, iterations, and versions. See Chapter III, Section A.1.b of the HomVEE Draft Version 2 Handbook, including Exhibit II.4, for more information on how HomVEE screens research for eligibility.

○ A sample encompasses both the entire intervention group and the entire comparison group of participants included in a study.

○ A subgroup is a subset of the sample examined in a study (that is, an analytic subgroup). For example, researchers may examine how a home visiting model affects teenage mothers when there are mothers with a range of ages in their study; hence, teenage mothers would be an analytic subgroup. Sometimes, researchers present subgroup findings in a manuscript alongside findings for the overall sample, and sometimes researchers prepare a manuscript based exclusively on subgroup findings from a broader study. (For HomVEE, results from teenage mothers would not be considered an analytic subgroup analysis when the overall study only enrolled teenage mothers.) See 2.7 below for more details on HomVEE's clarified subgroup definition.

• Manuscripts describe study results. Manuscripts may be published or unpublished research, such as journal articles, book chapters, or working papers. A single study may produce one, or many, manuscripts. Typically, one manuscript reports on only one study, although in rare cases one manuscript may include several studies, if it describes evaluations of multiple interventions or the same intervention evaluated in multiple distinct (non-overlapping) samples.
• Findings summarize the effect of a home visiting model on a specific sample or subgroup, on a specific eligible outcome measure (see Chapter III, Section A.4.a of the HomVEE Draft Version 2 Handbook), at a specific time point, from a specific analysis. A manuscript typically includes multiple findings.

HomVEE rates findings (according to standards proposed in Chapter III of the HomVEE Draft Version 2 Handbook) and sorts manuscripts according to the highest-rated finding in the manuscript (see Chapter II, Section B.2.b of the HomVEE Draft Version 2 Handbook). When determining which models meet HHS criteria for an “evidence-based early childhood home visiting service-delivery model,” HomVEE considers both whether the research that calculated the findings was well designed, and whether the findings come from different studies (with distinct samples). See Exhibit II.11 of the HomVEE Draft Version 2 Handbook for details.

2.2 Establish a 20-Year Moving Search Window for Reviewing Most Manuscripts

Searches in HomVEE's first 11 annual reviews were for manuscripts published in or after 1989. Generally, the HomVEE Draft Version 2 Handbook updates the description of how HomVEE identifies research.

Now, HomVEE proposes to implement a 20-year moving window for previously unreviewed manuscripts to be eligible for review. Beginning with the 2021 review, to keep the review current, HomVEE proposes establishing a 20-year moving search window for previously unreviewed manuscripts to be eligible for review. For example, for the 2021 review, HomVEE would consider manuscripts released or published in 2001 through 2020. However, HomVEE proposes that two categories of older research remain eligible for review (1) older research that HomVEE has already reviewed and (2) research submitted at any time (that is, since HomVEE's inception and moving forward) through the call for research. This change is described in Chapter II, Section A.1.a.1 of the HomVEE Draft Version 2 Handbook.

2.3 Adopt the PRESS Method for Systematic Searching

Professional librarians have always conducted annual HomVEE literature searches using a transparent process in which the databases and search terms are published on the HomVEE website. Beginning with the 2021 review, in recognition of accepted practice in the library science field, HomVEE proposes to use a modified Peer Review of Electronic Search Strategies (PRESS) method to refine the search terms (McGowan et al. 2016). This approach includes adjusting search terms and search databases in keeping with the recommendations of professional librarians. This change is described in more depth in Chapter II, Section A.1.a.1 of the HomVEE Draft Version 2 Handbook.

2.4 Add New “Grey Literature” Databases

To better capture research that is not published in academic journals, HomVEE proposes to expand its annual search, beginning in 2021, to include two new databases to identify this “grey literature”: Google Scholar and the Harvard Think Tank Search. See Chapter II, Section A.1.a.1 of the HomVEE Draft Version 2 Handbook.

2.5 Establish Rule for Accepting Supplemental Information

HomVEE will continue, as it always has, to accept submissions from the public, both during its call for research and at other times, and to consider those annually as part of prioritizing research for review. The HomVEE Draft Version 2 Handbook proposes a clarification stating that, if authors submit unpublished work to the HomVEE call for research, HomVEE will consider only unpublished full manuscripts with sufficient text describing the study's procedure, analysis approach, and findings.

As part of the HomVEE Draft Version 2 Handbook (Chapter II, Section B.1.b), HomVEE proposes establishing a new rule about accepting supplemental information from stakeholders. Under the new rule, HomVEE would accept supplemental information only under specific circumstances. HomVEE must maintain a strict review schedule for the annual review to ensure results are released on time.

Supplemental information can take two forms (1) new information about a study's methods or procedures, or (2) new research that supplements what HomVEE had on hand at the close of that year's call for research, such as additional findings or new analyses of research in a previously reviewed manuscript, or an entirely new set of findings.

2.5.1 HomVEE Rule About New Information

HomVEE proposes to incorporate new information about methods and procedures into the initial review of a manuscript only if (1) it is provided in direct response to an author query and (2) authors submit it in time for reviewers to examine it during the same annual review cycle in which HomVEE issued the query. Otherwise, HomVEE intends to require authors wait until HomVEE releases its annual review results for the model described in the manuscript in question. Then, authors could follow the process for requesting a reconsideration of evidence to ask HomVEE to examine supplemental information that authors provide, through the appeals process, about methods or procedure.

2.5.2 HomVEE Rule About New Research

HomVEE proposes to treat all new research as a submission to the following year's call for research, unless it consists of new analyses conducted at the explicit request of the HomVEE review team (see Sections 3.3.1 and 3.3.2, below on repeated measures studies and structural equation models, respectively).

2.6 Define Contrasts in Impact Research That Are Ineligible for Review by HomVEE

To date, HomVEE has placed no restrictions on services offered to the comparison condition in the impact studies it reviews. Beginning with the 2021 review, HomVEE proposes that the review generally exclude research that isolates the impact of model features. Research on specific features does not answer HomVEE's core question of whether an early childhood home visiting model is effective. Specifically, HomVEE proposes not to review studies about the impact of model features. However, studies isolating the impact of a curriculum module may be treated as evidence for an independent model if all of the following criteria are met:

• The curriculum module satisfies the definition of an early childhood home visiting model;
• The treatment group does not receive any other curriculum modules from the base model; and
• The curriculum module has a manual and implementation infrastructure independent from that of the base model.

This change is described in Chapter III, Section A.2 of the HomVEE Draft Version 2 Handbook.

2.7 Defining Subgroups and Protocol for Reporting Subgroup Analyses

In the HomVEE Draft Version 2 Handbook, HomVEE defines a subgroup as a subset of the sample that researchers choose to examine in a study; that is, an analytic subgroup.

Subgroup research is important for HomVEE because a model can meet the HHS criteria for “evidence-based early childhood home visiting service delivery models” based on findings from subgroups. The HHS criteria for an “evidence-based early childhood home visiting service delivery model” include special rules about subgroup findings: If favorable results that could form the evidence base for a model “[are] found for subgroups but not for the full sample for the study, [the findings must] be replicated in the same domain in two or more studies using non-overlapping analytic study samples.” (See Exhibit II.11 in the HomVEE Draft Version 2 Handbook.) Therefore, HomVEE exercises care in identifying subgroup research and understanding how the subgroup relates to the overall study sample.

HomVEE defines a subgroup as a subset of the overall sample examined in a study—that is, an analytic subgroup (see Section 2.1, above). Notably, this is different from defining subgroup as a subset of the overall population. Although researchers may examine an analytic subgroup in hopes of making inferences about a subset of the population, the goal of the HHS criteria is to ensure that program impacts are replicated consistently for an outcome domain. Such replication is what gives HomVEE confidence that evidence of effectiveness is not due simply to chance. Thus, if a model meets HHS criteria for evidence of effectiveness based on subgroup findings, this means that research in which that subgroup was similarly defined in relation to the broader sample had consistent, favorable (statistically significant) findings in distinct study samples.

Subgroup results may be nested within a manuscript (for example, results from teenage mothers when the overall results in the manuscript are from mothers with a range of ages), or they may be the main focus of a manuscript (for example, a manuscript focusing on results from teenage mothers when the overall study sample included mothers with a range of ages). HomVEE treats both of those as analytic subgroup analyses. HomVEE's definition means that not all analyses restricted to a certain characteristic are subgroup analyses. For example, results from teenage mothers are not an analytic subgroup analysis when the overall study only enrolled teenage mothers, even though teenage mothers are a subgroup of the population of mothers as a whole.

Because HomVEE's mission is to identify which models are effective according to the HHS criteria, and to use project resources judiciously, HomVEE proposes to only review research on replicable subgroups (if it meets other eligibility criteria defined in Chapter II, Section A of the HomVEE Draft Version 2 Handbook), and to only report review results for replicated subgroups. HomVEE proposes the following definitions for those terms:

• Replicable subgroups are defined by a characteristic that a different study could replicate with a non-overlapping sample. Most subgroups are replicable, in theory. However, HomVEE does not consider subgroups defined by cohort or time (for example, a subgroup of mothers enrolled in 1995 in a study that included mothers enrolled across several years) to be replicable in subsequent studies, and therefore does not review time-based subgroups. Similarly, HomVEE will only consider a subgroup defined by location to be replicable if the location was selected based on defined characteristics (for example, county with the highest teen birth rate in the state in a study conducted in several counties). Location-based subgroups defined by a location name (for example, Adams County in a study conducted in several counties) will not be reviewed because the HomVEE team cannot confidently verify whether the subgroup sample in a subsequent study in that county overlaps with the first study when the team applies HHS criteria.

HomVEE will report subgroup results only from a replicated subgroup, one that has an identical definition in two non-overlapping research samples. For example, a study examining a subgroup of primiparous teenagers is not replicated by a study examining primiparous women of all ages. This approach is consistent with the HHS criteria's emphasis on observing effects across independent samples.

2.8 Clarify HomVEE's Approach to Operationalizing the HHS Criteria for Randomized Controlled Trials

As specified in the statute that authorized the MIECHV Program and required HHS to establish criteria for evidence of effectiveness of home visiting models, the HHS criteria for an “evidence-based early childhood home visiting service delivery model” state that additional criteria apply when the research on home visiting models comes from randomized controlled trials (see Exhibit II.11 in the HomVEE Draft Version 2 Handbook). Specifically, one or more favorable impacts must be sustained for at least 1 year after program enrollment, and one or more favorable impacts must be reported in a peer-reviewed journal. The HomVEE Draft Version 2 Handbook clarifies the way that HomVEE has operationalized the additional criteria for randomized controlled trials (RCTs). Specifically, these two requirements can be satisfied by findings from different studies, provided the quality of these findings is rated as moderate or high.

3.0 Clarifications and Changes to HomVEE Standards

HomVEE proposes several updates to its standards for reviewing manuscripts about impact studies, including both clarifications and changes.

It is natural in the course of a systematic review for issues to arise that cannot be addressed by existing standards and rules. These issues require the HomVEE team to develop internal guidance to guide the review.

Clarifications to the standards added to the HomVEE Draft Version 2 Handbook, summarized in Section 3.1 below, represent HomVEE's attempt to formalize internal guidance generated over the course of conducting reviews so that the procedures are applied systematically.

Similarly, as research methods evolve, it is necessary for ongoing systematic reviews to change their standards to meet the needs of the field. Changes are HomVEE's attempt both to align with aspects of other ongoing, federally sponsored systematic evidence reviews (Section 3.2) and to specify standards for research designs that are becoming increasingly common in home visiting studies (Section 3.3).

3.1 Clarifications

3.1.1 Changes to Terminology Used

HomVEE reviews manuscripts about research that uses any of three types of quasi-experimental designs (QEDs)—regression discontinuity designs (RDDs), single-case designs (SCDs), and non-experimental group designs (NEDs). Previously, HomVEE used QED to refer only to NEDs. Other designs, including SCD and RDD, are also quasi-experimental, so HomVEE proposes labeling this category of research more precisely, as NED (see Chapter I, Section C of the HomVEE Draft Version 2 Handbook). HomVEE intends to use this new terminology to more accurately reflect the fact that HomVEE does not have (and does not propose to implement) requirements about statistical matching in NED designs.

3.1.2 Ineligible and Preferred Analyses

To date, as long as the underlying study design is an RCT or QED, HomVEE has not specified rules for identifying analyses as ineligible for review.

Starting with the 2021 review, HomVEE proposes to exclude certain analyses within manuscripts about RCTs and QEDs as ineligible, as described in Chapter III, Section A.3 of the HomVEE Draft Version 2 Handbook. HomVEE's mission is to determine whether research shows that a home visiting model improves outcomes for children and families. Questions about the mechanisms behind how a model works, the settings where it might work best, and the populations who benefit the most from the intervention are outside of the scope of the HomVEE review. Although answers to these questions are important for understanding and improving home visiting models, the primary aim of the HomVEE review is to identify currently available models that are effective. For this reason, certain types of analyses designed to answer questions other than whether a model is effective are not eligible for review.

In addition, analyses of how the home visiting model affected only sample members who received it are sometimes ineligible for review if other analyses in the manuscript better address HomVEE's mission.

3.1.2.1 Mediating and Moderating Analyses

First, HomVEE proposes that most mediating and moderating analyses (except some structural equation models, see Section 3.3.2 below), would be ineligible for review. HomVEE focuses on research that answers the following question: Is the home visiting model effective? Mediating and moderating analyses answer important, but slightly different questions of how, and for whom, the model works.

3.1.2.2 Endogenous Analyses

Second, HomVEE proposes that the review would exclude analyses that control for endogenous characteristics. These characteristics (1) are defined by behavior emerging after study participants know whether they will be in the intervention group or the comparison group and (2) could theoretically be affected by a home visiting model. Analyses that control for endogenous characteristics produce biased estimates of the effectiveness of an intervention. Analyses of subgroups defined by endogenous characteristics would also be ineligible for review.

3.1.2.3 Analyses of the Impact of the Treatment on the Treated

The HomVEE Draft Version 2 Handbook also specifies a proposed clarification to how HomVEE would review studies that examine the effect of the treatment (the home visiting model) on the treated (study sample members who receive the treatment). Specifically, when a study's researchers examine the effect of both the intent to treat (ITT) and the treatment on the treated (TOT), HomVEE proposes to focus its review on the ITT, because those estimates more realistically depict the average magnitude of the effect that a program replicating the model would cause. If those researchers report only TOT estimates, HomVEE reviews those using What Works Clearinghouse (WWC) Version 4.1 guidance on reviewing for Complier Average Causal Effects.

3.1.3 Eligible Outcomes and Baseline Assessability

Since its inception, HomVEE has reviewed findings of home visiting impact studies that fall into eight domains related to child, maternal, and family well-being. In the HomVEE Draft Version 2 Handbook, HomVEE proposes clarifying which specific findings within eligible analyses are eligible for review (see Chapter III, Section A.4). These clarifications formalize and expand HomVEE's existing internal guidance on eligible outcomes and baseline assessability.

First, the HomVEE Draft Version 2 Handbook clarifies that only unique findings would be eligible for review (those that report results on a different outcome, sample or subgroup, or time period, or with a different analytic approach, than findings reported in other manuscripts about the same home visiting model). In these cases, consistent with current HomVEE practices, the review simply would reference the other manuscript—the first or most complete one in which HomVEE encountered the finding—where HomVEE users could find those results and the review conclusions. The Draft Version 2 Handbook also clarifies that HomVEE would not consider simple transformations of analyses with the same sample, outcome, and time period to be unique findings within a manuscript if they (1) transform findings data from frequency to a ratio (such as percentage or per thousand) or (2) transform findings data across different ratio types (such as from percentage to per thousand) because these simple transformations do not constitute a different analytic approach. In manuscripts with such transformations, HomVEE proposes to review the finding that is calculated as a percentage, because it is an intuitive measure to many readers and can be easily compared across studies.

Second, the HomVEE Draft Version 2 Handbook specifies categorization practices and baseline equivalence requirements for outcomes that HomVEE reviews. See Exhibit III.2 and Chapter III, Section B.3 of the HomVEE Draft Version 2 Handbook for a summary and Appendix C of that document for a detailed listing. That appendix indicates which outcomes or outcome categories belong under each of the eight domains. Also, although HomVEE requires NEDs and certain RCTs (those with high attrition or compromised randomization) to establish that the intervention and comparison groups are equivalent at baseline, the review team recognizes that some measures cannot or should not be measured at baseline. Therefore, Appendix C of the HomVEE Draft Version 2 Handbook clarifies which outcomes HomVEE would expect authors to assess at baseline.

3.2 Changes To Align HomVEE With Standards of Other Federally Sponsored Systematic Reviews

HomVEE's initial standards aligned to WWC standards, Version 2.1, which were the latest standards implemented when the HomVEE review began. These standards define the criteria that research must meet to be assigned each of three ratings. HomVEE calls these ratings high, moderate, and low, although the WWC rates research as Meets Standards (HomVEE high), Meets Standards with Reservations (HomVEE moderate), and Does Not Meet Standards (HomVEE low). The WWC remains a prominent and influential federally sponsored systematic evidence review. In early 2020, WWC released Version 4.1 standards. Furthermore, in the time since HomVEE began, ACF has begun overseeing another, related systematic review: The Title IV-E Prevention Services Clearinghouse. That review focuses on child welfare research, some of which overlaps with home visiting research, and its standards are similar to those of WWC Version 4.1. ACF is interested in aligning standards for HomVEE and the Prevention Services Clearinghouse where appropriate.

In its HomVEE Draft Version 2 Handbook, HomVEE proposes to adopt many aspects of the latest WWC standards and some aspects of the Prevention Services Clearinghouse standards so that the review stays synchronized with accepted best practices in federally sponsored systematic reviews. The sections below describe proposed changes to the HomVEE review that would affect study ratings as the HomVEE criteria stand now. (HomVEE proposes to fully adopt WWC Version 4.1 criteria for regression discontinuity design studies, which are not described below because HomVEE has not, to date, reviewed any studies with this design.)

3.2.1 Requirement for Validity and Reliability of Outcome Measures

To date, HomVEE has had no stated validity and reliability requirements that outcomes must meet, although the review reports whether outcomes are primary (which HomVEE defines as an outcome measured through direct observation, direct assessment, or administrative data; or self-reported data collected using a standardized [normed] instrument) or secondary (for HomVEE, most self-reported data, excluding self-reports based on a standardized instrument). With the Draft Version 2 Handbook (see Chapter III, Section B.4), HomVEE proposes to introduce face validity and reliability standards. HomVEE reviewers will apply these new standards to all findings that are within one of HomVEE's eight outcome domains and to all measures HomVEE uses to assess baseline equivalence. Findings about outcomes that do not meet both the face validity and the reliability standard would rate low. With this change, HomVEE proposes to stop sorting outcomes as primary or secondary.

To meet the face validity standard, an outcome measure must be (1) clearly defined and (2) measure the construct it was designed to measure. This information could come from the manuscript reviewers examine, or from supplemental information that HomVEE requests from the author. HomVEE reviewers propose to consult with project leaders whenever it is not clear whether a measure meets the validity requirement, and project leaders would in turn consult with subject matter experts and with ACF about the validity of new and of modified standardized measures.

Some outcome measures are not appropriate to validate with psychometric tests. HomVEE proposes to assume that the following measures are reliable: (1) Administrative records obtained from child welfare or other social service agencies, hospitals or clinics, and schools; (2) demographic characteristics; and (3) medical or physical tests.

Otherwise, to demonstrate reliability, outcome measures must meet at least one of the following standards:

• Internal consistency (such as Cronbach's alpha) of 0.50 or higher.
• Test-retest reliability of 0.40 or higher.
• Inter-rater reliability (as indicated by percentage agreement, correlation, or kappa) of 0.50 or higher.

Under the proposed approach, HomVEE reviewers would prioritize reliability statistics on the sample of participants in the manuscript under review, but would also consider statistics from test manuals or studies of the psychometric properties of the measures. The review team may ask authors to provide additional information about the reliability of their measures.

3.2.2 In Some Cases, Some Sample Loss Does Not Count as Attrition

Attrition happens when outcome data are missing for some members of the intervention and comparison groups in a study. Previously, HomVEE counted all sample loss as attrition unless the authors had imputed findings (see Section 3.2.3, below). In alignment with Version 4.1 of the WWC Standards, the HomVEE Draft Version 2 Handbook (see Chapter III, Section B.1) proposes that some types of sample loss will not count as attrition in HomVEE. First, losing sample members after random assignment because of acts of nature, such as hurricanes, fires, or the COVID-19 pandemic, is not considered attrition if the loss affects the intervention and comparison conditions in the same way. However, if the sample loss due to an act of nature was concentrated in one of the conditions, then the sample loss would be considered attrition. Second, when researchers exclude a subsample of the randomly assigned sample from their analysis, HomVEE would not consider that excluded subsample to constitute attrition if (1) the subsample was randomly selected or (2) the subsampling was based on characteristics that were clearly determined before the start of the intervention and applied consistently across the intervention and comparison conditions.

3.2.3 Standards for Addressing Missing Data

The original HomVEE standards did not specify how reviewers would respond when study authors used various analytic strategies to account for missing data. The HomVEE Draft Version 2 Handbook (see Chapter III, C.2, as well as Appendix F) proposes to align HomVEE's practices to the way WWC Version 4.1 standards handle studies with missing data. Specifically, HomVEE proposes to first calculate attrition based on the analytic sample in the manuscript, treating any imputed values as lost sample. If baseline data in the analytic sample are missing or imputed, baseline equivalence would have to be established using the largest baseline difference accounting for missing or imputed baseline data. Second, manuscripts about studies with missing data would only be eligible for review by HomVEE if the authors had used the following specific approaches to address the missing data:

• Complete case analysis
• Maximum likelihood (including expectation maximization and full information maximum likelihood)
• Multiple imputation (must be conducted separately by treatment status)
• Nonresponse weights (must be conducted separately by treatment status; acceptable only for missing outcome data, not for missing baseline data)

In alignment with WWC version 4.1 standards, if the baseline data include imputed data, HomVEE would also apply other criteria when assessing baseline equivalence (see 3.2.5).

3.2.4 No Baseline Equivalence Requirement for Low-Attrition RCTs

Original HomVEE standards require authors of RCTs and non-experimental designs to establish baseline equivalence on race and ethnicity, socioeconomic status, and measures of outcomes that are feasible to assess when the study begins.

In low-attrition RCTs, the original standards allow authors to instead implement statistical controls for these characteristics. In alignment with WWC Version 4.1 Standards, the HomVEE Draft Version 2 Handbook proposes that HomVEE would no longer require that RCTs with low attrition establish equivalence or adjust for baseline differences. This is because proper randomization is expected to produce groups that are similar, and baseline differences that might be observed on one or more measures are not generally evidence of differences that will introduce bias into research findings.

3.2.5 Baseline Equivalence Depends on Difference in Effect Sizes, and Other Considerations

The original HomVEE standards based the assessment of equivalence on measuring statistically significant differences between intervention and comparison groups at baseline. In line with WWC Version 4.1 standards, HomVEE's Draft Version 2 Handbook proposes that HomVEE will assess baseline equivalence based on the magnitude of the difference in standard deviation units (effect size).

To limit bias that can arise from differences in the treatment and comparison group units used to measure the effect of a home visiting model on outcomes, the groups must appear similar on the relevant baseline characteristics that are thought to be related to the outcomes. This balance is best shown using the observed magnitude of differences in the sample.

Specifically, the new HomVEE criterion for baseline equivalence proposes to rely on effect size, computed as the absolute value of the difference between treatment and comparison groups in standard deviation units. HomVEE would require the following to be true for research to demonstrate baseline equivalence for a specified characteristic:

• A baseline effect size less than or equal to 0.05 meets the baseline equivalence requirement and requires no statistical adjustment.
• For a baseline effect size that is greater than 0.05 and less than or equal to 0.25, an acceptable statistical adjustment for the baseline characteristic is required to meet the baseline equivalence requirement.
• If the baseline effect size is greater than 0.25, HomVEE considers the intervention and comparison groups to be nonequivalent, that is, the intervention and comparison groups do not meet the baseline equivalence requirement for the specified characteristic.

Under the proposed new standards, HomVEE would also consider the following when assessing baseline equivalence:

• HomVEE would allow baseline data that include imputed data to be used to demonstrate baseline equivalence of the analytic sample in some cases. If the baseline data include imputed data, HomVEE would first estimate how large the baseline difference (in standard deviation units) between intervention and comparison groups might be under different assumptions about how the missing data are related to measured and unmeasured factors. Then HomVEE would use the largest of those estimates in absolute value as the effect size for assessing baseline equivalence.
• The measures used to establish baseline equivalence must be at the same level as the unit of analysis. For example, in an analysis at the individual or family level, measures of socioeconomic status at the ZIP code level may not be used to establish baseline equivalence between the individuals or families in the intervention and comparison groups.
• If the impact analyses use weights, then the baseline means must be calculated using the same weights.
• If the study conducted random assignment within blocks or strata, and the analyses include dummy variables that differentiate these blocks or strata, then these same dummy variables can be used to adjust the baseline means.

This criterion is described in the HomVEE Draft Version 2 Handbook, Chapter III, Section B.2.

3.2.6 Allowable Statistical Adjustment Techniques

HomVEE has always required that authors implement statistical adjustments for baseline differences if their studies use (1) an RCT design or (2) one type of quasi-experimental design, NED. To date, HomVEE has not specified allowable techniques for that adjustment. With the HomVEE Draft Version 2 Handbook (see Chapter III, Section B.2.b), HomVEE proposes to follow WWC Version 4.1 guidelines about which statistical adjustment procedures are acceptable. Those are:

• Acceptable analytic methods to adjust for baseline differences:

○ Regression adjustments

○ Analysis of covariance (ANCOVA) or multivariate analysis of covariance (MANCOVA)

○ Repeated measures analysis of variance (ANOVA) or multivariate analysis of variance (MANOVA)

○ Estimating impacts only for groups defined at baseline (for example, ever had a baby versus never had a baby)

○ Growth curve modeling (this approach to modeling repeated measures research is also subject to other requirements; see 3.3.1 below)

• Acceptable methods if baseline and follow-up measures of outcome are the same and have a strong relationship to each other

○ Gain or change scores (pre-post differences)

○ Difference-in-difference adjustments

○ Fixed effects for individuals

3.2.7 Cluster RCTs

HomVEE rarely encounters RCTs with a cluster design in which a group of sample members, such as a neighborhood, is assigned to be offered a home visiting model or some other condition. However, in such designs, limiting sample loss at both the cluster (for example, a neighborhood) and subcluster (for example, a family that received home visiting) levels is important to maintaining the integrity of the randomization design. Under HomVEE's original standards, reviewers would apply a cluster correction to findings if authors themselves had not done so, but no special requirements were in place for rating studies that used a cluster design.

Instead, in research reviewed under the HomVEE Draft Version 2 Handbook, HomVEE proposes to align to WWC Version 4.1 guidelines for calculating attrition and non-response of subcluster members (such as individuals or families) in cluster RCTs. Exhibit III.14 of the HomVEE Draft Version 2 Handbook specifies how studies would be rated based on their combination of attrition at the cluster and individual levels, and authors' decisions about implementing statistical controls. In brief, a cluster RCT would be eligible to rate high only if it has low sample loss at both the cluster level and the individual level. To rate moderate, research about cluster RCTs with high attrition and research about cluster NEDs would need to demonstrate baseline equivalence of the analytic sample. Additional detail about this new standard appears in Chapter III, Section C.1 of the HomVEE Draft Version 2 Handbook.

3.2.8 Adopt New WWC Version 4.1 Standards for Regression Discontinuity Designs

Regression discontinuity research has been eligible for review by HomVEE since the project's inception, using earlier WWC pilot criteria for this research. This design is rare in home visiting research, and, to date, HomVEE has not reviewed any research that uses this design.

HomVEE proposes to align its Version 2 standards to WWC's latest (Version 4.1) RDD standards. The updates to the RDD standards consist of:

• A new set of procedures for reviewing “fuzzy” RDDs (for example, those in which some intervention group members do not receive intervention services and the analysis adjusts for this nonparticipation),
• Expanded procedures for reviewing multi-site and multiple assignment variable regression discontinuity designs, and
• A preference for local bandwidth impact estimation over global impact regression with flexible functional forms.

Appendix D of the HomVEE Draft Version 2 Handbook thoroughly describes the new WWC Version 4.1 RDD standards that HomVEE propose to implement, and their corresponding reporting procedures.

3.3 Other Changes

The HomVEE Draft Version 2 Handbook proposes two other changes to standards for reviewing impact studies. The two changes would define a new approach to reviewing designs that are becoming increasingly common in home visiting studies—repeated measures analyses and structural equation models. The third change pertains to SCD research.

3.3.1 Repeated Measures Analyses

In repeated measures analyses, authors measure the research sample at several time points to chart its growth over the course of the intervention and sometimes beyond. To date, HomVEE has not specified any standards for reviewing repeated measures analyses in group-design studies (such as RCTs and NEDs), nor have other federally sponsored systematic evidence reviews thoroughly addressed this.

In the proposed new standard, HomVEE would only review and report findings from repeated measures analyses with multiple follow-ups in RCTs and NEDs when the findings are available for individual time points, relative to baseline. When rating each time point, HomVEE would apply its Version 2 RCT or NED standards. Generally, when gathering information to rate each time point, HomVEE would defer to what the author reported or what the review team could calculate based on details the author provided. As a last resort, when adjusted analyses are necessary in order to rate the study, and the HomVEE team cannot make the necessary calculations, the HomVEE team would ask authors to reanalyze their data to calculate adjusted time point findings. HomVEE would exclude from its review of a repeated measures analysis any time points for which an impact cannot be included in HomVEE reports because neither author-provided nor HomVEE-calculated estimates are available. Chapter III, Section C.3 of the HomVEE Draft Version 2 Handbook describes the proposed new approach to rating repeated measures analyses in detail.

3.3.2 Structural Equation Models

Structural equation models (SEMs) examine the relationship between a dependent variable and multiple independent variables, often incorporating multiple outcomes from different follow-up periods. To date, HomVEE standards did not define how the review would incorporate SEM research. Chapter III, Section C.4 of the HomVEE Draft Version 2 Handbook specifies how HomVEE proposes to approach the review of research with this design going forward. In brief, only SEMs that are accompanied by a path diagram (including one authors may submit in response to a query from HomVEE) and that are identified (that is, the degrees of freedom are greater than the parameters to be estimated) would be eligible for review. Within SEMs that are eligible for review, HomVEE would review only findings for which the answer to the following two questions is yes: (1) Is there a direct pathway from the intervention to the outcome? and (2) Are there no pathways leading to that outcome from another outcome? This approach is consistent with HomVEE's proposed new approach to mediated and moderated analyses. See also Section 3.1.2.1 above.

3.3.3 Review of Single-Case Design (SCD) Research

SCDs are quasi-experimental research designs in which an individual case serves as its own control, and the outcome is measured repeatedly within and across different conditions (as defined in What Works Clearinghouse [WWC] Version 4.1 standards). SCD research has been eligible for HomVEE review since its inception, using earlier WWC pilot criteria for this research. With Version 4.1, WWC has removed the “pilot” designation from its standards and has updated its procedures for reviewing SCD research in several ways; HomVEE proposes aligning to WWC's version 4.1. Although WWC previously standards instructed reviewers to only use visual analysis of changes in the outcome over time and across conditions to characterize the findings from an SCD study, the new standards from WWC also have reviewers calculate and use a design-comparable effect size to characterize the findings. Reviewers would still use visual analysis to assess whether a SCD study is well designed. To calculate a design-comparable effect size, the HomVEE contractor review team would use data presented in the study if possible, or (only if necessary) contact the study authors to request raw study data so the team could calculate that value. Appendix E of the HomVEE Draft Version 2 Handbook thoroughly describes the WWC Version 4.1 SCD standards that HomVEE propose to implement, and their newly updated reporting procedures.

4.0 Timeline for HomVEE To Apply New Procedures and Standards

HomVEE proposes to apply the new procedures and standards beginning with the 2021 review. HomVEE will not retroactively apply the new standards to previously reviewed research about evidence-based models unless it is SCD research about a model HomVEE prioritizes and selects for review.

To promote consistency in reporting across the review, clarifications about the outcomes that are eligible for review in each domain retroactively will apply to all models regardless of (1) their evidence-based status according to HHS criteria and (2) whether they are prioritized and selected for review. However, manuscripts that have findings excluded or moved to other domains will not be re-reviewed with HomVEE Version 2 standards (unless they are manuscripts about a SCD study). In addition, the HomVEE team will retroactively apply clarified definitions of study, manuscript, and subgroup, with the aim of relabeling HomVEE products so they use consistent language.

Also, HomVEE typically reviews eligible models every other year at the earliest. In 2021 (the first year that new procedures and standards are in effect), HomVEE will suspend this rule for one year only, so that models reviewed in 2020 are not excluded from consideration for the 2021 review.

4.1 HomVEE Will Not Retroactively Apply New Procedures and Standards to Inactive Models

For models that no longer provide implementation support, the HomVEE team generally does not plan to retroactively apply the new procedures and standards, except to apply the clarifications about the outcomes that are eligible for review in each domain and about the definitions of study, manuscript, and subgroup. The team proposes to update reports about those models on the HomVEE website (https://homvee.acf.hhs.gov) to indicate that they were reviewed under the initial HomVEE procedures and standards.

5.0 Request for Information (RFI)

Through this Federal Register Notice, ACF is soliciting information from a broad array of stakeholders on the proposed revisions to HomVEE's procedures. Federal, state, and local decision makers rely on HomVEE to know which home visiting models are effective. New definitions, rules, and procedures about model versions may affect which models are deemed effective by HomVEE. New procedures may affect which models are eligible for review and deemed effective by HomVEE. New standards may affect which studies constitute well-designed research that serves as an evidence base for models that meet HHS criteria for an “evidence-based early childhood home visiting service delivery model.”

Responses to this Federal Register notice will inform ACF's ongoing discussion about HomVEE's procedures and standards, with the aim of publishing a final HomVEE Version 2 Handbook by the end of 2020. This RFI is for information and planning purposes only and should not be construed as a solicitation or as an obligation on the part of ACF or HHS.

(Authority: Social Security Act Title V § 511 [42 U.S.C. 711], as extended by the Bipartisan Budget Act of 2018 (Pub. L. 115-123) through fiscal year 2022)

John M. Sweet Jr,

ACF/OPRE Certifying Officer.

References

U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, What Works Clearinghouse. What Works Clearinghouse Version 2.1 Evidence Standards. 2011. Available at https://ies.ed.gov/​ncee/​wwc/​Docs/​referenceresources/​wwc_​procedures_​v2_​1_​standards_​handbook.pdf. Accessed June 19, 2020.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, What Works Clearinghouse. What Works Clearinghouse Procedures Handbook: Version 4.1., 2020a. Available at https://ies.ed.gov/​ncee/​wwc/​Docs/​referenceresources/​WWC-Procedures-Handbook-v4-1-508.pdf. Accessed June 19, 2020.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, What Works Clearinghouse. What Works Clearinghouse Standards Handbook: Version 4.1., 2020b. Available at https://ies.ed.gov/​ncee/​wwc/​Docs/​referenceresources/​WWC-Standards-Handbook-v4-1-508.pdf. Accessed June 19, 2020.