2020-16992. Updated Defintions, Rules, and Procedures Related to Model Versions: Home Visiting Evidence of Effectiveness (HomVEE) Review

AGENCY:

Administration for Children and Families, U.S. Department of Health and Human Services.

ACTION:

Request for public comment.

SUMMARY:

The Administration for Children and Families (ACF), within the U.S. Department of Health and Human Services (HHS), oversees the Home Visiting Evidence of Effectiveness (HomVEE) review, which is proposing to revise the procedures and standards that guide its work. The revised procedures and standards will be presented in two separate Federal Register notices. The current Federal Register notice seeks comments on proposed updated definitions, rules, and procedures related to handling home visiting model versions (commonly referred to in the home visiting research literature as adaptations) in the HomVEE review. Another Federal Register notice summarizes proposed changes and clarifications to HomVEE's procedures and standards for rating the quality of impact studies and determining which home visiting models meet HHS criteria for evidence of effectiveness. Readers are referred to the full text of the HomVEE Draft Version 2 Handbook on the HomVEE website (https://homvee.acf.hhs.gov/​) for more details on all proposed changes.

DATES:

Send comments on or before September 1, 2020.

ADDRESSES:

Submit questions, comments, and supplementary documents to [e-mail] with “HomVEE model versions FRN comment” in the subject line.

SUPPLEMENTARY INFORMATION:

Invitation to Comment: HHS invites comments regarding this notice. To ensure that your comments are clearly stated, please identify the section of this notice that your comments address.

1.0 Background

To help policymakers, program administrators, model developers, researchers, and the public identify rigorous research and understand which early childhood home visiting models are effective, ACF's Office of Planning, Research, and Evaluation within HHS oversees the HomVEE review. HomVEE's mission is to conduct a thorough and transparent review of the research literature on home visiting for families with pregnant women and children from birth to kindergarten entry. The review team identifies well-designed research within that pool and extracts and summarizes the findings from that research.

One critical use of HomVEE's results is to determine which home visiting models meet the HHS criteria for an “evidence-based early childhood home visiting service delivery model” (see Exhibit II.11 in the HomVEE Draft Version 2 Handbook), a key requirement of eligibility for implementation with the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program funding. The MIECHV Program is administered by the Health Resources and Services Administration (HRSA) in partnership with ACF. Created in 2010, the MIECHV Program provides funding to states, territories, and tribal entities to implement home visiting models. MIECHV awardees have the flexibility to tailor the program to serve the specific needs of their communities. Through a needs assessment, awardees identify at-risk communities and select home visiting service delivery models that best meet state and/or local needs. As per MIECHV's authorizing statute, state and territory awardees must spend the majority of their MIECHV Program grants to implement evidence-based home visiting models, with up to 25 percent of funding available to implement promising approaches that will undergo rigorous evaluation.

For the first time since its inception in 2009, HomVEE is proposing substantial revisions to several procedures and standards that guide the systematic review. The proposed revisions include (1) clarifying and updating standards and procedures (presented in a separate Federal Register notice) for rating the quality of impact studies that are used to determine which home visiting models meet HHS criteria for an “evidence-based early childhood home visiting service delivery model” and (2) clarifying definitions, rules, and procedures for handling model versions (commonly referred to in the home visiting research literature as adaptations) in the HomVEE review process. The current Federal Register notice focuses on the latter set of revisions.

HomVEE routinely encounters research in which an individual home visiting model has been changed or altered for various reasons, including to fit a new or different context (for example, adding a new curriculum targeted to a specific population) or to implement a new feature (for example, adding text messaging between visits). In order to ensure the transparency and accuracy of the HomVEE review, it is important to have clear definitions, rules, and procedures to classify research on different versions of home visiting models. Over the course of the past 2 years, a HomVEE workgroup composed of ACF and HRSA staff with contractor support met to discuss and develop the draft definitions, rules, and procedures outlined in this Federal Register notice. Their work was informed by a close review of the procedures of other federally sponsored systematic evidence reviews.

Through this Federal Register notice, ACF seeks to provide a transparent account of how the review operates and to collect stakeholder input on draft definitions, rules, and procedures related to model versions. Because these procedural changes may affect critical decision making, and to better understand the implications of the changes for various stakeholders, ACF seeks public input on the draft changes. After a period of public comment, HomVEE will finalize and release the final definitions, rules, and procedures related to model versions.

2.0 Definitions, Rules, and Procedures for Reviewing Home Visiting Models and Model Versions

First, this Federal Register notice presents HomVEE's definition of “early childhood home visiting model.” Then, it discusses how HomVEE proposes to define and distinguish among various types of model versions.

2.1 Defining an Early Childhood Home Visiting Model

In order to identify eligible research, HomVEE intends to apply the following definition of an early childhood home visiting model: HomVEE defines an early childhood home visiting model as an intervention that delivers a specified set of services (through a specified set of interactions). A model has a set of fidelity standards that describe how the model is to be implemented. These voluntary programs are interventions, either designed or adapted and tested for delivery in the home, in which trained home visitors meet with expectant parents or families with young children on a schedule that is defined or can be tailored to meet family needs. During the visits, home visitors aim to build strong, positive relationships with families who want and need support to improve child and family outcomes. Models reviewed by HomVEE must serve pregnant women or families with children from birth to kindergarten entry. In a model eligible for review, the primary service delivery strategy must be home visiting and there must be research on the model that examines its effects in at least one of eight outcome domains: Child development and school readiness, child health, family economic self-sufficiency, linkages and referrals, maternal health, positive parenting practices, reductions in child maltreatment, and reductions in juvenile delinquency, family violence, and crime.^[1] This definition of a home visiting model is for the purpose of sorting and classifying the research literature as part of the HomVEE review. It is not for the purposes of determining eligibility for MIECHV funding.

2.1.1 Classifying Research on Early Childhood Home Visiting Models

2.1.1.1 Defining Model Versions

To date, the HomVEE team has relied on model developers and manuscript authors to identify the specific version of the model they are examining in a research study. However, manuscripts vary in the extent to which their authors clearly identify the model version being tested. Moving forward, HomVEE intends to implement definitions, rules, and procedures to systematically differentiate versions of a model. This will help ensure that HomVEE treats all models equally and classifies research on model effectiveness accurately. These definitions, rules, and procedures are for the purpose of sorting and classifying the research literature as part of the HomVEE review. It is not for the purposes of determining eligibility for MIECHV funding.

HomVEE proposes to apply the following definitions to differentiate models:

• Base model: The model as designed by the developer. (This includes implementations of the base model by the developer and replications of the base model by others.)
• Model version: Distinguished by a substantial change to a core feature of the base model (substantial change and core feature are defined below).
• Model iteration: Distinguished by a substantial change to a core feature of the base model that the model developer intends to represent a permanent update to the base model. The term model iteration is intended to capture changes that are part of the natural evolution of a model over time.
• Family of models: A base model and its iterations and versions.

As indicated by the definitions, identifying substantial changes to core features or the addition or removal of core features is critical for differentiating base models and model versions. For the purposes of HomVEE's review, HomVEE conceptualizes home visiting models as being composed of a set of core features that include both content features (such as the topics covered in a curriculum) and implementation features (such as home visitor training and staff qualifications). Core features are integral to the model and will be defined as the following:

• Visit format (in-person, virtual, mixed)
• Home visit frequency
• Home visit length
• Number of home visits
• Program duration
• Required staffing structure
• Home visitor education and experience
• Home visitor supervision requirements
• Home visitor caseload
• Requirements for program certification
• Training requirements for home visitors (initial and ongoing)
• Trainer qualifications
• Curriculum
• Other features in the model's theory of change

HomVEE further proposes to differentiate core features by whether the feature has a fixed requirement or flexible requirement. Core features with fixed requirements have specific guidelines that guide the implementation of the core feature, such as requiring a specific curriculum or delivering home visits with a specific frequency. Fixed requirements may also be a specific range, such as requiring home visits of 60-90 minutes, or minimum thresholds, such as a staff qualification of at least a B.A. degree. Flexible requirements do not have specific guidelines about implementation, such as no requirement for implementers to use a specific curriculum.

2.1.1.2 Identifying Core Features

For model families that are candidates for review in a given year, HomVEE proposes to generate a list of core features for each model and determine whether each core feature of each model has fixed or flexible requirements. HomVEE will first generate the list based on information available through manuscripts, the developer website, and/or the existing HomVEE implementation profile. HomVEE will then send the list to model developers to invite them to verify HomVEE's understanding of the model's core features.

2.1.1.3 Using Information on Core Features To Identify Model Versions

Using a set of predetermined decision rules and the list of core features verified by model developers, HomVEE intends to identify substantial changes to core features to distinguish research on model versions and iterations from research on the base model. HomVEE welcomes feedback from the field on these decision rules. The rules draw on general guidance from the implementation science literature and current practices of other evidence reviews.

Specifically, HomVEE will consider a change substantial if a fixed requirement related to a core feature is added or removed (see Table 1). For example, if researchers or model implementers remove a curriculum module or add text message check-ins with families in between in-person visits, those changes would be considered substantial. In addition, HomVEE will consider as substantial those changes in which a fixed requirement becomes flexible or a flexible requirement becomes fixed. To illustrate, a base model may allow implementers to select their own curriculum that best fits their context. Researchers and model implementers may decide to modify that feature and require a specific curriculum. This would be considered a substantial change because a flexible requirement (no required curriculum) is now fixed (that is, a specific curriculum is now required).

In addition, for core features that involve a specific frequency or number (such as number of home visits or home visitor caseloads), changes that modify the frequency by more than 50 percent will, in general, be considered substantial.^[2] To illustrate, a base model may require home visitors to offer families a minimum of 40 visits. Researchers and model implementers may change the home visit frequency requirements of the base model to offer families a minimum of 10 visits to better serve a population where 40 visits are not feasible. Given HomVEE's proposed decision rules, this would be considered a substantial change, because the visit frequency was changed by more than 50 percent.

Table 1—Proposed Decision Rules for Identifying Model Versions

Core feature	Non-substantial change to a core feature include:	Substantial changes to a core feature (model version or iteration) include:	Changes that requires consultation with experts include:
Curriculum	None	—Addition or subtraction of curriculum modules	Any other changes.
		—Flexible requirement becomes fixed
		—Fixed requirement becomes flexible
		—Change in required curriculum (from one to another)
Home visit frequency	—Intended frequency changes by less than 25 percent	—Intended frequency changes by more than 50 percent	—Intended frequency changes by 25-50 percent.
		—Flexible requirement becomes fixed (such as from no required frequency to required monthly visits)
		—Fixed requirement becomes flexible
Home visitor caseloads	—Intended caseload changes by less than 25 percent	—Intended caseload size changes by more than 50 percent	—Intended caseload size changes by 25-50 percent.
		—Flexible requirement becomes fixed (such as from no caseload guidance to requiring one visitor per 10 families)
		—Fixed requirement becomes flexible
Home visitor education and experience	None	—Required degree type changes (such as from R.N. to M.S.W.)	None.
		—Flexible requirement becomes fixed (such as from no education requirement to requiring a Bachelor's degree)
		—Fixed requirement becomes flexible
Home visitor supervision	—Less than 25 percent change in intended supervision frequency	—More than 50 percent change in intended supervision frequency	—25-50 percent change in intended supervision frequency.
		—Flexible requirement becomes fixed such as from no requirement for the frequency of supervision to requiring monthly supervision meetings)	—Mode of supervision (one-on-one, reflective supervision) added or dropped.
		—Fixed requirement becomes flexible
Home visitor pre-service training	—Less than 25 percent change in required training hours	—25-50 percent change in required training hours	—25-50 percent change in required training hours.
	—Change in training mode (in person to virtual) that does not change training content	—Addition or subtraction of training modules
		—Flexible requirement becomes fixed (such as from no training requirement to specified training requirement)
		—Fixed requirement becomes flexible (such as from specified training requirement to no training requirement)
Home visitor in-service training	—Less than 25 percent change in required training hours	—25-50 percent change in required training hours	—25-50 percent change in required training hours.
	—Change in training mode (in person to virtual) that does not change training content	—Addition or subtraction of training modules
		—Flexible requirement becomes fixed (such as from no training requirement to specified training requirement)
		—Fixed requirement becomes flexible (such as from specified training requirement to no training requirement)
Program certification	None	—Program certification requirement added or dropped	Changes to certification standards or procedures.
Number of home visits	—Intended dosage changes by less than 25 percent	—Intended dosage changes by more than 50 percent	—Intended dosage changes 25-50 percent.
		—Flexible requirement becomes fixed (such as from tailored number of visits to fixed number of visits)
		—Fixed requirement becomes flexible (such as from fixed number of visits to tailored number of visits)
Program duration	—Intended duration changes by less than 25 percent	—Intended duration changes by more than 50 percent	—Intended duration changes 25-50 percent.
		—Flexible requirement becomes fixed (such as from duration tailored to family need to a fixed number of months)
		—Fixed requirement becomes flexible (such as from a fixed number of months to duration tailored to family need)
Required staffing structure	—Less than 25 percent change in recommended supervisor caseload	—More than 50 percent change in recommended supervisor caseload	—25-50 percent change in recommended supervisor caseload.
		—Flexible requirement becomes fixed (such as from no required structure to specified structure)	—Addition or elimination of required non-supervisory positions
		—Fixed requirement becomes flexible (such as from specified structure to no required structure)
Trainer qualifications	None	—Addition or subtraction of certification requirement	—Any other changes.
		—Flexible requirement becomes fixed (such as from no required qualifications to specified qualifications)
		—Fixed requirement becomes flexible (such as from specified qualifications to no required qualifications)
Visit format (in person, virtual, mixed)	None	—Change from one fixed format to another	—Any changes involving phone visits.
	—Less than 25 percent change in required number of in-person visits	—More than 50 percent change in number of in-person visits	—25-50 percent change in number of in-person visits.
		—Flexible requirement becomes fixed (such as from no required visit format to required in-person format)
		—Fixed requirement becomes flexible (such as from required in-person format to no required visit format)
Visit length	None	—More than 50 percent change in required visit length	—25-50 percent change in required visit length.
		—Flexible requirement becomes fixed (such as from no required length to required length of one hour)
		—Fixed requirement becomes flexible (such as from required length of one hour to no required length)
Other features of the model's theory of change	None	None	—Any change.

2.1.2 Prioritizing Early Childhood Home Visiting Models

HomVEE proposes that early childhood home visiting models (including base models, model versions, and model iterations) continue to be eligible for review as long as they meet all of HomVEE's other criteria for inclusion as described in the HomVEE Draft Version 2 Handbook. HomVEE will continue to select families of models for review by creating a prioritization score for each family of models using a combination of manuscript and model characteristics (see the HomVEE Draft Version 2 Handbook, Chapter II, Section A, for additional details). Also, after a family of models is selected for review, HomVEE will continue to review all previously unreviewed research on the base model, model iterations, and model versions.

2.2 Applying the HHS Criteria

HomVEE proposes to review research on a model version and its base model separately. Therefore, a model version would need to independently meet HHS criteria for “an evidence-based early childhood home visiting service delivery model.” In contrast, HomVEE proposes that research on the base model and research on model iterations be reviewed together and contribute to HomVEE's determination of whether the base model meets HHS criteria.

2.3 Applying the New Definitions, Rules, and Procedures

The new definitions, rules, and procedures about model versions will apply to all models that HomVEE considers for review, regardless of whether the model already meets HHS criteria for an “evidence-based early childhood home visiting service delivery model.” HomVEE anticipates implementing the new definitions, rules, and procedures to the HomVEE review beginning with the models prioritized for review in 2021. The new definitions, rules, and procedures will be implemented with other models over time.

3.0 Request for Information

Through this Federal Register notice, ACF is soliciting information from a broad array of stakeholders on the proposed revisions to HomVEE's procedures. Federal, state, and local decision makers rely on HomVEE to know which home visiting models are effective. New definitions, rules, and procedures about model versions may affect which models HomVEE determines to meet HHS criteria for “an evidence-based early childhood home visiting service delivery model.”

Responses to this Federal Register notice will inform ACF's ongoing discussion about HomVEE's procedures and standards, with the aim of publishing a final HomVEE Version 2 Handbook by the end of 2020. This Federal Register notice is for information and planning purposes only and should not be construed as a solicitation or as an obligation on the part of ACF or HHS.

(Authority: Social Security Act Title V § 511 [42 U.S.C. 711], as extended by the Bipartisan Budget Act of 2018 (Pub. L. 115-123) through fiscal year 2022)

John M. Sweet, Jr.,

ACF/OPRE Certifying Officer.

Footnotes

1.  These domains are inclusive of the benchmark domains and individual outcomes listed in the statute that authorized the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program (Social Security Act, Section 511 [42 U.S.C. 711]).

2.  If a model under study lowers a required minimum threshold for a core feature, such as requiring at least 10 home visits instead of 20, this is a substantial change to the core feature and, therefore, is a model version or iteration. If a model under study exceeds a minimum requirement, this is not a substantial change to the core feature and therefore is a model replication.