2018-25835. Statewide Data Indicators for Child and Family Services Reviews: Request for Public Comment

AGENCY:

Children's Bureau (CB), Administration for Children and Families (ACF), Administration on Children, Youth and Families (ACYF), Department of Health and Human Services (HHS).

ACTION:

Request for public comment on revised syntax used to calculate Statewide Data Indicators for Child and Family Services Reviews (CFSRs).

SUMMARY:

In October 2014, the Administration for Children and Families (ACF) published a notice in the Federal Register (79 FR 61241) with the final plan to replace the statewide data indicators used to determine a state's substantial conformity with titles IV-B and IV-E of the Social Security Act through the Child and Family Services Reviews (CFSRs). In May 2015, ACF published a notice in the Federal Register (80 FR 27263) to correct some of the calculations and language used in the October 2014 notice. After the May 2015 notice, additional technical errors in the syntax and formulation of the statewide data indicators were identified. Some of those errors were discovered by states and other interested parties that operationalized the indicators for continuous quality improvement. Based on the amount of time required to complete a comprehensive review of the syntax, make revisions and validate the accuracy of the calculations; the Children's Bureau (CB) decided to suspend use of the indicators in determinations of substantial conformity during the third round of CFSRs. In October 2016, CB published CFSR Technical Bulletin #9 to inform states of the decision to limit use of the CFSR statewide data indicators and national performance to context information for round three of CFSRs. Since then, CB has thoroughly reviewed, revised, tested, and obtained independent review and validation of the syntax used to perform data quality checks and calculate state performance on the statewide data indicators. Before CB finalizes the revised syntax, and publishes revised performance results for all states and revised national standards, we invite state child welfare agencies, partner organizations, and the public to review, test, and provide comments on the revised syntax used to perform data quality checks and calculate observed performance on the statewide data indicators.

DATES:

Comments are due by February 25, 2019.

ADDRESSES:

You may send comments, identified by Docket Number ACF-2018-0007, by one of the following methods:

• Federal eRulemaking Portal: https://www.regulations.gov. Follow the instructions for submitting comments.
• Email: [e-mail]. Include Docket Number ACF-2018-0007 in subject line of the message.

FOR FURTHER INFORMATION CONTACT:

James Gregory, [phone], [e-mail].

SUPPLEMENTARY INFORMATION:

This Federal Register announcement has four sections: Background describes the purpose, history, and authority for the CFSRs and the use of statewide data indicators; Overview of Review and Validation of Syntax to Calculate CFSR 3 Statewide Data Indicator Performance describes the internal and independent review and validation process and findings; Supporting Documents provides a brief description of five resources available to aid interested parties in the syntax review process; and Invitation to Comment wherein we solicit public comment on the revised syntax that is pending finalization.

Background

CB implemented the CFSRs in 2001 in response to a mandate in the Social Security Amendments of 1994. The reviews are required for CB to determine if such programs are in substantial conformity with title IV-B and IV-E plan requirements. The review process, as regulated at 45 CFR 1355.31-37, grew out of extensive consultation with interested groups, individuals, and experts in the field of child welfare and related areas.

The CFSRs enable CB to: (1) Ensure conformity with federal child welfare requirements; (2) determine what is actually happening to children and families as they are engaged in child welfare services; and (3) assist states to enhance their capacity to help children and families achieve positive outcomes. CB conducts the reviews in partnership with state child welfare agency staff and other partners and stakeholders involved in the provision of child welfare services. We have structured the reviews to help states identify strengths as well as areas needing improvement within their agencies and programs.

We use the CFSR to assess state performance on seven outcomes and seven systemic factors. The seven outcomes focus on key items measuring safety, permanency, and well-being. The seven systemic factors focus on key state plan requirements of titles IV-B and IV-E that provide a foundation for child outcomes. If we determine a state has not achieved substantial conformity in one or more of the areas assessed in the review, the state is required to develop and implement a program improvement plan addressing the areas of nonconformity within 2 years. CB supports the states with technical assistance and monitors implementation of their program improvement plans. If the state is unable to complete its program improvement plan successfully, a portion of the state's federal title IV-B and IV-E funds is withheld.

In the April 23, 2014, Federal Register (79 FR 22604), we provided a detailed review of the consultation with the field and information considered in developing the third round of CFSRs, and proposed a set of statewide data indicators for public comment. We considered all public comments and issued a final plan in the October 10, 2014, Federal Register (79 FR 61241) that included national standards for state performance on statewide data indicators and criteria CB would use to determine whether a state is in substantial conformity with certain child welfare outcomes. Simultaneously, CB released CFSR Technical Bulletin #8 which provided more details on calculation methods, and a workbook that showed individual state performance on the indicators and preliminary findings of whether the state met national standards. In responding to state and other stakeholder questions after the release of those publications, we discovered errors in our descriptions and calculations. As a result, on May 13, 2015, we released a notice in the Federal Register (80 FR 27263) to correct some calculations and language used in the October 2014 final notice. We also amended Technical Bulletin #8 that provided technical detail on calculations, how CB determined whether a state met national standards, and how program improvement goals relative to the statewide data indicators would be established for states not meeting national standards.

After the May 2015 notice, we discovered additional technical errors in the syntax and formulation of the statewide data indicators. Some of the changes were identified by states and other interested parties that operationalized the indicators for continuous quality improvement. CB determined additional time was necessary to review, correct, and thoroughly test revised syntax used to calculate national and state performance on the data indicators. Recognizing challenges associated with the delay and implementation of revised indicators, especially for states already involved in the CFSR and program improvement planning process, CB decided to suspend use of the indicators in determinations of substantial conformity and as a basis for potentially imposing financial penalties during the third round of CFSRs. We are authorized by the regulations at 45 CFR 1355.34(b)(4) and (5) to add, amend, or suspend any of the statewide data indicators and to adjust the national standards when appropriate. CB issued Technical Bulletin #9 on October 11, 2016, to notify states of the change in the use of the statewide data indicators. Technical Bulletin #9, which rescinded Technical Bulletin #8A, provided guidance on how state performance on the CFSR 3 statewide data indicators would be used as context, revised related program improvement plan requirements, and re-issued the description of program improvement measurement methods.

We recently completed the comprehensive syntax review, revision, and validation process for calculating statewide data indicator performance information. The revisions do not change the statewide data indicator measures as defined in prior Federal Register notices. Revisions are limited to operationalizing the measures and calculation methods contained in the statistical syntax used to generate performance information.

Following the commencement of this Federal Register Announcement, we will review public comments to determine readiness to finalize the revised syntax or the potential need for additional revisions. We will issue a Federal Register to provide notice when the revised syntax is finalized. The notification will address applicable revisions and clarifications necessary to the Federal Register document from May 2015, and include a summary of, and our responses to, public comments we received. We will also publish revised performance results for all states, revised national standards, and the final revised syntax and supporting files to calculate data quality, observed performance, and risk-standardized performance for the CFSR 3 statewide data indicators.

Overview of Review and Validation of Syntax To Calculate CFSR 3 Statewide Data Indicator Performance

After the discovery of technical errors in the syntax and formulation of the statewide data indicators, federal staff completed a comprehensive internal review to identify and make syntax revisions. The internal in-depth line-level review of all syntax related to computing performance on the statewide indicators was completed in May 2017. Staff verified each step of indicator computation to determine whether the syntax was functioning accurately and as designed. Additional data quality tests were performed to confirm the integrity of the population of child records eligible for analysis, and syntax functioned correctly across a variety of scenarios, including outlier cases with high error risk. The comprehensive review resulted in the identification of additional syntax revisions, changes to how measurement methods are operationalized, and new methods to screen and address data quality problems in AFCARS and NCANDS submissions.

In addition to our internal review, we contracted for an independent review and validation of the syntax used to complete data quality checks and calculate observed and risk-standardized performance on the seven statewide data indicators. The independent reviewers used the following multi-step process to systematically review and verify all of the syntax:

1. Reviewed syntax for completeness, functionality on systems external to the Children's Bureau

2. Performed reliability checks against performance indicators and state data profiles

3. Performed line-level validity checks of SPSS and STATA syntax files for accurate task execution, data quality verification

a. Reviewed the syntax and assessed its function/purpose

b. Selected and applied the appropriate validity check(s)

c. Ran all syntax and inspected dataset(s) and output

d. Debriefed on similar code segments across syntax files

The independent reviewers determined the syntax can validly and reliably develop source data files, perform data quality checks, and calculate state observed performance and risk-standardized performance on the seven statewide data indicators according to the definitions set forth in the Federal Register Notices and associated CB guidance documents. The reviewers also verified there are no errors in the revised syntax for observed performance and data quality checks, or the original syntax for risk-standardized performance that would cause inaccuracies in the calculations. The independent review also helped demonstrate the feasibility for states and partner agencies to replicate the syntax with several additional steps to calculate data quality and observed performance on their own. Additional information about the independent review and verification process and associated findings are detailed in the 2018 report, Review and Validation of the Analytic Syntax Used to Produce CFSR Statewide Data Indicators (https://www.acf.hhs.gov/​cb/​resource/​cfsr-round3-sdi-syntax-validation).

Supporting Documents

The documents identified below are provided to assist states and other interested parties with the independent syntax review process:

A. List of CFSR Round 3 Statewide Data Indicator Syntax Revisions: Provides a summary of errors, corrections, and changes made to the syntax used to calculate performance on the statewide data indicator(s) and data quality checks. (https://www.acf.hhs.gov/​cb/​resource/​cfsr-round3-sdi-syntax).

B. CFSR Round 3 Statewide Data Indicator Data Dictionary: Provides a description of each statewide data indicator and data quality check, including the numerators, denominators, risk adjustments, exclusions, and corresponding data notes. (https://www.acf.hhs.gov/​cb/​resource/​cfsr-round3-sdi-data-dictionary).

C. Statewide Data Indicators—PDF Version of Syntax: Provides the syntax used to perform data quality checks, calculate observed performance, and calculate risk-standardized performance on the seven statewide data indicators. This document may help individuals develop a better understanding of the statistical calculations or provide assistance for individuals who do not have SPSS and/or STATA statistical software to run the syntax files. (https://www.acf.hhs.gov/​cb/​resource/​cfsr-round3-sdi-syntax-pdf).

D. CFSR Round 3 Statewide Data Indicator Syntax Zip File: Provides syntax and supporting files in SPSS format that are used to calculate data quality and observed performance. Note: Syntax to calculate risk-standardized performance (RSP) for each state uses a national, risk-adjusted model that requires child-level data from all states (i.e., national datasets) and thus cannot be replicated by states and interested parties. (https://www.acf.hhs.gov/​cb/​resource/​cfsr-round3-sdi-syntax-zip).

E. Instructions to Run Data Quality (DQ) Checks and Observed Performance Syntax for Statewide Data Indicators: Provides an overview and brief description of the folders, files, and syntax included in the CFSR round 3 zip file; considerations and steps for running the syntax; and software requirements. (https://www.acf.hhs.gov/​cb/​resource/​cfsr-round3-sdi-dq-instructions).

Invitation To Comment

We are inviting state child welfare agencies, partner organizations, and the public to review, test, and provide comments on the revised syntax to perform data quality checks and calculate observed performance on the statewide data indicators. The purpose of this invitation is to provide an opportunity for additional independent review prior to finalizing the revised syntax. Interested parties may choose to review a written version of the statistical syntax and supporting documents, and/or follow the instructions to replicate the syntax files to generate CFSR data indicator performance information using state-generated AFCARS and NCANDS files.

We are interested in receiving comments on the revised syntax and related considerations and recommendations to strengthen our statistical calculations for generating CFSR statewide data indicator performance information. We encourage states and interested parties to share the outcome of their review and/or results from running the revised syntax files. Please be specific in identifying the corresponding statewide data indicator or data quality check being referenced in the comment; the syntax revision, question, or recommendation; and the rationale for considerations or recommendation(s), as applicable.

Comments containing suggested revisions should be limited to operationalizing the measures and calculation methods contained in the statistical syntax used to generate performance data. We are not making changes or requesting comments at this time pertaining to the statewide data indicator measures as defined in prior Federal Register notices and the Data Dictionary available with this invitation.

Lynn A. Johnson,

Assistant Secretary for Children and Families.