2015-11515. Statewide Data Indicators and National Standards for Child and Family Services Reviews

AGENCY:

Children's Bureau (CB), Administration for Children and Families (ACF), Administration on Children, Youth and Families (ACYF), Department of Health and Human Services (HHS).

ACTION:

Final Notice of Statewide Data Indicators and National Standards for Child and Family Services Reviews; correction.

SUMMARY:

On October 10, 2014, the Administration of Children and Families (ACF) published a document in the Federal Register (79 FR 61241). The document provided CB's final plan to replace the statewide data indicators used to determine a state's substantial conformity with titles IV-B and IV-E of the Social Security Act through the Child and Family Services Reviews (CFSRs). This document provides corrections to errors and misstatements in that document and some of the calculations of the statewide data indicators.

DATES:

Effective: May 13, 2015.

FOR FURTHER INFORMATION CONTACT:

Miranda Lynch Thomas, Children's Bureau, 1250 Maryland Ave. SW., 8th Floor, Washington, DC 20024, [phone].

SUPPLEMENTARY INFORMATION:

Background

CB implemented the CFSRs in 2001 in response to a mandate in the Social Security Amendments of 1994. The reviews are required for CB to determine whether such programs are in substantial conformity with title IV-B and IV-E plan requirements. The review process, as regulated at 45 CFR 1355.31-37, grew out of extensive consultation with interested groups, individuals, and experts in the field of child welfare and related areas.

The CFSRs enable CB to: (1) Ensure conformity with federal child welfare requirements; (2) determine what is actually happening to children and families as they are engaged in child welfare services; and (3) assist states to enhance their capacity to help children and families achieve positive outcomes. CB conducts the reviews in partnership with state child welfare agency staff and other partners and stakeholders involved in the provision of child welfare services. We have structured the reviews to help states identify strengths as well as areas needing improvement within their agencies and programs.

We use the CFSR to assess state performance on seven outcomes and seven systemic factors. The seven outcomes focus on key items measuring safety, permanency, and well-being. The seven systemic factors focus on key state plan requirements of titles IV-B and IV-E that provide a foundation for child outcomes. If we determine that a state has not achieved substantial conformity in one or more of the areas assessed in the review, the state is required to develop and implement a program improvement plan addressing the areas of nonconformity within 2 years. CB supports the states with technical assistance and monitors implementation of their program improvement plans. If the state is unable to complete its program improvement plan successfully, a portion of the state's federal title IV-B and IV-E funds is withheld.

Most relevant to this document are the national standards for state performance on statewide data indicators CB uses to determine whether a state is in substantial conformity with certain child outcomes. We are authorized by the regulations at 45 CFR 1355.34(b)(4) and (5) to add, amend, or suspend any of the statewide data indicators and to adjust the national standards when appropriate. If we determine that a state is not in substantial conformity with a related outcome due to its performance on an indicator, the state will include that indicator in its program improvement plan. The improvement a state must achieve is relative to the state's baseline performance at the beginning of the program improvement plan period.

In an April 23, 2014, Federal Register document (79 FR 22604), we provided a detailed review of the consultation with the field and information considered in developing the third round of the CFSRs and proposed a set of statewide data indicators for public comment. We considered all public comments and issued a final plan in the October 10, 2014, Federal Register (79 FR 61241). Simultaneously, CB released CFSR Technical Bulletin #8, which provided more details on calculation methods and a workbook that showed individual state performance on the indicators and preliminary findings of whether the state met the national standards at that time based on data submitted as of July 2014. In responding to state and other stakeholder questions since the release of those publications, we have found errors in our descriptions and calculations that we are correcting here. We will release an amended technical bulletin and workbook concurrently with this Federal Register document to make applicable corrections to those documents. We will also release the associated syntax in SPSS and STATA format so that states and other interested parties can review the detail related to the indicators. Although we intend to provide tools that allow the state to monitor its performance results on the indicators on a periodic basis, we know the additional detail is helpful to states that want to monitor themselves more frequently or in more depth. Finally, since this document focuses on just the revisions and clarifications necessary to the Federal Register document from October 2014, we will also publish a document that incorporates these revisions and clarifications into the original document.

Language Errors and Clarifications

This section discusses the language errors and clarifications we are making to the original Federal Register document.

Clarification of the Trial Home Visit Adjustment to Permanency Performance Areas 1, 2, and 3

On page 61244, we provided a description of how we calculated the permanency in 12 months for children entering foster care indicator. In part, we explained that we had applied a trial home visit adjustment to this indicator. We stated that this meant that if a child discharges from foster care during the 12-month period to reunification with parents or other caretakers after a placement setting of a trial home visit, any time in that trial home visit that exceeds 30 days is discounted from the child's length of stay in foster care. We use six 6-month Adopting and Foster Care Analysis and Reporting System (AFCARS) reporting periods of data (3 years) to calculate the numerator in this indicator. We are clarifying that the trial home visit adjustment is applied to all AFCARS reporting periods used for the indicator.

If a child discharges from foster care to reunification with parents or other caretakers after a placement setting of a trial home visit during any of the six report periods used for the indicator, any time in that trial home visit that exceeds 30 days is discounted from the length of stay in foster care. In other words, the actual date of discharge to permanency could occur at any time during the 3 years used to calculate this indicator, and the trial home visit would then be applied to see if it may result in a reduction in the length of time in foster care for the purposes of this data indicator.

On pages 61244-61245, we explained that the trial home visit adjustment was also applied to the following indicators: Permanency in 12 months for children in foster care 12 to 23 months and permanency in 12 months for children in foster care 24 months or more. However, this is not accurate. We do not apply the trial home visit adjustment to these indicators because it has no impact on the outcomes for children in care on the first day and followed for only 12 months. This is because these indicators rely on only two AFCARS report periods (one year of data) and do not look beyond the 12-month period to see whether a child has discharged to permanency.

Revisions to Attachment A—Statewide Data Indicators

Attachment A provided a summary of each final statewide data indicator including the numerators, denominators, risk adjustments, and data periods used to calculate the national standards.

In describing the permanency in 12-months indicator for children entering foster care, we said that we used the AFCARS periods 2011B through 2013A for calculating the national standard for this indicator. This was a typographical error. We used six AFCARS report periods for a total of 3 years of data: 2011B through 2014A.

Also, the applicable exclusions and notes were partly in error for the following indicators: Permanency in 12 months for children in foster care 12 to 23 months and permanency in 12 months for children in foster care 24 months. We carried forward the same error described in the previous section with regard to the trial home visits adjustment. We do not apply the trial home visit adjustment to these indicators.

Revisions to Attachment D—Data Quality Items, Limits, and Applicable Measures

Attachment D provided information on the data quality limits applied in determining whether to include state data for calculating the indicators.

Data quality limits are applied to avoid skewing results. There may be a number of reasons why a state's data exceeds a data quality limit, including outliers that exist in the data. Therefore, not all exclusions are necessarily the result of poor data quality. The data quality limits outlined in Attachment D are intended to be a guide to avoid misrepresenting state performance or national standards.

Two listed data quality items had typographical errors that changed their meaning. The AFCARS Within-file data quality check, “Percent of children on 1st removal,” is applied to all indicators with the exception of recurrence of maltreatment. The limit noted was less than 95 percent but it should read more than 95 percent. The NCANDS Cross File Check named “Child IDs don't match across years” should read “Child IDs match across years.” This means that the state has not met the item limit if less than 1 percent of the Child IDs match across years, as we expect them to be based on patterns of recurrence.

In addition, the term “Dropped cases” was used in the section on AFCARS Cross File Checks. This term refers to instances in which a child who is reported during one 6-month period is not reported in the next period, and there is no record that the child exited. However, this term is technically incorrect, as it is the record of the case that drops from the file. For clarity, this cross file check should instead be referred to as “Dropped records.”

For all NCANDS data quality items outlined in Attachment D, it should be noted that these data quality items were applied to victims only. This includes all NCANDS cross file and within-file checks.

With regard to the data quality items applied to the indicator of maltreatment in foster care, the table indicates that we apply the NCANDS cross file check, “Child IDs match across years, but dates of birth and sex do not match,” to this indicator. We do not apply this check to the maltreatment in foster-care indicator because it requires only 1 year of data.

Finally, for clarification, the NCANDS data quality item “Some victims have AFCARS IDs” was previously located in the “NCANDS Data—Cross File Checks” section. It should be in the “NCANDS Within file checks” section, because it requires only the NCANDS child file and does not match with AFCARS.

Changes to National Standards and State Performance

This section discusses the changes or clarifications in the methods of calculating the national standards and state performance that affect the national standards we provided in the original document.

We made one change to the application of data quality items that has an implication for the calculation of national standards and state performance. There was an oversight in the application of the data quality items for the measure of permanency in 12 months for children entering foster care. For this indicator, we originally applied the data quality items only to the first four data periods, but upon further investigation we determined that the data quality items should be applied to all six periods used in the calculation of this measure.

There was a slight modification made to several calculations that require the date of discharge. When calculating the length of stay in foster care, age at exit, or other variables that require the data of discharge, we previously used an imputed version of the date of discharge. An imputed date of discharge was used when the date of discharge was missing for a child in one report period, but in the subsequent 6-month period the child was reported as being in a new removal episode with a value for his or her date of discharge from the prior foster care episode. We are no longer using the date of discharge from the prior foster care episode to impute missing dates of discharge and are using only the date of discharge from foster care submitted to us by the state. If that date is missing, it is treated as a missing value and no attempt is made to impute the value using subsequent files.

In determining the national standards and the state-by-state performance outlined in the workbook, we inadvertently did not use the most recent submission for all periods of data for three states. We have re-run the analysis and national standards to incorporate these resubmissions and all data are now current as of July 10, 2014.

Due to the above noted changes, Table 1 on page 61249 should be replaced as follows:

Table 1—National Standards for CFSR Round 3 Statewide Data Indicators

	National standard
Statewide Data Indicators for Safety Outcome 1:
Maltreatment in Foster Care	8.50 victimizations per 100,000 days in foster care.
Recurrence of Maltreatment	9.1 percent.
Statewide Data Indicators for Permanency Outcome 1:
Permanency in 12 Months for Children Entering Foster Care	40.5 percent.
Permanency in 12 Months for Children in Foster Care 12 to 23 Months	43.6 percent.
Permanency in 12 Months for Children in Foster Care 24 Months or More	30.3 percent.
Re-Entry to Foster Care in 12 Months	8.3 percent.
Placement Stability	4.12 moves per 1,000 days in foster care.

On page 61254, we provided an overview of the number of states excluded from the national standards for each data indicator. Based on the changes noted here, these should be updated as follows:

• Permanency in 12 months for first day cohorts with 12-23 months and 2 or more years prior time in care: Three states are now excluded instead of one.
• Permanency in 12 months for children entering foster care indicator: Four states are now excluded instead of three.
• Recurrence of maltreatment: Five states are now excluded instead of four.

There was no change to the number of states excluded for the indicators of re-entry to foster care in 12 months, maltreatment in foster care, or placement stability.

Changes to Monitoring Statewide Data Indicators in Program Improvement Plans

The changes noted in relation to the calculation of national standards also have relevance to the calculation of each state's observed and risk-standardized performance, as well as the improvement factors used to set program improvement plan goals. Revised state-level data, including changes to results indicating the need for a program improvement plan, are reflected in the revised workbook.

The changes in the workbook regarding setting program improvement plan targets also reflect a revision to the bootstrapping process outlined in Technical Bulletin #8. We began with three observed values for 3 years of data, aggregated at the state level. From those three values, we averaged them in different combinations to get seven values. From those 7 values, a bootstrapping process was used to get 30 values. These 30 values were then resampled 1,000 times. After careful review, we have determined that we do not have justification for bootstrapping to 30 values. We have eliminated that step and are now bootstrapping the 7 values to get 1,000 resamples. This yields a grand mean and improvement factor that is very similar to the original set, but is more reflective of the true parameters.

Because of these changes, Table 2 has also been revised. In addition, upon further reflection we believe that in order to avoid confusion, it is best to use the terminology of floors and caps versus minimum and maximum amounts of improvement. Using the terms of floors and caps is also consistent with the previously issued Technical Bulletin #8. All other references to minimum and maximum levels of improvement in the original Federal Register document should be read as floors and caps. Table 2 should be replaced as follows:

Table 2—Caps and Floors on Program Improvement Plan Improvement Factors Relevant to the Statewide Data Indicators

	Floor	Cap
Statewide Data Indicators for Safety Outcome 1:
Maltreatment in Foster Care	0.904	0.812
Recurrence of Maltreatment	0.951	0.902
Statewide Data Indicators for Permanency Outcome 1:
Permanency in 12 Months for Children Entering Foster care	1.031	1.063
Permanency in 12 Months for Children in Foster Care 12 to 23 months	1.046	1.082
Permanency in 12 Months for Children in Foster Care 24 Months or More	1.042	1.091
Re-Entry to Foster Care in 12 Months	0.891	0.834
Placement Stability	0.959	0.904

Dated: May 5, 2015.

Mark H. Greenberg,

Acting Assistant Secretary for Children and Families.