2018-24949. Family Self-Sufficiency Performance Measurement System (“Composite Score”)

AGENCY:

Office of Public and Indian Housing, HUD.

ACTION:

Notice of new performance measurement system (“Composite Score”) for the Family Self-Sufficiency Program.

SUMMARY:

This notice describes and responds to comments on a performance measurement system that HUD plans to implement for Public Housing Agencies (PHAs) that receive HUD Family Self-Sufficiency (FSS) program coordinator grants. The desired effect of this notice is to notify the public regarding the criteria for evaluating FSS programs.

DATES:

Applicability Date: December 17, 2018.

FOR FURTHER INFORMATION CONTACT:

Questions on this notice may be addressed to [e-mail] or by contacting Anice Chenault at 502-618-8163 (email strongly preferred).

Electronic Data Availability. This Federal Register notice and a spreadsheet containing scores using the methodology for FSS programs funded in any of the last three years will be available electronically from the HUD FSS web page: https://www.hud.gov/​program_​offices/​public_​indian_​housing/​programs/​hcv/​fss. Federal Register notices also are available electronically at https://www.federalregister.gov/​,, the U.S. Government Printing Office website.

SUPPLEMENTARY INFORMATION:

I. Background

On December 12, 2017, HUD published a notice in the Federal Register (FR-6046-N-01, 82 FR 58434) (2017 Notice) describing and requesting comment on a performance measurement system that HUD plans to implement for public housing agencies (PHAs) that receive HUD Family Self Sufficiency (FSS) program coordinator grants. Through this notice, HUD is implementing the FSS performance measurement system, as proposed in the 2017 Notice. Additionally, in response to public comments, HUD is revising the methodology it uses to compute FSS Performance Scores under the new system; these revisions are described below, in section III of this notice. Henceforth, HUD will use the new system to evaluate the performance of PHAs receiving HUD program coordinator funding in a strictly advisory manner. Beginning with Fiscal Year (FY) 2019 appropriations, HUD intends to use the performance measurement system in the determination of FSS funding awards. The complete, updated methodology can be found on HUD's website at: https://www.hud.gov/​program_​offices/​public_​indian_​housing/​programs/​hcv/​fss.

Under section 23(i) of the Housing Act of 1937 (42 U.S.C. 1437u(i)), HUD is required to establish criteria to evaluate eligible entities' implementation of local FSS programs. HUD has developed this new FSS performance measurement system to provide HUD, Congress, public housing agencies (PHAs), and other eligible entities with information on the performance of individual FSS programs. The information will help grantees determine how their programs compare to others across the country in efforts to help participants to successfully graduate from the program and make progress toward economic security. The information will also help HUD understand the extent to which FSS program performance—individually and collectively—improves or declines over time.

Initially, HUD plans to use the performance measures to identify high performing and low performing FSS programs, which could inform its understanding of best practices and its delivery of technical assistance. Toward these goals, at least once per year, HUD will analyze data collected through the Public Housing Information Center (PIC) to calculate FSS performance scores for each FSS program that received an FSS coordinator grant in one or more of the past three fiscal year NOFA competitions. Beginning in Fiscal Year 2019, HUD plans to consider the FSS performance score of an FSS program in determining FSS funding awards.

HUD developed the approach described in this Notice based in part on feedback received on an earlier performance measurement approach proposed in the FY 2014 FSS Notice of Funding Availability (NOFA). In the FY 2014 NOFA, HUD proposed, and asked for feedback on, evaluating FSS programs based on the share of FSS participants that experience an increase in earned income (also known as “earnings growth”) over a specified time period. Some commenters raised concerns that this approach did not adequately account for differences in local economic conditions and differences in the approaches of local FSS programs. While some FSS programs encourage participants to increase their earnings immediately, others encourage FSS participants to build skills and credentials first and then seek higher paying jobs. The FSS performance measurement system proposed in the December 2017 Notice was developed to address these issues, as well as many others, and to allow for a more nuanced evaluation of the performance of local FSS programs.

A PHA's FSS performance score will be calculated based on three measures, weighted as follows:

A. Earnings Performance Measure (50 percent);

B. FSS Graduation Rate (30 percent);

C. Participation Rate (20 percent).

HUD has selected these measures because they are important indicators of program performance and are verifiable using the data HUD collects through the PIC data system. No outside or additional reporting will be required, which ensures that the system will not increase the reporting burden of PHAs. No new Paperwork Reduction Act (PRA) Information Collection will be required for the scoring, as proposed.

The Earnings Performance Measure represents the difference between the earnings growth of FSS participants and the earnings growth of similar non-FSS households assisted by the PHA within a specified time frame. This approach, along with a statistical adjustment described below, helps to control for variations in local economic conditions. The program was envisioned and designed for the purpose of increasing employment and earnings for its participants. Therefore, the performance score assigns the Earnings Performance Measure a high weight.

HUD has assigned the next highest weight to the Graduation Rate indicator—which represents the rate of FSS participants who successfully “graduate” from the program—to encourage PHAs to work closely with individual FSS participants to increase graduation rates. To graduate from FSS, a participant must be employed, be independent of cash welfare assistance for at least one year, and achieve the other goals set forth in the participant's contract of participation.

Finally, the FSS performance score looks at the local program's Participation Rate, which reflects the extent to which a PHA exceeds the minimum number of households that HUD requires the PHA to serve as a condition of receiving an FSS grant. PHAs with higher Participation Rates are serving more households than required, which is a desired output, provided the PHAs are serving those households effectively. Because the Earnings Performance Measure is weighted more heavily than the Participation Rate, however, PHAs should be careful not to execute more Contracts of Participation than they can serve effectively, because doing so would likely reduce their scores on the Earnings Performance Measure. Together, the Earnings Performance Measure, Graduation Rate, and Participation Rate are expected to provide a balanced measurement of the performance of an individual FSS program.

As indicated in the 2017 Notice soliciting public comment, HUD does not intend to use this performance measurement system for Tribes/Tribally Designated Housing Entities (TDHEs), who do not report into Public and Indian Housing Information Center (PIC), or for PHAs with a Moving to Work (MTW) designation, as they report differently into PIC, using Form HUD-50058-MTW. However, HUD is presently exploring a change to the reporting processes for MTW agencies, in order to include them in the FSS performance scoring process. Nor does HUD intend, after considering public comment, to use this performance measurement system for unfunded PHAs, and PHAs and private owners that serve Project-based Rental Assistance (PBRA) residents at this time.^[1] The Agency will continue to explore options for modifying the scoring system for those sub-groups.

II. HUD's Responses to Public Comments

HUD received 68 unique public comments on the planned measures, which are summarized below along with HUD's responses. HUD's responses to comments are organized into five categories: (A) Overall Comments; (B) Comments on Earnings Performance Measure; (C) Comments on FSS Graduation Rate Measure; (D) Comments on Participation Rate Measure; and (E) Comments on Weighting of the Measures. At the conclusion of this Notice, in Section III., Final Thresholds, HUD provides the final FSS performance measurement system thresholds that it intends to adopt to calculate FSS performance scores.

A. Overall Comments

1. Comment: Data Quality. Many commenters raised concerns about the quality of data from the PIC system used to calculate the FSS performance scores, particularly with regard to data entered prior to HUD's 2016 guidance. Some requested that PHAs be allowed to examine and correct all data used for calculating their measures prior to HUD calculating the FSS performance measures. Others suggested that this might not be possible or that there would not be resources to correct the data.

HUD Response: Data Quality. On May 6, 2016, HUD issued PIH Notice 2016-08 to help PHAs understand how to submit timely and accurate PIC data regarding FSS, along with a series of webinars to help PHAs apply the guidance to improve their PIC data quality for both current and past participants. Further, HUD has emphasized the importance of PHAs submitting accurate PIC data for many years. HUD believes it is reasonable to rely on existing PIC data in calculating FSS performance scores.

It is important to note that each time the FSS performance scores are calculated, HUD will retrieve a new data report from the PIC system. This ensures that if a PHA has made changes to improve the accuracy of its reporting on any metric, for current or past participants, all of these changes will be reflected in its performance score.

2. Comment: Limitations on Included Measures. Many commenters expressed the view that the measures in the planned performance measurement system do not address the variations in participants' goals. Some participants or programs may have interim goals related to addressing barriers to work (e.g., treating psychiatric illness or barriers, accessing medical care, securing childcare, or completing training, or education), which would not immediately result in higher earnings, even if participants make important progress. Several commenters suggested that participation in/provision of services or progress toward Individual Training and Services Plan (ITSP) goals should be included as a measure. Some suggested that changes in educational attainment also be included as a measure.

Several commenters also stated that inputs and outputs should be included in the measures, such as the work associated with serving participants, meeting with participants, connecting participants to services, making referrals, etc. Some indicated that, without these measures, they are not given adequate “credit” for serving high-needs participants or that they may be penalized for participant performance issues that are beyond their control (through the earnings and FSS graduation measures).

HUD Response: Limitations on Included Measures. HUD agrees that there is tremendous variety in the ITSP goals of individual FSS participants, which go beyond the statutorily mandated goals of employment and being welfare-free. It is precisely this variety, however, that makes these goals extremely difficult to factor into a performance measurement system. Since each ITSP is set up individually, it would be both impracticable and unwise to standardize ITSP goals across all programs. While HUD could potentially measure the share of ITSP goals achieved for each participant, this would not represent a direct comparison across local FSS programs if some programs set goals that were easy to attain while others set more difficult targets. This approach could also create an incentive for PHAs to change how they are defining individuals' goals to increase their FSS performance scores, without necessarily improving outcomes for participants. Finally, HUD does not currently collect data on the goals set nor the share of ITSP goals that participants attain, so the inclusion of ITSP goal data in a performance measurement system for FSS would require additional reporting by PHAs, which would add to their administrative burden.

HUD recognizes the importance and value of setting a range of goals for participants, including goals other than employment. Over time, however, HUD believes the achievement of these goals will support the ultimate goal of the program, which is increased earnings, which will then be captured in the performance measurement system. This is one of the benefits of having five (or more) years to work with participants. The long duration of the FSS program provides PHAs an opportunity to work with participants on a range of issues—including education, training, work readiness, etc.—that will, over time, contribute to earnings gains that can be measured and reflected in the FSS performance measurement system. The earnings and FSS graduation rate measures accommodate this long time-frame, examining data for FSS participants that entered the program as far back as 7.5 to 8 years ago, respectively.

3. Comment: Homeownership. A few commenters expressed concern that the measures do not support homeownership goals for FSS participants and stated that progress toward homeownership should be included as a measure in the performance measurement system.

HUD Response: Homeownership. HUD commends PHAs that work with participants on homeownership and recognizes that the achievement of homeownership is an important outcome for many FSS participants. At the same time, it is clear that homeownership is a more realistic goal in some parts of the U.S. than others, due to variations in the local economy. This makes it difficult and inequitable to use homeownership as a performance measure in comparing FSS programs on a national basis.

4. Comment: Reliance on Past Performance Data. Some commenters opined that it is unfair to base an assessment of FSS performance on data from prior periods during which FSS coordinators were unaware of the performance measures and could not change their programs accordingly.

HUD Response: Reliance on Past Performance Data. The performance measurement system recognizes that it takes considerable time for an individual FSS participant to make material progress in increasing his or her earnings and to graduate from the program. This requires measurements that span years, rather than months. To implement such a system prospectively, without relying on data from prior periods, would require HUD to wait many years before having valid measures of FSS program performance. Such a delay would undermine HUD's ability to achieve the key purposes of the FSS performance measurement system. In order to ensure that FSS funds are spent responsibly and that FSS participants have access to high-quality programs, HUD needs the ability to recognize the achievements of high-performing FSS programs and identify struggling FSS programs in need of improvement.

The goals of improving earnings and helping FSS participants graduate successfully from the program should not come as a surprise to PHAs administering FSS programs. These goals have been clear since the program's inception and NOFAs have been announcing HUD's intent to use increased earnings as an evaluation metric since FY 2014. The participation rate also should not come as a surprise to PHAs, as HUD has historically based funding decisions on the number of FSS families served by PHAs. HUD's interest in PHAs serving more families (so long as they can do so without undermining earnings growth and FSS graduation rates), as reflected in the participation rate, is a factor that PHAs can influence going forward by adjusting their caseloads.

5. Comment: Real-Time Data. Some commenters requested a way to monitor their programs' progress with respect to the measures periodically or in real time.

HUD Response: Real-Time Data. HUD plans to provide updated scores at least once each year so PHAs can track their progress. In addition, PHAs can calculate their own participation rates and FSS graduation rates at any time.

6. Comment: Small PHAs/Small FSS Programs. Several commenters raised the concern that the measures could disadvantage small PHAs or small FSS programs because volatility in the data would be more likely and factors beyond the FSS program's control could drive results.

HUD Response: Small PHAs/Small FSS Programs. HUD recognizes that there may be greater volatility in the data for small FSS programs, which could be affected by the outcomes for one or more participants with unusual characteristics or experiences. Accordingly, in assigning earnings scores, HUD has built in protection for small FSS programs by using a test of statistical significance that makes it more difficult for smaller FSS programs than larger programs to receive a zero (0) score on the earnings measure. See the Dec. 12, 2017 Federal Register Notice (at page 82 FR 58437) for more details on the statistical test.

HUD has also examined the FSS performance composite scores of PHAs to determine if small programs systematically receive lower composite scores and determined that, there is not a strong relationship between program size and composite FSS performance score. In fact, the decile of PHAs with the second smallest FSS programs (10th through 19th percentile) had the second highest median composite scores of any decile (the highest was the group of PHAs in the 70th through the 79th percentile in size). PHAs with the very smallest FSS programs (0 to 9th percentile) did have the lowest median composite score, but the next lowest score was recorded by PHAs in the 80th to 89th percentile in size. This is an indication that there is not a strong relationship between program size and composite FSS performance score. However, HUD may continue to monitor scores to determine if there are any patterns that might help with the targeting of technical assistance efforts or the interpretation of performance data.

7. Comment: Joint Applicants. One commenter suggested that it would be more appropriate to pool joint applicant data for all measures, not just for participation.

HUD Response: Joint Applicants. HUD agrees, and is changing the methodology accordingly.

8. Comment: Initial Funding Period. Some commenters thought that FSS programs should not be assessed during their initial 12-month funding period or directly after receiving additional funding for the first time.

HUD Response: Initial Funding Period. HUD agrees with the need to be careful in interpreting the FSS performance scores of newly funded FSS programs and will take this into account in determining how to use the scores. However, HUD believes it is important to measure the performance of all FSS programs that receive HUD coordinator funding so that programs have a way of tracking their performance over time. Also, since HUD has not funded new applicants in several years, all PHAs currently being scored have had programs funded since at least FY2012.

9. Comment: Minimum Standards. A few commenters said that HUD should consider setting minimum standards for performance rather than rating FSS programs on a curve.

HUD Response: Minimum Standards. FSS programs will not be graded on a curve, but rather based on whether or not they exceed the specific fixed standards (or thresholds) adopted in the final FSS performance measures. While HUD used percentiles of the distribution to determine the initial thresholds for each score, those thresholds have now been fixed. This means that over time, a PHA's scores may move up or down, based on where the PHA's earnings, FSS graduation, and participation measures fall relative to the thresholds. In other words, a PHA's performance will determine in which performance category the PHA falls, since there is not a set number of “high” or “low” performers.

10. Comment: Zero Housing Assistance Payments (HAP). Some commenters suggested that attainment of a zero HAP amount (either at FSS graduation or in general) should be added as a performance measure.

HUD Response: Zero Housing Assistance Payments (HAP). The ability of an FSS participant to reach a level of earnings at which his or her HAP amount drops to zero will depend to a significant degree on the local labor market and the level of the voucher payment standard, which is a function of the rental housing market as well as a PHA's policies. Since FSS participants in some markets have a much greater likelihood of achieving zero HAP than others, this measure does not provide a useful basis for comparing the performance of PHAs in different labor and housing markets.

11. Comment: Unfunded PHAs, MTW PHAs, and PHAs that serve PBRA residents. HUD requested comments on the treatment of these types of PHAs and received many thoughtful comments on the development of performance measures for such PHAs.

HUD Response: Unfunded PHAs, MTW PHAs, and PHAs that serve PBRA residents. HUD appreciates all the thoughtful comments received on these subjects and will be considering these comments as HUD works to determine how best to evaluate the performance of these programs.

12. Comment: Portability. Some commenters were concerned about which PHA gets “credit” for FSS participants who port out of their PHA or into their PHA, although there was no consensus on how this should be addressed.

HUD Response: Portability. If a family ports, for the Participation Measure, each PHA (the receiving and the initial PHA) will benefit from the family's FSS enrollment. For the earnings and FSS graduation measures, the composite score will count the family as a participant in the FSS program at the PHA who currently administers the FSS contract and thus has final influence on the family's outcomes.

B. Comments on Earnings Performance Measure

1. Comment: Complexity of Earnings Performance Measure. Several commenters expressed a concern that the measures (especially the earnings measure) are too complicated or confusing. They indicated that PHAs will not understand them and will not be able to track their own progress. A few asked for information on which comparison households are included for their PHA so that they can track progress and correct data for those comparison households if needed. A few commenters expressed confusion about how comparison households are chosen and who chooses them.

HUD Response: Complexity of Earnings Performance Measure. HUD acknowledges that the methodology for computing the earnings performance score is somewhat complex, but believes the complexity is justified as a means of adjusting for variations in local economic conditions and approaches (e.g., human capital development or “work first” or some combination) at different PHAs. Fortunately, however, the measure produces a single clear data point—the earnings performance measure—that PHAs can use to track their progress over time. To the extent that FSS programs are successful in helping participants to increase their earnings—whether in the short-term or in the long-term—they should be able to achieve a strong earnings performance score. For information on how the measure works and how comparison households are selected, see the December 12, 2017 Federal Register Notice (at pages 82 FR 58435-37) and comments below.

2. Comment: Elderly Individuals and Persons with Disabilities. A few commenters suggested that excluding households headed by elderly persons or persons with disabilities from the earnings performance measure would discourage FSS programs from serving these households.

HUD Response: Elderly Individuals and Persons with Disabilities. This comment provides a good opportunity to clarify that the methodology is designed to achieve the opposite effect. Although program regulations require FSS programs to serve any resident who desires to participate and is able to “seek and maintain employment,” see 24 CFR 984.303(b)(4), some FSS programs may be concerned that serving elderly persons and persons with disabilities would lower their earnings performance score because this population may be less likely to experience large earnings gains than other individuals. The methodology excludes households headed by elderly persons or persons with disabilities from the earnings performance measure, which ensures that PHAs can serve these households without worrying about the possibility that this might reduce their earnings performance score. All households served through FSS (regardless of age category or disability status) will be counted in the participation and FSS graduation measures.

3. Comment: Changes in Elderly or Disability Status. One commenter asked how HUD will account for FSS participants who age out of the non-elderly category while enrolled in FSS and those that acquire a disability while participating in the program. Will they be included or excluded from the analysis used to calculate the earnings performance measure?

HUD Response: Changes in Elderly or Disability Status. Given the strong interest in and capacity for work of many adults in the 60 to 65 age range, HUD believes it is appropriate to retain in the earnings analysis FSS participants who begin their FSS tenure below the age of 62 but achieve that age during their participation. On the other hand, HUD agrees that a person whose status changes to “disabled” during the course of participation in FSS should be excluded from the earnings analysis in order to be consistent with the inclusion of data for other persons with disabilities in the earnings analysis. The methodology for calculating the earnings performance measure has thus been changed to exclude people who are or become disabled while participating in FSS from the analysis.

4. Comment: Selecting Comparison Households. Many commenters expressed concern that the variables used to select comparison households were not sufficient to account for important life circumstances that may affect the potential for employment and increased earnings. The most common variables they recommended be included were: Language, education level, childcare availability, family composition (including children of all ages and workable adults or presence of a household member with a disability), mental health, and additional information about household composition. Some commenters also noted that FSS participants are different than non-FSS participants in terms of motivation, resources, or barriers to employment, though there was disagreement among commenters on whether FSS participants are more likely to have high barriers or low barriers.

HUD Response: Selecting Comparison Households. As described in the December 12, 2017 Federal Register Notice, in selecting comparison households for purposes of calculating the earnings performance measure, HUD considered the following household characteristics: Earnings as of the time of the FSS household's entry into FSS, age of head of household, length of time in the voucher or public housing program, number of adults in the household and number of children under age 5. While some of the additional factors recommended by commenters are not available in the PIC dataset used to compute the FSS performance measures, several are, including: presence of children of any age and presence of a household member with a disability.

In response to this comment, HUD has considered whether the increased precision of adding additional comparison factors would outweigh the dilution of the weight of the existing factors and lead to an insufficient number of comparison households. Further analysis has determined that number of children under 18 is better than presence of children under age 5 in predicting whether a household would join FSS and therefore is a better factor in choosing comparison households. HUD will therefore remove presence of children under age 5 from the factors used to match comparison households and instead include number of children under 18.

After further analysis, it has been determined that the presence of a child with a disability and presence of a non-head of household adult with a disability are not substantial factors predicting a household's choice to participate in FSS, but each of these factors is associated with a large and significant difference in a household's future earnings change. As a result, HUD will include both factors in selecting comparison households.

5. Comment: Location of Comparison Households. A few commenters stated that households selected as comparisons for purposes of the earnings performance measure should be matched by similar census tract, neighborhood, or other measure of geography, to account for local variations in opportunity.

HUD Response: Location of Comparison Households. HUD agrees that it would be preferable to select comparison households from the same geography as the FSS participants to which they are being compared but notes that this may be impossible to achieve at a very small level of geography, such as census tract or ZIP code, due to an insufficient number of comparison households, especially at small PHAs. Moreover, households in neighboring census tracts or ZIP codes are likely to still be in the same labor market, and thus can still be effective comparators.

In PHAs that serve a very large geographical area, such as statewide PHAs, however, this point may not hold true since the economic conditions may be very different in different parts of the state. Accordingly, HUD plans to modify the protocol to require, under certain circumstances, that comparison households be in the same county and PHA as the FSS participants to which they are being compared. HUD will apply this protocol to all state PHAs and to non-State PHAs serving three or more counties where at least 10 percent of the PHA's housing choice voucher (HCV) or public housing households are leased in each of those counties. To ensure this approach does not unduly dilute the ability to find comparable households, HUD will require that FSS participants be matched to comparison households in the same county only in counties where there are at least four times as many non-FSS households as FSS households being served by the PHA.

6. Comment: Shifts in Enrollment. Many commenters were concerned that the performance measures would encourage PHAs to recruit or enroll participants with a high probability of increases in earnings or chances of FSS graduation. This comment arose most often for the earnings measure, though commenters differed on whether this would lead to recruiting minimally employed participants so that they had room to grow or participants who are already somewhat financially successful and have high potential to increase salaries without much intervention. A few commenters raised the concern that FSS programs will stop serving participants with substantial barriers who are riskier for the earnings and FSS graduation measures and require more intensive intervention.

HUD Response: Shifts in Enrollment. HUD appreciates these concerns and would remind PHAs of the requirement to open the program equally to all residents and administer the program for the residents who sign up for it, without trying to adjust enrollment to gain a higher score. As the commenters note, earnings gains among both unemployed participants and already employed participants can help boost a program's earnings performance score. It is also important to note that by regulation, FSS programs may screen families for interest and motivation to participate in the FSS program, but such programs are only permitted to screen for permissible motivational screening factors, i.e., those which solely measure the family's interest and motivation to participate in the FSS program. They may not exclude interested households based on other, prohibited characteristics.^[2]

7. Comment: Variations in Economic Conditions. Some commenters raised the concern that the earnings measure advantages communities with higher wages and stronger employment opportunities (primarily urban areas) and disadvantages communities with lower wages and weaker employment opportunities (primarily rural and suburban areas).

HUD Response: Variations in Economic Conditions. Because the earnings performance score is calculated based on the difference between the earnings growth of FSS participants and comparison households at the same PHA, it already controls to some extent for difference in economic conditions. Presumably, the comparison households at a PHA in a stronger economic market will experience greater earnings growth than the comparison households at a PHA in a weaker economic market, setting up a higher bar for FSS programs to exceed in the stronger market.

Based on these comments, however, HUD has conducted additional analysis to determine if there are some residual effects of strong economic conditions that are not accounted for in this methodology and therefore a need to account for it in assigning earnings performance scores. This analysis found that there is in fact still a relationship between the earnings performance measures and county median income. Accordingly, HUD has decided to apply an adjustment factor to the earnings performance measure to account for the residual effect of local economic conditions.

To compute this adjustment factor, HUD first used a linear regression model to examine the relationship between the earnings growth of comparison households within a PHA and the average county median income of those households. On average, earnings growth of comparison households was higher in counties with high median incomes, and lower in counties with low median incomes. HUD developed an adjustment factor that eliminated this relationship and then applied this adjustment factor to the earnings performance measure for each PHA, resulting in an adjusted earnings performance measure.

Using these adjusted earnings performance measures, HUD has recalculated the thresholds for awarding a 10, 7.5, or 0 earnings performance score by focusing on the 80th, 60th, and 20th percentile, respectively, of the distribution of adjusted measures. In selecting the revised thresholds, HUD has analyzed the distribution of scores across all funded PHAs, rather than the narrower universe described in the December 12, 2017 Federal Register Notice at 82 FR 58437 (the earlier notice included only PHAs whose earnings performance measures have a significant likelihood of being different from $0, per a statistical test). This makes the methodology more consistent with how HUD is calculating thresholds for the FSS graduation rate.

8. Comment: Interim Earnings. Many commenters expressed the view that the results of interim reexaminations of income should be included in analyzing earnings growth because they capture seasonal income, and the most recent progress toward higher earnings. Several were also concerned that if participants reach a level of earnings where they no longer receive any HAP, this increase in earnings may only be captured by interim reexaminations and FSS exit reports.

HUD Response: Interim Earnings. As noted in the December 12, 2017 Federal Register Notice, HUD did not consider the earnings reported through interim reexaminations of income in the analysis of earnings gains because some PHAs conduct such reexaminations when income increases between annual reexaminations and others do not. Excluding these interim results thus facilitates a direct comparison of local FSS programs. Further, participants' incomes are not reexamined at the time of exit from FSS. While excluding interim reexaminations will mean missing certain earnings changes, such as when a family's earnings increase to the point where they are paying zero HAP, HUD has determined that their inclusion would make it difficult to compare results across PHAs, an essential element of the performance measurement system.

9. Comment: Other Comments on the Earnings Measure. Most commenters agreed that averages were more appropriate than medians for the earnings measure. A few commenters stated that new employment and/or employment retention should be included as part of the earnings measure or in addition to the earnings measure. A few commenters suggested that escrow accumulation be included as part of or in addition to the earnings measure.

HUD Response: Other Comments on the Earnings Measure. As noted in the December 12, 2017 Federal Register Notice (at page 82 FR 58438-39), HUD chose to focus on average earnings growth rather than median earnings growth to ensure that PHAs received credit for the major, transformative earnings gains experienced by some FSS participants, even if this experience was not typical of the whole population of FSS participants. HUD appreciates that most commenters agreed with this approach. However, HUD disagrees with adding new employment, employment retention, and escrow accumulation as additional measures or as part of the earnings measure. Households that experience new employment and escrow accumulation are likely to also experience increased earnings, since these measures are strongly related. Accordingly, the inclusion of these measures as additional measures would provide even heavier weight to earnings than is already the case, which HUD does not believe to be appropriate. HUD also notes that data on “new employment” is not currently collected (though HUD could make inferences about this from the PIC data) and that this measure could disadvantage PHAs that serve a population that generally enters FSS employed. Escrow is driven largely by earnings gains, though it is also affected by the loss of welfare assistance or other non-earnings income and thus is less precise than earnings in measuring earnings growth. Escrow accumulation also does not take into account earnings gains for households above 50 percent of Area Medium Income (AMI), which is taken into consideration by the earnings measure currently in place. Additionally, until HUD has published a regulation or notice that implements Section 102 of Housing Opportunity Through Modernization Act of 2016 (HOTMA), residents who are subject to the Earned Income Disregard will have their escrow affected by that policy (in that escrow will not grow while income is disregarded for rent calculation purposes). While the current measure does not directly measure employment retention, it does factor it in since an FSS participant who retains his or her job while a comparison household does not will experience greater gains in earnings (zero) than the comparison household (a negative number), boosting the PHAs' average earnings performance score.

C. Comments on FSS Graduation Rate Measure

1. Comment: FSS Graduation Rate. Many commenters were concerned that the inclusion of an FSS graduation rate measure would encourage PHAs to graduate families quickly instead of encouraging families to set ambitious employment goals in addition to the necessary requirements of maintaining entry level employment and being free of welfare cash assistance for twelve (12) months. Others noted that PHAs define/operationalize some of the FSS graduation standards differently from one another, so this measure would not be consistent across PHAs. A few commenters said that the FSS graduation measure penalizes programs for terminating non-compliant participants.

HUD Response: FSS Graduation Rate. FSS graduation is an important milestone in the FSS program. FSS graduation marks the point at which FSS participants attain both their individual goals and the required program goals of employment and independence from welfare cash assistance. It also is the prerequisite for participants to receive the final disbursement from their escrow accounts.

Together, the Earnings Performance Measure, Graduation Rate, and Participation Rate provide a balanced measurement of the performance of an individual FSS program. Because the Earnings Performance Measure is weighted more heavily than the Graduation Rate, PHAs should balance the need to graduate participants with setting ambitious employment goals so participants can maximize their earnings growth while in the program. In addition, while PHAs have the discretion to terminate the FSS participation of non-compliant participants, HUD would encourage PHAs to first work with participants to determine if their challenges can be addressed so participants can successfully complete the FSS program. Additional guidance can be found in the FSS Promising Practices Guidebook.

D. Comments on Participation Rate Measure

1. Comment: Top Participation Scores. Many commenters expressed the view that having the top scores for participation substantially higher than the minimum a PHA is expected to serve with HUD funding is unfair and encourages PHAs to enroll more people than they can effectively serve. A few saw it as an unfunded mandate.

HUD Response: Top Participation Scores. All PHAs that serve the minimum number of participants expected based on the level of HUD coordinator funding will receive at least a 5 as a participation score. If a PHA can attain strong earnings and FSS graduation results while exceeding this minimum, however, HUD wishes to encourage them to do so as this helps to maximize the number of families benefitting from the FSS program. This is the reason for assigning higher participation scores to PHAs that achieve higher participation levels. Since earnings is weighted much more heavily than participation, however, HUD emphasizes that PHAs should only increase their caseloads if and to the extent they can do so without undermining their earnings and FSS graduation results.

HUD examined FSS performance data to determine if there is a correlation between a PHA's participation rate and its earnings and FSS graduation rate, paying particular attention to the participation rate threshold for obtaining a score of 10 points (80th percentile). This analysis did not find a strong relationship between participation rate and earnings performance measure. In fact, PHAs with participation rates between the 80th and 90th percentile had the highest average earnings performance measure of any decile and a median earnings performance measure that was typical for the sample as a whole, confirming that the threshold for obtaining a score of 10 points is not one that leads to lower earnings performance scores.

In terms of FSS graduation rates, the median FSS graduation rate was fairly similar for most deciles of participation rate, except for the very highest and lowest deciles, which both had lower FSS graduation rates than the other deciles. However, the threshold for qualifying for 10 points on the participation rate is set at the 80th percentile and not the 90th percentile (the starting point for the highest decile) and PHAs with participation rates between the 80th and 90th percentile had median and average FSS graduation rates that were typical for the sample as a whole, confirming that this threshold does not inherently lead to sub-par performance.

Based on this analysis, HUD has determined that it is appropriate to encourage PHAs to adopt higher participation rates, so long as they can do so without compromising their earnings performance and FSS graduation rates. However, HUD has decided to change the final scoring so as to reward incremental improvements in participation rates, rather than only participation rates that exceed one of two specific thresholds. Accordingly, HUD will assign PHAs with participation rates above .95 a score of 5, 6, 7, 8, 9 or 10, depending on their participation rate, as specified in Section III of this notice. A score of 10 will be awarded for a participation rate at or above 2.0, which is close to the 80th percentile level HUD previously identified.

2. Comment: Participation Rate and PHA Size. A few commenters said that the participation rate measure disadvantages either large PHAs/programs or small ones. For small programs in small PHAs, there may be less opportunity to recruit participants and smaller economies of scale for the coordinator. For large programs, increases in the number of participants enrolled would have to be very large in order to increase the participation score.

HUD Response: Participation Rate and PHA Size. The commenters are split about whether the participation rate calculation benefits smaller or larger PHAs. HUD believes this reflects the reality that all PHAs (regardless of size) have the potential to obtain either a high or a low participation rate, depending on how they manage their FSS program. This is confirmed by the fact that, in the initial spreadsheet of PHA scores, PHAs of all sizes are well represented at each of the participation score levels. While all PHAs must comply with the minimum enrollment requirements associated with the receipt of HUD coordinator funding, each PHA should make a determination of how many families they can serve effectively above this minimum based on their staff capacity, the intensity of participants' needs, and other resources available at the PHA and from partner organizations. HUD encourages PHAs to serve as many households as they can, so long as they do not exceed the level they can effectively support. Additionally, as explained above, there is no clear correlation between a PHA's size and the overall composite score.

E. Comments on Weighting of the Measures

1. Comment: Weighting. Several commenters felt that the weights are appropriate and did not comment further. Many commenters expressed the view that the earnings measure is weighted too highly. Commenters who suggested this were often concerned that the earnings measure would not show progress for FSS participants in longer-running education or training programs and so, did not account for variations in participant goals. Some commenters felt that FSS graduation and participation should have the same weight, regardless of the weight of the earnings measure. One reason given for this is that participation is essential for FSS graduation. Another was that weighting FSS graduation rate too highly compared to participation would encourage PHAs to graduate families before they had met ambitious goals.

HUD Response: Weighting. HUD appreciates the range of views expressed on this matter. After considering the comments, HUD plans to retain the weighting specified in the December 12, 2017 Federal Register Notice. Earnings represent by far the most powerful and objective measure available to HUD. While there are many goals to which FSS participants aspire, the achievement of most of these should lead to higher earnings which can then be measured through the earnings performance measure. Accordingly, HUD believes that a weight of 50 percent is appropriate.

While there is a case for weighting FSS graduation rate and participation rate equally, HUD believes weights of 30 percent for the FSS graduation rate and 20 percent of the participation rate are appropriate. As noted above, FSS graduation is an important milestone for the FSS program and HUD would like to see PHAs raise FSS graduation rates. HUD would also like to see PHAs serve more families if and to the extent they can do so without jeopardizing their achievement of strong earnings and FSS graduation rates. Weighting FSS graduation rate more heavily than participation rate is consistent with HUD's goal of not creating incentives for PHAs to raise caseloads beyond the point where families can be served effectively.

III. Final Thresholds

A. Summary of Adjustments to FSS Performance Score Methodology

After considering all of the public comments, HUD is adopting the proposed FSS performance measurement system, with the adjustments noted above, which will henceforth be used by HUD to evaluate the performance of PHAs receiving HUD program coordinator funding. These adjustments are summarized in the table below:

Changes to Methodology for Computing FSS Performance Scores

Overall	• Where a family ports, each PHA (the receiving and the initial PHA) will benefit from the family's FSS enrollment as it relates to the PHA's participation measure. For the earnings and FSS graduation measures, HUD will include the family for the PHA who currently administers the FSS contract. • HUD will treat joint applicants as a single PHA for purposes of computing all three components of the FSS performance score.
Earnings Performance Score	• In calculating the earnings performance score, HUD will exclude FSS participants who become classified as disabled at any point during their participation. • HUD will include within the earnings measure FSS participants that begin the FSS contract below age 62, even if they reach or exceed the age of 62 during their Contract of Participation.
	• In selecting comparison households, HUD will match FSS families with comparison families based on the number of children under the age of 18, rather than the presence of child under age 5. HUD will also match FSS families with comparison families based on presence of a child with a disability and presence of a non-head of household adult with a disability.
	• Under certain circumstances, HUD will require that comparison households be in the same county and PHA as the FSS participants to which they are being compared. HUD will apply this protocol to all state PHAs and to any additional PHAs where three or more counties are each home to at least 10 percent of households receiving housing assistance from the PHA (through HCV or public housing). To ensure this approach does not unduly dilute the ability to find comparable households, HUD will require that FSS participants be matched to comparison households in the same county only in counties where there are at least four times as many non-FSS households as FSS households being served by the PHA.
	• HUD will apply an adjustment factor to the earnings performance measure to account for variations in local economic conditions.

After making these adjustments to the methodology, HUD has recalculated the thresholds for translating the FSS performance measures into individual component scores and the final composite score and notes the final thresholds below.

B. Updated Thresholds for FSS Performance Scores

The following are the updated thresholds HUD will use to compute an FSS Performance Score for each PHA. See the December 12, 2017 Federal Register Notice and the updated complete methodology, which can be found on HUD's website at https://www.hud.gov/​program_​offices/​public_​indian_​housing/​programs/​hcv/​fss,, for more information on each of the two steps in this process.

1. Step One: Assigning Scores to Each of the Three Measures

In Step One, HUD will assign a score of 0 to 10 to each PHA's FSS program for each of the three measures. Scores will be assigned using the thresholds and procedures described below. The ranges for awarding points between two values include those values as well as all intermediary values.

a. Earnings Performance Measure (50 percent of final score):

• 10 points: Earnings performance measure of $8,700 or higher.
• 7.5 points: Earnings performance measure between $6,950 and $8,699.99.
• 0 points: Earnings performance measure below $4,050 and a p-value of <.10 on a statistical test measuring the likelihood that a PHA's earnings performance measure is significantly lower than the median measure of $6,302 (see December 12, 2017 Federal Register Notice at page 82 FR 58437 for an explanation of this statistical test).
• 5 points: All PHAs that do not qualify for a 10, 7.5, or a 0.

b. FSS Graduation Rate (30 percent of final score):

• 10 points: FSS graduation rate of 38 percent or higher.
• 7.5 points: FSS graduation rate between 28 percent and 37.9 percent.
• 0 points: FSS graduation rate below 10 percent.
• 5 points: All PHAs that do not qualify for a 10, 7.5, or a 0

c. Participation Rate (20 percent of final score):

• 10 points: Participation rate of 2.0 or higher.
• 9 points: Participation rate between 1.8 and 1.99.
• 8 points: Participation rate between 1.6 and 1.79.
• 7 points: Participation rate between 1.4 and 1.59.
• 6 points: Participation rate between 1.2 and 1.39.
• 5 points: Participation rate between .96 and 1.19.
• 0 points: Participation rate of .95 or lower.

2. Step Two: Developing the Final FSS Performance Score and Grade

In Step Two, after computing individual scores for each of the three measures, HUD will aggregate each PHA's scores using the weights noted above to develop a final FSS Performance Score from 0 to 10. Based on this score, HUD will assign the following ranking to the PHA's performance:

• Category 1: FSS Performance score of 8.0 or higher.
• Category 2: FSS Performance score between 4.26 and 7.99.
• Category 3: FSS Performance score between 3.26 and 4.25.
• Category 4: FSS Performance score of 3.25 or lower.

IV. Environmental Impact

This notice does not direct, provide for assistance or loan and mortgage insurance for, or otherwise govern or regulate, real property acquisition, disposition, leasing, rehabilitation, alteration, demolition, or new construction, or establish, revise or provide for standards for construction or construction materials, manufactured housing, or occupancy. Accordingly, under 24 CFR 50.19(c)(1), this notice is categorically excluded from environmental review under the National Environmental Policy Act of 1969 (42 U.S.C. 4321).

Dated: November 7, 2018.

Dominique Blom,

General Deputy Assistant Secretary, Public and Indian Housing.

Footnotes

1.  Section 306 of the Economic Growth, Regulatory Relief, and Consumer Protection Act (Pub. L. 115-174, Approved May 24, 2018) amended the United States Housing Act of 1937. Among various provisions, this law extended FSS program eligibility to tenants of certain privately-owned properties subsidized with project-based rental assistance (PBRA).

2.  24 CFR 984.203(c).