2021-17339. HUD Program Evaluation Policy-Policy Statement

AGENCY:

Office of Policy Development and Research, HUD.

ACTION:

Notice.

SUMMARY:

This policy statement articulates the core principles and practices of the Department of Housing and Urban Development's evaluation and research activities. This policy reaffirms HUD's commitment to conducting rigorous, relevant evaluations and to using evidence from evaluations to inform policy and practice.

DATES:

August 13, 2021.

FOR FURTHER INFORMATION CONTACT:

For information about this notice, contact Todd M. Richardson, Evaluation Officer, Office of Policy Development and Research, U.S. Department of Housing and Urban Development, 451 7th Street SW, Washington, DC 20410, telephone [phone]. The listed telephone number is not a toll-free number. Persons with hearing- or speech-impairments may access this number through TTY by calling Federal Relay Service at 1-800-877-8339 (this is a toll-free number).

SUPPLEMENTARY INFORMATION:

I. Background

Evaluation activity occurs in several offices at HUD, but the special mission of HUD's Office of Policy Development and Research (PD&R) is to inform HUD policy development and implementation to improve life in American communities through conducting, supporting, and sharing research, surveys, demonstrations, program evaluations, and best practices. Within HUD, PD&R is responsible for most, but not all, program evaluations. The office provides reliable and objective data and analysis to help inform policy decisions.

In July 2016, GAO issued a report entitled “Department of Housing and Urban Development: Actions Needed to Incorporate Key Practices into Management Functions and Program Oversight,” (GAO 16-497) in which GAO presented a broad assessment of HUD's management of its operations and programs.^[1] In the report, GAO examined HUD efforts to: (1) Meet Federal requirements and implement key practices for management functions, including performance planning and reporting and human capital, financial, acquisition, and information technology (IT) management; and (2) oversee and evaluate programs.

PD&R is the primary office within HUD responsible for data analysis, research, program evaluations, and policy studies that inform the development and implementation of programs and policies across HUD offices. PD&R undertakes program evaluations, often by using a process that includes convening expert panels. However, GAO found that PD&R had neither developed agency-wide, written policies for its program evaluations, nor documented the criteria used to select the expert panels and review the quality of program evaluations.

On December 6, 2016 (81 FR 87949), HUD issued a policy statement in the Federal Register ^[2] responding to the GAO report by setting out the core principles and practices of PD&R's evaluation and research activities. This statement incorporated some language from a policy statement by the Office of Policy, Research, and Evaluation of the Administration for Children and Families of the U.S. Department of Health and Human Services.

On January 14, 2019, the Foundations for Evidence-Based Policymaking Act of 2018 (“Evidence Act”), Public Law 115-435, was enacted.^[3] Section 101 of the Evidence Act created 5 U.S.C. 311-315 and mandated that the head of each agency appoint an Evaluation Officer, including at HUD. This officer must establish common standards for all HUD evaluations, whether performed by PD&R or another office. This issuance articulates department-wide evaluation standards and states other new principles based on PD&R's experience since the November 2016 publication.

II. HUD Program Evaluation Policy

Section 101 of the Evidence Act defines “evaluation” to mean “an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency.”

HUD has identified the following core principles and practices as fundamental to ensuring high-quality and consistent evaluation results: Rigor, relevance, transparency, independence, ethics, and technical innovation. This policy applies to all HUD-sponsored evaluations and regulatory impact analyses; they also apply to the selection of projects, contractors, and HUD staff that are involved in evaluations.

Rigor

HUD is committed to using the most rigorous methods that are appropriate to the evaluation questions and feasible within budget and other constraints. The need for rigor is not restricted to impact evaluations; rigor is also necessary in implementation or process evaluations, descriptive studies, outcome evaluations, and formative evaluations; in both qualitative and quantitative approaches. Rigor requires ensuring that inferences about cause and effect are well founded (internal validity); requires clarity about the populations, settings, or circumstances to which results can be generalized (external validity); requires that researchers seek to understand and correct for implicit bias in the formulation of research questions and methods; and requires the use of measures that accurately capture the intended information (measurement reliability and validity). Implicit biases are discriminatory biases that reflect unidentified traces of past experience, including implicit attitudes that affect feelings, thoughts or actions, and implicit stereotypes that affect how others are characterized. Survey instruments are pre-tested with members of the population to be studied to increase measurement validity. When statistically appropriate, and particularly if the number of hypotheses being tested is large, HUD will require the use of commonly accepted adjustments to classical statistical testing to reduce the probability that random outliers are presented as meaningful.

In assessing the effects of programs or services, HUD evaluations use methods that isolate to the greatest extent possible the impacts of the programs or services from other influences, such as trends over time, geographic variation, or pre-existing differences between participants and non-participants. Where feasible, research should employ a treatment group and a counterfactual. A treatment group is a population that has received an intervention. A counterfactual is a population that did not receive the intervention and can credibly represent what would have happened to the treatment group in the absence of the intervention according to the above standard for rigor. For such causal questions, experimental approaches are preferred. When experimental approaches are not feasible, HUD uses the most rigorous approach that is feasible.

In both quantitative and qualitative research, rigor means having clear research questions and an explicit analytic framework; justification for case selection and sampling methods in relation to research goals; and transparent, verifiable methods of systematic data collection and analysis, auditable records, and attention to possible alternative interpretations during analysis and writing.

HUD ensures that contractors and grantees conducting evaluations have appropriate expertise through emphasizing the requirement for rigor in requests for proposals and funding opportunity announcements, noting that applicants' capacity for rigor will be evaluated in the selection process. In addition, HUD will judge research teams with equal capacity for rigor to be more qualified if the team includes researchers demographically similar to or knowledgeable about the perspectives and lived experiences of the populations studied.

HUD employs a strategic human capital development plan to hire, train, and retain a workforce that ensures staff have the tools and resources to accomplish the mission.

Relevance

The HUD evaluation agenda reflects the legislative requirements and policy issues related to HUD's mission. HUD solicits input from stakeholders, both internal and external, including stakeholders with lived experience, such as program participants, and grantees, on the selection of programs to be evaluated, initiatives, demonstrations, and research questions. For new initiatives and demonstrations in particular, evaluations will be more feasible and useful when planned in advance, in concert with the development of the initiative or demonstration, rather than as an afterthought. HUD strives to understand the relevance of its completed research through concerted stakeholder engagement, including with people and grantees affected by HUD programs, to continuously improve its research agenda.

Expert panels include research and other subject matter experts and are diverse in ways tailored to the study, including racial and ethnic diversity and representatives of the studied populations.

HUD strives to design program evaluations and other analyses to better understand structural racism and to reveal unequal benefits and harms across social groups as relevant, with special attention to race, national origin, color, familial status, religion, disability, age, and sex (including gender identity and sexual orientation).

To support this goal, insofar as feasible, HUD collects and reports data on race, ethnicity, gender, and income, and other characteristics of underrepresented and underserved communities relevant for research and analysis efforts.

HUD also encourages research to engage studied populations. “Engagement” means the deliberate and intentional inclusion of the thoughts and perspectives of studied groups, such as program participants, grantees, and underrepresented and underserved populations. This includes collecting people's thoughts and perspectives through standard (valid and rigorous) research methods such as surveys, focus groups, in-depth interviews, or ethnography to produce published research that conveys studied populations' thoughts and perspectives. In another type of engagement, people influence the research that is about them. This includes a wide range of activities that lie on a continuum from simple input (which should be documented and published) to full co-creation of any aspect of the research, from topic selection to research design, data collection, data analysis, interpretation, writing, or dissemination, or even being on the research team. An example of simple input is to include in the final report a summary of comments on the research by members of studied groups. Examples of collaboration include, but are not limited to, co-creating the list of topics to be covered in a survey or having members of a studied group on the research team.

To raise awareness of and spur creative approaches to engagement of studied populations in program evaluations and HUD-sponsored research, HUD may require contractors and grantees to explain how their research will and will not engage studied populations. HUD recognizes that engagement must be tailored to particular research efforts.

HUD retains the right to determine research methods.

HUD disseminates findings in ways that are accessible and useful to policymakers, practitioners, and members of communities affected by HUD programs and policies. Published findings will be accessible to individuals with disabilities pursuant to Section 508 of the Rehabilitation Act. PD&R partners with other HUD program offices to inform internal and external stakeholders through disseminating evidence from HUD-sponsored evaluations.

Transparency

HUD will release methodologically valid evaluations without regard to the findings. Evaluation reports must describe the methods used, including strengths and weaknesses, and discuss the generalizability of the findings. Evaluation reports must present comprehensive results, including favorable, unfavorable, and null findings. HUD will publish interim findings, as projected in the initial research design. If there are indications that the findings of the final report may differ, HUD will provide appropriate qualifications accompanying the publication to guard against misunderstanding or misuse of the interim findings. If there are interim findings, HUD will publish those findings even if there are indications that the findings of the final report may differ. When findings are highly relevant to current policy, HUD evaluations carry a foreword articulating the policy position of the Department with respect to those findings.

If the findings of a HUD evaluation will have broad public interest and includes a counterfactual, PD&R will publish a synopsis of the research design, data collection and analysis plan soon after it is approved and will require interim and final reports that deviate from that document to explain how they deviate and why.

HUD publishes a 5-year Learning Agenda ^[4] that outlines the research and evaluation that it believes would be of greatest value to public policy. PD&R lists all ongoing evaluation projects at the HUDUSER.gov website ^[5] and updates it quarterly. PD&R will release evaluation results timely, usually within 4 months of receiving the final report.

HUD will, where possible, archive administrative and evaluation data for secondary use by interested researchers. HUD typically builds requirements into contracts to prepare data sets for secondary use. Access for external researchers may be provided directly through data licenses or indirectly through inter-agency agreements. This policy may not apply for data that has obvious commercial value, such as mortgage performance data. HUD staff may publish the results of their scholarship and analysis in any forum, so long as they do not claim to speak for the Department.

HUD evaluation contracts will generally permit contractors to publish their findings within 6 months of the termination of the contract if HUD has not already published them.

Independence

Independence and objectivity are core principles of evaluation. Agency and program leadership, program staff, service providers, and others participate actively in setting evaluation priorities, identifying evaluation questions, and assessing the implications of findings. However, it is important to insulate evaluation functions from undue influence and from both the appearance and the reality of bias. To promote objectivity, HUD protects independence in the design, conduct, and analysis of evaluations. To this end:

• HUD conducts evaluations through the competitive award of grants and contracts to external experts who are free from conflicts of interest.
• HUD also conducts evaluations in-house and supports unsolicited external evaluation proposals with funding, data, or both.
• The Evaluation Officer will consult with the HUD office with lead responsibility on the design of evaluation projects and analysis plans and will advise that office on whether to publish evaluation reports.

Ethics

HUD-sponsored evaluations must be conducted in an ethical manner and safeguard the dignity, rights, safety, and privacy of participants. HUD-sponsored evaluations must comply with both the spirit and the letter of relevant requirements such as regulations governing research involving human subjects. In particular, PD&R protects the privacy of HUD-assisted households and HUD-insured borrowers through its Rule of Eleven; that is, PD&R allows no disclosure of information about the characteristics of any group of individuals or households numbering fewer than eleven by PD&R staff, contractors, grantees, or licensees.

HUD is a signatory to the Federal Policy for the Protection of Human Subjects, generally known as the “Common Rule.” 24 CFR part 60, which includes its own requirements for ensuring adequate provisions to protect the privacy of human subjects research.^[6]

HUD does not tolerate plagiarism, or fabrication or deliberate mischaracterization of data by staff, contractors or grantees who are engaged in evaluation activity.

Technical Innovation

PD&R supports and employs new methods of data collection and analysis that more reliably and efficiently answer research questions than old methods do.

Application of These Principles to Economic Analysis of Regulations

Economic analysis of both existing and proposed regulations, properly conducted, is a critical tool in improving public policy. Economists at HUD rely on the insights, data, and empirical estimates from rigorous program evaluations when predicting the economic impact of an incremental change to the program. In any HUD Regulatory Impact Analysis:

• HUD analyzes whether the issues addressed by the regulation stem from a market failure, government failure, or other systemic problem, and whether the regulation addresses the root causes of those problems.
• HUD uses and as necessary produces the best objective estimates of the benefits, costs, and transfers resulting from the regulation, taking into account gaps and uncertainties in the available data and methodologies.
• HUD assesses the economic benefits, costs, and transfers of proposed regulatory actions as required by Executive Order 12866.^[7] HUD provides additional analysis of impacts across groups defined by race, ethnicity, and other characteristics that may define underrepresented and underserved groups when such analyses are relevant and feasible.

• Where clear alternatives to the regulatory actions exist, HUD objectively estimates the benefits, costs, and transfers of those alternatives, and additional analysis of impacts of those alternatives across underrepresented or underserved groups as well.

Todd M. Richardson,

Evaluation Officer.

Footnotes

1.  See http://www.gao.gov/​assets/​680/​678551.pdf.

2.  https://www.federalregister.gov/​d/​2016-29215.

3.  https://www.congress.gov/​115/​plaws/​publ435/​PLAW-115publ435.pdf. See also the website, at https://www.congress.gov/​bill/​115th-congress/​house-bill/​4174,, on the bill, H.R. 4174 of the 115th Congress, that became the Act.

4.  https://www.huduser.gov/​portal/​about/​pdr_​learningagenda.html.

5.  https://www.huduser.gov/​portal/​about/​PDR-Research.html.

6.  https://www.ecfr.gov/​cgi-bin/​retrieveECFR?​n=​pt24.1.60.

7.  See https://www.archives.gov/​files/​federal-register/​executive-orders/​pdf/​12866.pdf for the original order, and see https://www.archives.gov/​federal-register/​executive-orders/​1993-clinton.html for citations of, and links to, other executive orders that amended or supplemented this order.