2020-21745. Definitions and Selection Criteria That Apply to Direct Grant Programs

AGENCY:

Department of Education.

ACTION:

Final rule; incorporation by reference.

SUMMARY:

The Secretary is issuing this rule in order to update the versions of the What Works Clearinghouse Standards Handbook and What Works Clearinghouse Procedures Handbook incorporated by reference into the Department's applicable regulations.

DATES:

Effective date: These regulations are effective October 5, 2020.

Applicability date: These regulations are applicable for competitions announced on or after October 5, 2020.

Incorporation by reference: The incorporation by reference of the What Works Clearinghouse Standards Handbook, Versions 4.0 and 4.1 and What Works Clearinghouse Procedures Handbook, Versions 4.0 and 4.1 is approved by the Director of the Federal Register as of October 5, 2020. The incorporation by reference of the other material in §  77.1 was approved by the Director of the Federal Register as of July 31, 2017.

FOR FURTHER INFORMATION CONTACT:

Jonathan Jacobson, U.S. Department of Education, 550 12th Street SW, PCP-4158, Washington, DC 20202-5900. Telephone: [phone]. Email: [e-mail].

If you use a telecommunications device for the deaf (TDD) or text telephone (TTY), call the Federal Relay Service (FRS), toll free, at 1-800-877-8339.

SUPPLEMENTARY INFORMATION:

Final Regulatory Changes

In these final regulations, we update relevant provisions of the Education Department General Administrative Regulations to include the current versions of the What Works Clearinghouse (WWC) Standards Handbook and What Works Clearinghouse Procedures Handbook (the Handbooks). We also incorporate these Handbooks, which provide a detailed description of the standards and procedures of the WWC, by reference. The Handbooks are available to interested parties at https://ies.ed.gov/​ncee/​wwc/​Handbooks. The Version 3.0 Handbook added reviewer guidance not included in the Version 2.1 Handbook, and described procedures for WWC practice guides, single study reviews, and quick reviews. More details are available at https://ies.ed.gov/​ncee/​wwc/​Docs/​referenceresources/​wwc_​procedures_​v3_​0_​standards_​handbook_​updates.pdf. The Version 4.0 Handbooks separated “procedures” for reviewing, reporting, and synthesizing study findings from “standards” of internal validity. They also removed the “pilot” designation from the standards for regression discontinuity designs (RDDs) and updated the WWC's standards for “fuzzy” RDDs, complier average causal effects, cluster-level assignment studies, and studies with missing data. More details are available at https://ies.ed.gov/​ncee/​wwc/​Docs/​referenceresources/​wwc_​handbook_​summary_​v4.0.pdf. The Version 4.1 Handbooks removed the “pilot” designation from the WWC's standards for single-case designs (SCDs) and added new procedures for estimating design-comparable effect sizes from SCD studies for synthesis with findings from group design studies. These Handbooks also removed the “substantively important” designation based on the magnitude of effect size reported in a study and revised WWC procedures for synthesizing findings across studies to use a meta-analytic approach rather than counting studies that found positive effects. More details are available at https://ies.ed.gov/​ncee/​wwc/​Docs/​referenceresources/​WWCHandbookSummary-v4-1-508.pdf.

The WWC is an initiative of the U.S. Department of Education's (the Department's) National Center for Education Evaluation and Regional Assistance, within the Institute of Education Sciences (IES), which was established under the Education Sciences Reform Act of 2002 (Title I of Pub. L. 107-279). The WWC is an important part of the Department's strategy to use rigorous and relevant research, evaluation, and statistics to inform decisions in the field of education. The WWC provides critical assessments of scientific evidence on the effectiveness of education programs, policies, products, and practices (referred to as “interventions”) and a range of publications and tools summarizing this evidence. The WWC meets the need for credible, succinct information by reviewing research studies; assessing the quality of the research; summarizing the evidence of the effectiveness of interventions on student outcomes and other outcomes related to education; and disseminating its findings broadly.

In addition, we make other minor, technical updates to 34 CFR 77.1, to correct numbering and cross-references.

34 CFR Part 77

Section 77.1 Definitions That Apply to All Department Programs

Current Regulations: Section 77.1(c) establishes definitions that, unless a statute or regulation provides otherwise, apply to the regulations in title 34 of the Code of Federal Regulations and can be used in Department grant competitions.

Final Regulations and Reasons: We are revising the definitions of “experimental study,” “moderate evidence,” “quasi-experimental design study,” and “strong evidence” to refer to Versions 4.0 and 4.1 of the Handbooks, in addition to the current references to Versions 2.1 and 3.0, following the publication of Version 4.0 in October 2017 and Version 4.1 in January 2020. We refer to Versions 2.1, 3.0, 4.0, and 4.1 so that applicants can cite practice guides, intervention reports, and study reviews that the WWC has prepared under those Handbook versions. We also are updating the definition of “What Works Clearinghouse Handbook (WWC Handbook)” to be “What Works Clearinghouse Handbooks (WWC Handbooks)” and to include Versions 4.0 and 4.1 of the Handbooks.

This update will help ensure that future evidence reviews reflect the most recent standards and procedures used elsewhere by the WWC. Improvements included in the Version 4.0 and Version 4.1 WWC Handbooks are described at https://ies.ed.gov/​ncee/​wwc/​Docs/​referenceresources/​wwc_​handbook_​summary_​v4.0.pdf and https://ies.ed.gov/​ncee/​wwc/​Docs/​referenceresources/​WWCHandbookSummary-v4-1-508.pdf. In addition, this technical update will allow for a sufficient number of certified reviewers to conduct WWC reviews for Department grant competitions, since the WWC offers training and certification in Version 4.1 standards at https://ies.ed.gov/​ncee/​wwc/​OnlineTraining,, and there are fewer reviewers certified in the older Version 3.0 standards where there is no longer training.

Section 77.2 Incorporation By Reference

Current Regulations: Section 77.2 incorporates Version 2.1 and 3.0 of the What Works Clearinghouse Procedures and Standards Handbook by reference.

Final Regulations and Reasons: For the reasons stated above, we are updating this section to incorporate the WWC Standards Handbook, Versions 4.0 and 4.1 and WWC Procedures Handbook, Versions 4.0 and 4.1 by reference as well.

Waiver of Proposed Rulemaking and Delayed Effective Date

Under the Administrative Procedure Act (APA) (5 U.S.C. 551 et seq.), the Department generally offers interested parties the opportunity to comment on proposed regulations. However, these regulations make technical changes only and do not establish substantive policy. The regulations are therefore exempt from notice and comment rulemaking under 5 U.S.C. 553(b)(3)(B).

The APA also generally requires that regulations be published at least 30 days before their effective date, unless the agency has good cause to implement its regulations sooner (5 U.S.C. 553(d)(3)). Again, because these final regulations are merely technical, there is good cause to make them effective on the day they are published. However, these technical changes will only apply to competitions announced on or after the date of the publication of this final rule.

Executive Orders 12866, 13563, and 13771

Regulatory Impact Analysis

Under Executive Order 12866, the Office of Management and Budget (OMB) must determine whether this regulatory action is “significant” and, therefore, subject to the requirements of the Executive order and subject to review by OMB. Section 3(f) of Executive Order 12866 defines a “significant regulatory action” as an action likely to result in a rule that may—

(1) Have an annual effect on the economy of $100 million or more, or adversely affect a sector of the economy, productivity, competition, jobs, the environment, public health or safety, or State, local, or Tribal governments or communities in a material way (also referred to as an “economically significant” rule);

(2) Create serious inconsistency or otherwise interfere with an action taken or planned by another agency;

(3) Materially alter the budgetary impacts of entitlement grants, user fees, or loan programs or the rights and obligations of recipients thereof; or

(4) Raise novel legal or policy issues arising out of legal mandates, the President's priorities, or the principles stated in the Executive order.

This final regulatory action is not a significant regulatory action subject to review by OMB under section 3(f) of Executive Order 12866.

Under Executive Order 13771, for each new regulation that the Department proposes for notice and comment or otherwise promulgates that is a significant regulatory action under Executive Order 12866 and that imposes total costs greater than zero, it must identify two deregulatory actions. For Fiscal Year 2020, any new incremental costs associated with a new regulation must be fully offset by the elimination of existing costs through deregulatory actions. Because the proposed regulatory action is not significant, the requirements of Executive Order 13771 do not apply. Moreover, even if this were a significant regulatory action, Executive Order 13771 would not apply because it is a “transfer rule,” i.e., regulations that cause only income transfers between taxpayers and program beneficiaries, such as those regarding discretionary grant programs.

We have also reviewed these regulations under Executive Order 13563, which supplements and explicitly reaffirms the principles, structures, and definitions governing regulatory review established in Executive Order 12866. To the extent permitted by law, Executive Order 13563 requires that an agency—

(1) Propose or adopt regulations only on a reasoned determination that their benefits justify their costs (recognizing that some benefits and costs are difficult to quantify);

(2) Tailor its regulations to impose the least burden on society, consistent with obtaining regulatory objectives and taking into account—among other things and to the extent practicable—the costs of cumulative regulations;

(3) In choosing among alternative regulatory approaches, select those approaches that maximize net benefits (including potential economic, environmental, public health and safety, and other advantages; distributive impacts; and equity);

(4) To the extent feasible, specify performance objectives, rather than the behavior or manner of compliance a regulated entity must adopt; and

(5) Identify and assess available alternatives to direct regulation, including economic incentives—such as user fees or marketable permits—to encourage the desired behavior, or provide information that enables the public to make choices.

Executive Order 13563 also requires an agency “to use the best available techniques to quantify anticipated present and future benefits and costs as accurately as possible.” The Office of Information and Regulatory Affairs of OMB has emphasized that these techniques may include “identifying changing future compliance costs that might result from technological innovation or anticipated behavioral changes.”

We are issuing these final regulations only on a reasoned determination that their benefits justify their costs. In choosing among alternative regulatory approaches, we selected those approaches that maximize net benefits. Based on an analysis of anticipated costs and benefits, the Department believes that these final regulations are consistent with the principles in Executive Order 13563.

We also have determined that this regulatory action does not unduly interfere with State, local, and Tribal governments in the exercise of their governmental functions.

Potential Costs and Benefits

Under Executive Order 12866, we have assessed the potential costs and benefits of this regulatory action and have determined that these regulations would not impose additional costs. We believe any additional costs imposed by these final regulations will be negligible, primarily because they reflect technical changes that do not impose additional burden on those submitting evidence for the Department to assess. Updating definitions in 34 CFR 77.1(c) to allow WWC reviews and reports reviewed under Versions 4.0 or 4.1 standards will expand the body of WWC-reviewed evidence available to inform decision-makers. Evidence previously reviewed and reported by the WWC under Version 2.1 or Version 3.0 standards may still satisfy the requirements of strong or moderate evidence, provided all other aspects of those definitions are addressed. When the Department must review evidence that has not previously been rated by the WWC, using the most up-to-date WWC standards and recently trained and certified WWC reviewers will increase the integrity of the review process. We believe any costs will be significantly outweighed by the potential benefits of making necessary updates and ensuring that regulated parties are aware of and may refer to the current versions of the Handbooks.

Regulatory Flexibility Act Certification

The Regulatory Flexibility Act does not apply to this rulemaking because there is good cause to waive notice and comment under the APA.

Paperwork Reduction Act of 1995

These regulations do not contain any information collection requirements.

Intergovernmental Review

This program is not subject to Executive Order 12372 and the regulations in 34 CFR part 79.

Accessible Format: Individuals with disabilities can obtain this document in an accessible format (e.g., braille, large print, audiotape, or compact disc) on request to the program contact person listed under FOR FURTHER INFORMATION CONTACT.

Electronic Access to This Document: The official version of this document is the document published in the Federal Register. You may access the official edition of the Federal Register and the Code of Federal Regulations at www.govinfo.gov. At this site you can view this document, as well as all other documents of this Department published in the Federal Register, in text or Portable Document Format (PDF). To use PDF you must have Adobe Acrobat Reader, which is available free at the site.

You may also access documents of the Department published in the Federal Register by using the article search feature at www.federalregister.gov. Specifically, through the advanced search feature at this site, you can limit your search to documents published by the Department.

List of Subjects in 34 CFR Part 77

• Education
• Grant programs-education
• Incorporation by reference

Betsy DeVos,

Secretary of Education.

For the reasons discussed in the preamble, the Secretary amends part 77 of title 34 of the Code of Federal Regulations as follows:

PART 77—DEFINITIONS THAT APPLY TO DEPARTMENT REGULATIONS

1. The authority citation for part 77 continues to read as follows:

Authority: 20 U.S.C. 1221e-3 and 3474, unless otherwise noted.

2. Section 77.1 is amended by:

a. In paragraph (b) introductory text, removing the words “ title 34 of the Code of Federal Regulations” and adding the words “subtitles A and B of this title” in their place.

b. In paragraph (c):

i. In the introductory text, adding the words “subtitles A and B of” before the words “this title”.

ii. In the second sentence of the introductory text of the definition of “Experimental study”, removing the word “Handbook” and adding in its place the word “Handbooks”.

iii. In the definition of “Grant”, redesignating paragraphs (1) through (4) as paragraphs (i) through (iv), respectively.

iv. In the definition of “Local educational agency”, redesignating paragraphs (a) introductory text, (a)(1) and (2), (b), and (c) as paragraphs (i) introductory text, (i)(A) and (B), (ii), and (iii), respectively.

v. Revising the definition of “Moderate evidence”.

vi. In the second sentence of the definition of “Quasi-experimental design study,” removing the word “Handbook” and adding in its place the word “Handbooks”.

vii. In the definition of “Service function”, redesignating paragraphs (a) introductory text, (a)(1)(i) and (ii), (a)(2), and (b), as paragraphs (i) introductory text, (i)(A)(1) and (2), (i)(B), and (ii), respectively.

viii. Revising the definitions of “Strong evidence” and “What Works Clearinghouse Handbook (WWC Handbook)”.

c. Remove the parenthetical authority at the end of the section.

The revisions read as follows:

§ 77.1

Definitions that apply to all Department programs.

* * * * *

(c) * * *

Moderate evidence means that there is evidence of effectiveness of a key project component in improving a relevant outcome for a sample that overlaps with the populations or settings proposed to receive that component, based on a relevant finding from one of the following:

(i) A practice guide prepared by the WWC using version 2.1, 3.0, 4.0, or 4.1 of the WWC Handbooks reporting a “strong evidence base” or “moderate evidence base” for the corresponding practice guide recommendation;

(ii) An intervention report prepared by the WWC using version 2.1, 3.0, 4.0, or 4.1 of the WWC Handbooks reporting a “positive effect” or “potentially positive effect” on a relevant outcome based on a “medium to large” extent of evidence, with no reporting of a “negative effect” or “potentially negative effect” on a relevant outcome; or

(iii) A single experimental study or quasi-experimental design study reviewed and reported by the WWC using version 2.1, 3.0, 4.0, or 4.1 of the WWC Handbooks, or otherwise assessed by the Department using version 4.1 of the WWC Handbooks, as appropriate, and that—

(A) Meets WWC standards with or without reservations;

(B) Includes at least one statistically significant and positive (i.e., favorable) effect on a relevant outcome;

(C) Includes no overriding statistically significant and negative effects on relevant outcomes reported in the study or in a corresponding WWC intervention report prepared under version 2.1, 3.0, 4.0, or 4.1 of the WWC Handbooks; and

(D) Is based on a sample from more than one site (e.g., State, county, city, school district, or postsecondary campus) and includes at least 350 students or other individuals across sites. Multiple studies of the same project component that each meet requirements in paragraphs (iii)(A), (B), and (C) of this definition may together satisfy the requirement in this paragraph (iii)(D).

* * * * *

Strong evidence means that there is evidence of the effectiveness of a key project component in improving a relevant outcome for a sample that overlaps with the populations and settings proposed to receive that component, based on a relevant finding from one of the following:

(i) A practice guide prepared by the WWC using version 2.1, 3.0, 4.0, or 4.1 of the WWC Handbooks reporting a “strong evidence base” for the corresponding practice guide recommendation;

(ii) An intervention report prepared by the WWC using version 2.1, 3.0, 4.0, or 4.1 of the WWC Handbooks reporting a “positive effect” on a relevant outcome based on a “medium to large” extent of evidence, with no reporting of a “negative effect” or “potentially negative effect” on a relevant outcome; or

(iii) A single experimental study reviewed and reported by the WWC using version 2.1, 3.0, 4.0, or 4.1 of the WWC Handbooks, or otherwise assessed by the Department using version 4.1 of the WWC Handbooks, as appropriate, and that—

(A) Meets WWC standards without reservations;

(B) Includes at least one statistically significant and positive (i.e., favorable) effect on a relevant outcome;

(C) Includes no overriding statistically significant and negative effects on relevant outcomes reported in the study or in a corresponding WWC intervention report prepared under version 2.1, 3.0, 4.0, or 4.1 of the WWC Handbooks; and

(D) Is based on a sample from more than one site (e.g., State, county, city, school district, or postsecondary campus) and includes at least 350 students or other individuals across sites. Multiple studies of the same project component that each meet requirements in paragraphs (iii)(A), (B), and (C) of this definition may together satisfy the requirement in this paragraph (iii)(D).

* * * * *

What Works Clearinghouse (WWC) Handbooks (WWC Handbooks) means the standards and procedures set forth in the WWC Standards Handbook, Versions 4.0 or 4.1, and WWC Procedures Handbook, Versions 4.0 or 4.1, or in the WWC Procedures and Standards Handbook, Version 3.0 or Version 2.1 (all incorporated by reference, see § 77.2). Study findings eligible for review under WWC standards can meet WWC standards without reservations, meet WWC standards with reservations, or not meet WWC standards. WWC practice guides and intervention reports include findings from systematic reviews of evidence as described in the WWC Handbooks documentation.

* * * * *

3. Section 77.2 is amended by:

a. In paragraph (a):

i. In the second sentence, removing the word “below” and adding the words “in paragraph (b) of this section” in its place; and

ii. Revising the last sentence of the paragraph; and

b. Revising paragraph (b).

The revisions read as follows:

§ 77.2

Incorporation by Reference.

(a) * * * For information on the availability of this material at NARA, email [e-mail], or go to http://www.archives.gov/​federal-register/​cfr/​ibr-locations.html.

(b) Institute of Education Sciences, 550 12th Street SW, Washington, DC 20202, [phone], http://ies.ed.gov/​ncee/​wwc/​Handbooks.

(1) What Works Clearinghouse Standards Handbook, Version 4.1, January 2020, IBR approved for § 77.1.

(2) What Works Clearinghouse Procedures Handbook, Version 4.1, January 2020, IBR approved for § 77.1.

(3) What Works Clearinghouse Standards Handbook, Version 4.0, October 2017, IBR approved for § 77.1.

(4) What Works Clearinghouse Procedures Handbook, Version 4.0, October 2017, IBR approved for § 77.1.

(5) What Works Clearinghouse Procedures and Standards Handbook, Version 3.0, March 2014, IBR approved for § 77.1.

(6) What Works Clearinghouse Procedures and Standards Handbook, Version 2.1, September 2011, IBR approved for § 77.1.