2017-15989. Definitions and Selection Criteria That Apply to Direct Grant Programs

AGENCY:

Department of Education.

ACTION:

Final rule with request for comments.

SUMMARY:

The Secretary is issuing this rule in order to better align the regulations with the definition of “evidence-based” in the statutory authority. These changes mean that all competitive grant programs in the Department can continue to use the same provisions for evidence-based grant-making.

DATES:

Effective date: These regulations are effective July 31, 2017. The incorporation by reference of certain publications listed in these regulations is approved by the Director of the Federal Register as of July 31, 2017.

Comment due date: We will accept comments on or before August 30, 2017. We will consider the comments received and may conduct additional rulemaking based on the comments.

ADDRESSES:

Submit your comments through the Federal eRulemaking Portal or via postal mail, commercial delivery, or hand delivery. We will not accept comments submitted by fax or by email or those submitted after the comment period. To ensure that we do not receive duplicate copies, please submit your comments only once. In addition, please include the Docket ID at the top of your comments.

• Federal eRulemaking Portal: Go to www.regulations.gov to submit your comments electronically. Information on using Regulations.gov, including instructions for accessing agency documents, submitting comments, and viewing the docket, is available on the site under “How to use Regulations.gov.”
• Postal Mail, Commercial Delivery, or Hand Delivery. If you mail or deliver your comments about these final regulations, address them to Kelly Terpak, U.S. Department of Education, 400 Maryland Avenue SW., Room 4W312, Washington, DC 20202-5900.

Privacy Note: The Department's policy for comments received from members of the public is to make these submissions available for public viewing in their entirety on the Federal eRulemaking Portal at www.regulations.gov. Therefore, commenters should be careful to include in their comments only information that they wish to make publicly available.

FOR FURTHER INFORMATION CONTACT:

Kelly Terpak, U.S. Department of Education, 400 Maryland Avenue SW., Room 4W312, Washington, DC 20202-5900. Telephone: [phone] or by email: [e-mail].

If you use a telecommunications device for the deaf (TDD) or text telephone (TTY), call the Federal Relay Service (FRS), toll free, at 1-800-877-8339.

SUPPLEMENTARY INFORMATION:

As noted above, these regulations are effective on July 31, 2017. However, for grant award competitions announced by the Department in the Federal Register prior to the effective date of these regulations, unless the notice specifies otherwise, the provisions of 34 CFR parts 75 and 77 revised or removed through this notice of final regulations continue to apply to competitions and grants awarded under those notices inviting applications.

Invitation To Comment

These regulations do not establish substantive policy changes, but instead make technical changes to existing regulations. Since these regulations make only technical changes, a comment period is not required. However, we are interested in whether you think we should make any changes in these regulations and thus we are inviting your comments. We will consider these comments in determining whether to make further technical changes to the regulations or engage in additional rulemaking. To ensure that your comments have maximum effect, we urge you to identify clearly the specific section or sections of the regulations that each of your comments addresses and to arrange your comments in the same order as the regulations. See ADDRESSES for instructions on how to submit comments.

We invite you to assist us in complying with the specific requirements of Executive Orders 12866 and 13563 and their overall requirements of reducing regulatory burden that might result from these regulations. Please let us know of any additional ways we could reduce potential costs or increase potential benefits while preserving the effective and efficient administration of the Department's programs and activities.

During and after the comment period, you may inspect all public comments about these regulations by accessing Regulations.gov. You may also inspect the comments in person in Room 6W245, 400 Maryland Avenue SW., Washington, DC, between the hours of 8:30 a.m. and 4:00 p.m., Eastern time, Monday through Friday of each week except Federal holidays. If you want to schedule time to inspect comments, please contact the person listed under FOR FURTHER INFORMATION CONTACT.

Assistance to Individuals with Disabilities in Reviewing the Rulemaking Record: On request, we will provide an appropriate accommodation or auxiliary aid to an individual with a disability who needs assistance to review the comments or other documents in the public rulemaking record for these regulations. If you want to schedule an appointment for this type of aid, please contact the person listed under FOR FURTHER INFORMATION CONTACT.

Final Regulatory Changes

I. Selection Criteria

Background: The regulations in subpart D of 34 CFR part 75 set forth the general requirements that govern the Department's selection of grantees for direct grant awards. For those direct grant programs that make discretionary grant awards, the Secretary uses selection criteria to evaluate applications submitted under those programs. The regulations establish a menu of selection criteria that the Secretary may use in any Department discretionary grant competition.

34 CFR Part 75

§ 75.210 General Selection Criteria

Current Regulations: Current § 75.210(c) lists 29 factors under the “Quality of the Project Design” selection criterion. Section 75.210(h) includes 12 factors under the “Quality of the Project Evaluation” selection criterion.

Final Regulations and Reasons: We make the following changes to the selection criteria in § 75.210(c) and (h):

(1) Add one selection factor under the “Quality of the Project Design” criterion (§ 75.210(c)) to clarify that the Department may assess the extent to which an applicant's proposed project would represent a faithful adaptation of the evidence cited in support of its project. This factor is designed to assess whether projects would in fact implement the evidence cited as support, such that the project is “evidence-based” as described in section 8101(21)(A) of the Elementary and Secondary Education Act (ESEA), as amended by the Every Student Succeeds Act (ESSA).

(2) For clarification, add two selection factors under the “Quality of the Project Evaluation” criterion (§ 75.210(h)) focused on (a) the qualifications of an applicant's evaluator; and (b) the sufficiency of resources to carry out the project evaluation.

We also revise two factors under the “Quality of the Project Design” criterion (§ 75.210(c)) and four factors under the “Quality of the Project Evaluation” criterion (§ 75.210(h)) to align terminology with the revised evidence definitions in 34 CFR part 77. Specifically, the regulations:

(1) Replace references to “evidence of promise” and “strong theory” with “promising evidence” and “demonstrates a rationale,” respectively.

(2) Align terminology with the revised definitions in 34 CFR 77.1(c) to include the term “project component” and clarify that the What Works Clearinghouse standards are described in the What Works Clearinghouse Handbook.

We are making these revisions to improve the menu of selection criteria and factors by better aligning them to the evidence-related definitions in 34 CFR part 77. We make these revisions in conjunction with the amendments to the definitions in 34 CFR part 77, which, as discussed elsewhere in this document, we also revise to align with the evidence provisions in section 8101(21) of the ESEA, as amended by the ESSA, and for clarity. The final regulations do not change the way the Secretary uses the current and new selection criteria and factors. The Secretary will continue to use selection criteria that are consistent with the purpose of the program and permitted under the applicable statutes and regulations.

II. Evidence Preferences and Priorities

§ 75.226 What procedures does the Secretary use if the Secretary decides to give special consideration to applications supported by strong evidence of effectiveness, moderate evidence of effectiveness, or evidence of promise?

Current Regulations: Under § 75.226, the Secretary may establish a competitive preference or absolute priority for projects supported by strong evidence of effectiveness, moderate evidence of effectiveness, or evidence of promise, as those terms are currently defined in 34 CFR part 77.

Final Regulations and Reasons: The Secretary makes technical revisions to the title and text of this section to describe procedures for giving special consideration to applications supported by strong, moderate, or promising evidence, which are the evidence-related terms used in the ESEA. We include definitions for these terms elsewhere in this document.

These technical changes ensure that discretionary grant programs authorized by the ESEA, as amended by the ESSA, can establish evidence-based priorities under § 75.226 and allow the Department the option to use one set of uniform evidence standards for all discretionary grant programs across each program's authorizing statute.

III. Evidence Definitions

Background: Section 77.1(c) establishes definitions that, unless a statute or regulation provides otherwise, apply to the regulations in title 34 of the Code of Federal Regulations and can be used in Department grant competitions. This section includes a number of definitions that support the Department's use of evidence in grant competitions. The ESSA amended the ESEA to include a new definition of “evidence-based” that necessitates changes to these definitions.

34 CFR Part 77

§ 77.1 Definitions That Apply to All Department Programs

Current Regulations: Section 77.1(c) establishes definitions that, unless a statute or regulation provides otherwise, apply to the regulations in title 34 of the Code of Federal Regulations and can be used in Department grant competitions.

Final Regulations and Reasons: We establish new, and revise some existing, definitions to (1) ensure alignment with provision in the ESEA, as amended by the ESSA, providing a single set of evidence definitions; and (2) make minor clarifying revisions to existing provisions. In these final regulations, we:

(1) Add a definition of “evidence-based” that incorporates the four levels of evidence in section 8101(21)(A) of the ESEA, as amended by the ESSA.

(2) Add a definition for “project component” as a single, clarifying term for what may be included in a project. The term clarifies that “policy” may be one component of a project; encompasses “an activity, strategy, or intervention,” to be consistent with the definition of “evidence-based” in section 8101(21) of the ESEA, as amended by the ESSA; and includes “process,” “product,” and “practice,” which were in the evidence definitions in 34 CFR 77.1(c) (e.g., strong evidence of effectiveness) prior to these final regulations.

(3) Remove the definitions of “large sample” and “multi-site sample” and instead incorporate them into the new “moderate evidence” and “strong evidence” definitions, to streamline these definitions.

(4) Replace the term “strong theory” with the term “demonstrates a rationale,” as this is the fourth level of evidence in the definition of “evidence-based” in section 8101(21) of the ESEA, as amended by the ESSA.

(5) Replace the term “evidence of promise” with the term “promising evidence,” to align with the definition of “evidence-based” in section 8101(21) of the ESEA, as amended by the ESSA. In the definition of “promising evidence,” we clarify—

• How practice guides and intervention reports prepared by the What Works Clearinghouse (WWC), in alignment with the WWC standards incorporated in the definition, can provide promising evidence;
• How the Department already reviews single studies to determine whether they qualify under this level of evidence; and
• That certain quasi-experimental studies and experimental studies that do not meet WWC standards can qualify as promising evidence, as the previous “evidence of promise” definition implied.
• That correlational studies with statistical controls for selection bias must be well-designed and well-implemented to qualify as promising evidence, as the ESEA, as amended by the ESSA, provides.

(6) Replace the term “moderate evidence of effectiveness” with the term “moderate evidence,” which is used in the ESEA definition of “evidence-based.” In the definition of “moderate evidence,” we clarify—

• How practice guides and intervention reports prepared by the WWC, in alignment with the WWC standards incorporated in the definition, can provide moderate evidence;
• How the Department already reviews single studies to determine whether they qualify under this level of evidence; and
• Through language regarding “relevant findings,” that there must be a link between the proposed activities, strategies, and interventions and specific statistically significant effects, as required under the definition of “evidence-based” in section 8101(21) of the ESEA, as amended by the ESSA.

(7) Replace the term “randomized controlled trial” with the term “experimental study,” to align with the definition of “evidence-based,” in section 8101(21) specifically with regard to “strong evidence.” In this new definition of “strong evidence,” we clarify the types of studies that can qualify as experimental studies—including, but not limited to, randomized controlled trials—as provided in the applicable WWC Handbook.

(8) Replace the term “strong evidence of effectiveness” with the term “strong evidence,” which is used in the definition of “evidence-based” in section 8101(21) of the ESEA, as amended by the ESSA. In the definition of “strong evidence,” we clarify—

• How practice guides and intervention reports prepared by the WWC, in alignment with the WWC standards incorporated in the definition, can provide promising evidence under the definition of “evidence-based” in section 8101(21) of the ESEA, as amended by the ESSA;
• How the Department already reviews single studies to determine whether they qualify under this level of evidence; and
• Through language regarding “relevant findings,” that there must be a link between the proposed activities, strategies, and interventions and specific statistically significant effects, as required under the definition of “evidence-based” in section 8101(21) of the ESEA, as amended by the ESSA.

(9) Replace the term “What Works Clearinghouse Evidence Standards” with the term “What Works Clearinghouse Handbook,” to clarify that the Handbook's procedures—not just standards—are relevant to evidence determinations, consistent with current practice. We also incorporate this Handbook, which provides a detailed description of the standards and procedures of the WWC, by reference. The WWC is an initiative of the U.S. Department of Education's National Center for Education Evaluation and Regional Assistance, within the Institute of Education Sciences (IES), which was established under the Education Sciences Reform Act of 2002. The WWC is an important part of IES's strategy to use rigorous and relevant research, evaluation, and statistics to inform decisions in the field of education. The WWC provides critical assessments of scientific evidence on the effectiveness of education programs, policies, products, and practices (referred to as “interventions”) and a range of publications and tools summarizing this evidence. The WWC meets the need for credible, succinct information by reviewing research studies; assessing the quality of the research; summarizing the evidence of the effectiveness of programs, policies, products, and practices on student outcomes and other outcomes related to education; and disseminating its findings broadly. This Handbook is available to interested parties at the Web site address included in the regulation (https://ies.ed.gov/​ncee/​wwc/​Handbooks).

(10) Make minor clarifying changes to the definition of “logic model” so it is more easily understood.

(11) Make minor clarifying changes to the definition of “quasi-experimental design study” to align with terminology in the revised § 77.1(c).

(12) Make minor clarifying changes to the definition of “relevant outcome” to align with terminology in the revised § 77.1(c).

Waiver of Proposed Rulemaking and Delayed Effective Date

Under the Administrative Procedure Act (APA) (5 U.S.C. 553), the Department generally offers interested parties the opportunity to comment on proposed regulations. However, these regulations make technical changes only and do not establish substantive policy. The regulations are therefore exempt from notice and comment rulemaking under 5 U.S.C. 553(b)(3)(B). However, the Department is providing a 30-day comment period and invites interested persons to participate in this rulemaking by submitting written comments. The Department will consider the comments received and may conduct additional rulemaking based on the comments.

The APA also generally requires that regulations be published at least 30 days before their effective date, unless the agency has good cause to implement its regulations sooner (5 U.S.C. 553(d)(3)). Again, because these final regulations are merely technical, there is good cause to make them effective on the day they are published.

Executive Orders 12866, 13563, and 13771

Regulatory Impact Analysis

Under Executive Order 12866, the Secretary must determine whether this regulatory action is “significant” and, therefore, subject to the requirements of the Executive order and subject to review by the Office of Management and Budget (OMB). Section 3(f) of Executive Order 12866 defines a “significant regulatory action” as an action likely to result in a rule that may—

(1) Have an annual effect on the economy of $100 million or more, or adversely affect a sector of the economy, productivity, competition, jobs, the environment, public health or safety, or State, local, or Tribal governments or communities in a material way (also referred to as an “economically significant” rule);

(2) Create serious inconsistency or otherwise interfere with an action taken or planned by another agency;

(3) Materially alter the budgetary impacts of entitlement grants, user fees, or loan programs or the rights and obligations of recipients thereof; or

(4) Raise novel legal or policy issues arising out of legal mandates, the President's priorities, or the principles stated in the Executive order.

This final regulatory action is not a significant regulatory action subject to review by OMB under section 3(f) of Executive Order 12866.

Under Executive Order 13771, for each new regulation that the Department proposes for notice and comment or otherwise promulgates that is a significant regulatory action under Executive Order 12866, it must identify two deregulatory actions. For Fiscal Year 2017, any new incremental costs associated with a new regulation must be fully offset by the elimination of existing costs through deregulatory actions. The final regulations are not a significant regulatory action. Therefore, the requirements of Executive Order 13771 do not apply.

We have also reviewed these regulations under Executive Order 13563, which supplements and explicitly reaffirms the principles, structures, and definitions governing regulatory review established in Executive Order 12866. To the extent permitted by law, Executive Order 13563 requires that an agency—

(1) Propose or adopt regulations only on a reasoned determination that their benefits justify their costs (recognizing that some benefits and costs are difficult to quantify);

(2) Tailor its regulations to impose the least burden on society, consistent with obtaining regulatory objectives and taking into account—among other things and to the extent practicable—the costs of cumulative regulations;

(3) In choosing among alternative regulatory approaches, select those approaches that maximize net benefits (including potential economic, environmental, public health and safety, and other advantages; distributive impacts; and equity);

(4) To the extent feasible, specify performance objectives, rather than the behavior or manner of compliance a regulated entity must adopt; and

(5) Identify and assess available alternatives to direct regulation, including economic incentives—such as user fees or marketable permits—to encourage the desired behavior, or provide information that enables the public to make choices.

Executive Order 13563 also requires an agency “to use the best available techniques to quantify anticipated present and future benefits and costs as accurately as possible.” The Office of Information and Regulatory Affairs of OMB has emphasized that these techniques may include “identifying changing future compliance costs that might result from technological innovation or anticipated behavioral changes.”

We are issuing these final regulations only on a reasoned determination that their benefits justify their costs. In choosing among alternative regulatory approaches, we selected those approaches that maximize net benefits. Based on an analysis of anticipated costs and benefits, the Department believes that these final regulations are consistent with the principles in Executive Order 13563.

We also have determined that this regulatory action does not unduly interfere with State, local, and Tribal governments in the exercise of their governmental functions.

Potential Costs and Benefits

Under Executive Order 12866, we have assessed the potential costs and benefits of this regulatory action and have determined that these regulations would not impose additional costs. We believe any additional costs imposed by these final regulations will be negligible, primarily because they reflect technical changes which do not impose additional burden. Moreover, we believe any costs will be significantly outweighed by the potential benefits of making necessary clarifications and ensuring consistency among the Education Department General Administrative Regulations and section 8101(21) of ESEA, as amended by the ESSA.

Clarity of the Regulations

Executive Order 12866 and the Presidential memorandum “Plain Language in Government Writing” require each agency to write regulations that are easy to understand.

The Secretary invites comments on how to make these regulations easier to understand, including answers to questions such as the following:

• Are the requirements in the regulations clearly stated?
• Do the regulations contain technical terms or other wording that interferes with their clarity?
• Does the format of the regulations (grouping and order of sections, use of headings, paragraphing, etc.) aid or reduce their clarity?
• Would the regulations be easier to understand if we divided them into more (but shorter) sections? (A “section” is preceded by the symbol “§ ” and a numbered heading; for example, § 75.210.)
• Could the description of the regulations in the SUPPLEMENTARY INFORMATION section of this preamble be more helpful in making the regulations easier to understand? If so, how?
• What else could we do to make the regulations easier to understand?

To send any comments that concern how the Department could make these regulations easier to understand, see the instructions in the ADDRESSES section.

Regulatory Flexibility Act Certification

The Secretary certifies that these regulations do not have a significant economic impact on a substantial number of small entities.

Paperwork Reduction Act of 1995

The Paperwork Reduction Act of 1995 does not require you to respond to a collection of information unless it displays a valid OMB control number. We display the valid OMB control number assigned to a collection of information in final regulations at the end of the affected section of the regulations.

Intergovernmental Review

This program is not subject to Executive Order 12372 and the regulations in 34 CFR part 79.

Accessible Format: Individuals with disabilities can obtain this document in an accessible format (e.g., braille, large print, audiotape, or compact disc) on request to the program contact person listed under FOR FURTHER INFORMATION CONTACT.

Electronic Access to This Document: The official version of this document is the document published in the Federal Register. Free internet access to the official edition of the Federal Register and the Code of Federal Regulations is available via the Federal Digital System at: www.gpo.gov/​fdsys. At this site you can view this document, as well as all other documents of this Department published in the Federal Register, in text or Portable Document Format (PDF). To use PDF you must have Adobe Acrobat Reader, which is available free at the site.

You may also access documents of the Department published in the Federal Register by using the article search feature at: www.federalregister.gov. Specifically, through the advanced search feature at this site, you can limit your search to documents published by the Department.

List of Subjects

34 CFR Part 75

• Accounting
• Copyright
• Education
• Grant programs—education
• Inventions and patents
• Private schools
• Reporting and recordkeeping requirements
• Youth organizations

34 CFR Part 77

• Education
• Grant programs—education
• Incorporation by reference

Dated: July 25, 2017.

Betsy DeVos,

Secretary of Education.

For the reasons discussed in the preamble, the Secretary amends parts 75 and 77 of title 34 of the Code of Federal Regulations as follows:

PART 75—DIRECT GRANT PROGRAMS

1. The authority citation for part 75 continues to read as follows:

Authority: 20 U.S.C. 1221e-3 and 3474, unless otherwise noted.

2. Section 75.210 is amended by:

a. Revising paragraphs (c)(2)(xxviii) and (xxix);

b. Adding paragraph (c)(2)(xxx); and

c. Revising paragraphs (h)(2)(viii) through (xii); and

d. Adding paragraph (h)(2)(xiii).

The revisions and addition read as follows:

§ 75.210

General selection criteria.

* * * * *

(c) * * *

(2) * * *

(xxviii) The extent to which the proposed project is supported by promising evidence (as defined in 34 CFR 77.1(c)).

(xxix) The extent to which the proposed project demonstrates a rationale (as defined in 34 CFR 77.1(c)).

(xxx) The extent to which the proposed project represents a faithful adaptation of the evidence cited in support of the proposed project.

* * * * *

(h) * * *

(2) * * *

(viii) The extent to which the methods of evaluation will, if well implemented, produce evidence about the project's effectiveness that would meet the What Works Clearinghouse standards without reservations as described in the What Works Clearinghouse Handbook (as defined in 34 CFR 77.1(c)).

(ix) The extent to which the methods of evaluation will, if well implemented, produce evidence about the project's effectiveness that would meet the What Works Clearinghouse standards with or without reservations as described in the What Works Clearinghouse Handbook (as defined in 34 CFR 77.1(c)).

(x) The extent to which the methods of evaluation will, if well implemented, produce promising evidence (as defined in 34 CFR 77.1(c)) about the project's effectiveness.

(xi) The extent to which the evaluation plan clearly articulates the key project components, mediators, and outcomes, as well as a measurable threshold for acceptable implementation.

(xii) The qualifications, including relevant training, experience, and independence, of the evaluator.

(xiii) The extent to which the proposed project plan includes sufficient resources to conduct the project evaluation effectively.

* * * * *

3. Revise § 75.226 to read as follows:

§ 75.226

What procedures does the Secretary use if the Secretary decides to give special consideration to applications supported by strong, moderate, or promising evidence?

(a) As used in this section, “strong evidence” is defined in 34 CFR 77.1(c).

(b) As used in this section, “moderate evidence” is defined in 34 CFR 77.1(c).

(c) As used in this section, “promising evidence” is defined in 34 CFR 77.1(c).

(d) If the Secretary determines that special consideration of applications supported by strong, moderate, or promising evidence is appropriate, the Secretary may establish a separate competition under the procedures in 34 CFR 75.105(c)(3), or provide competitive preference under the procedures in 34 CFR 75.105(c)(2), for applications supported by—

(1) Evidence that meets the conditions in the definition of “strong evidence”;

(2) Evidence that meets the conditions in the definition of “moderate evidence”; or

(3) Evidence that meets the conditions in the definition of “promising evidence.”

PART 77—DEFINITIONS THAT APPLY TO DEPARTMENT REGULATIONS

3. The authority citation for part 77 continues to read as follows:

Authority: 20 U.S.C. 1221e-3 and 3474, unless otherwise noted.

4. Section 77.1(c) is amended by:

a. Adding, in alphabetical order, a definition for “Demonstrates a rationale”.

b. Removing the definition of “Evidence of promise”.

c. Adding, in alphabetical order, definitions for “Evidence-based” and “Experimental study”.

d. Removing the definition of “Large sample”.

e. Revising the definition of “Logic model”.

f. Adding, in alphabetical order, a definition for “Moderate evidence”.

g. Removing the definitions of “Moderate evidence of effectiveness” and “Multi-site sample”.

h. Adding, in alphabetical order, definitions for “Project component” and “Promising evidence”.

i. Revising the definitions of “Quasi-experimental design study” and “Relevant outcome”.

j. Adding, in alphabetical order, a definition for “Strong evidence”.

k. Removing the definitions of “Strong evidence of effectiveness”, “Strong theory”, and “What Works Clearinghouse Evidence Standards”.

l. Adding, in alphabetical order, a definition for “What Works Clearinghouse Handbook”.

The additions and revisions read as follows:

§ 77.1

Definitions that apply to all Department programs.

* * * * *

(c) * * *

Demonstrates a rationale means a key project component included in the project's logic model is informed by research or evaluation findings that suggest the project component is likely to improve relevant outcomes.

* * * * *

Evidence-based means the proposed project component is supported by one or more of strong evidence, moderate evidence, promising evidence, or evidence that demonstrates a rationale.

Experimental study means a study that is designed to compare outcomes between two groups of individuals (such as students) that are otherwise equivalent except for their assignment to either a treatment group receiving a project component or a control group that does not. Randomized controlled trials, regression discontinuity design studies, and single-case design studies are the specific types of experimental studies that, depending on their design and implementation (e.g., sample attrition in randomized controlled trials and regression discontinuity design studies), can meet What Works Clearinghouse (WWC) standards without reservations as described in the WWC Handbook:

(i) A randomized controlled trial employs random assignment of, for example, students, teachers, classrooms, or schools to receive the project component being evaluated (the treatment group) or not to receive the project component (the control group).

(ii) A regression discontinuity design study assigns the project component being evaluated using a measured variable (e.g., assigning students reading below a cutoff score to tutoring or developmental education classes) and controls for that variable in the analysis of outcomes.

(iii) A single-case design study uses observations of a single case (e.g., a student eligible for a behavioral intervention) over time in the absence and presence of a controlled treatment manipulation to determine whether the outcome is systematically related to the treatment.

* * * * *

Logic model (also referred to as a theory of action) means a framework that identifies key project components of the proposed project (i.e., the active “ingredients” that are hypothesized to be critical to achieving the relevant outcomes) and describes the theoretical and operational relationships among the key project components and relevant outcomes.

* * * * *

Moderate evidence means that there is evidence of effectiveness of a key project component in improving a relevant outcome for a sample that overlaps with the populations or settings proposed to receive that component, based on a relevant finding from one of the following:

(i) A practice guide prepared by the WWC using version 2.1 or 3.0 of the WWC Handbook reporting a “strong evidence base” or “moderate evidence base” for the corresponding practice guide recommendation;

(ii) An intervention report prepared by the WWC using version 2.1 or 3.0 of the WWC Handbook reporting a “positive effect” or “potentially positive effect” on a relevant outcome based on a “medium to large” extent of evidence, with no reporting of a “negative effect” or “potentially negative effect” on a relevant outcome; or

(iii) A single experimental study or quasi-experimental design study reviewed and reported by the WWC using version 2.1 or 3.0 of the WWC Handbook, or otherwise assessed by the Department using version 3.0 of the WWC Handbook, as appropriate, and that—

(A) Meets WWC standards with or without reservations;

(B) Includes at least one statistically significant and positive (i.e., favorable) effect on a relevant outcome;

(C) Includes no overriding statistically significant and negative effects on relevant outcomes reported in the study or in a corresponding WWC intervention report prepared under version 2.1 or 3.0 of the WWC Handbook; and

(D) Is based on a sample from more than one site (e.g., State, county, city, school district, or postsecondary campus) and includes at least 350 students or other individuals across sites. Multiple studies of the same project component that each meet requirements in paragraphs (iii)(A), (B), and (C) of this definition may together satisfy this requirement.

* * * * *

Project component means an activity, strategy, intervention, process, product, practice, or policy included in a project. Evidence may pertain to an individual project component or to a combination of project components (e.g., training teachers on instructional practices for English learners and follow-on coaching for these teachers).

* * * * *

Promising evidence means that there is evidence of the effectiveness of a key project component in improving a relevant outcome, based on a relevant finding from one of the following:

(i) A practice guide prepared by WWC reporting a “strong evidence base” or “moderate evidence base” for the corresponding practice guide recommendation;

(ii) An intervention report prepared by the WWC reporting a “positive effect” or “potentially positive effect” on a relevant outcome with no reporting of a “negative effect” or “potentially negative effect” on a relevant outcome; or

(iii) A single study assessed by the Department, as appropriate, that—

(A) Is an experimental study, a quasi-experimental design study, or a well-designed and well-implemented correlational study with statistical controls for selection bias (e.g., a study using regression methods to account for differences between a treatment group and a comparison group); and

(B) Includes at least one statistically significant and positive (i.e., favorable) effect on a relevant outcome.

* * * * *

Quasi-experimental design study means a study using a design that attempts to approximate an experimental study by identifying a comparison group that is similar to the treatment group in important respects. This type of study, depending on design and implementation (e.g., establishment of baseline equivalence of the groups being compared), can meet WWC standards with reservations, but cannot meet WWC standards without reservations, as described in the WWC Handbook.

* * * * *

Relevant outcome means the student outcome(s) or other outcome(s) the key project component is designed to improve, consistent with the specific goals of the program.

* * * * *

Strong evidence means that there is evidence of the effectiveness of a key project component in improving a relevant outcome for a sample that overlaps with the populations and settings proposed to receive that component, based on a relevant finding from one of the following:

(i) A practice guide prepared by the WWC using version 2.1 or 3.0 of the WWC Handbook reporting a “strong evidence base” for the corresponding practice guide recommendation;

(ii) An intervention report prepared by the WWC using version 2.1 or 3.0 of the WWC Handbook reporting a “positive effect” on a relevant outcome based on a “medium to large” extent of evidence, with no reporting of a “negative effect” or “potentially negative effect” on a relevant outcome; or

(iii) A single experimental study reviewed and reported by the WWC using version 2.1 or 3.0 of the WWC Handbook, or otherwise assessed by the Department using version 3.0 of the WWC Handbook, as appropriate, and that—

(A) Meets WWC standards without reservations;

(B) Includes at least one statistically significant and positive (i.e., favorable) effect on a relevant outcome;

(C) Includes no overriding statistically significant and negative effects on relevant outcomes reported in the study or in a corresponding WWC intervention report prepared under version 2.1 or 3.0 of the WWC Handbook; and

(D) Is based on a sample from more than one site (e.g., State, county, city, school district, or postsecondary campus) and includes at least 350 students or other individuals across sites. Multiple studies of the same project component that each meet requirements in paragraphs (iii)(A), (B), and (C) of this definition may together satisfy this requirement.

* * * * *

What Works Clearinghouse Handbook (WWC Handbook) means the standards and procedures set forth in the WWC Procedures and Standards Handbook, Version 3.0 or Version 2.1 (incorporated by reference, see 34 CFR 77.2). Study findings eligible for review under WWC standards can meet WWC standards without reservations, meet WWC standards with reservations, or not meet WWC standards. WWC practice guides and intervention reports include findings from systematic reviews of evidence as described in the Handbook documentation.

* * * * *

5. Section 77.2 is added to read as follows:

§ 77.2

Incorporation by Reference.

(a) Certain material is incorporated by reference into this part with the approval of the Director of the Federal Register under 5 U.S.C. 552(a) and 1 CFR part 51. All approved material is available for inspection at Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance by email at [e-mail], and is available from the sources listed below. It is also available for inspection at the National Archives and Records Administration (NARA). For information on the availability of this material at NARA, call 202-741-6030 or go to www.archives.gov/​federal_​register/​code_​of_​federal_​regulations/​ibr_​locations.html.

(b) Institute of Education Sciences, 550 12th Street SW., Washington, DC 20202, [phone], http://ies.ed.gov/​ncee/​wwc/​Handbooks.

(1) What Works Clearinghouse Procedures and Standards Handbook, Version 3.0, March 2014, IBR approved for § 77.1.

(2) What Works Clearinghouse Procedures and Standards Handbook, Version 2.1, September 2011, IBR approved for § 77.1.