2012-29897. Direct Grant Programs and Definitions That Apply to Department Regulations

AGENCY:

Department of Education.

ACTION:

Notice of proposed rulemaking.

SUMMARY:

The Secretary proposes to amend the regulations in 34 CFR parts 75 and 77 of the Education Department General Administrative Regulations (EDGAR) in order to improve the Department's ability to promote projects supported by evidence; evaluate the performance of discretionary grant programs and grantee projects; review grant applications using selection factors that promote reform objectives related to project evaluation, sustainability, productivity, and capacity to scale; and reduce burden on grantees in selecting implementation sites, implementation partners, or evaluation service providers for their proposed projects. These proposed changes would allow the Department to be more effective and efficient when selecting discretionary grantees, provide higher-quality data to Congress and the public, and better focus applicants on the particular goals and objectives of the programs to which they apply for grants.

DATES:

We must receive your comments on or before February 12, 2013.

ADDRESSES:

Submit your comments through the Federal eRulemaking Portal or via postal mail, commercial delivery, or hand delivery. We will not accept comments by fax or by email. Please submit your comments only one time, in order to ensure that we do not receive duplicate copies. In addition, please include the Docket ID at the top of your comments.

• Federal eRulemaking Portal: Go to http://www.regulations.gov to submit your comments electronically. Information on using Regulations.gov, including instructions for accessing agency documents, submitting comments, and viewing the docket, is available on the site under “How To Use This Site.”
• Postal Mail, Commercial Delivery, or Hand Delivery. If you mail or deliver your comments about these proposed regulations, address them to Margo Anderson, U.S. Department of Education, 400 Maryland Avenue SW., Room 4W313, Washington, DC 20202-5900.

Privacy Note:

The Department's policy for comments received from members of the public is to make these submissions available for public viewing in their entirety on the Federal eRulemaking Portal at www.regulations.gov. Therefore, commenters should be careful to include in their comments only information that they wish to make publicly available.

FOR FURTHER INFORMATION CONTACT:

Erin McHugh, U.S. Department of Education, 400 Maryland Avenue SW., Room 4W319, Washington, DC 20202. Telephone: [phone] or by email: [e-mail].

If you use a telecommunications device for the deaf (TDD) or text telephone (TTY), call the Federal Relay Service (FRS) toll free at 1-800-877-8339.

SUPPLEMENTARY INFORMATION:

Invitation to Comment on Proposed Regulations

We invite you to submit comments regarding these proposed regulations. To ensure that your comments have maximum effect in developing the final regulations, we urge you to identify clearly the specific section or sections of the proposed regulations that each of your comments addresses and to arrange your comments in the same order as the proposed regulations.

We invite you to assist us in complying with the specific requirements of Executive Order 12866 and 13563 and their overall requirement of reducing regulatory burden that might result from these proposed regulations. Please let us know of any ways we could reduce potential costs or increase potential benefits while preserving the effective and efficient administration of the Department's programs and activities.

Because Executive Order 12866 and the presidential memorandum on “Plain Language in Government Writing” require each agency to write regulations that are easy to understand, we invite you to comment on how to make these proposed regulations easier to understand, including answers to questions such as the following:

• Are the requirements in the proposed regulations clearly stated?
• Do the proposed regulations contain technical terms or other wording that interferes with their clarity?
• Does the format of the proposed regulations (grouping and order of sections, use of headings, paragraphing, etc.) aid or reduce their clarity?
• Would the proposed regulations be easier to understand if we divided them into more (but shorter) sections? (A “section” is preceded by the symbol “§ ” and a numbered heading; for example, § 75.210 General selection criteria.
• Could the description of the proposed regulations in this preamble be more helpful in making the proposed regulations easier to understand? If so, how?
• What else could we do to make the proposed regulations easier to understand?

Retrospective Review of EDGAR

On January 21, 2011, President Obama issued Executive Order 13563, “Improving Regulation and Regulatory Review” (76 FR 3821). The order requires all Federal agencies to “consider how best to promote retrospective analysis of rules that may be outmoded, ineffective, insufficient, or excessively burdensome, and to modify, streamline, expand, or repeal them in accordance with what has been learned.” Accordingly, on August 22, 2011, the Department issued its Plan for Retrospective Analysis of Existing Regulations. (See ed.gov/policy/gen/reg/retrospective-analysis/index.html).

Our plan identified a number of regulatory initiatives for retrospective review and analysis. One of those initiatives, already begun in 2010, was a review of the Department's discretionary grants process. Part of that initiative was a close retrospective review of the Education Department General Administrative Regulations (EDGAR), which govern discretionary grantmaking and administration.

As part of this retrospective review of EDGAR, we identified key provisions that required substantive changes to improve transparency and the efficiency and effectiveness of our grant-making functions. These included our regulations on establishing and collecting data on measures of grantee performance, the selection criteria that peer reviewers use to evaluate applications, and the procedures grantees must use to select research sites and evaluators. This notice is the result of the Department's regulatory review of those provisions.

On May 10, 2012, President Obama issued Executive Order 13610, “Identifying and Reducing Regulatory Burdens.” (77 FR 28469). Among other things and as part of their retrospective review, this order requires Federal agencies to invite “public suggestions about regulations in need of retrospective review and about appropriate modifications to such regulations.”

Therefore, in addition to your comments on the specific regulations proposed in this notice, we seek input on other regulations within EDGAR that may be in need of modification and amendments to those regulations that you would suggest. We are particularly interested in your feedback on the following questions:

• Are the regulations achieving their intended outcomes, e.g., do they establish a fair and equitable process for selecting applications for funding while ensuring transparency in the selection process and enhancing accountability for funding decisions?
• Have changes in the economy or other external factors led to an increase or decrease in costs imposed on applicants for, and recipients of, discretionary grants?
• Are any of the regulations outmoded, unnecessary, or out of date?
• Do the regulations cause confusion or create other questions? If so, how could we amend the regulations to address this problem?
• What do relevant data show about the effectiveness and benefits of the regulations in comparison to their costs?

Although the Department may or may not respond to comments that we receive on the retrospective review of these other provisions of EDGAR, we will use that feedback to further inform and plan our retrospective review efforts.

During and after the comment period, you may inspect all public comments about these proposed regulations by accessing Regulations.gov. You may also inspect the comments, in person, in Room 4W335, 400 Maryland Avenue SW., Washington, DC, between the hours of 8:30 a.m. and 4:00 p.m., Washington, DC time, Monday through Friday of each week except Federal holidays. Please contact the person listed under FOR FURTHER INFORMATION CONTACT.

Assistance to Individuals With Disabilities in Reviewing the Rulemaking Record

On request, we will provide an appropriate accommodation or auxiliary aid to an individual with a disability who needs assistance to review the comments or other documents in the public rulemaking record for these proposed regulations. If you want to schedule an appointment for this type of aid, please contact the person listed under FOR FURTHER INFORMATION CONTACT.

Summary of Proposed Changes

In this notice, the Secretary proposes amendments that would:

1. Allow the Secretary, in an application notice for a competition, to establish performance measurement requirements;

2. Revise requirements regarding project evaluations submitted to the Department by grantees;

3. Authorize applicants to use simplified procurement procedures to select implementation sites and procure services from implementation and evaluation service providers, but only if the site or service provider is named in the grant application;

4. Allow the Secretary, through an announcement in the Federal Register, to authorize grantees under particular programs to award subgrants to directly carry out programmatic activities. The subgrantees and programmatic activities must be identified and described in the grantees' applications;

5. Add one new selection criterion and revise two existing criteria that the Department could choose to use to evaluate applications. The new criterion would be used to assess the extent to which a proposed project could be brought to scale. We would add five new factors to the criterion “Quality of the Project Evaluation” that could be used to assess how well a proposed project evaluation would produce evidence about the project's effectiveness. Finally, we would revise one factor and add five new factors to the criterion “Quality of the Project Design”;

6. Authorize program offices to consider the effectiveness of proposed projects under a new priority that could be used as either an absolute, competitive preference, or invitational priority; and

7. Allow the Secretary to fund data collection periods after the end of the substantive work of a project so that project outcomes could be assessed using data from the entire project period.

As discussed in more detail later in this notice, the proposed changes would strengthen the Secretary's authority to: (a) Evaluate grantee performance; (b) provide applicants and grantees with greater flexibility in selecting implementation sites, implementation partners, and evaluation service providers; (c) allow the Secretary to authorize subgrants for particular programs; (d) improve the targeting of selection criteria and factors so that applicants are better informed and able to focus their application narratives on specific program objectives; and (e) allow consideration of the strength of evidence supporting the proposed project when evaluating grant applications.

Significant Proposed Regulations

We group major issues according to subject, with appropriate sections of the proposed regulations referenced in parentheses. We discuss other substantive issues under the sections of the proposed regulations to which they pertain.

Generally, we do not address proposed regulatory provisions that are technical or otherwise minor in effect.

I. Performance Measurement

Background

Congress passed the Government Performance and Results Act of 1993 (GPRA) (Pub. L. 103-62) in order to hold Federal agencies accountable for achieving program results. Under GPRA, agencies are required to report to Congress on the effectiveness of the programs they administer, based on performance measures established for those programs.

The purposes of GPRA are to improve Federal program effectiveness and accountability to the public by: Focusing on results, service quality, and customer satisfaction; giving Federal program managers information about program results and service quality; and providing objective information to Congress and the public on the relative effectiveness and efficiency of Federal programs and spending. The GPRA Modernization Act of 2010 (Pub. L. 111-352) supports additional improvements in Federal agencies' performance planning and reporting. Federal agencies are required to make their strategic and annual plans publicly available and post quarterly updates via a central, Government-wide Web site. The goal of the GPRA Modernization Act is to improve the use of data in policy, budget, and management decision-making.

GPRA requires Federal agencies to establish performance measures and targets for programs they administer and to report annually to the Office of Management and Budget (OMB) on the extent to which those programs are meeting their targets. For discretionary (non-formula) grant programs, the Department establishes performance measures to address the extent to which the program as a whole is effective in achieving its goals through the projects it funds. However, we have found that grantees' performance data do not consistently correspond to overall program performance measures because grantees typically only report on and measure data related to project-specific outcomes.

The Secretary therefore proposes the following amendments to improve the Department's ability to collect reliable, valid, and meaningful data for evaluating the outcomes of Department programs and the performance of individual grantees.

Proposed Regulatory Changes

34 CFR Part 75

Section 75.110 Information Regarding Performance Measurement

Current Regulations: None.

Proposed Regulations: Proposed § 75.110 would allow the Secretary to establish performance measurement requirements in an application notice for a competition. These requirements could include performance measures, baseline data, performance targets, and performance data. This proposed section would also allow the Secretary to establish in an application notice a requirement that applicants propose performance measures for their projects, as well as the baseline data and performance targets for each proposed measure.

Reasons: To improve the likelihood that grantees collect and report data that effectively measure the outcomes of each grant, the proposed amendments would allow the Secretary to require discretionary grant applicants to include program-level and project-specific performance measures, baseline data, and targets in their applications. Requiring this information improves the ability of the Department to measure program effectiveness under GPRA performance measures, clarifies that grantees will be required to report on their project-specific performance measures, and stresses that the extent to which grantees meet performance targets will be considered in making continuation grants.

II. Procurement and Subgrant Processes for Entities Named in Applications

Proposed Regulatory Changes

34 CFR Part 75

Background:

From our experience, many applicants find it useful to describe elements of their proposed evaluations in their applications, including implementation sites or the provider that would conduct the project evaluation should the proposed project be funded. This information is often an important factor in the Department's peer review of discretionary grant applications, particularly in instances when the quality of the project evaluation is a selection criterion.

The Department's procurement regulations in 34 CFR 74.43 and 34 CFR 80.36(c) provide that a grantee must conduct its procurement transactions in a manner that provides, to the maximum extent practical, full and open competition. This requirement is intended to ensure that grantees consider contractor performance objectively and offer an opportunity for providers to compete for the contract. While the Department values full and open competition, the Department also recognizes that this requirement presents challenges for applicants whose applications would be strengthened by including details about the implementation sites and the evaluation service provider. The Secretary proposes to reduce this burden by simplifying the procurement process used to select implementation sites, implementation partners, and evaluation service providers.

Section 75.135 Competition Exception for Implementation Sites, Implementation Partners, or Evaluation Service Providers

Current Regulations: There is no current § 75.135. The Department's procurement regulations in 34 CFR 74.43 and 34 CFR 80.36(c) provide that a grantee must conduct its procurement transactions in a manner that provides full and open competition. In many cases, grantees must use formal competition procedures to select contractors. Under these current provisions, an applicant for a grant requiring an evaluation would need, in many cases, to conduct a formal bidding process to select implementation sites, implementation partners, or evaluation service providers before submitting its application to the Department or following award of the grant. These types of procurement requirements can be very costly and time consuming at a time when the applicant cannot be sure it will be selected for a grant. Because the selection of implementation sites or partners and evaluation service providers is often an important factor in designing a project and submitting a high-quality application, we propose an exception to the Department's procurement regulations for entities named in a grant application.

Proposed Regulations: The Secretary proposes to add a new § 75.135 that would exempt certain applicants from the full competitive contracting requirements in 34 CFR 74.43 and 80.36(c). Specifically, an applicant for a grant that must be conducted at multiple sites or that requires an external evaluation would not be required to comply with the applicable formal competition requirements in 34 CFR 74.43 and 80.36(c) when entering into a contract if—

(1) The contract is with an entity that agrees to provide a site or sites where the applicant would conduct the project activities under the grant or the contract is with the evaluation service provider that would conduct the project evaluation;

(2) The implementation sites, implementation partners, or evaluation service providers are identified in the application for the grant; and

(3) The implementation sites, implementation partners, or evaluation service providers are included in the application in order to meet a regulatory, statutory, or priority requirement related to the competition.

A successful applicant would need to certify that any employee, officer, or agent participating in the selection, award, or administration of a contract is free of any real or apparent conflict of interest.

In the case of a contract for a provider to conduct the project evaluation, the proposed amendment would permit the applicant or grantee to use the informal competition requirements for small purchases that are currently applicable only to governments under 34 CFR 80.36(d)(1), regardless of whether the applicant or grantee is a government entity and regardless of whether the purchase meets the small purchase threshold.

During the course of the grant, a successful applicant would be required to obtain the Department's permission to change any implementation site, implementation partner, or evaluation service provider that the applicant specified in the application and selected under proposed § 75.135. The exception also would not relieve an applicant of the obligation to conduct an informal review of evaluation service providers in order to determine the best available provider or from its obligations under the Department's other procurement requirements.

A successful applicant that does not meet the three criteria above would not be exempt from complying with the applicable formal competition requirements in 34 CFR 74.43 and 80.36(c) when entering into a contract. For example, an applicant that does not identify its implementation sites, implementation partners, or evaluation service provider in its application would be required to comply with the applicable formal competition requirements in 34 CFR 74.43 and 80.36(c).

Reasons: This proposed new section addresses a difficulty many applicants face when selecting implementation sites, partners, and evaluation service providers prior to submitting their applications. Requiring grantees to use formal competitive procedures to select implementation sites and partners could significantly diminish both the ability of many applicants to compete for grants and the quality of project evaluations. For example, without this proposed regulation, a successful applicant would be limited in its ability to select implementation sites that include specific populations that it proposed to serve through the project or to work with the evaluation service provider that assisted in designing the applicant's evaluation plan.

Formal competition requirements also inhibit the ability of many applicants to select evaluators who would work with the applicants to design project evaluation plans. Some of the best evaluations of projects may be conducted by evaluation service providers that are involved in the initial design of a project. Such work generally takes place during the development of an application, before the applicant knows whether it will receive a grant. Thus, requiring an applicant to hold a formal competition involving sealed bids or competitive proposals in order to select an evaluation service provider (either before or after it receives a grant) can have major negative consequences. For example, an evaluation service provider would be excluded from the competition to select the project evaluator under the procurement requirements in parts 74 and 80 if it helped prepare an application and helped the applicant set up the standards used to select an evaluation service provider or contractor (see 34 CFR 74.42, 74.43, and 80.36). High-quality evaluation of a project funded by the Department may be hindered if an evaluation service provider that designed the evaluation strategy for an application is excluded from the evaluation procurement competition for that project. Given the uncertainty of the competitive process, the Secretary also believes that applicants should not be required to use formal competition procedures to select an evaluation service provider at the time they prepare their applications.

While the Secretary proposes to remove the competition requirement for selecting sites and implementation partners and thus permit applicants and grantees to use informal procedures instead, the Secretary would continue to require all applicants to comply with the other procurement requirements in parts CFR 74 and 80, including the requirements for cost price analysis, standards of conduct, conflicts of interest, and the prohibition of contingent payment for services. Additionally, the proposed amendment does not supersede any State laws regarding procurement.

Finally, based on the other procurement requirements in CFR parts 74 and 80, these exceptions would not relieve an applicant of its responsibility to document that it made genuine efforts to select the best implementation sites, implementation partners, or evaluation service providers for the project, considering qualifications, capabilities, availability, price, and other important factors.

§ 75.708 Prohibition on Subgrants.

Current Regulations: Section 75.708(a) prohibits grantees from awarding subgrants unless specifically authorized by statute.

Proposed Regulations: The Secretary proposes to revise the prohibition on subgrants in § 75.708(a) to allow subgrants when authorized by statute or as provided for by a new § 75.708(b). Under this proposed new § 75.708(b), the Secretary could, through an announcement in the Federal Register, authorize subgrants when necessary to meet the purposes of a particular program. In addition, the Federal Register announcement would identify the types of entities (e.g., State or local educational agencies, institutions of higher education, or non-profit organizations) that could receive subgrants under the program.

We would add § 75.708(c) to provide that subgrants, if authorized under § 75.708(b), could be awarded to entities identified in a grantee's application. The subgrant must be used to directly carry out activities described in the application.

We would add a new § 75.708(d), which would establish requirements grantees would have to follow in awarding subgrants authorized under § 75.708(b). We would re-designate the current § 75.708(b) as § 75.708(e).

Reasons: The revision of § 75.708(a) is necessary to provide grantees with flexibility to work with partners or other entities to carry out project activities. The prohibition on subgrants, in conjunction with the requirement on full and open competition for procurement transactions in 34 CFR 74.43 and 34 CFR 80.36(c), unduly restricts grantees from working with partners or other entities identified in their applications as being directly responsible for carrying out project-related activities.

In order to ensure appropriate subgranting by Department grantees, our proposed revision authorizes subgrants only when approved by the Secretary for a particular program and only to the types of entities (e.g., State or local educational agencies, institutions of higher education, or non-profit organizations) designated by the Secretary. In addition the proposed revision would limit the entities that may receive subgrants to those that: (1) Are identified in a grantee's application, or (2) are competitively selected using the grantee's procedures for selecting subgrants and, (3) will use the subgrant directly to carry out project activities described in the grantee's application. In all cases where a grantee is working with an organization or entity that is not identified in its application, not selected through a competitive process, or not an organization or entity directly responsible for carrying out an activity or activities described in the grantee's application, the grantee would be required to follow the procurement procedures set out in 34 CFR Parts 74 and 80. Additionally, the grantee—as the fiscal agent—would remain responsible to the Department for the proper use of all grant funds, including those subgranted to another entity.

In addition, we would add a new § 75.708(d) requiring grantees to ensure that: (1) Subgrants are awarded on the basis of an approved budget that is consistent with the grantee's approved application and all applicable Federal statutory, regulatory, and other requirements; (2) subgrants include all conditions required by Federal law; and (3) subgrantees are aware of requirements imposed upon them by Federal law, including the Federal anti-discrimination laws enforced by the Department.

This revision provides grantees, in programs and to entities designated by the Secretary, with the flexibility to award subgrants in specific circumstances where necessary to ensure proper implementation of an approved project without diminishing accountability for Federal funds or project outcomes.

III. Selection Criteria

Background

The regulations in subpart D of 34 CFR part 75 set forth the general requirements that govern the Department's selection of grantees for direct grant awards. For those direct grant programs that make discretionary grant awards, the Secretary uses selection criteria to evaluate applications submitted under those programs. The regulations specify certain selection criteria from which the Secretary may choose (general EDGAR criteria). They allow the Secretary to use program-specific selection criteria and the general EDGAR selection criteria, as well as to develop other criteria based on the statutory provisions for the funding program. However, some program regulations currently do not provide that the Secretary may use program-specific selection criteria in conjunction with EDGAR and statutory criteria. The regulations also describe how the Secretary determines which criteria and which factors within those criteria are used in a particular competition and how the Secretary may weight the criteria and factors.

As we have managed competitions under the general regulations governing selection criteria, we have found that some of the regulations on the selection of grantees do not provide the Department the discretion it needs, absent a lengthy rulemaking process, to conduct grant competitions closely aligned with Department, legislative, and program objectives and priorities that can change from year to year in response to new and unanticipated circumstances. These proposed regulations, therefore, would provide the Department additional flexibility to establish criteria based on program regulations, in addition to the current authority to base criteria on statutory provisions. The proposed regulations would also specifically authorize program offices to establish additional selection criteria in § 75.210 based on statutory and regulatory provisions.

These proposed regulations would also add new selection factors under the “Quality of Project Design” criterion on organizational and programmatic sustainability and organizational productivity. The proposed regulations would also add to the “Quality of the Project Evaluation” criterion five new selection factors on the types of evidence the evaluation designs would produce on the performance and implementation of the project. Finally, the proposed regulations would establish a new criterion to evaluate the extent to which an applicant proposes a project that could be brought to scale.

The addition of these selection factors would ensure that the Department's discretionary grant programs would more effectively promote the development and implementation of effective and sustainable practices, and support adoption and implementation of necessary reforms. These proposed regulations would not change the way the Secretary uses the current and new selection criteria and factors. The Secretary would continue to use those selection criteria and factors that are consistent with the purpose of the program and permitted under the applicable statutes and regulations.

Proposed Regulatory Changes

34 CFR Part 75

Section 75.209 Selection Criteria Based on Statutory or Regulatory Provisions

Current Regulations: Current § 75.209 provides that the Secretary may evaluate applications by establishing selection criteria based on the statutory provisions for the authorized program. These provisions include, but are not limited to, those related to specific statutory selection criteria, allowable activities, application content requirements, and other pre-award and post-award conditions.

Proposed Regulations: We propose to revise § 75.209 to allow the Secretary to use selection criteria, the factors in program regulations, and those based on program statute, along with the selection criteria in § 75.210 (often referred to as the EDGAR selection criteria) to produce more focused selection criteria. Thus, § 75.209 would allow the Secretary to establish selection criteria, and factors for considering those criteria, based on statutory or regulatory provisions that apply to the authorized program, which may include, but are not limited to, criteria and factors that reflect:

• Criteria contained in the program statute or regulations;
• Criteria in § 75.210;
• Allowable activities specified in the program statute or regulations;
• Application content requirements specified in the program statute or regulations;
• Program purposes, as described in the program statute or regulations; or
• Other pre-award and post-award conditions specified in the program statute or regulations.

Reasons: The Secretary proposes amending this section so that the Department can establish selection criteria based both on a program's statute and regulations. Program regulations are used to help clarify and fill in the gaps of more general statutory requirements and provide further detail about authorized activities for a program.

Under this proposed amendment, the Secretary would be able to use the more specific regulatory provisions to establish selection criteria that are focused more closely on the intended outcomes of the competition and, thereby, help applicants to structure their applications so as to more accurately and concisely describe how they will achieve those outcomes. In addition to providing for establishment of criteria based on program regulations, this amendment would allow the Secretary to use a combination of criteria from the program statute, its established regulations, or the general selection criteria in § 75.210.

§ 75.210 

General Selection Criteria

Current Regulations: Current § 75.210 contains a list of eight selection criteria: “Need for Project” in paragraph (a); “Significance” in paragraph (b); “Quality of the Project Design” in paragraph (c); “Quality of Project Services” in paragraph (d); “Quality of Project Personnel” in paragraph (e); “Adequacy of Resources” in paragraph (f); “Quality of the Management Plan” in paragraph (g); and “Quality of the Project Evaluation” in paragraph (h). Under each of these selection criteria, the Secretary may select from a number of factors to focus each criterion.

Proposed Regulations: The Secretary proposes to revise the introductory paragraph of § 75.210, add selection factors to the criteria in § 75.210(c) and (h), and add a new criterion as paragraph (i) to address the ability of an applicant to bring a project to scale.

Introductory Text: We propose to amend the introductory paragraph of § 75.210 so that the Secretary may select factors that could be considered under a criterion both from the factors listed under that criterion and factors listed under other criteria. For example, the proposed amendment would allow the Secretary to establish “Quality of the Project Design” as a selection criterion and include selection factors from “Need for Project” (§ 75.210(a)) or “Significance” (§ 75.210(b)) in the factors that will be considered under the “Quality of the Project Design” criterion. Currently, to use a single selection factor under the “Need for Project” criterion, the Department, in most cases, would need to include the “Need for Project” criterion, even if the factor in question could be appropriately grouped with factors from another selection criterion, such as “Significance.”

Reasons: The purpose of this amendment is to provide the Secretary with the flexibility to choose and combine selection factors established in § 75.210 under various selection criteria. This would enable the Department to align the selection criteria and factors with the goals and objectives of a particular discretionary grant competition in a more coherent and effective fashion than is currently permitted. Selection criteria and factors that are concise and are aligned as closely as possible with the goals and objectives of a particular grant competition would more effectively guide applicants in preparing clearer and more focused applications that in turn can be more effectively evaluated and rated by peer reviewers. The current regulations, by contrast, do not allow this close focus. Including a greater number of selection criteria in application notices, solely to include particular selection factors, makes it more likely that applications will be less focused and more difficult for peer reviewers to accurately evaluate and score.

Current Regulations: Section 75.210(c) establishes the selection criterion “Quality of the Project Design.” The Secretary may consider one or more of the 23 factors listed under this criterion in determining the quality of the project design, including the extent to which the project design will build capacity that extends beyond the project period and establish linkages to services provided by other programs.

Proposed Regulations: The Secretary proposes to add new factors to the criterion in paragraph (c), (xxiv and xxv) relating to the sustainability of the proposed project after the end of the project period.

Reasons: Adding these selection factors would help ensure that the Department's discretionary grant programs effectively promote the development and implementation of effective and sustainable practices and support adoption and implementation of necessary reforms. By promoting the development of a multi-year plan for incorporation into the applicant's ongoing work, the proposed factors would better encourage applicants to develop sustainability plans than do the related selection factors in current § 75.210(c). The proposed factors also would allow the Secretary to consider a proposed project's potential for sustainability over time, including the extent to which the project has the support of various stakeholders and adequate resources to continue the project after the grant period ends.

Proposed Regulations: The Secretary proposes to revise § 75.210(c) to add a new selection factor (xxvi) regarding the extent to which the proposed project will increase efficiency in the use of time, staff, money, or other resources in order to improve results and increase productivity.

Reasons: Current § 75.210(c) does not include a factor that promotes increased productivity. Considering the budget challenges that State and local educational agencies, institutions of higher education, non-profit organizations, and other entities working in education face during economic downturns, and given the potential for new knowledge and capabilities to improve efficiency, the Department believes that it is appropriate to consider the potential for increasing productivity, i.e., the extent to which a proposed project includes a strategy to make more efficient use of time, money, and staff, when assessing an application.

Proposed Regulations: The Secretary proposes to revise § 75.210(c)(xvi) to read “The extent to which the proposed project will integrate with or build on similar or related efforts to improve relevant outcomes, using existing funding streams from other programs or policies supported by community, State, and Federal resources.”

In addition, the Secretary proposes to add a new selection factor (xxvii) regarding the extent to which the proposed project will integrate with or build on similar or related efforts in order to improve relevant outcomes, using nonpublic funds or resources.

Reasons: Given the budget challenges facing State and local educational agencies, institutions of higher education, and other entities working in education, there is a need for strategies and practices to improve relevant outcomes while controlling costs. Moreover, “silos” within and between agencies at the local, State, and Federal levels often impede program integration and result in less efficient and effective efforts. The purpose of revising this selection factor and adding a factor focused on nonpublic investments is to improve levels of program integration, to facilitate shared agendas for actions focused on common outcomes, and to leverage public and private sector investments in education. In addition, the Department believes that leveraging existing programs and policies that are supported by other funds, including other Federal, State, local, or private funds, increases the likelihood that selected projects will be sustained beyond the grant period.

Proposed Regulations: The Secretary proposes to add two new selection factors (xxviii and xxix) regarding the extent to which the proposed project is supported by evidence of promise or strong theory. Later in this notice, we propose adding definitions to Part 77, including evidence of promise, strong theory, and other terms to ensure consistent understanding of the selection factors we propose in this notice.

Reasons: The Department recognizes that at the various stages of a proposed project's development, different types of evidence are available to assess the effectiveness of a project. The proposed selection factors would permit the Secretary to use strength of evidence as a selection factor in determining the projects the Department will fund while maintaining the flexibility to consider a wider variety of studies or data an applicant might present that is appropriate to the goals of the project. The flexibility provided by the proposed selection factor would be particularly beneficial for innovative areas where strong or moderate evidence of effectiveness is not yet available because it would allow the Secretary to consider strength of evidence appropriate to a project's stage of development.

Current Regulations: Section 75.210(h) establishes the selection criterion “Quality of the Project Evaluation.” The Secretary may consider one or more of the seven factors listed under this criterion in determining the quality of the project evaluation design, such as the extent to which the project proposes feasible and appropriate evaluation methods, uses objective performance measures, and permits periodic assessment.

Proposed Regulations: The Secretary proposes to revise § 75.210(h) to add five new selection factors. Two of the new selection factors address the extent to which the methods of evaluation will, if well-implemented, produce evidence about the project's effectiveness that would meet the What Works Clearinghouse Evidence Standards.^[1] The other three proposed selection factors address the extent to which the methods of evaluation will produce evidence of promise about the grant-supported intervention, valid and reliable performance data on relevant outcomes of the project, and the extent to which the evaluation plan articulates key components as well as measureable thresholds for acceptable implementation of the project.

Reasons: Although current § 75.210(h) includes selection factors regarding proposed evaluation methods, it does not include a selection factor that promotes use of the strongest possible study designs for estimating a program's effect or a selection factor that assesses the extent to which the proposed evaluation will articulate information that can be used to assess whether the project was implemented with fidelity.

Linking two of the proposed new selection factors to the What Works Clearinghouse Evidence Standards ^[2] reflects the predominant view among research experts that the randomized controlled trial (also referred to as an experimental design study) is the most rigorous and defensible method for producing unbiased evidence of project effectiveness. Random assignment of entities (students, teachers, schools, or other units of analysis) to a treatment or control group is the most effective way to eliminate plausible competing explanations for observed differences between treated and non-treated individuals or groups (i.e., the estimated treatment effect). Adding these selection factors will allow the Secretary to consider the extent to which applicants propose evaluations that will contribute to a strong body of evidence on the effectiveness of the proposed project.

Additionally, the other three proposed selection factors allow the Secretary to consider evaluation methods that will produce data on a project's evidence of promise, performance on relevant outcomes, and fidelity of implementation. Each of these factors would improve the Department's ability to assess evaluation plans for projects at various stages of their development.

Current Regulations: None.

Proposed Regulations: The Secretary proposes to add selection criterion § 75.210(i), “Strategy to Scale.” “Scale” refers to expanding the use or implementation of a proposed practice, strategy, or program to provide services at a State, regional, or national level while maintaining the demonstrated effectiveness of the approach. Under the proposed new criterion, the Secretary would consider the applicant's strategy to scale the proposed project. In determining the applicant's strategy to scale the proposed project, the Secretary would consider one or more factors, including the extent to which the applicant's strategy to scale addresses a particular barrier or barriers that prevented the applicant, in the past, from reaching the level of scale that is proposed in the application; the applicant's capacity (e.g., in terms of qualified personnel, financial resources, or management capacity) to bring the proposed project to scale; and the extent to which the applicant demonstrates there is unmet demand for the proposed project that will enable the applicant to reach the level of scale that is projected in the application. In addition, the Secretary could consider the feasibility of replicating the project and the mechanisms for broadly disseminating information on the project so as to support further development or replication.

Reasons: It is important that successful best practices be shared and implemented more broadly. The addition of this selection criterion would allow the Secretary to consider the proposed scaling methodology and the feasibility of successfully replicating the proposed project in a variety of settings and with other populations. The proposed selection criterion would allow the Department to consider whether applicants have the potential to serve more groups in a variety of settings, which would be important in estimating the likelihood of a proposed project's success at scale and in considering applications for activities that include broad sharing of best practices. Additionally, Department programs could use the proposed criterion, in conjunction with the proposed priority regarding evidence of effectiveness, to encourage the field to focus its attention and resources on projects that are effective.

IV. Evidence of Effectiveness

Background

To support effective projects and provide incentives to the field for building an evidence base on the effectiveness of interventions, the Secretary proposes a priority for projects that can cite and build upon an existing base of strong or moderate evidence of effectiveness. This priority would be a critical part of the Department's efforts to fund and increase the use of programs with evidence of effectiveness.

Section 75.226 Consideration of Applications Supported by Strong or Moderate Evidence of Effectiveness

Current Regulations: None.

Proposed Regulations: The Secretary proposes to establish procedures for giving special consideration to applications supported by strong or moderate evidence of effectiveness. Proposed § 75.226 would establish that if the Secretary determines to give special consideration to applications supported by strong or moderate evidence of effectiveness for a particular grant competition, the Secretary could either establish a separate competition or give a competitive preference to applications supported by strong or moderate evidence of effectiveness under the procedures in 34 CFR 75.105(c)(2).

Reasons: By expanding the number of Department programs awarding grants to those projects supported by strong or moderate evidence of effectiveness, the Department could better ensure that discretionary grant funds are used to support effective interventions and activities.

V. Program Budgets

Background

So that the Department can learn as much as possible from successful discretionary grants and its programs as a whole, we propose amendments regarding budget periods. We would:

• Establish that a project may receive an extension of the funding period for the purpose of collecting, analyzing, and reporting performance data;
• Clarify that a multi-year data collection may be funded through separate budget periods; and
• Clarify that any information relevant to the grantee's performance during the project period should be considered when determining whether a grantee receives a continuation award.

Proposed Regulatory Changes

34 CFR Part 75

Section 75.250 Maximum Funding Period

Current Regulations: Current § 75.250 is titled “Project period can be up to 60 months.” This section provides that the Secretary may approve a project period of up to 60 months, but it does not specifically authorize funding grants for periods longer than 60 months. Other regulations in part 75 prohibit the use of Federal funds for projects extending past 60 months. See current § 75.261, which addresses the circumstances under which a grantee may request a no-cost extension of a project period.

Proposed Regulations: We propose to amend § 75.250 to provide that the Secretary may approve a data collection period of up to 72 months—if not inconsistent with any statutory limits on the grant award period—after the end of the project period and provide funding during this period for the sole purpose of collecting, analyzing, and reporting data regarding project performance. During a data collection period, a grantee could use the funds only for data collection, analysis, and reporting purposes. Section 75.250(b)(2) would give the Secretary discretion to notify applicants in the notice inviting applications for a competition or later, after grantees have started their projects of the intent to fund data collection periods.

Given these proposed changes, the Secretary also proposes to change the title of this section to “Maximum funding period.”

Reasons: It is the Department's experience that the effectiveness of a project cannot always be determined on the date that the substantive work of the project is complete. For example, a four-year project designed to increase the ability of certain high school students to successfully complete college may require data collection for up to six years after the date the substantive work of the project ends. With the discretion to approve a data collection period after the end of a project period and offer continued funding for data collection, the Department could ensure that performance data are collected and are used to evaluate both the project and program performance. The Secretary would expect to fund any data collection period of a grant at a much lower level than the original substantive work of the grant.

Section 75.251 The Budget Period

Current Regulations: Current § 75.251 describes how the Secretary may fund multi-year projects through separate budget periods, generally of 12 months each.

Proposed Regulations: The Secretary proposes to add a new paragraph (c) to this section to clarify that multi-year data collection periods may be funded through budget periods in the same manner as project periods are funded.

Reasons: We are proposing to revise § 75.251 to correspond to the proposed revisions to § 75.250.

Section 75.253 Continuation of a Multi-Year Project After the First Budget Period

Current Regulations: Under current § 75.253(a), a grantee may only receive a continuation award if the grantee has met certain requirements, including the requirement that the grantee make substantial progress toward the objectives of the grant. If a grantee does not make substantial progress, it must obtain permission from the Department to make changes to the project that would help the grantee make substantial progress during the remainder of the project period.

Proposed Regulations: The Secretary proposes to amend § 75.253 by adding a new paragraph (b) to clarify that in deciding whether to make a continuation grant, the Secretary could consider any information relevant to the grantee's performance during the project period. This could include information relevant to the authorizing statute, a criterion, a priority, or a performance measure, or any financial or other requirement that applied to the selection of applications for new awards. While this proposed standard for granting continuation awards is implicit under the current regulations, the Secretary believes that this standard should be explicit so that grantees have a clearer understanding of how the Department decides to make a continuation award.

In addition, we propose to amend paragraph (a)(2) so that in making continuation awards, the Secretary could consider not only the extent to which a grantee has made substantial progress in achieving the goals and objectives of the project, but also whether a grantee met the performance targets in the approved application, if the Secretary established performance measurement requirements for the grant in the application notice. If a grantee fails to meet these targets, proposed paragraph (a)(2) would require the grantee to obtain the Secretary's approval for changes to the project that enable the grantee to achieve the project's goals, objectives, and performance targets, if any, without changing the project scope or objectives. The Secretary would retain the requirement in the current regulation that any such changes may not increase the amount of funds obligated to the project by the Department.

Reasons: Current § 75.253 does not describe the standards used to determine whether a grantee has made substantial progress on its grant. Therefore, we propose these amendments to clarify the standards that the Department considers when determining whether a grant will receive a continuation award. The proposed amendments would establish that the Secretary may also consider whether a grantee has met the performance targets in its approved application when making continuation awards.

Section 75.590 Evaluation by the Grantee

Current Regulation: Current § 75.590 requires a grantee to submit performance reports to the Department that evaluate at least annually the grantee's progress in achieving the objectives in its approved application, the effectiveness of the project in meeting the purposes of the program, and the effect of the project on participants being served by the project. This provision does not currently provide any standards for evaluating the progress in achieving performance targets.

Proposed Regulation: The Secretary proposes to revise § 75.590 to add a new paragraph (a) to provide that if an application notice for a competition requires applicants to describe how they would evaluate their projects, any evaluation must meet the standards set in the approved application for the project. The performance measurement data collected by the grantee and used in the evaluation must meet the performance measurement requirements in the approved application.

We also propose to designate the current regulatory text in § 75.590 as new paragraph (b) and revise that text to conform to the other changes we are proposing regarding performance measurement. Specifically, we propose that if the application notice for a competition did not require an applicant to submit an evaluation plan, the grantee must provide information in its performance report to the Department demonstrating (1) The progress made by the grantee in the most recent budget period; (2) the effectiveness of the project; and (3) the effect of the project on the participants served by the project. If the application notice required applicants to propose how they would meet performance requirements, the performance report would also need to address the extent to which the grantee met the project's performance targets and other performance measurement requirements for the budget period addressed by the performance report.

Reasons: The proposed revisions to § 75.590 strengthen the Department's authority to monitor the quality of grantees' project evaluations. Additionally, these revisions complement other proposed regulations in this notice regarding performance measurement requirements.

VI. Definitions

Background

These proposed regulations include references to terms that are not currently defined in EDGAR. To ensure a common understanding of these terms, we propose establishing the following definitions.

Proposed Regulatory Changes

34 CFR Part 77

Section 77.1 Definitions That Apply to All Department Programs

Current Regulations: Section 77.1(c) establishes definitions that unless a statute or regulation provides otherwise, apply to parts 34 CFR 74 and 80.

Proposed Regulations: The Secretary proposes to incorporate the definitions for the following terms into § 77.1(c): “ambitious,” “baseline data,” “evidence of promise,” “large sample,” “logic model,” “moderate evidence of effectiveness,” “multi-site sample,” “national level,” “performance measure,” “performance target,” “randomized controlled trial,” “regional level,” “relevant outcome,” “quasi-experimental study,” “strong evidence of effectiveness,” and “strong theory.”

Reasons: The Secretary proposes establishing these definitions to ensure consistent understanding of the selection factors and priority we propose in this notice.

Executive Orders 12866 and 13563

Regulatory Impact Analysis

Under Executive Order 12866, the Secretary must determine whether this regulatory action is “significant” and, therefore, subject to the requirements of the Executive order and subject to review by the Office of Management and Budget (OMB). Section 3(f) of Executive Order 12866 defines a “significant regulatory action” as an action likely to result in a rule that may—

(1) Have an annual effect on the economy of $100 million or more, or adversely affect a sector of the economy, productivity, competition, jobs, the environment, public health or safety, or State, local, or tribal governments or communities in a material way (also referred to as an “economically significant” rule);

(2) Create serious inconsistency or otherwise interfere with an action taken or planned by another agency;

(3) Materially alter the budgetary impacts of entitlement grants, user fees, or loan programs or the rights and obligations of recipients thereof; or

(4) Raise novel legal or policy issues arising out of legal mandates, the President's priorities, or the principles stated in the Executive Order.

This proposed regulatory action is a significant regulatory action subject to review by OMB under section 3(f) of Executive Order 12866.

We have also reviewed these regulations under Executive Order 13563, which supplements and explicitly reaffirms the principles, structures, and definitions governing regulatory review established in Executive Order 12866. To the extent permitted by law, Executive Order 13563 requires that an agency—

(1) Propose or adopt regulations only upon a reasoned determination that their benefits justify their costs (recognizing that some benefits and costs are difficult to quantify);

(2) Tailor its regulations to impose the least burden on society, consistent with obtaining regulatory objectives and taking into account—among other things and to the extent practicable—the costs of cumulative regulations;

(3) In choosing among alternative regulatory approaches, select those approaches that maximize net benefits (including potential economic, environmental, public health and safety, and other advantages; distributive impacts; and equity);

(4) To the extent feasible, specify performance objectives, rather than the behavior or manner of compliance a regulated entity must adopt; and

(5) Identify and assess available alternatives to direct regulation, including economic incentives—such as user fees or marketable permits—to encourage the desired behavior, or provide information that enables the public to make choices.

Executive Order 13563 also requires an agency “to use the best available techniques to quantify anticipated present and future benefits and costs as accurately as possible.” The Office of Information and Regulatory Affairs of OMB has emphasized that these techniques may include “identifying changing future compliance costs that might result from technological innovation or anticipated behavioral changes.”

We are issuing these regulations only on a reasoned determination that their benefits justify their costs. In choosing among alternative regulatory approaches, we selected those approaches that maximize net benefits. Based on the analysis that follows, the Department believes that these regulations are consistent with the principles in Executive Order 13563.

We also have determined that this regulatory action would not unduly interfere with State, local, and tribal governments in the exercise of their governmental functions.

In accordance with both Executive orders, the Department has assessed the potential costs and benefits of this regulatory action. The potential costs associated with this regulatory action are those resulting from statutory requirements and those we have determined as necessary for administering the Department's programs and activities.

Summary of Potential Costs and Benefits

Under the proposed regulations, applicants would have to use performance measures, baseline data, and performance targets established by the Department or establish their own performance measures, and determine baseline data performance targets for each performance measure. Although these proposed regulations would explicitly require such determinations and data collections, these requirements are implicit under the current regulations and grantees are already required to report on the extent to which they are meeting performance targets under the performance report ED 524B, which is approved under OMB control number 1894-0003. Therefore, we do not expect an increase in reporting burden on grantees under the proposed amendments.

The benefits of the proposed regulations would be that the Department would have explicit authority to collect meaningful data that we could use to assess the success of individual projects and report to Congress and OMB about the success of Department programs in achieving their legislative objectives. The ability to determine the success of Department programs could help improve the effectiveness of Department programs, without imposing additional costs on grantees or other parties.

The proposed regulations would also permit the Department to provide an exception for certain applicants from the full competitive contracting requirements in 34 CFR 74.43 and 80.36(c) for a grant that requires an external evaluation. Additionally, the proposed regulations would remove the prohibition on subgrants and allow for subgrants to any entity that is identified in a grantee's application and uses the subgrant directly to carry out activities described in the application. This action would reduce costs and increase benefits.

The benefits are that the proposed rule would remove a barrier for these grantees to contracting with the same evaluator both in the grant application stage and after receiving a grant award (and similarly, to selecting evaluation sites and implementation partners both pre-grant award and post-award), and thereby potentially enhance the quality of these projects. At the same time the proposed regulations would relieve grantees of the costs of administering competitions without reducing accountability or increasing the risk of improper use of or accounting for grant expenditures.

Additionally, under the proposed regulations, the Department would have greater flexibility in conducting grant competitions to use selection criteria that (1) are closely aligned with program objectives and priorities, and (2) promote reform objectives related to project evaluation, sustainability, productivity, and capacity to scale. This change would benefit applicants as well as the Department because it allows the Secretary to establish selection criteria that are concise and closely aligned with the goals and objectives of a particular grant competition and are focused more closely and coherently on the intended outcomes of the competition. The regulations would generate these benefits without increasing the costs for applicants, grantees, or the Department that already exist for creating and reviewing grant applications.

Elsewhere in this section under the heading Paperwork Reduction Act of 1995, we identify and explain burdens specifically associated with information collection requirements.

Clarity of the Regulations

Executive Order 12866 and the Presidential memorandum on “Plain Language in Government Writing” require each agency to write regulations that are easy to understand.

The Secretary invites comments on how to make these proposed regulations easier to understand, including answers to questions such as the following:

• Are the requirements in the proposed regulations clearly stated?
• Do the proposed regulations contain technical terms or other wording that interferes with their clarity?
• Does the format of the proposed regulations (grouping and order of sections, use of headings, paragraphing, etc.) aid or reduce their clarity?
• Would the proposed regulations be easier to understand if we divided them into more (but shorter) sections? (A “section” is preceded by the symbol “§ ” and a numbered heading; for example, § 75.210 General selection criteria.)
• Could the description of the proposed regulations in the SUPPLEMENTARY INFORMATION section of this preamble be more helpful in making the proposed regulations easier to understand? If so, how?
• What else could we do to make the proposed regulations easier to understand?

To send any comments that concern how the Department could make these proposed regulations easier to understand, see the instructions in the ADDRESSES section of this preamble.

Regulatory Flexibility Act Certification

Paperwork Reduction Act of 1995

The Secretary certifies that these proposed regulations would not have a significant economic impact on a substantial number of small entities because the proposed regulations would affect only slightly the overall burden on applicants and grantees, as explained in the Paperwork Reduction Act of 1995 discussion in this SUPPLEMENTARY INFORMATION.

Paperwork Reduction Act of 1995

As part of its continuing effort to reduce paperwork and respondent burden, the Department conducts a preclearance consultation program to provide the general public and Federal agencies with an opportunity to comment on proposed and continuing collections of information in accordance with the Paperwork Reduction Act of 1995 (PRA) (44 U.S.C. 3506(c)(2)(A)). This helps ensure that: The public understands the Department's collection instructions, respondents can provide the requested data in the desired format, reporting burden (time and financial resources) is minimized, collection instruments are clearly understood, and the Department can properly assess the impact of collection requirements on respondents.

Section 75.110 contains an information collection requirement. Under the PRA the Department has submitted a copy of this section to OMB for its review.

A Federal agency may not conduct or sponsor a collection of information unless OMB approves the collection under the PRA and the corresponding information collection instrument displays a currently valid OMB control number. Notwithstanding any other provision of law, no person is required to comply with, or is subject to penalty for failure to comply with, a collection of information if the collection instrument does not display a currently valid OMB control number.

In the final regulations we will display the control number assigned by OMB to any information collection requirement proposed in this NPRM and adopted in the final regulations.

Collection of Information: The proposed regulations would affect applicants and grantees of the Department's discretionary grant programs, including State, local, and tribal governments and non-profit organizations, such as institutions of higher education.

Applications: OMB has approved the Department's Generic Application Package under OMB Control number 1894-0006, which applies to those competitions that use the current EDGAR selection criteria in § 75.210 and statutory criteria that have been developed under the EDGAR procedures in § 75.209.

Regarding the burden imposed by the Generic Application Package, the Department proposes to add proposed § 75.110 to the other sections already identified as creating burden related to that package. While § 75.110 is new, it would not impose any new data collection requirements for the Generic Application Package because performance measurement burden for that package has already been calculated under the selection criteria in § 75.210. The amendments proposed in this NPRM would not increase the existing paperwork burdens under the Generic Application Package. The Secretary also proposes to cover the burden associated with the EDGAR selection criteria from § 75.209 and § 75.210 under § 75.200, which fully details the sources that program offices can use to establish selection criteria under EDGAR.

Each fiscal year, the Department receives over XX,000 applications under competitions covered by the Generic Application Package. Applicants that apply to programs that use the EDGAR criteria would be affected by the proposed changes to the selection criteria that would require applications to address evaluation and performance measurement more specifically.

The Department already has selection criteria that ask applicants to describe the evaluation plans for their projects; the burden associated with the proposed regulations is currently covered under § 75.210(h). However, an applicant for a discretionary grant would only have to respond once to provide the following information regarding the project: The performance baselines; the performance measures; the performance targets; and the methodology for collecting performance data. Thus, we do not expect greater burden under these proposed regulations and the Generic Application Package because that burden is already covered under existing criteria. Instead, we expect that as a result of these proposed regulations, applicants would provide greater clarity on the methodologies they would use to collect and report data.

Because these proposed regulations would expand the number of programs that could use proposed § 75.209 to create criteria based on statutory and regulatory requirements, there is a potential under the proposed regulations that more program offices would use the EDGAR process to establish criteria for their competitions. If more competitions use the Generic Application Package, the overall hours of burden under the Generic Application Package and OMB Control number 1894-0006 would grow. However, any “new” burden covered by the Generic Application Package would result from fewer programs using program-specific application packages, so the total burden covered by program-specific application packages would be reduced in an amount equivalent to the burden increase associated with the Generic Application Package. If the amendments to the sections regarding the selection criteria become final, we would work closely with OMB to monitor the extent to which burden currently covered by separate program-specific application packages would shift to the Generic Application Package and request appropriate changes in the total burden covered by the Generic Application Package.

The current Generic Application Package was approved by OMB based on an estimate of 9,861 responses over three years and an estimate of 447,089 total hours required to prepare applications.

Performance reports: OMB has also approved the U.S. Department of Education Grant Performance Report (ED 524B) under OMB Control number 1894-0003.

Over three years, the Department receives ED 524B performance reports from approximately 5,900 discretionary grantees. A grantee would have to respond on an annual basis to prepare performance reports throughout the course of the project period, including any no-cost extensions of the grant or funded data collection extensions, and respond once to prepare a final performance and financial report. These burdens have already been accounted for under the ED 524B.

The number of reports estimated annually under the ED 524B is 5,900 and the estimated reporting burden-hours for that report is 132,200. We do not expect any change in burden under these proposed regulations. However, there is some potential that more programs might be able to use the ED 524B performance report as a result of more programs using the EDGAR selection criteria. We will monitor that potential and work with OMB to determine if the Department needs to revisit the total burden covered by the ED 524B performance report.

Intergovernmental Review

These proposed regulations affect Direct Grant programs of the Department that are subject to Executive Order 12372 and the regulations in 34 CFR part 79. One of the objectives of the Executive order is to foster an intergovernmental partnership and a strengthened federalism. The Executive order relies on processes developed by State and local governments for coordination and review of proposed Federal financial assistance.

This document provides early notification of our specific plans and actions for these programs.

Assessment of Educational Impact

In accordance with section 411 of the General Education Provisions Act, 20 U.S.C. 1221e-4, the Secretary particularly requests comments on whether these proposed regulations would require transmission of information that any other agency or authority of the United States gathers or makes available.

Accessible Format: Individuals with disabilities can obtain this document in an accessible format (e.g., braille, large print, audiotape, or compact disc) on request to the program contact person listed under FOR FURTHER INFORMATION CONTACT.

Electronic Access to This Document: The official version of this document is the document published in the Federal Register. Free Internet access to the official edition of the Federal Register and the Code of Federal Regulations is available via the Federal Digital System at: www.gpo.gov/​fdsys. At this site you can view this document, as well as all other documents of this Department published in the Federal Register, in text or Adobe Portable Document Format (PDF). To use PDF you must have Adobe Acrobat Reader, which is available free at the site.

You may also access documents of the Department published in the Federal Register by using the article search feature at: www.federalregister.gov. Specifically, through the advanced search feature at this site, you can limit your search to documents published by the Department.

List of Subjects

34 CFR Part 75

• Accounting
• Copyright
• Education
• Grant programs-education

34 CFR Part 77

• Education
• Grant programs-education

Dated: December 6, 2012.

Arne Duncan,

Secretary of Education.

For the reasons discussed in the preamble, the Secretary proposes to amend parts 75 and 77 of title 34 of the Code of Federal Regulations as follows:

PART 75—DIRECT GRANT PROGRAMS

1. The authority citation for part 75 continues to read as follows:

Authority: 20 U.S.C. 1221e-3 and 3474, unless otherwise noted.

2. Add a new § 75.110 to read as follows:

§ 75.110

Information regarding performance measurement.

(a) The Secretary may establish in an application notice for a competition one or more performance measurement requirements, including requirements for performance measures, baseline data, or performance targets, and a requirement that applicants propose in their applications one or more of their own performance measures, baseline data, or performance targets.

(b) If an application notice requires applicants to propose project-specific performance measures, baseline data, or performance targets, the application must include the following, as required by the application notice:

(1) Performance measures. How each proposed performance measure would accurately measure the performance of the project and how the proposed performance measure would be consistent with the performance measures established for the program funding the competition.

(2) Baseline data. (i) Why each proposed baseline is valid; or

(ii) If the applicant has determined that there are no established baseline data for a particular performance measure, an explanation of why there is no established baseline and of how and when, during the project period, the applicant would establish a valid baseline for the performance measure.

(3) Performance targets. Why each proposed performance target is ambitious yet achievable compared to the baseline for the performance measure and when, during the project period, the applicant would meet the performance target.

(c) If the application notice establishes performance measurement requirements, the applicant must also describe in the application—

(1)(i) The data collection and reporting methods the applicant would use and why those methods are likely to yield reliable, valid, and meaningful performance data; and

(ii) If the Secretary requires applicants to collect data after the substantive work of a project is complete regarding the attainment of certain performance targets, the data collection and reporting methods the applicant would use during the post-performance period and why those methods are likely to yield reliable, valid, and meaningful performance data.

(2) The applicant's capacity to collect and report reliable, valid, and meaningful performance data, as evidenced by high-quality data collection, analysis, and reporting in other projects or research.

(Authority: 20 U.S.C. 1221e-3 and 3474)

3. Add a new undesignated center heading “Competition Exceptions” in subpart C immediately before the undesignated center heading “State Comment Procedures”.

4. Add a new § 75.135 to subpart C under the undesignated center heading “Competition Exceptions” to read as follows:

S75.135

Competition exception for proposed implementation sites, implementation partners, or evaluation service providers.

(a) When entering into a contract with implementation sites or partners, an applicant is not required to comply with the competition requirements in 34 CFR 74.43 or 80.36(c), as applicable, if—

(1) The contract is with an entity that agrees to provide a site or sites where the applicant would conduct the project activities under the grant;

(2) The implementation sites or partner entities that the applicant proposes to use are identified in the application for the grant; and

(3) The implementation sites or partner entities are included in the application in order to meet a regulatory, statutory, or priority requirement related to the competition.

(b) When entering into a contract for data collection, data analysis, or evaluation services, an applicant may select a provider using the informal, small-purchase procurement procedures in 34 CFR 80.36(d)(1), regardless of whether that applicant would otherwise be subject to that part or whether the evaluation contract would meet the standards for a small purchase order, if—

(1) The contract is with the data collection, data analysis, or evaluation service provider that would conduct the project services;

(2) The evaluation service provider that the applicant proposes to use is identified in the application for the grant; and

(3) The evaluation service provider is included in the application in order to meet a statutory, regulatory, or priority requirement related to the competition.

(c) If the grantee relied on the exceptions under paragraph (a) or (b) of this section, the grantee must certify that any employee, officer, or agent participating in the selection, award, or administration of a contract is free of any real or apparent conflict of interest.

(d) A grantee must obtain the Secretary's prior approval for any change to an implementation site, implementation partner, or evaluation service provider, if the grantee relied on the exceptions under paragraph (a) or (b) of this section to select the entity or evaluator.

(e) The exceptions in paragraphs (a) and (b) of this section do not extend to the other procurement requirements in 34 CFR part 74 and 34 CFR part 80 regarding contracting by grantees and subgrantees.

(Authority: 20 U.S.C. 1221e-3 and 3474)

5. Revise § 75.209 to read as follows:

§ 75.209

Selection criteria based on statutory or regulatory provisions.

The Secretary may establish selection criteria and factors based on statutory or regulatory provisions that apply to the authorized program, which may include, but are not limited to criteria and factors that reflect—

(a) Criteria contained in the program statute or regulations;

(b) Criteria in § 75.210;

(c) Allowable activities specified in the program statute or regulations;

(d) Application content requirements specified in the program statute or regulations;

(e) Program purposes, as described in the program statute or regulations; or

(f) Other pre-award and post-award conditions specified in the program statute or regulations.

(Authority: 20 U.S.C. 1221e-3 and 3474)

6. Amend § 75.210 by:

A. Revising the introductory text.

B. Revising paragraph (c)(2)(xvi).

C. Adding paragraphs (c)(2)(xxiv) through (xxix).

D. Adding paragraphs (h)(2)(viii) through (xii).

And

E. Adding a new paragraph (i).

The revisions and additions read as follows.

§ 75.210

General selection criteria.

In determining the selection criteria to evaluate applications submitted in a grant competition, the Secretary may select one or more of the following criteria and may select from among the list of optional factors under each criterion. The Secretary may define a selection criterion by selecting one or more specific factors within a criterion or assigning factors from one criterion to another criterion.

* * * * *

(c) * * *

(2) * * *

(xvi) The extent to which the proposed project will integrate with or build on similar or related efforts to improve relevant outcomes (as defined in 34 CFR 77.1(c)), using existing funding streams from other programs or policies supported by community, State, and Federal resources.

* * *

(xxiv) The extent to which the applicant demonstrates that it has the resources to operate the project beyond the length of the grant, including a multi-year financial and operating model and accompanying plan; the demonstrated commitment of any partners; evidence of broad support from stakeholders (e.g., State educational agencies, teachers' unions) critical to the project's long-term success; or more than one of these types of evidence.

(xxv) The potential and planning for the incorporation of project purposes, activities, or benefits into the ongoing work of the applicant beyond the end of the grant.

(xxvi) The extent to which the proposed project will increase efficiency in the use of time, staff, money, or other resources in order to improve results and increase productivity.

(xxvii) The extent to which the proposed project will integrate with or build on similar or related efforts in order to improve relevant outcomes (as defined in 34 CFR 77.1(c)), using nonpublic funds or resources.

(xxviii) The extent to which the proposed project is supported by evidence of promise (as defined in 34 CFR 77.1(c)).

(xxix) The extent to which the proposed project is supported by strong theory (as defined in 34 CFR 77.1(c)).

* * * * *

(h) * * *

(2) * * *

(viii) The extent to which the methods of evaluation will, if well-implemented, produce evidence about the project's effectiveness that would meet the What Works Clearinghouse Evidence Standards without reservations.^[3]

(ix) The extent to which the methods of evaluation will, if well-implemented, produce evidence about the project's effectiveness that would meet the What Works Clearinghouse Evidence Standards with reservations.^[4]

(x) The extent to which the methods of evaluation will, if well-implemented, produce evidence of promise (as defined in 34 CFR 77.1(c)).

(xi) The extent to which the methods of evaluation will provide valid and reliable performance data on relevant outcomes.

(xii) The extent to which the evaluation plan clearly articulates the key components, mediators, and outcomes of the grant-supported intervention, as well as a measurable threshold for acceptable implementation.

(i) Strategy to Scale

(1) The Secretary considers the applicant's strategy to scale the proposed project.

(2) In determining the applicant's capacity to scale the proposed project, the Secretary considers one or more of the following factors:

(i) The applicant's capacity (e.g., in terms of qualified personnel, financial resources, or management capacity) to bring the proposed project to scale on a national or regional level (as defined in 34 CFR 77.1(c)) working directly, or through partners, during the grant period.

(ii) The applicant's capacity (e.g., in terms of qualified personnel, financial resources, or management capacity) to further develop and bring to scale the proposed process, product, strategy, or practice, or to work with others to ensure that the proposed process, product, strategy, or practice can be further developed and brought to scale, based on the findings of the proposed project.

(iii) The feasibility of successful replication of the proposed project, if favorable results are obtained, in a variety of settings and with a variety of populations.

(iv) The mechanisms the applicant will use to broadly disseminate information on its project so as to support further development or replication.

(v) The extent to which the applicant demonstrates there is unmet demand for the process, product, strategy, or practice that will enable the applicant to reach the level of scale that is proposed in the application.

(vi) The extent to which the applicant identifies a specific strategy or strategies that address a particular barrier or barriers that prevented the applicant, in the past, from reaching the level of scale that is proposed in the application.

7. Add § 75.266 to subpart D to read as follows:

§ 75.266

What procedures does the Secretary use if the Secretary decides to give special consideration to applications supported by strong or moderate evidence of effectiveness?

(a) As used in this section, “strong evidence of effectiveness” is defined in 34 CFR 77.1(c);

(b) As used in this section, “moderate evidence of effectiveness” is defined in 34 CFR 77.1(c); and

(c) If the Secretary determines that special consideration of applications supported by strong or moderate evidence of effectiveness is appropriate, the Secretary may establish a separate competition under the procedures in 34 CFR 75.105(c)(3), or provide competitive preference under the procedures in 34 CFR 75.105(c)(2), for applications supported by:

(1) Evidence of effectiveness that meets the conditions set out in paragraph (a) of the definition of “strong evidence of effectiveness” in 34 CFR 77.1;

(2) Evidence of effectiveness that meets the conditions set out in either paragraph (a) or (b) of the definition of “strong evidence of effectiveness” in 34 CFR 77.1; or

(3) Evidence of effectiveness that meets the conditions set out in the definition of “moderate evidence of effectiveness.”

(Authority: 20 U.S.C. 1221e-3 and 3474)

8. Revise § 75.250 to read as follows:

§ 75.250

Maximum funding period.

(a) The Secretary may approve a project period to fund the substantive work of a grant and a data collection period to fund data collection, analysis, and reporting related to a grant after the end of the project period.

(b) The Secretary may approve a project period of up to 60 months to perform the substantive work of a grant.

(1) The Secretary may approve a data collection period for a grant for a period of up to 72 months after the end of the project period and provide funding for the data collection period for the sole purpose of collecting, analyzing, and reporting performance measurement data regarding the project.

(2) The Secretary may inform applicants of the Secretary's intent to approve data collection periods in the application notice published for a competition or may decide to fund data collection periods after grantees have started their project periods.

(Authority: 20 U.S.C. 1221e-3 and 3474)

9. Amend § 75.251 by adding a new paragraph (c) to read as follows:

§ 75.251

Budget Periods.

* * * * *

(c) If the Secretary funds a multi-year data collection period, the Secretary may fund the data collection period through separate budget periods and fund those budget periods in the same manner as those periods are funded during the project period.

10. Amend § 75.253 by—

A. Revising paragraph (a)(2).

B. Adding a new paragraph (a)(5).

C. Re-designating paragraphs (b) through (e) as paragraphs (c) through (f).

D. Adding a new paragraph (b).

And

E. Revising newly re-designated paragraph (f).

The revisions and additions read as follows:

§ 75.253

Continuation of a multi-year project after the first budget period.

(a) * * *

(2) The grantee has either—

(i) Made substantial progress in achieving—

(A) The goals and objectives of the project; and

(B) If the Secretary established performance measurement requirements for the grant in the application notice, the performance targets in the grantee's approved application; or

(ii) Obtained the Secretary's approval for changes to the project that—

(A) Do not increase the amount of funds obligated to the project by the Secretary; and

(B) Enable the grantee to achieve the goals and objectives of the project and meet the performance targets of the project, if any, without changing the scope or objectives of the project.

* * *

(5) The grantee has maintained financial and administrative management systems that meet the requirements in 34 CFR 74.21 or 80.20, as appropriate.

(b) In deciding whether a grantee has made substantial progress, the Secretary may consider any information relevant to the authorizing statute, a criterion, a priority, or a performance measure, or to a financial or other requirement that applies to the selection of applications for new grants.

* * * * *

(f) Unless prohibited by the program statute or regulations, a grantee that is in the final budget period of its project period may seek continued assistance for the project as required under the procedures for selecting new projects for grants.

11. Revise § 75.590 to read as follows.

§ 75.590

Evaluation by the grantee.

(a) If the application notice for a competition required applicants to describe how they would evaluate their projects, each grantee under that competition must demonstrate to the Department that—

(1) The evaluation meets the standards of the evaluation in the approved application for the project; and

(2) The performance measurement data collected by the grantee and used in the evaluation meet the performance measurement requirements of the approved application.

(b) If the application notice for a competition did not require applicants to describe how they would evaluate their projects, each grantee must provide information in its performance report demonstrating—

(1) The progress made by the grantee in the most recent budget period, including progress based on the performance measurement requirements for the grant, if any;

(2) The effectiveness of the grant, including fulfilling the performance measurement requirements of the approved application, if any; and

(3) The effect of the project on the participants served by the project, if any.

(Authority: 20 U.S.C. 1221e-3 and 3474)

12. Amend §  75.708 by:

A. Revising paragraph (a).

B. Re-designating paragraph (b) as paragraph (e); and

C. Adding new paragraphs (b), (c) and (d).

The revision and additions read as follows.

§ 75.708

Subgrants.

(a) A grantee may not make a subgrant under a program covered by this part unless authorized by statute or by paragraph (b) of this section.

(b) The Secretary may, through an announcement in the Federal Register, authorize subgrants when necessary to meet the purposes of a program. In this announcement, the Secretary will—

(1) Designate the types of entities, e.g., State educational agencies, local educational agencies, institutions of higher education, and non-profit organizations, to which subgrants can be awarded; and

(2) Indicate whether subgrants can be made to entities identified in an approved application or, without regard to whether the entity is identified in an approved application, have to be selected through a competitive process set out in subgranting procedures established by the grantee.

(c) If authorized under paragraph (b) of this section, a subgrant is allowed if it will be used by that entity to directly carry out project activities described in that application.

(d) The grantee, in awarding subgrants under paragraph (b) of this section, must—

(1) Ensure that subgrants are awarded on the basis of an approved budget that is consistent with the grantee's approved application and all applicable Federal statutory, regulatory, and other requirements;

(2) Ensure that every subgrant includes any conditions required by Federal statute and executive orders and their implementing regulations; and

(3) Ensure that subgrantees are aware of requirements imposed upon them by Federal statute and regulation, including the Federal anti-discrimination laws enforced by the Department; and

* * * * *

PART 77—DEFINITIONS THAT APPLY TO DEPARTMENT REGULATIONS

13. The authority citation for part 77 continues to read as follows:

Authority: 20 U.S.C. 1221e-3 and 3474, unless otherwise noted.

14. Amend § 77.1(c) by adding the following definitions in alphabetical order:

§ 77.1

Definitions that apply to all Department programs.

* * * * *

(c) * * *

Ambitious means promoting continued, meaningful improvement for program participants or for other individuals or entities affected by the grant, or representing a significant advancement in the field of education research, practices, or methodologies. When used to describe a performance target, whether a performance target is ambitious depends upon the context of the relevant performance measure and the baseline for that measure.

* * * * *

Baseline means the starting point from which performance is measured and targets are set.

* * * * *

Evidence of promise means there is empirical evidence to support the theoretical linkage(s) between at least one critical component and at least one relevant outcome presented in the logic model for the proposed process, product, strategy, or practice. Specifically, evidence of promise means the conditions in paragraphs (a) and (b) are met:

(a) There is at least one study that is a—

(1) Correlational study with statistical controls for selection bias;

(2) Quasi-experimental study that meets the What Works Clearinghouse Evidence Standards with reservations; ^[1] or

(3) Randomized controlled trial that meets the What Works Clearinghouse Evidence Standards with or without reservations.^[2]

(b) The study referenced in paragraph (a) found a statistically significant or substantively important (defined as a difference of 0.25 standard deviations or larger), favorable association between at least one critical component and one relevant outcome presented in the logic model for the proposed process, product, strategy, or practice.

* * * * *

Large sample means an analytic sample of 350 or more students (or other single analysis units) who were randomly assigned to a treatment or control group or 50 or more groups (such as classrooms or schools) that contain 10 or more students (or other single analysis units) and that were randomly assigned to a treatment or control group.

* * * * *

Logic model (also referred to as theory of action) means a well-specified conceptual framework that identifies key components of the proposed process, product, strategy, or practice (i.e., the active “ingredients” that are hypothesized to be critical to achieving the relevant outcomes) and describes the relationships among the key components and outcomes, theoretically and operationally.

* * * * *

Moderate evidence of effectiveness means one of the following conditions is met:

(a) There is at least one study of the effectiveness of the process, product, strategy, or practice being proposed that meets the What Works Clearinghouse Evidence Standards without reservations,^[3] found a statistically significant favorable impact on a relevant outcome (with no statistically significant unfavorable impacts on that outcome for relevant populations in the study or in other studies of the intervention reviewed by and reported on by the What Works Clearinghouse), and includes a sample that overlaps with the populations or settings proposed to receive the process, product, strategy, or practice.

(b) There is at least one study of the effectiveness of the process, product, strategy, or practice being proposed that meets the What Works Clearinghouse Evidence Standards with reservations,^[4] found a statistically significant favorable impact on a relevant outcome (with no statistically significant unfavorable impacts on that outcome for relevant populations in the study or in other studies of the intervention reviewed by and reported on by the What Works Clearinghouse), includes a sample that overlaps with the populations or settings proposed to receive the process, product, strategy, or practice, and includes a large sample and a multi-site sample (Note: multiple studies can cumulatively meet the large and multi-site sample requirements as long as each study meets the other requirements in this paragraph).

* * * * *

Multi-site sample means more than one site, where site can be defined as an LEA, locality, or State.

* * * * *

National level describes the level of scope or effectiveness of a process, product, strategy, or practice that is able to be effective in a wide variety of communities, including rural and urban areas, as well as with different groups (e.g., economically disadvantaged, racial and ethnic groups, migrant populations, individuals with disabilities, English learners, and individuals of each gender).

* * * * *

Performance measure means any quantitative indicator, statistic, or metric used to gauge program or project performance.

* * * * *

Performance target means a level of performance that an applicant would seek to meet during the course of a project or as a result of a project.

* * * * *

Quasi-experimental design study means a study using a design that attempts to approximate an experimental design by identifying a comparison group that is similar to the treatment group in important respects. These studies, depending on design and implementation, can meet What Works Clearinghouse Evidence Standards with reservations ^[5] (they cannot meet What Works Clearinghouse Evidence Standards without reservations).

* * * * *

Randomized controlled trial means a study that employs random assignment of, for example, students, teachers, classrooms, schools, or districts to receive the intervention being evaluated (the treatment group) or not to receive the intervention (the control group). The estimated effectiveness of the intervention is the difference between the average outcome for the treatment group and for the control group. These studies, depending on design and implementation, can meet What Works Clearinghouse Evidence Standards without reservations.^[6]

* * * * *

Regional level describes the level of scope or effectiveness of a process, product, strategy, or practice that is able to serve a variety of communities within a State or multiple States, including rural and urban areas, as well as with different groups (e.g., economically disadvantaged, racial and ethnic groups, migrant populations, individuals with disabilities, English learners, and individuals of each gender). For an LEA-based project, to be considered a regional-level project, a process, product, strategy, or practice must serve students in more than one LEA, unless the process, product, strategy, or practice is implemented in a State in which the State educational agency is the sole educational agency for all schools.

* * * * *

Relevant outcome means the student outcome(s) (or the ultimate outcome if not related to students) the proposed process, product, strategy, or practice is designed to improve; consistent with the specific goals of a program.

* * * * *

Strong evidence of effectiveness means one of the following conditions is met:

(a) There is at least one study of the effectiveness of the process, product, strategy, or practice being proposed that meets the What Works Clearinghouse Evidence Standards without reservations,^[7] found a statistically significant favorable impact on a relevant outcome (with no statistically significant unfavorable impacts on that outcome for relevant populations in the study or in other studies of the intervention reviewed by and reported on by the What Works Clearinghouse), includes a sample that overlaps with the populations and settings proposed to receive the process, product, strategy, or practice, and includes a large sample and a multi-site sample (Note: multiple studies can cumulatively meet the large and multi-site sample requirements as long as each study meets the other requirements in this paragraph).

(b) There are at least two studies of the effectiveness of the process, product, strategy, or practice being proposed, each of which: Meets the What Works Clearinghouse Evidence Standards with reservations,^[8] found a statistically significant favorable impact on a relevant outcome (with no statistically significant unfavorable impacts on that outcome for relevant populations in the studies or in other studies of the intervention reviewed by and reported on by the What Works Clearinghouse), includes a sample that overlaps with the populations and settings proposed to receive the process, product, strategy, or practice, and includes a large sample and a multi-site sample.

* * * * *

Strong theory means a rationale for the proposed process, product, strategy, or practice that includes a logic model.

Footnotes

1.  See What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

2.  See What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

3.  See What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

4.  See What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

1.  See What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

2.  See What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

3.  See What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

4.  See What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

5.  See What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

6.  See What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

7.  See What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

8.  See What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.