2013-19390. Direct Grant Programs and Definitions That Apply to Department Regulations

AGENCY:

Department of Education.

ACTION:

Final regulations.

SUMMARY:

The Secretary amends the regulations in the Education Department General Administrative Regulations (EDGAR) to: improve the Department's ability to evaluate the performance of discretionary grant programs and grantee projects; support, where appropriate, projects supported by evidence of effectiveness; review grant applications using selection factors that promote the Secretary's policy objectives related to project evaluation, sustainability, productivity, and strategy to scale; and reduce burden on grantees in selecting implementation sites, implementation partners, or evaluation service providers for their proposed projects. These amendments will allow the Department to be more effective and efficient when selecting grantees in discretionary grant competitions, provide higher-quality data to the Congress and the public, and better focus applicants on the goals and objectives of the programs to which they apply for grants.

DATES:

These regulations are effective September 12, 2013.

FOR FURTHER INFORMATION CONTACT:

Erin McHugh, U.S. Department of Education, 400 Maryland Avenue SW., Room 4W319, LBJ, Washington, DC 20202. Telephone: [phone] or by email: [e-mail].

If you use a telecommunications device for the deaf (TDD) or a text telephone (TTY), call the Federal Relay Service (FRS), toll free, at 1-800-877-8339.

SUPPLEMENTARY INFORMATION:

 

Executive Summary

Purpose of This Regulatory Action: The purpose of this action is to amend EDGAR to improve the quality and effectiveness of grant-making decisions and reduce the burden on applicants and grantees. These amendments will help align the Department's grant process with the Secretary's policy objectives and allow Department programs to design grant competitions to achieve those objectives. These amendments will also increase the flexibility for applicants and grantees to both reduce burden on applicants and grantees and improve the quality of data generated and reported by grantees. The authority to amend EDGAR is 20 U.S.C. 1221e-3 and 3474.

Summary of the Major Provisions of This Regulatory Action: These rules:

1. Allow the Secretary, in the application notice for a grant competition, to establish performance measurement requirements for grantees (New § 75.110);

2. Revise requirements for project evaluations submitted to the Department by grantees and for continuation of a multi-year project to incorporate performance measurement requirements for grantees (Amended §§ 75.253 and 75.590);

3. Authorize grantees to procure implementation sites without regard to the procurement procedures in parts 74 and 80 and use small purchase procedures to procure evaluation service providers and providers of services that are essential to the success of a proposed grant, provided the site or service provider is identified in the grant application (New § 75.135);

4. Allow the Secretary, through an announcement in the Federal Register, to authorize grantees under particular programs to award subgrants to directly carry out programmatic activities. The possible subgrantees and the program activities they would carry out must be identified and described in the grantees' applications or selected through a competitive process set out in subgranting procedures established by the grantee (New § 75.708);

5. Add one new selection criterion and amend two existing selection criteria that the Department may use to evaluate applications. The new criterion is used to assess the extent to which a proposed project could be brought to scale. The amendments to the general selection criteria also include the addition of five new factors to § 75.210(h) (Quality of the Project Evaluation) that could be used to assess how well a proposed project evaluation would produce evidence about the project's effectiveness. We also revised one factor and added six new factors to § 75.210(c) (Quality of the Project Design) (Amended §§ 75.209 and 75.210);

6. Authorize program offices to consider the effectiveness of proposed projects under a new priority that could be used as an absolute, competitive preference, or invitational priority (New § 75.266); and

7. Allow the Secretary to fund data collection periods after the end of the substantive work of a project so that project outcomes could be assessed using data from the entire project period (Amended §§ 75.250 and 75.251).

Costs and Benefits: The Secretary believes that these regulations do not impose significant costs on entities that would receive assistance through Department of Education programs. Any costs imposed on applicants by these regulations are limited to the paperwork burden involved in preparing an application and keeping records needed to track progress on meeting performance measures. The benefits of implementing them outweigh any costs incurred by applicants.

The benefits of the amendments in these regulations for the use of performance measures, baseline data, and performance targets established by the Department or by grantees themselves are that the Department would collect meaningful data that could be used to select applications for funding and assess the success of individual projects. The Department will also use these data to report to the Congress and the Office of Management and Budget (OMB) on the success of the grant programs in achieving their legislative objectives. The Department's strengthened capability to evaluate the success of Department programs should help improve the effectiveness of those programs and improve transparency about how public funds are expended, without imposing additional costs on grantees or other parties.

Additionally, these final regulations add a new § 75.135 and amend § 75.708 regarding subgranting and competition exceptions. These sections will reduce costs, increase benefits, and potentially improve project quality by removing barriers that impede grantees from working with, either through a contract or a subgrant, implementation partners and service providers identified in funded applications. These final regulations will relieve grantees of the costs of administering competitions without reducing accountability or increasing the risk of improper use of or accounting for grant expenditures.

These regulations also provide the Department with greater flexibility in conducting grant competitions by allowing for the use of selection criteria that: (1) Are closely aligned with program objectives and priorities, and (2) promote policy objectives such as project evaluation, sustainability, productivity, and strategy to scale. Thus, these amendments will benefit applicants, the Department, and the public by allowing the Secretary to establish selection criteria that are concise and closely aligned with the goals and objectives of a particular grant competition and are focused more closely and coherently on the intended programmatic and policy outcomes. Because the new, more specific criteria will be used instead of the more generic criteria currently in EDGAR, the regulations will generate these benefits without increasing the costs for applicants, grantees, or the Department.

On December 14, 2012, the Secretary published a notice of proposed rulemaking (NPRM) for these amendments in the Federal Register (77 FR 74392).

This document includes three revisions from the NPRM. We discuss changes from the NPRM in greater detail in the Analysis of Comments and Changes. Specifically, we have revised § 75.135 to make it clear that grantees may exercise the competition exception when procuring services from entities in instances where the entity is identified in the funded application. We have also revised the definitions of “strong evidence of effectiveness” and “moderate evidence of effectiveness” in § 77.1 to clarify that only studies with unfavorable outcomes that were so substantial as to call into question the potential effectiveness of the proposed project would disqualify the evidence from meeting the condition in the definitions. We do not discuss minor technical or editorial changes.

Public Comment: In response to our invitation in the NPRM, 38 parties submitted comments on the proposed regulations. We group major issues according to subject. Analysis of Comments and Changes: An analysis of the comments and of any changes in the regulations since publication of the NPRM follows.

Information Regarding Performance Measurement—§ 75.110

Comments: One commenter agreed with the proposed amendment to create § 75.110, stating that establishing performance measurement requirements in a notice inviting applications for a competition would both increase the likelihood of obtaining more robust data on grantee performance and increase the number of rigorous evaluation studies in the field.

Some commenters agreed with the proposed amendment but requested clarification on key points. One commenter expressed concerns that performance measures beyond those related to student achievement would not be considered. Another commenter suggested differentiating between performance measures and outcomes data, indicating that performance measures help grantees continuously improve their projects, while outcomes data are useful in evaluating the success of their projects. The commenter also suggested developing a list of indicators for applicants to use when defining and adopting their own measures of success. Another commenter noted the importance of aligning performance measures with program goals and taking into account the size and scope of each proposed project when evaluating the quality of the performance measures.

Two commenters expressed support for the proposed amendment, but they suggested that special considerations be made for applicants with limited capacity to analyze and collect data and recommended that these applicants be permitted to use grant funds and additional planning time in order to meet the performance measurement requirement.

One commenter expanded on the idea of allowing grantees to use grant funds for performance measurement by suggesting the inclusion of a provision for performance measurement expenses in part 75, subpart F.

Discussion: We agree with the commenter that the proposed amendment on performance measurement will strengthen the quality of data provided by grantees on their projects. Grantees typically report performance measures specific to their projects. Because those performance measures vary significantly, even among projects supported under one competition, it is very difficult for the Department to track the overall success of a program without performance measures that apply to all projects funded under a particular program's competition. By requiring standard performance measurements in a notice inviting applications, and by retaining the applicant's ability to set additional project-specific measures, we are more likely to obtain data that are meaningful both to evaluate the overall program and the quality of each grant funded under a competition for that program. This allows us to more effectively measure each program's effectiveness, as required by the Government Performance and Results Act of 1993 (GPRA), and will provide the Department a mechanism to hold grantees accountable for their performance and their success in implementing their grants.

We also agree that it is important to consider performance measures beyond those related to student achievement. While Department programs share a common focus on improving academic achievement for students, many programs focus on factors that may not directly relate to students, such as professional development for teachers or engaging parents and family members in the school community. With this in mind, § 75.110 gives the Department flexibility to set standard performance measurement requirements for all types of programs, not just programs that measure student performance, while continuing to invite applicants to set additional project-specific measures. Therefore, this regulation will allow the use of a variety of performance measures. While we explicitly require that grantees collect and report on GPRA-mandated performance measures, which may be focused on student achievement, grantees retain the discretion to establish additional performance measures uniquely related to the objectives of their proposed projects.

We recognize that some grantees may have limited capacity to meet the performance measurement requirement and acknowledge that this may appear to disadvantage small local educational agencies (LEAs), rural LEAs, community colleges, and small nonprofit organizations in particular. We suggest that when preparing an application, an applicant assess its needs and develop its proposed budget accordingly. For example, an applicant that lacks sufficient resources to collect and analyze data on its own may request funding to obtain data collection and evaluation services from external providers. Neither current regulations nor these new regulations prohibit an applicant from including in its project budget support for data collection and analysis. If an applicant decides to procure these services from a contractor, the applicant must meet the procurement requirements authorized under new § 75.135(b), including identifying the proposed contractor in the application. See discussion of § 75.135 under Procurement and Subgrant Process for Entities Named in Applications.

We agree that performance measures should be aligned with the goals of the Department program and that useful measures will take into account the expected scope and size of each proposed project. Regarding the comment suggesting that we amend § 75.110 to list specific program-aligned performance measures, we do not think § 75.110 is the most appropriate platform for enumerating specific, program-aligned performance measures. The purpose of the amendments to § 75.110 is to permit the Department to establish performance measures in the notice inviting applications and to establish standard performance measurement requirements that all applicants for a particular Department program must use, while still allowing applicants the flexibility to suggest other, more project-specific, performance measures. Also, given the variety of programs to which these regulations apply, we do not think it is appropriate to prescribe a list of indicators in the regulation. We think that Department program officials are in the best position to establish appropriate performance measurement indicators for particular grant competitions and need the discretion to change the measures as the program evolves. In addition, more detailed information on indicators for a particular Department program will be provided in each notice inviting applications than can be provided through the use of generic performance measures listed in a regulation.

Finally, we agree that performance measures and outcomes data are two separate terms, but we want to clarify that both are necessary and important to the continuous improvement and success of a grant. “Performance measure” is defined as any quantitative indicator, statistic, or metric used to gauge program or project performance. Thus, a performance measure is a unit for measuring outcome data. By selecting the appropriate measures, we can ensure that the outcome data collected by grantees are relevant to program performance and that the Department has the data needed to report program performance information to the Congress under GPRA. Further, we expect that grantees will collect outcome data not only at the end of a project, but in the interim as well. Formative outcome data are collected and analyzed throughout the project period and are useful for the continuous improvement of the project, while summative outcome data are collected and analyzed at the end of the project period and are useful when evaluating the project's overall impact. Performance measures are expected to inform both types of outcome data.

Changes: None.

Comments: Some commenters expressed concern that the proposed changes to § 75.110 would unfairly disadvantage small, rural, and economically disadvantaged LEAs that may have limited access to data or limited resources to collect data. Commenters requested that we clarify how performance measures, baseline data, performance targets, and performance data will be set, and by whom they will be set.

Some commenters expanded on this suggestion, requesting that we solicit LEAs or field experts for input on defining performance measures, baseline data, performance targets, and performance data.

Some commenters did not agree with the proposed change and expressed concern that it would prove too costly and burdensome for grantees. One commenter did not agree with the proposed regulation because, according to the commenter, the performance measurement requirement would be too costly and would not ultimately improve services for students.

Discussion: While we recognize that all applicants may not have equal resources to collect and report performance measurement data prior to receiving a grant, each applicant should assess its capacity when writing its application and develop its budget proposal accordingly. An applicant may include funds in its project budget to support data collection and analysis. Applicants can use the exception in § 75.135 to procure the needed expertise to collect the appropriate data and evaluate the outcomes under the measures established for the competition.

Performance measures must be aligned to the goals of the program, which are based on the law and the Department's regulations and policies. As such, performance measures for a particular program are generally set by the Department officials responsible for the program. We appreciate the opinions of LEA representatives and field experts and encourage interested parties to comment on notices of proposed priorities; however, performance measures must ultimately align with program goals so the Department can measure the effectiveness of its programs.

Gathering reliable and valid information on project outcomes is an integral part of determining which processes, products, strategies, and practices are working for students and which are not. While these final regulations may require grantees to use a portion of project funds on measuring performance, we consider it to be an important investment that will ultimately lead grantees to more successful results and thereby improve results for students and help the Department report more meaningful information to the Congress on the benefits of the Department's programs, as required under GPRA.

Changes: None.

Procurement and Subgrant Process for Entities Named in Applications—§ 75.135 Competition Exception for Implementation Sites, Implementation Partners, or Evaluation Service Providers; and § 75.708 Prohibition on Subgrants

Comments: Several commenters expressed strong support for the proposed changes to §§ 75.135 and 75.708. Many of these commenters recommended that the Secretary allow grantees to make subgrants without approval from the Secretary. The commenters stated that individual grantees are better positioned than the Secretary to determine whether they need to make subgrants to carry out their projects, what types of entities may receive subgrants, and how the subgrants would be made. One commenter suggested revising the regulation to provide that subgrants should always be allowed unless the Department decides to prohibit it in certain circumstances. The commenter thought that formulating the regulation in this manner would encourage public-private partnerships while preserving the Secretary's authority to prohibit subgranting when necessary. One commenter argued that providing direct authority to grantees to identify and administer subgrants would reduce the administrative burden of seeking approval from the Department. Another commenter indicated this flexibility is necessary to mitigate implementation delays in instances when the publication of the notice inviting applications in the Federal Register is not timely. The commenter noted that State educational agencies (SEAs) particularly need this flexibility and suggested adding a new paragraph that allows a State's Chief School Officer to determine the types of entities that may receive subgrants and the procedures for making subgrants within the State.

Some commenters also recommended that the regulation specifically identify SEAs, institutions of higher education (IHEs), and nonprofit organizations as types of entities that may be awarded a subgrant. One commenter proposed adding for-profit entities as a type of entity that may be awarded a subgrant. The commenter noted the inclusion of for-profit entities is particularly important considering that many grants are designed around a product or service that will be provided by a for-profit entity and emphasized that grantees should identify partners or providers based on the needs of their projects without consideration for the corporate status of a partner or provider.

Discussion: We appreciate the commenters' support for the changes regarding subgranting. However, we decline to make the revisions suggested because it is prudent and necessary for the Department to maintain control over when § 75.708 is used. The Department must ensure that subgrants are only authorized and used in a manner and under circumstances that are consistent with the requirements and purposes of authorizing statutes. This objective can only be met if the Department retains control over authorizing the grantee's use of subgrants. We note however, that under § 75.708 the Secretary will indicate through an announcement in the Federal Register whether subgrants can be made to entities identified in an approved application or can be made to entities selected through a competitive process included in the grantee's application. Thus, in lieu of requiring a grantee to identify all entities that will receive subgrants in the approved application, the Secretary may allow a grantee to use a competitive process that it describes in the approved application to determine the entities that will receive subgrants.

With regard to the comment suggesting that for-profit entities be allowed to receive subgrants, we note that grantees already have the authority to enter into contracts with for-profit entities. Additionally, we think that procurements are the appropriate vehicle for grantees to use to secure goods and services from for-profit entities. For that reason, we decline to revise the regulations to allow subgrants to for-profit entities. However, we agree that there may be circumstances under which a product or service provided by a for-profit entity is integral to implementation of a project. As a result, we have revised § 75.135 to include entities that will provide a product or service that would, if removed from the grant, have a detrimental effect on the successful implementation of the grant.

Changes: We have revised § 75.135(b) to clarify that when entering into a contract for data collection, data analysis, evaluation services, or essential services, as defined in paragraph (f) of this section, an applicant may select a provider using the informal, small-purchase procurement procedures in 34 CFR 80.36(d)(1), regardless of whether that applicant would otherwise be subject to that part or whether the evaluation contract would meet the standards for a small purchase order, if—

(1) The contract is with the data collection, data analysis, evaluation service, or essential service provider;

(2) The data collection, data analysis, evaluation service, or essential service provider that the applicant proposes to use is identified in the application for the grant; and

(3) The data collection, data analysis, evaluation service, or essential service provider is identified in the application in order to meet a statutory, regulatory, or priority requirement related to the competition.

We have also added paragraph (f) to state that, for the purposes of this section, essential service means a product or service directly related to the grant that would, if not provided, have a detrimental effect on the grant.

Comment: One commenter recommended that the Department broadly implement the authority to allow subgrants. The commenter suggested that the Ready To Learn (RTL) program is particularly well-suited for the use of subgrants given that subrecipients of RTL grantees are often responsible for the development and production of educational programming that is integral to the grant. According to the commenter, subgranting will both continue to ensure close monitoring of funds and foster close collaboration that will further project objectives.

Discussion: As noted elsewhere in our responses to comments in this section, the Department cannot establish a universal rule allowing the use of subgrants because program statutes define differently whether subgrants may be used and in what circumstances they may be used. Additionally, subgrants, even when not prohibited by a statute, may not always be appropriate for a particular program.

Changes: None.

Comments: Two commenters stated that, although the proposed amendments in § 75.135 that permit exceptions to the procurement procedures are beneficial and they support them as written, some grantees may be subject to State or local laws that require specific procurement procedures. One commenter explained that such State laws and requirements negate the benefits of the proposed amendment and suggested revising the language to minimize this consequence.

Discussion: The proposed amendments in § 75.135 relax requirements that otherwise apply to grantees under parts 74 and 80. They do not, as the commenters noted, eliminate a grantee's responsibilities to comply with their own procurement requirements and State and local laws that exceed those required by Department regulations. State governments may follow their own procurement requirements, subject only to the requirement that they must include in their contracts all clauses required by Federal statutes, Executive orders, and implementing regulations. We note that some SEAs have adopted some of the requirements in § 80.36. Other State, local, and Indian tribal government grantees must comply with the minimum requirements in 34 CFR 80.36. Non-governmental grantees must comply with the minimum procurement requirements in 34 CFR 74.41-74.48. These final regulations do not change other applicable financial management and procurement requirements in 34 CFR parts 74 and 80, including those that require State agencies to follow their own procurement policies and procedures (34 CFR 80.36(a)) or that generally require grantees to maintain procurement procedures that prohibit conflicts of interest. The continued applicability of these requirements in parts 74 and 80 of EDGAR is crucial to ensuring accountability for the use of Federal funds by grantees.

Changes: None.

Comments: Many commenters expressed strong support for the revisions to § 75.135 and § 75.708 and agreed that the exemption from procurement requirements for selecting implementation sites or partner entities, and the use of small purchase procedures to select evaluation service providers identified in grant applications will improve the implementation and outcome of grants funded by the Department. One commenter noted that the proposed amendment would be particularly beneficial to SEAs because it will support more efficient use of resources and ensure grant activities are implemented on a timely basis.

One commenter expressed general support for the proposed amendment but recommended revising it to include a competition exception for products or services identified in the application that are unique and essential, meaning that the use of an alternative product or service would be detrimental to the implementation of the project. The commenter also suggested adding a condition that allows the use of simplified procurement procedures for products or services that are not core to the implementation of the project when the costs of conducting a competition would be excessive in relation to the amount of grant funds that would be awarded in the contract. The commenter also stated that local and State procurement requirements would still apply, so these revisions would eliminate an additional Federal requirement in these instances but would not remove those existing protections.

Discussion: We appreciate the commenters' support for the proposed amendment to § 75.135 and agree it will be beneficial to grantees. With regard to the recommendation that we revise the regulation to allow grantees to use the competition exception for products and services identified in the application that are unique, we recognize that grant projects may be designed around such products or services. Therefore, we have revised paragraph (b) of § 75.135 to include an exception for entities that will provide a product or service that would, if not provided, have a detrimental effect on the grant. However, we decline to revise the regulation to reduce the competition requirements for products or services that are not identified in the application or core to the implementation of the project because we do not think such a revision is consistent with the intent of the change. We do not intend this change to limit competition in instances when full and open competition is practical. We note, however, that the simplified acquisition threshold already provides grantees some flexibility in competition requirements for procurements under $100,000. OMB has proposed to raise this threshold to $150,000 in its proposed amendments to title 2 of the CFR. See the OMB proposal, Reform of Federal Policies Relating to Grants and Cooperative Agreements; Cost Principles and Administrative Requirements (Including Single Audit Act), published on Friday, February 1, 2013, at 78 FR 7282. Thus, regardless of the exemption authorized in these final regulations, applicants will have greater flexibility to use small purchase procedures when the procurement threshold is raised. Therefore, it is not necessary or appropriate to make the changes suggested.

Also, as noted earlier, these regulations relax certain procurement requirements that otherwise apply to grantees under parts 74 and 80. Grantees should be aware, however, that these amendments do not eliminate a grantee's responsibilities to comply with its own procurement requirements and State and local laws to the extent that those requirements and laws exceed the minimum requirement in parts 74 and 80.

Changes: We have revised § 75.135(b) to clarify that when entering into a contract for data collection, data analysis, evaluation services, or essential services, as defined in paragraph (f) of this section, an applicant may select a provider using the informal, small-purchase procurement procedures in 34 CFR 80.36(d)(1), regardless of whether that applicant would otherwise be subject to that part or whether the evaluation contract would meet the standards for a small purchase order, if—

(1) The contract is with the data collection, data analysis, evaluation service, or essential service provider;

(2) The data collection, data analysis, evaluation service, or essential service provider that the applicant proposes to use is identified in the application for the grant; and

(3) The data collection, data analysis, evaluation service, or essential service provider is identified in the application in order to meet a statutory, regulatory, or priority requirement related to the competition.

We have also added paragraph (f) to state that, for the purposes of this section, essential service means a product or service directly related to the grant that would, if not provided, have a detrimental effect on the grant.

Comment: None.

Discussion: Based on the comments received, we revised § 75.135(b) to expand the circumstances under which grantees may use small purchase procedures. We have also revised § 75.135(c) to require applicants, who utilize this exception, to certify that they followed the small purchase procedures. The small purchase procedures, while not as extensive as the full procurement requirements set out in Parts 74 and 80, provide important protections to the Federal interest in the prudent and allowable use of grant funds. By requiring applicants that utilize this exception to certify that they followed the small purchase procedures, we provide increased assurance that the protections to Federal grant funds offered by those procedures are, in fact, in place.

Changes: We have revised the certification requirement in § 75.135(c) to require grantees that relied on the exceptions of § 75.135(b) to certify that they used small purchase procedures to obtain a product or service if the applicant relied on the exception authorized in this section to procure the product or service.

Amendments Relating to Evidence—§ 75.210(c) Quality of the Project Design, § 75.210(h) Quality of the Evaluation, § 75.266 Consideration for Applications Supported by Strong or Moderate Evidence, § 75.590 Evaluation by the Grantee, and § 77.1 Definitions That Apply to All Department Programs

Comments: Many commenters strongly supported the proposed definitions of “strong evidence of effectiveness” and “moderate evidence of effectiveness” in § 77.1(c). One commenter applauded the Department for expanding the focus on evidence-based practices and stated that this effort will result in higher quality grant applications and outcomes. One commenter noted that the strength of evidence used to support a project's effectiveness should be rigorous, objective, and pertinent to the goals of the project.

A few of these commenters suggested amendments or clarifications to these definitions. One commenter recommended clarifying that a study, in order to meet these definitions, need only meet the appropriate standards outlined in the What Works Clearinghouse (WWC) and not necessarily be reviewed by the WWC or posted on the WWC Web site. Three commenters noted that these definitions, as proposed, present a risk that a study could meet the definition even if the effects are: (1) On trivial or developer-created outcomes; (2) artificially inflated or likely a result of chance; or (3) so small in size as to be of little importance. These commenters recommended revising the definitions to clarify that the study must be of sufficient duration and sample size to represent a valid test and to require that the study find a significant favorable outcome based on a measure of clear policy importance. One of these commenters further suggested requiring that the study have a substantial and important effect on improving student achievement or student growth, closing achievement gaps, decreasing dropout rates, increasing high school graduation rates, or increasing college enrollment and competition rates. The commenters stated that such changes avoid the loophole of classifying programs that lack policy or practical importance as evidence-based programs.

Discussion: We agree with the commenters that the evidence used to support a project's effectiveness should be objective and pertinent to the goals of the project. However, we also recognize that at the various stages of a proposed project's development, different types of evidence are available to assess the effectiveness of a project. That is why we include definitions for four levels of evidence: “Strong evidence of effectiveness,” “moderate evidence of effectiveness,” “evidence of promise,” and “strong theory.” We establish these definitions in order to develop an understanding for applicants of what is required to meet each level of evidence. Combined, these four levels of evidence allow the Department to support effective projects (strong or moderate evidence of effectiveness) or projects that have a high potential to be effective (evidence of promise or strong theory).

We appreciate the commenters' concerns regarding the definitions of “strong evidence of effectiveness” and “moderate evidence of effectiveness.” However, we do not consider the proposed changes necessary as the definitions already safeguard against the risks identified by the commenters.

With regard to ensuring the outcome has policy and practical importance, both of these definitions refer to the effect on a “relevant outcome.” The definition of “relevant outcome” explains that it is the ultimate outcome of the proposed process, product, strategy, or practice and should be consistent with the specific goals of the Department program. Thus, the references to “relevant outcome” would not allow studies that only show effects on trivial outcomes to meet the requirements of the definitions.

Further, the WWC Procedures and Standards Handbook ^[1] explains that a study that does not include a valid or reliable outcome measure, or does not provide adequate information to determine whether it uses an outcome that is valid or reliable, would not meet WWC Evidence Standards. Because the WWC Evidence Standards are incorporated in these definitions, a study that only includes an outcome measure created by the evaluator or developer with weak or no validity or reliability data would not meet the requirements of the definitions.

With regard to requiring that a study has an adequate sample size to ensure that the effect is not artificially inflated, a result of chance, or so small it is of little importance, both definitions refer to the definitions of a “large sample” and a “multi-site sample.” In order for any study to meet the requirements of “strong evidence of effectiveness,” that study would need to have used a “large sample” and a “multi-site sample.” Under the definition of “moderate evidence of effectiveness,” a study must meet, among other requirements, one of the following two conditions: (1) WWC Evidence Standards without reservations; or (2) WWC Evidence Standards with reservations.^[2] Although a small study that meets WWC Evidence Standards “without reservations” would meet one of these two conditions, a study that meets WWC Evidence Standards “with reservations” could only meet the requirements of “moderate evidence of effectiveness” if it used a “large sample” and a “multi-site sample.” We think the inclusion of “large sample” and “multi-site sample” in the definitions of “strong evidence of effectiveness” and “moderate evidence of effectiveness” appropriately mitigate the risks identified by the commenters.

With regard to ensuring that studies are of sufficient duration to meet the requirements, we note that the WWC Evidence Standards do not require a minimum study length. More importantly, because it is not clear that requiring a minimum study length is appropriate or necessary, we decline to revise the definitions to include such a requirement.

Changes: None.

Comment: None.

Discussion: In § 77.1(c), we have revised the definitions of “moderate evidence of effectiveness” and “strong evidence of effectiveness” by adding the phrase “and overriding” to the second parenthetical in paragraphs (a) and (b) of both definitions. We add this phrase to clarify that only studies with unfavorable outcomes that were so substantial as to call into question the potential effectiveness of the proposed project would disqualify the evidence from meeting the condition in the definitions.

Changes: We have revised the second parenthetical in paragraphs (a) and (b) of the definitions of “moderate evidence of effectiveness” and “strong evidence of effectiveness” to add the phrase “and overriding.” The parenthetical now reads “with no statistically significant and overriding unfavorable impacts on that outcome for relevant populations in the study or in other studies of the intervention reviewed by and reported on by the What Works Clearinghouse.”

Comments: One commenter expressed concern that the proposed definitions related to evidence would stifle innovation and that providing special consideration for projects supported by evidence of effectiveness would limit the pool of applications for a competition. Another commenter stated that such consideration is not appropriate for all programs and the proposed amendment appears to be an attempt to turn all projects funded by the Department into Investing in Innovation (i3) projects.

Discussion: The establishment of procedures to provide special consideration for projects supported by strong or moderate evidence of effectiveness provides the Secretary a mechanism to support effective projects and offer incentives to the field for building an evidence base on the effectiveness of the processes, products, strategies, and practices that are, or will be used, in education. However, as noted in our response to other comments in this discussion, we recognize that different types of evidence are available at the various stages of a proposed project's development and that there are some areas where strong or moderate evidence of effectiveness is not yet available. As such, we agree that it would not be appropriate for the Secretary to consider whether a project is supported by strong or moderate evidence of effectiveness for all Department programs. The Secretary will only provide special consideration for projects supported by strong or moderate evidence of effectiveness in programs where such evidence exists or where such incentives are meaningful. When such levels of evidence do not exist, Department program officials may consider whether using “evidence of promise” or “strong theory” would be more appropriate for spurring innovation. Thus, we do not think providing special consideration in program areas that do have these levels of evidence would preclude robust competition or stifle innovation.

Changes: None.

Comments: Three commenters expressed support for the establishment of procedures to provide special consideration for projects supported by strong or moderate evidence of effectiveness. However, these commenters suggested clarifying that the special consideration be given to both existing projects supported by strong or moderate evidence of effectiveness and new projects that are proposing to adopt or adapt models supported by strong or moderate evidence of effectiveness.

Discussion: The definitions of “strong evidence of effectiveness” and “moderate evidence of effectiveness” in § 77.1(c) indicate that the study needs to be of the effectiveness of the proposed process, product, strategy, or practice. These definitions also clarify that the study must overlap with the populations and settings in the proposed project. Therefore, a new project that is adopting the model of the process, product, strategy, or practice in the study meets the definitions. An applicant proposing a new process, product, strategy, or practice that is adapting or changing the model from what was in the study would need to explain how the study supports the adapted version of the model. Thus, a study may be used to support an adaptation of the model in the study so long as the applicant can provide a justification that the proposed project's efficacy necessitates the adaptation, and is based upon the evidence and theory supported by the original study.

Given the variety of programs to which these regulations apply, we do not think it is appropriate for the Department to determine at what single point an adaptation would make the study no longer credible for supporting the effectiveness of the proposed project. However, any programs providing special consideration for projects supported by strong or moderate evidence of effectiveness would provide instructions to applicants on the information they need to submit to demonstrate that they meet the applicable evidence level.

Changes: None.

Comment: One commenter recommended providing special consideration only for projects supported by strong or moderate evidence of effectiveness through the establishment of a separate competition, as opposed to “an across the board competitive preference.”

Discussion: Section 75.266 authorizes the Secretary to establish a separate competition or provide a competitive preference for applications supported by strong or moderate evidence of effectiveness. We decline to limit the Secretary to providing special consideration through a separate competition because that process may not be appropriate for all Department programs. Given the variety of programs to which these regulations apply, it is important that we provide sufficient flexibility for determining which programs require, and how the Secretary would consider, strong or moderate evidence of effectiveness.

Change: None.

Comment: One commenter noted that projects funded by the Department should produce evaluations that meet defined standards but questioned whether the WWC Evidence Standards were appropriate considering the burden associated with conducting evaluations that are designed to meet those standards. Specifically, the commenter expressed concern that small or rural LEAs would not have the capacity to conduct such evaluations and that the Department's use of selection factors promoting WWC Evidence Standards would favor large research organizations over LEAs. The commenter further stated that it is contradictory for the Department to use selection factors that promote evaluations more rigorous than required by the program. To address these concerns, the commenter recommended revising § 75.210(h)(2)(viii)-(x) to require that the proposed project evaluation meets the next level higher or equivalent level of the prior evidence supporting the proposed project's effectiveness.

Discussion: The WWC is an initiative of the Department's Institute of Education Sciences (IES) and serves as a central and trusted source of scientific evidence for what works in education. Although we recognize the WWC Evidence Standards primarily refer to randomized controlled trial (RCT) and quasi-experimental design (QED) studies, we also note that these designs are the most rigorous and defensible methods for producing unbiased evidence of project effectiveness.

We agree with the commenter that conducting project evaluations that are designed to meet the WWC Evidence Standards requires planning and resources. However, because an applicant may obtain an evaluation service provider to conduct the project evaluation through a contract and may include these activities and costs in its proposed project budget, the use of the factors in § 75.210(h)(2)(viii)-(x) would not inherently disadvantage an applicant that lacks the internal capacity to conduct such evaluations. We also note that § 75.210(h) (Quality of the Project Evaluation) is only one criterion among multiple criteria used to evaluate applications. The Department considers each program's purpose, goals, and applicant pool when deciding which selection criteria and factors to use in a given year's competition. By creating these factors under § 75.210(h) (Quality of the Project Evaluation), the Department has the option—not the obligation—to use them to encourage applicants to propose project evaluations that would meet WWC Evidence Standards. Consequently, the Department will use these factors when appropriate for a particular competition and will not use them when doing so would conflict with required program evaluations.

We decline to replace these factors with a factor that would allow a proposed project evaluation to be the equivalent level of the prior evidence supporting the proposed project's effectiveness. In general, to provide the public the greatest return on its investment, evaluations funded by the Department should build on prior research, as appropriate. Although we recognize the importance of replicating results of a past study, we think it is important for applicants to propose project evaluations that increase the level of evidence of the proposed project's effectiveness, as appropriate. By providing the flexibility to select among the various factors under § 75.210(h) (Quality of the Project Evaluation), the Department has the discretion to select factors that are appropriate for the areas of study and research goals for a particular program.

Changes: None.

Comments: Two commenters indicated they had no objections to the proposed changes to §§ 75.210, 75.266, and 77.1 regarding evidence of effectiveness and WWC Evidence Standards, but cautioned the Department to be prudent in their use in discretionary grant competitions. One commenter stated that lack of evidence should not be the sole rationale for deciding not to make a grant to a particular applicant and suggested that the new regulations should not be used to establish a high threshold for evidence of effectiveness in areas where the amount of evidence on existing practice is not strong, particularly in areas that are difficult to measure, such as school climate or efforts to reduce administrative burden or build capacity. Similarly, another commenter recommended that programs establish thresholds for evidence of effectiveness that are commensurate with the quantity and quality of existing evidence in the field.

Discussion: We agree that the new regulations in §§ 75.210, 75.266, and 77.1 regarding evidence of effectiveness should only be used when appropriate for a particular program. We are making changes to these regulations to support evidence-based grant making in areas where evidence exists and to provide incentives and opportunities to build the body of evidence of effectiveness in education.

Changes: None.

Comments: Several commenters noted the distinction between a “project” and a “strategy within a project.” The commenters stated that it might be more appropriate for the Department to evaluate the effectiveness of an individual strategy used by a grantee rather than attempt to measure the effectiveness of a project as a whole. The commenters suggested revising the proposed selection criteria to clarify that programs or strategies could be used when evaluating a project's effectiveness.

Discussion: An applicant may propose to evaluate different strategies within a project using different evaluation design methods. For example, an applicant may propose a pre-post analysis to assess progress of one strategy within its project and a more rigorous evaluation design for another strategy within its project. Despite the flexibility we allow an applicant in designing the proposed project evaluation, under § 75.590 (Evaluation by the grantee), the entire project being supported by Federal funds must be evaluated.

Changes: None.

Comments: A few commenters stated that they could not support the inclusion of selection factors that consider evidence of effectiveness because it would competitively disadvantage certain types of applicants, including those with limited resources or those that serve student populations that have unique needs.

Two commenters specifically suggested that the consideration of effectiveness would present a disadvantage to community colleges. One commenter discussed three challenges for community colleges that make it difficult for them to conduct rigorous evaluations. According to the commenter, rigorous evaluation designs (1) distract from community colleges' missions to provide access to education for all students; (2) often require approval of an Institutional Review Board (IRB), and many community colleges have not established IRBs; and (3) require signed consent from participants, which creates additional complications and concerns regarding student access to educational programs or support services. Both commenters stated that most studies posted on the WWC Web site focus on K-12 education and that existing research around community colleges is insufficient for them to compete if factors related to evidence of effectiveness are used by the Department. To address this concern, one commenter recommended creating a special track of priority funding for empirical research on community colleges in all of the Department's postsecondary programs.

One commenter suggested modifying the new selection factors to exempt programs with “historical evidence of benefit to students.” The commenter stated that programs that equalize educational opportunity among low-income, first-generation college students who, in large part, are from underrepresented groups, should not be required to use quantitative research to determine their effectiveness.

One commenter stated that programs providing parental training and engagement services would be disadvantaged by selection factors related to evidence of effectiveness because such programs require a focus on individual parent and family needs. The commenter expressed concern that the use of these selection factors, or any special consideration given for evidence of effectiveness, would limit which entities could apply to a particular program without providing a clear benefit to children and their families. Another commenter suggested that the selection factors referring to “evidence of promise” and “strong theory” be the only selection factors related to evidence of effectiveness used for implementation-based grants.

Discussion: As noted elsewhere in our response to comments in this discussion, we agree that the selection factors relating to evidence of effectiveness, whether they fall under § 75.210(c) (Quality of the Project Design) or § 75.210(h) (Quality of the Project Evaluation), should only be used when appropriate for a particular program. We include these selection factors to support evidence-based grant making in areas where evidence exists and to provide incentives and opportunities to build the body of evidence of effectiveness in education.

Because the Department has the discretion to select factors that are appropriate for the areas of study and research goals of a particular program, and therefore would not select factors that would require applicants to provide evidence of effectiveness in areas that have not been widely researched, we decline to remove these factors. Additionally, we do not think the amount of research for a particular area of education on the WWC Web site is a reason not to add these factors to the regulations. A study does not need to be reviewed by the WWC or posted on the WWC Web site to meet the WWC Evidence Standards. Department program officials could use research available on the WWC Web site or from other sources to inform their decision on whether these selection factors are appropriate for the particular program.

Further, we note that these factors address evidence of effectiveness and evaluation of effectiveness at various levels. Two of the factors refer to grantees proposing evaluation designs that meet WWC Evidence Standards with or without reservations, but we also include two other factors that refer to “evidence of promise” and “strong theory.” Including four levels of evidence provides the Department the opportunity to consider the level of evidence available in the field for the types of projects to be funded by the relevant program and the capacity of potential applicants to design evaluations that would assess the effectiveness of a project at these different levels.

With regard to the other issues raised by the commenters, we recognize that rigorous evaluation designs require grantees to compare individuals participating in the project to those who are not participating in the evaluation. However, requiring more rigorous evaluation designs does not contradict the educational mission of serving all students because evaluating the effectiveness of a particular intervention is necessary to understand which interventions effectively improve student outcomes. Although funds that support evaluation services cannot also support direct services to students, investing in evaluation allows entities to focus finite resources on only those processes, products, strategies, or practices that are most effective in improving student outcomes. Therefore, we do not think evaluating the effectiveness of a project using a rigorous design would impede an entity from carrying out its educational mission. Further, because an applicant may contract to obtain an evaluation service provider that has access to an IRB to conduct the project evaluation, and because the applicant may include these activities or activities related to accessing an independent IRB or establishing its own IRB to support the project evaluation and their costs in its proposed budget, we do not think applicants that lack their own IRBs are disadvantaged. Similarly, because under the Common Rule for the Protection of Human Subjects, an IRB can modify or waive requirements for written consent, and the costs for activities to obtain written consent from participants may be included in the proposed budget, we do not think a specific type of entity is inherently disadvantaged by the use of selection factors that encourage applicants to propose rigorous evaluations of their projects' effectiveness.

With regard to the comments recommending that these factors not be used for programs that historically have benefited students and that we only allow the use of “evidence of promise” and “strong theory” for implementation grants, we reiterate the importance of the Department supporting the improvement of information available to practitioners and policymakers about which practices work, for which types of students, and in which contexts. These selection factors support that goal by providing incentives to applicants for grants to build an evidence base on the effectiveness of the processes, products, strategies, and practices that are, or will be used, in education.

Changes: None.

Comments: A few commenters stated that the Department should not support evaluations using research designs that include control groups. Two commenters stated that experimental designs are unethical because they require grantees to withhold treatment from students in public education. Another commenter felt that it was inappropriate to deny services to students simply to accommodate research and evaluation, particularly when the Congress authorized and funded the program to provide services to students. One commenter further expressed concern that favoring such designs would provide an advantage to large research organizations over LEAs that lack the capacity to conduct such evaluations.

In addition to concerns about placing students or teachers in “experimental” versus “control” groups, one commenter cited challenges regarding the use of RCTs. Specifically, the commenter noted that a pure control condition is rare because fidelity of implementation can significantly impact the effectiveness of a process, product, strategy, or practice. The commenter indicated that how well a particular process, product, strategy, or practice works depends on the conditions under which it is implemented, and the costs of observation and metrics to determine the fidelity of that implementation are significant. The commenter also noted that, because products and services are constantly changing and improving, the products or services are frequently no longer available in the format or version that was studied by the time an evaluation is complete. The commenter concluded that equal consideration of alternative study designs, such as regression discontinuity designs, is needed to ensure the Department does not limit the development, implementation, and evaluation of innovative projects.

One commenter expressed concern that the selection factors that refer to “evidence of promise” and “strong theory” would be used to make all implementation projects into randomized research projects with control groups. According to the commenter, these selection factors would skew successful applications toward projects conducting research studies and away from projects providing services to students and teachers. The commenter stated that to require such project evaluations is not consistent with legislative intent and would not result in improved student outcomes. Another commenter made a similar statement that the proposed amendments regarding evidence of effectiveness and evaluation should not be used for the TRIO programs because they would undermine the intent of the Higher Education Opportunity Act of 2008.

Discussion: We appreciate the commenters' concerns about whether it is ethical to evaluate the effectiveness of a project using a randomized experimental design. In order to ensure ethical research, the Department, under the Common Rule for the Protection of Human Subjects in Research (34 CFR 97), does not permit covered human subjects research to be initiated until it has been reviewed by an IRB and receives the Department's protection of human subjects clearance. Although we recognize that conducting these types of evaluations can be difficult, we also recognize that random assignment of entities (students, teachers, schools, or other units of analysis) to a treatment or control group is the most effective way to eliminate plausible competing explanations for observed differences between treated and non-treated individuals or groups (i.e., the estimated treatment effect). Two common strategies used in randomized experiments in education that are designed to address this ethical concern are the use of a “wait-list” control group and the assignment of schools, rather than students, to the treatment group. Despite the challenges in conducting such evaluations, we consider it important to provide a mechanism for the Department, when appropriate, to use these selection factors to encourage grantees to conduct evaluations of their projects that will improve the information available to practitioners and policymakers about which processes, products, strategies, and practices work, for which types of students, and in which contexts.

We disagree that the regulation would favor large research institutions over other types of applicants. Applicants that do not feel they possess adequate resources to carry out a rigorous evaluation of their proposed projects may contract with an evaluation service provider. Applicants can use the exception in § 75.135 to procure the needed expertise to implement a rigorous evaluation.

We disagree with the comment that the use of selection factors referring to “evidence of promise” and “strong theory” requires all implementation projects to become randomized research projects with control groups. We define “strong theory” to mean a rationale for the proposed process, product, strategy, or practice that includes a “logic model” (as defined in § 77.1(c)). The development of a logic model and the associated rationale does not require a grantee to conduct a randomized experiment. Similarly, although a grantee may use a QED or RCT to meet the “evidence of promise” definition, a grantee could also use a correlational study with statistical controls. Thus, neither evidence level requires the use of a treatment and control group. See § 77.1(c) for definitions of these terms.

Although we appreciate that products and services change over time and may no longer be available in the format or version that was studied by the time an evaluation is complete, these selection factors are intended to provide incentives to applicants for grants to build an evidence base on the effectiveness of the processes, products, strategies, and practices that are, or will be used, in education. As noted elsewhere in our discussion of comments related to evidence of effectiveness, an applicant may use a study of a product or service to support an adaptation of it so long as the applicant can provide a justification that the proposed project's efficacy necessitates the adaptation, and is based upon the evidence and theory supported by the original study. This same concept applies to the potential for learning from a project evaluation of a product or service that may no longer be available in the format or version that was studied by the time an evaluation is complete. Moreover, the selection factor regarding “evidence of promise” does allow consideration of alternative study designs.

With regard to commenters' concerns about the Department requiring the use of evaluation designs that are in conflict with a program's statute, the Department does not propose or require grantees to use grant funds in a manner that is prohibited by statute. As noted elsewhere in our responses to comments in this section, the Department has discretion in determining which selection factors are most appropriate for a particular program's purpose and goals. Therefore, the Department would not use a selection factor that is in conflict with a program's governing statute, purpose, or goals.

Changes: None.

Comments: Several commenters recommended including a selection factor under § 75.210(h) (Quality of the Project Evaluation) to promote evaluative methods for small service delivery programs. Specifically, the commenters requested the addition of a selection factor for studies that assess or use best practices for service delivery strategies using small-scale pilots.

Discussion: We agree that a selection factor that encourages project evaluations of pilot strategies that may be best practices is of value, and we have included § 75.210(h)(2)(xii) for this purpose. This selection factor supports project evaluations that clearly articulate the key components and outcomes of the grant-supported process, product, strategy, or practice, as well as the measurable threshold for acceptable implementation. Implementation studies that articulate the key components of the proposed project and the measureable threshold for acceptable implementation of the key components are necessary to disseminate information about and replicate best practices. These studies also could be used to evaluate a pilot of service delivery strategy. Because § 75.210(h) (Quality of the Project Evaluation) and other existing factors under this criterion provide for the type of evaluation proposed by the commenter, we do not think it is necessary to create a factor for the evaluation of a specific type of project.

Changes: None.

Selection Criteria Based on Statutory or Regulatory Provisions—§ 75.209

Comment: One commenter agreed with the proposed changes to § 75.209 but suggested including a clause that explicitly allows for successful applicants to be able to adjust their projects based on a continuous improvement model. The commenter noted that this change would allow grantees to use formative evidence and research to adjust their projects as needed, resulting in better outcomes overall.

Discussion: We agree that continuous improvement models are useful. In fact, grantees currently address their lessons learned during the implementation of the project and discuss how they can continuously improve their projects in their annual performance reports to the Department. The regulations do not prohibit a grantee from adjusting its project as needed, provided that the scope or objectives of the project are not changed. Our intent in the changes to § 75.209 is to provide the Secretary the flexibility to use selection criteria related to a program's statute or regulations in notices inviting applications.

Changes: None.

General Selection Criteria—§ 75.210

Comments: Many commenters expressed general concern over the proposed changes to § 75.210(c) (Quality of the Project Design) without focusing on any one proposed factor. Some noted that the proposed selection factors under § 75.210(c) are not widely applicable to all Department programs and that some selection factors may unfairly disadvantage some applicants. For example, one commenter asserted that the proposed selection factors will not improve student outcomes and are therefore unnecessary. Another commenter expressed concern that the proposed selection factors allow the Secretary too much discretion when designing competitions and that the intent and purpose of some already-established programs could be compromised.

Alternatively, one commenter suggested that many of the proposed selection factors rely too much on peer reviewer subjectivity and further that inter-rater reliability between peer reviewers would be difficult to achieve if these factors are used in a competition.

Discussion: We agree that each selection factor in § 75.210(c) (Quality of the Project Design) is not applicable to each Department program. There is no requirement that the Department use each selection factor listed in § 75.210(c) (Quality of the Project Design) for every program or competition. We rely on Department program officials to choose the selection factors for their programs prudently, with the capacity of applicants in mind, such that the selection factors used will appropriately match the goals of the program.

As part of the discretionary grant process, we depend on peer reviewers for their objectivity and professional expertise. The Department trains peer reviewers on the details of the particular program, and monitors peer reviewer discussions to ensure that reviewers make scoring decisions based solely on the selection criteria provided in the notice inviting applications. While the Department takes these steps to support inter-rater reliability, we also rely on the professional judgment and expertise of peer reviewers when evaluating applications.

Although some factors may not directly relate to student achievement, we disagree with the comment that the new selection factors in § 75.210(c) (Quality of the Project Design) will not improve student achievement. Each factor requires applicants to approach the design of their projects in ways that will increase efficiency, productivity, and overall success. Increased student achievement will result from a Department competition's use of selection factors that better allow grantees to implement their projects effectively.

Changes: None.

Comment: One commenter agreed with the proposed change to § 75.210(c)(2)(xvi) regarding integration of a proposed project with similar or related efforts. The commenter stated that emphasizing integration efforts within the grantee's community would increase the likelihood of a successful project. The commenter noted that this proposed factor is particularly appropriate for public charter schools, given their unique positions in their communities.

Some commenters expressed concern that the proposed change to § 75.210(c)(2)(xvi) would disadvantage nonprofit entities. These commenters noted that nonprofit entities do not necessarily have control over State or Federal funding streams and may have difficulty securing willing community partners. One commenter expanded on this concern, and suggested that we clarify that the integration of existing funding streams only be considered to the extent practicable for applicants. Similarly, another commenter noted that in some high-need areas, opportunities to partner with funding organizations simply do not exist.

One commenter suggested that § 75.210(c)(2)(xvi) regarding increased efficiency and productivity and (xxvii) regarding using nonpublic funds or resources to build on similar or related efforts be used only as competitive preference priorities. The commenter stressed that the proposed selection factors could disadvantage small and rural LEAs, and potentially eliminate applicants with otherwise strong responses to the criterion due solely to their inability to secure other sources of funding. The commenter also warned that a nonpublic entity may have its own agenda when agreeing to partner with an applicant, possibly complicating the nature of the partnership.

One commenter expressed concern that nonpublic funds and resources were given favor in § 75.210(c)(2)(xxvii). The commenter argued that integration of resources is important regardless of their source, be it public or private. Based on this argument, the commenter suggested the selection factor be removed.

Discussion: We agree that the proposed selection factors will increase the likelihood of grantee success in implementing their projects. Requiring an applicant to create partnerships with community, State, or Federal partners establishes early on that project sustainability is an important factor for success, in the event that the proposed project is awarded grant funds.

We recognize that establishing partnerships with community organizations may be challenging for some grantees, particularly those based in high-need areas. However, we would like to clarify that, under amended selection factor § 75.210(c)(2)(xvi), a grantee is not limited to using organizations based in its community when selecting partners. Grantees may choose to integrate or build on the related efforts of other programs based anywhere in the country, assuming that the goals and efforts of such programs align appropriately with those of the grant.

We also recognize that small and rural LEAs may experience challenges when responding to the selection factor. We would like to stress that this change to the selection criterion in § 75.210(c) (Quality of Project Design) does not require its use in each Department competition. The Department chooses appropriate selection factors based on the intended goals of the program and the expected capacity of applicants.

We disagree that changing the proposed selection factors into competitive preference priorities would prove more favorable for small and rural LEAs. Competitive preference priorities, by providing grantees who address them with an advantage over those who choose not to do so, can be decisive in determining which applicants receive grants. Therefore, it is not clear that considering the integration of other funding streams as a competitive preference priority, as opposed to a selection factor, would address the commenter's concern.

Nonprofit entities, while typically not in a position to control State or Federal funding streams, do have options available to them to address § 75.210(c)(2)(xvi). This selection factor considers the extent to which an applicant has secured partners to build on similar efforts. A nonprofit entity, if it were eligible to apply for funding in a Department program, could collaborate with an LEA, which is likely receiving State and Federal funding, on a mutually beneficial project that aligns with the goals of the Department program.

While we agree that § 75.210(c)(2)(xxvii) regarding integrating with or building on related efforts may not be appropriate for some Department programs, we are retaining it because there are Department programs in which it would be beneficial. Because the use of this selection factor is not required for use in all Department programs or competitions, we think the best approach to addressing this concern is for the Department to use the selection factor in only those programs for which it is appropriate.

In response to the concern that a nonpublic entity may take advantage of an applicant and complicate the nature of the partnership to promote its own agenda, applicants applying to a competition in which § 75.210(c)(2)(xxvii) is a selection factor should use their best judgment in evaluating potential partners and only enter into formal relationships with entities that share their goals. This is generally a prudent practice, whether the applicant is choosing to partner with a public or a nonpublic entity, and should be followed in any competition that requires an applicant to work with a partner, even if § 75.210(c)(2)(xxvii) is not a selection factor.

Changes: None.

Comment: Some commenters praised the proposed selection factor § 75.210(c)(2)(xxiv) regarding resources for operating a project beyond the length of the grant. Commenters also supported § 75.210(c)(2)(xxv), which asks applicants to describe the potential and planning for the incorporation of project purposes and activities into the ongoing work of the grant. These commenters stated that grant funds should not be awarded in cases where long-term funding is needed but not secured and that asking an applicant to explicitly address how it plans to continue the project after the completion of a grant award will help to ensure long-term success.

Two commenters expressed agreement with the proposed changes to § 75.210(c)(2)(xxiv) and (xxv), but suggested some further modifications. One commenter suggested that we consider whether applicants have effectively worked to build a market for educational services. Another commenter suggested that when considering the extent to which an applicant has secured resources to sustain the project beyond the grant period, we also consider whether the applicant has demonstrated evidence of broad stakeholder commitment to the project.

One commenter agreed that it is critical that grantees plan their projects with sustainability in mind but did not agree with the addition of selection factors § 75.210(c)(2)(xxiv) and (xxv), arguing that the current state of the economy is not stable enough to ensure that resources committed at the time an award is made would still be available at the end of a grant period. Another commenter noted that the proposed changes could disadvantage community colleges and proposed that we consider an applicant's cost per student when using proposed selection factors related to sustainability. The commenter stated that applicants working with fewer resources per student need more flexibility in adhering to the requirements outlined in selection factor § 75.210(c)(2)(xxiv).

Discussion: We agree that long-term planning and broad stakeholder support are integral to a grantee's successful project. The amendments to § 75.210(c) (Quality of the Project Design) will allow for flexibility when assessing an applicant's plan to sustain its project after the grant period ends. With added flexibility in § 75.210(c), the Department may choose to fund applications that have a strong focus in effective and sustainable practices.

We recognize that some applicants, such as community colleges, may operate with fewer resources per student than other types of applicants. However, the regulations do not prohibit such an applicant from requesting funds in its budget proposal to support the proposed project fully. If an applicant assesses its resources and finds that it requires more funds per student to carry out the project and address selection factor § 75.210(c)(2)(xxiv), that applicant should plan its budget accordingly. It is important that an applicant have the discretion to determine how best to address its sustainability needs. For example, an applicant may design its project to include strategies that build its capacity to implement project activities more efficiently, which in turn would support sustainability after the grant.

The proposed selection factors related to sustainability are designed with the current economic climate in mind. As a few commenters noted, applicants should be actively planning on how those realities will affect their proposed projects. The intent of selection factors § 75.210(c)(2)(xxiv) and (xxv) is to encourage applicants to engage in this planning process and comprehensively plan how their projects could be implemented beyond the grant period if such projects were funded.

Finally, regarding the recommendation that we include a factor considering whether an applicant effectively worked to build a market for educational services, we note that we have added a new selection criterion, § 75.210(i) (Strategy to Scale), which includes selection factor § 75.210(i)(2)(v), that considers whether an applicant demonstrates an unmet demand for the process, product, strategy, or practice that will enable the applicant to reach the level of scale that is proposed in the application. This factor recognizes work that an applicant would do in advance of the project to build a market for educational services.

Changes: None.

Comment: Two commenters noted specifically the importance of productivity and efficiency, stating that selection factor § 75.210(c)(2)(xxvi) is especially appropriate given the current climate of limited resources and high expectations for success.

Discussion: We agree that productivity and efficiency have become increasingly important factors to consider in recent years. Allowing the Secretary to evaluate whether a proposed project is efficient and productive will ensure that Department funds are used as effectively and prudently as possible.

Changes: None.

Comments: Two commenters agreed with the addition of a new selection criterion, § 75.210(i) (Strategy to Scale), to consider an applicant's ability to successfully scale a project at the regional or national level. One commenter noted that the proposed addition would specifically benefit charter management organizations and support them in scaling successful strategies, and the other noted the importance of sharing best practices broadly.

Another commenter expressed support for the selection criterion in § 75.210(i) but requested that we allow for-profit entities, as well as nonprofit entities, to partner with grantees to bring their projects to scale during the grant period. The commenter stated that scaling has not historically been an area of expertise for entities providing educational services and that for-profit entities are well-suited to provide needed support.

Discussion: We agree that, in many Department programs, an applicant's ability to scale a proposed process, product, strategy, or program is very important. As the Department begins and continues to use this selection criterion, we expect potential applicants will devote resources and supports to focus on the processes, products, strategies, and programs that have greater potential to scale.

Should a grantee decide that it needs additional help in the area of scalability, that grantee is not obligated to seek assistance from only nonprofit entities. The proposed selection criterion, as written, does not explicitly refer to the types of entities with which a grantee may choose to work. We recognize that some for-profit entities may be particularly well-positioned to help grantees achieve scale, and encourage each grantee, to the extent it requires additional support, to seek out partners that are best suited to meet the needs of their projects.

Changes: None.

Comments: Many commenters noted that while the strategy to scale is an important criterion to consider for new projects, it is not applicable to programs that have already established successful practices at a national scale or to programs that are already widely available to students.

Conversely, some commenters expressed concern that the proposed selection criterion § 75.210(i) (Strategy to Scale) would not be feasible for small LEAs, rural LEAs, or community colleges. One commenter requested revising the language of the proposed criterion to “Feasibility of Replication” and placing it as a selection factor under § 75.210(h) (Quality of the Project Evaluation). This commenter also suggested that the subject of scaling a project is better suited to an IES grant. Another commenter noted that the selection criterion should instead be used only as a selection factor in specific circumstances because an applicant's capacity to scale is not a useful consideration for all Department programs. Another commenter added that some programs are, by definition, small and community based and that the use of this criterion would adversely affect such programs.

One commenter did not agree with the proposed selection criterion concerning an applicant's strategy to scale, noting that increasing focus on scaling projects to regional and national levels would decrease focus on student outcomes at the local level. The commenter also points out that many projects are effective because they are planned with a specific place in mind, and scaling such projects could prove ineffective.

Discussion: We recognize that the proposed selection criterion may not be applicable to Department programs that are already well established. We stress that Department program officials are in the best position to decide which selection criteria and factors fit the goals of their programs. When preparing notices inviting applications, the Department will continue to consider the strengths and needs of likely applicants, and will choose selection criteria that are appropriate to the program's purpose, goals, and applicant pool. Likewise, if the Department concludes that the nature of the program or types of applicants are not conducive to scaling, then the Department may decide not to include the selection criterion in the notice inviting applications for the program.

If the Department concludes the use of this criterion is consistent with the program's purpose and goals then applicants that better address the criterion will likely receive more points for the criterion than applicants that address it poorly. We recognize that some types of applicants may not typically design or implement projects that include activities to support effectively scaling a proposed process, product, strategy, or practice; however, any applicant responding to a notice inviting applications that includes this criterion may consider partnering with others to take the proposed process, product, strategy, or practice to scale.

We do not agree with the suggestion to change selection criterion § 75.210(i) (Strategy to Scale) into a selection factor under selection criterion § 75.210(h) (Quality of the Project Evaluation). The nuances needed to make the free-standing selection criterion useful and adaptable to a variety of Department programs would be lost if it were re-written as a selection factor under another criterion. It is important that a grantee experiencing success be able to share information about its project and support broad implementation of it to ensure that best practices are widely accessible and more easily replicated in the field. We think that by including § 75.210(i) (Strategy to Scale) as a selection criterion, as opposed to a selection factor within a selection criterion, we are able to provide clearer guidance to applicants on effective scaling methodology and feasible replication.

We disagree that including a selection criterion that considers an applicant's ability to effectively scale its proposed process, product, strategy, or practice would decrease focus on student outcomes at the local level. By choosing to consider one selection criterion, the Department does not diminish the influence of other selection criteria under consideration. For example, if Department program officials choose to consider § 75.210(c) (Quality of the Project Design) and § 75.210(i) (Strategy to Scale), a successful applicant would be expected to respond effectively to both criteria. That applicant would therefore need to explain why its project design is effective in increasing student outcomes in its current setting and explain its capacity to scale. While § 75.210(i) (Strategy to Scale) primarily considers how well an applicant could implement its proposed process, product, strategy, or practice in a variety of settings and populations, it remains one piece among many to be considered as part of the competition process.

Changes: None.

Maximum Funding Period—§ 75.250

Comments: Many commenters expressed support for the change to this regulation because it will improve the quality of the data available to determine whether educational activities improve teaching and learning. However, one of these commenters stated that the option for funding for continued evaluation should be guaranteed. The commenter also suggested that grantees be allowed to use funds to hire qualified data management personnel and consultants to develop data architecture and data storage capacity.

Discussion: We appreciate the commenters' support for this regulation. However, we cannot guarantee continued data collection periods for all programs and grantees because this option is only possible in cases where there is authority for evaluation activities and sufficient funds have been appropriated for the program. Because these conditions may not be met for all programs or in all years, we cannot guarantee a continued data collection period for all programs and grantees.

With regard to the recommendation that the Department allow grantees to use grant funds to support personnel or contracts to assist with data collection, we note that, under the current regulations and cost principles, applicants may include such costs in their proposed budgets to contract for these services so long as they are necessary to the performance measurement and evaluation of the project.

Changes: None.

Comment: One commenter recommended clarifying whether the Secretary could approve a data collection period without providing additional funds. The commenter explained that in some cases grantees may need both additional time and funds in order to complete performance measurement activities but that there are also instances when a grantee may need only additional time.

Discussion: Under § 75.261, a grantee may request a no-cost extension of its project period to complete approved project activities. Thus, the regulations already allow grantees to request additional time to complete performance measure and other project activities without additional funding, so long as the appropriation accounts remain available. Funds obligated on a fiscal year basis remain available in grant accounts for five fiscal years after the expiration of the fiscal year for which the funds were obligated by the Federal government. 31 U.S.C. 1552(a). Thus, both obligated and unobligated grant funds generally remain available to grantees during no cost extensions to fund continued collection of data after the end of a project period.

The amended regulations in § 75.250 allow the Secretary to approve a data collection period with or without additional funds. Prior to the approval of a data collection period, we would assess with the grantee the appropriate duration of the data collection period and whether additional funds are necessary to complete the data collection, reporting, and analysis that would occur during that period.

Changes: None.

Comment: One commenter stated that data collection is not the only valid reason for extending a grant period and suggested revising the regulations to allow extensions on the basis of effectiveness and to aid in a project's transition to a new funding stream.

Discussion: These amendments apply to discretionary grant programs that award funds on the basis of a competitive process. As such, it would not be appropriate for the Department to award additional funds to a grantee to conduct a new project or transition to a new funding stream outside of the competitive process.

Changes: None.

Continuation of a Multi-Year Project After the First Budget Period—§ 75.253

Comments: Three commenters expressed support for the change in this regulation. One commenter stated that the change would improve the use of performance measurement and evaluation. However, one of these commenters requested additional information on the meaning of “substantial progress.” Another commenter urged establishing program-specific evaluation requirements that balance the need for valid evidence of effectiveness with the need to limit burden on grantees.

Discussion: We appreciate the commenters' support for the amended regulation. Given the variety of programs to which these regulations apply, a more detailed definition of “substantial progress” would not be practical or helpful. The Department will establish program-specific evaluation requirements in the notice inviting applications and will consider the program's purpose, goals, and applicant pool when deciding the evaluation requirements to use in a given year's competition. As part of this process, the Department must consider the burden of the information collection associated with the application and program requirements and receive approval under the Paperwork Reduction Act of 1995 from OMB to collect that information. Because current law requires programs to consider the burden associated with information collection, we do not think it is necessary to make the change requested by the commenter.

Changes: None.

Other Comments Not Directly Related to the Proposed Rule

Comment: One commenter stated that the amendments were unclear and would produce nothing of value for students in this country.

Discussion: Although we recognize these amendments may not directly affect students, we disagree with the assessment that they would not produce anything of value. These amendments are designed to support the successful implementation of projects funded by the Department and improve their performance measures, which will in turn benefit students served by the projects. The proposed amendments also allow the Department to be more effective and efficient when selecting discretionary grantees, to provide higher-quality data to the Congress and the public about the effectiveness of Department programs, and to reduce administrative burden on applicants and grantees.

Changes: None.

Comment: One commenter expressed concern that students with disabilities are underserved.

Discussion: We appreciate the commenter's concern and note that section 504 of the Rehabilitation Act of 1973 and the Department's section 504 implementing regulations prohibit discrimination on the basis of disability for entities receiving financial assistance from the Department. In addition, the Department enforces Title II of the Americans with Disabilities Act (ADA), as well as the regulations implementing Title II of the ADA, which prohibit discrimination on the basis of disability by public entities. Finally, section 427 of the General Education Provisions Act (GEPA) addresses equitable access by requiring all applicants to provide a statement that identifies access barriers to participation, which can include barriers to participation by individuals with disabilities, in their projects and identifies solutions to overcome those barriers.

Moreover, the Department's Office of Special Education and Rehabilitative Services (OSERS) is committed to improving results and outcomes for people with disabilities of all ages. OSERS provides a wide array of supports to parents and individuals, school districts, and states in three main areas: Special education, vocational rehabilitation, and research. Within OSERS, the Office of Special Education Programs (OSEP) supports a comprehensive array of programs and projects authorized by the Individuals with Disabilities Education Act (IDEA) that improve results for infants, toddlers, children, and youth with disabilities.

Changes: None.

Comment: One commenter recommended adding language to §§ 76.722 and 80.40 to clarify that, although a grantee may require subrecipients to submit reports in a manner and format that enable the grantee to comply with Department requirements, an SEA must not do so in a manner that would place financial or programmatic burden on the subrecipient or require a subrecipient to provide data that is readily available to the SEA by other means. The commenter noted that monitoring subrecipients is vital to the successful implementation of a grant, but a grantee should not use it to usurp autonomy of subrecipients or to require the use of specific financial software that could be costly or burdensome to small entities.

Discussion: In the preamble of the NPRM, we discussed on page 74392 the Department's retrospective analysis of existing regulations and requested comment on other regulations within EDGAR that may be in need of modification. We appreciate this commenter's concerns regarding §§ 76.722 and 80.40 and will use the feedback to further inform and plan our retrospective review efforts.

Changes: None.

Executive Orders 12866 and 13563

Regulatory Impact Analysis

Under Executive Order 12866, the Secretary must determine whether this regulatory action is “significant” and, therefore, subject to the requirements of the Executive order and subject to review by the Office of Management and Budget (OMB). Section 3(f) of Executive Order 12866 defines a “significant regulatory action” as an action likely to result in a rule that may—

(1) Have an annual effect on the economy of $100 million or more, or adversely affect a sector of the economy, productivity, competition, jobs, the environment, public health or safety, or State, local, or tribal governments or communities in a material way (also referred to as an “economically significant” rule);

(2) Create serious inconsistency or otherwise interfere with an action taken or planned by another agency;

(3) Materially alter the budgetary impacts of entitlement grants, user fees, or loan programs or the rights and obligations of recipients thereof; or

(4) Raise novel legal or policy issues arising out of legal mandates, the President's priorities, or the principles stated in the Executive order.

This final regulatory action is a significant regulatory action subject to review by OMB under section 3(f) of Executive Order 12866.

We have also reviewed these regulations under Executive Order 13563, which supplements and explicitly reaffirms the principles, structures, and definitions governing regulatory review established in Executive Order 12866. To the extent permitted by law, Executive Order 13563 requires that an agency—

(1) Propose or adopt regulations only on a reasoned determination that their benefits justify their costs (recognizing that some benefits and costs are difficult to quantify);

(2) Tailor its regulations to impose the least burden on society, consistent with obtaining regulatory objectives and taking into account—among other things and to the extent practicable—the costs of cumulative regulations;

(3) In choosing among alternative regulatory approaches, select those approaches that maximize net benefits (including potential economic, environmental, public health and safety, and other advantages; distributive impacts; and equity);

(4) To the extent feasible, specify performance objectives, rather than the behavior or manner of compliance a regulated entity must adopt; and

(5) Identify and assess available alternatives to direct regulation, including economic incentives—such as user fees or marketable permits—to encourage the desired behavior, or provide information that enables the public to make choices.

Executive Order 13563 also requires an agency “to use the best available techniques to quantify anticipated present and future benefits and costs as accurately as possible.” The Office of Information and Regulatory Affairs of OMB has emphasized that these techniques may include “identifying changing future compliance costs that might result from technological innovation or anticipated behavioral changes.”

We are issuing these final regulations only on a reasoned determination that their benefits justify their costs. In choosing among alternative regulatory approaches, we selected those approaches that maximize net benefits. Based on the analysis that follows, the Department believes that these proposed regulations are consistent with the principles in Executive Order 13563.

We also have determined that this regulatory action does not unduly interfere with State, local, or tribal governments in the exercise of their governmental functions.

In accordance with both Executive orders, the Department has assessed the potential costs and benefits, both quantitative and qualitative, of this regulatory action. The potential costs associated with this regulatory action are those resulting from statutory requirements and those we have determined as necessary for administering the Department's programs and activities.

Paperwork Reduction Act of 1995

These regulations do not contain any information collection requirements.

Intergovernmental Review

These regulations subject to the requirements of Executive Order 12372 and the regulations in 34 CFR part 79. One of the objectives of the Executive order is to foster an intergovernmental partnership and a strengthened federalism. The Executive order relies on processes developed by State and local governments for coordination and review of proposed Federal financial assistance.

This document provides early notification of our specific plans and actions for these regulations.

Assessment of Educational Impact

In the NPRM we requested comments on whether the proposed regulations would require transmission of information that any other agency or authority of the United States gathers or makes available.

Based on the response to the NPRM and on our review, we have determined that these final regulations do not require transmission of information that any other agency or authority of the United States gathers or makes available.

Accessible Format: Individuals with disabilities can obtain this document in an accessible format (e.g., braille, large print, audiotape, or compact disc) on request to the program contact person listed under FOR FURTHER INFORMATION CONTACT.

Electronic Access to This Document: The official version of this document is the document published in the Federal Register. Free Internet access to the official edition of the Federal Register and the Code of Federal Regulations is available via the Federal Digital System at: www.gpo.gov/​fdsys. At this site you can view this document, as well as all other documents of this Department published in the Federal Register, in text or Adobe Portable Document Format (PDF). To use PDF you must have Adobe Acrobat Reader, which is available free at the site.

You may also access documents of the Department published in the Federal Register by using the article search feature at: www.federalregister.gov. Specifically, through the advanced search feature at this site, you can limit your search to documents published by the Department. You may also view this document in text [or PDF] at the following site:

List of Subjects

34 CFR Part 75

• Accounting
• Copyright
• Education
• Grant programs—education

34 CFR Part 77

• Education
• Grant programs—education

Dated: August 6, 2013.

Arne Duncan,

Secretary of Education.

For the reasons discussed in the preamble, the Secretary amends parts 75 and 77 of title 34 of the Code of Federal Regulations as follows:

PART 75—DIRECT GRANT PROGRAMS

1. The authority citation for part 75 continues to read as follows:

Authority: 20 U.S.C. 1221e-3 and 3474, unless otherwise noted.

2. Add a new § 75.110 to read as follows:

§ 75.110

Information regarding performance measurement.

(a) The Secretary may establish in an application notice for a competition one or more performance measurement requirements, including requirements for performance measures, baseline data, or performance targets, and a requirement that applicants propose in their applications one or more of their own performance measures, baseline data, or performance targets.

(b) If an application notice requires applicants to propose project-specific performance measures, baseline data, or performance targets, the application must include the following, as required by the application notice:

(1) Performance measures. How each proposed performance measure would accurately measure the performance of the project and how the proposed performance measure would be consistent with the performance measures established for the program funding the competition.

(2) Baseline data. (i) Why each proposed baseline is valid; or

(ii) If the applicant has determined that there are no established baseline data for a particular performance measure, an explanation of why there is no established baseline and of how and when, during the project period, the applicant would establish a valid baseline for the performance measure.

(3) Performance targets. Why each proposed performance target is ambitious yet achievable compared to the baseline for the performance measure and when, during the project period, the applicant would meet the performance target(s).

(c) If the application notice establishes performance measurement requirements, the applicant must also describe in the application—

(1)(i) The data collection and reporting methods the applicant would use and why those methods are likely to yield reliable, valid, and meaningful performance data; and

(ii) If the Secretary requires applicants to collect data after the substantive work of a project is complete regarding the attainment of certain performance targets, the data collection and reporting methods the applicant would use during the post-performance period and why those methods are likely to yield reliable, valid, and meaningful performance data.

(2) The applicant's capacity to collect and report reliable, valid, and meaningful performance data, as evidenced by high-quality data collection, analysis, and reporting in other projects or research.

(Authority: 20 U.S.C. 1221e-3 and 3474)

3. Add a new undesignated center heading “Competition Exceptions” in subpart C immediately before the undesignated center heading “State Comment Procedures”.

4. Add a new § 75.135 to subpart C under the undesignated center heading “Competition Exceptions” to read as follows:

§ 75.135

Competition exception for proposed implementation sites, implementation partners, or service providers.

(a) When entering into a contract with implementation sites or partners, an applicant is not required to comply with the competition requirements in 34 CFR 74.43 or 80.36(c), as applicable, if—

(1) The contract is with an entity that agrees to provide a site or sites where the applicant would conduct the project activities under the grant;

(2) The implementation sites or partner entities that the applicant proposes to use are identified in the application for the grant; and

(3) The implementation sites or partner entities are included in the application in order to meet a regulatory, statutory, or priority requirement related to the competition.

(b) When entering into a contract for data collection, data analysis, evaluation services, or essential services, an applicant may select a provider using the informal, small-purchase procurement procedures in 34 CFR 80.36(d)(1), regardless of whether that applicant would otherwise be subject to that part or whether the evaluation contract would meet the standards for a small purchase order, if—

(1) The contract is with the data collection, data analysis, evaluation service, or essential service provider;

(2) The data collection, data analysis, evaluation service, or essential service provider that the applicant proposes to use is identified in the application for the grant; and

(3) The data collection, data analysis, evaluation service, or essential service provider is identified in the application in order to meet a statutory, regulatory, or priority requirement related to the competition.

(c) If the grantee relied on the exceptions under paragraph (a) or (b) of this section, the grantee must certify in its application that any employee, officer, or agent participating in the selection, award, or administration of a contract is free of any real or apparent conflict of interest and, if the grantee relied on the exceptions of paragraph (b) of this section, that the grantee used small purchase procedures to obtain the product or service.

(d) A grantee must obtain the Secretary's prior approval for any change to an implementation site, implementation partner, or data collection, data analysis, evaluation service, or essential service provider, if the grantee relied on the exceptions under paragraph (a) or (b) of this section to select the entity.

(e) The exceptions in paragraphs (a) and (b) of this section do not extend to the other procurement requirements in 34 CFR part 74 and 34 CFR part 80 regarding contracting by grantees and subgrantees.

(f) For the purposes of this section, essential service means a product or service directly related to the grant that would, if not provided, have a detrimental effect on the grant.

(Authority: 20 U.S.C. 1221e-3 and 3474)

5. Revise § 75.209 to read as follows:

§ 75.209

Selection criteria based on statutory or regulatory provisions.

The Secretary may establish selection criteria and factors based on statutory or regulatory provisions that apply to the authorized program, which may include, but are not limited to criteria and factors that reflect—

(a) Criteria contained in the program statute or regulations;

(b) Criteria in § 75.210;

(c) Allowable activities specified in the program statute or regulations;

(d) Application content requirements specified in the program statute or regulations;

(e) Program purposes, as described in the program statute or regulations; or

(f) Other pre-award and post-award conditions specified in the program statute or regulations.

(Authority: 20 U.S.C. 1221e-3 and 3474)

6. Amend § 75.210 by:

A. Revising the introductory text.

B. Revising paragraph (c)(2)(xvi).

C. Adding paragraphs (c)(2)(xxiv) through (xxix).

D. Adding paragraphs (h)(2)(viii) through (xii).

E. Adding a new paragraph (i).

The revisions and additions read as follows.

§ 75.210

General selection criteria.

In determining the selection criteria to evaluate applications submitted in a grant competition, the Secretary may select one or more of the following criteria and may select from among the list of optional factors under each criterion. The Secretary may define a selection criterion by selecting one or more specific factors within a criterion or assigning factors from one criterion to another criterion.

* * * * *

(c) * * *

(2) * * *

(xvi) The extent to which the proposed project will integrate with or build on similar or related efforts to improve relevant outcomes (as defined in 34 CFR 77.1(c)), using existing funding streams from other programs or policies supported by community, State, and Federal resources.

* * * * *

(xxiv) The extent to which the applicant demonstrates that it has the resources to operate the project beyond the length of the grant, including a multi-year financial and operating model and accompanying plan; the demonstrated commitment of any partners; evidence of broad support from stakeholders (e.g., State educational agencies, teachers' unions) critical to the project's long-term success; or more than one of these types of evidence.

(xxv) The potential and planning for the incorporation of project purposes, activities, or benefits into the ongoing work of the applicant beyond the end of the grant.

(xxvi) The extent to which the proposed project will increase efficiency in the use of time, staff, money, or other resources in order to improve results and increase productivity.

(xxvii) The extent to which the proposed project will integrate with or build on similar or related efforts in order to improve relevant outcomes (as defined in 34 CFR 77.1(c)), using nonpublic funds or resources.

(xxviii) The extent to which the proposed project is supported by evidence of promise (as defined in 34 CFR 77.1(c)).

(xxix) The extent to which the proposed project is supported by strong theory (as defined in 34 CFR 77.1(c)).

* * * * *

(h) * * *

(2) * * *

(viii) The extent to which the methods of evaluation will, if well-implemented, produce evidence about the project's effectiveness that would meet the What Works Clearinghouse Evidence Standards without reservations.^[1]

(ix) The extent to which the methods of evaluation will, if well-implemented, produce evidence about the project's effectiveness that would meet the What Works Clearinghouse Evidence Standards with reservations.^[2]

(x) The extent to which the methods of evaluation will, if well-implemented, produce evidence of promise (as defined in 34 CFR 77.1(c)).

(xi) The extent to which the methods of evaluation will provide valid and reliable performance data on relevant outcomes.

(xii) The extent to which the evaluation plan clearly articulates the key components, mediators, and outcomes of the grant-supported intervention, as well as a measurable threshold for acceptable implementation.

(i) Strategy to scale. (1) The Secretary considers the applicant's strategy to scale the proposed project.

(2) In determining the applicant's capacity to scale the proposed project, the Secretary considers one or more of the following factors:

(i) The applicant's capacity (e.g., in terms of qualified personnel, financial resources, or management capacity) to bring the proposed project to scale on a national or regional level (as defined in 34 CFR 77.1(c)) working directly, or through partners, during the grant period.

(ii) The applicant's capacity (e.g., in terms of qualified personnel, financial resources, or management capacity) to further develop and bring to scale the proposed process, product, strategy, or practice, or to work with others to ensure that the proposed process, product, strategy, or practice can be further developed and brought to scale, based on the findings of the proposed project.

(iii) The feasibility of successful replication of the proposed project, if favorable results are obtained, in a variety of settings and with a variety of populations.

(iv) The mechanisms the applicant will use to broadly disseminate information on its project so as to support further development or replication.

(v) The extent to which the applicant demonstrates there is unmet demand for the process, product, strategy, or practice that will enable the applicant to reach the level of scale that is proposed in the application.

(vi) The extent to which the applicant identifies a specific strategy or strategies that address a particular barrier or barriers that prevented the applicant, in the past, from reaching the level of scale that is proposed in the application.

7. Revise § 75.250 to read as follows:

§ 75.250

Maximum funding period.

(a) The Secretary may approve a project period of up to 60 months to perform the substantive work of a grant.

(b) The Secretary may approve a data collection period for a grant for a period of up to 72 months after the end of the project period and provide funding for the data collection period for the sole purpose of collecting, analyzing, and reporting performance measurement data regarding the project. The Secretary may inform applicants of the Secretary's intent to approve data collection periods in the application notice published for a competition or may decide to fund data collection periods after grantees have started their project periods.

(Authority: 20 U.S.C. 1221e-3 and 3474.)

8. Amend § 75.251 by revising the section heading and adding a new paragraph (c) to read as follows:

§ 75.251

Budget periods.

* * * * *

(c) If the Secretary funds a multi-year data collection period, the Secretary may fund the data collection period through separate budget periods and fund those budget periods in the same manner as those periods are funded during the project period.

9. Amend § 75.253 by—

A. Revising paragraph (a)(2).

B. Adding a new paragraph (a)(5).

C. Redesignating paragraphs (b) through (e) as paragraphs (c) through (f).

D. Adding a new paragraph (b).

E. Revising newly redesignated paragraph (f).

The revisions and additions read as follows:

§ 75.253

Continuation of a multi-year project after the first budget period.

(a) * * *

(2) The grantee has either—

(i) Made substantial progress in achieving—

(A) The goals and objectives of the project; and

(B) If the Secretary established performance measurement requirements for the grant in the application notice, the performance targets in the grantee's approved application; or

(ii) Obtained the Secretary's approval for changes to the project that—

(A) Do not increase the amount of funds obligated to the project by the Secretary; and

(B) Enable the grantee to achieve the goals and objectives of the project and meet the performance targets of the project, if any, without changing the scope or objectives of the project.

* * * * *

(5) The grantee has maintained financial and administrative management systems that meet the requirements in 34 CFR 74.21 or 80.20, as appropriate.

(b) In deciding whether a grantee has made substantial progress, the Secretary may consider any information relevant to the authorizing statute, a criterion, a priority, or a performance measure, or to a financial or other requirement that applies to the selection of applications for new grants.

* * * * *

(f) Unless prohibited by the program statute or regulations, a grantee that is in the final budget period of its project period may seek continued assistance for the project as required under the procedures for selecting new projects for grants.

* * * * *

10. Add § 75.266 to subpart D to read as follows:

§ 75.266

What procedures does the Secretary use if the Secretary decides to give special consideration to applications supported by strong or moderate evidence of effectiveness?

(a) As used in this section, “strong evidence of effectiveness” is defined in 34 CFR 77.1(c);

(b) As used in this section, “moderate evidence of effectiveness” is defined in 34 CFR 77.1(c); and

(c) If the Secretary determines that special consideration of applications supported by strong or moderate evidence of effectiveness is appropriate, the Secretary may establish a separate competition under the procedures in 34 CFR 75.105(c)(3), or provide competitive preference under the procedures in 34 CFR 75.105(c)(2), for applications supported by:

(1) Evidence of effectiveness that meets the conditions set out in paragraph (a) of the definition of “strong evidence of effectiveness” in 34 CFR 77.1;

(2) Evidence of effectiveness that meets the conditions set out in either paragraph (a) or (b) of the definition of “strong evidence of effectiveness” in 34 CFR 77.1; or

(3) Evidence of effectiveness that meets the conditions set out in the definition of “moderate evidence of effectiveness.”

(Authority: 20 U.S.C. 1221e-3 and 3474.)

11. Revise § 75.590 to read as follows.

§ 75.590

Evaluation by the grantee.

(a) If the application notice for a competition required applicants to describe how they would evaluate their projects, each grantee under that competition must demonstrate to the Department that—

(1) The evaluation meets the standards of the evaluation in the approved application for the project; and

(2) The performance measurement data collected by the grantee and used in the evaluation meet the performance measurement requirements of the approved application.

(b) If the application notice for a competition did not require applicants to describe how they would evaluate their projects, each grantee must provide information in its performance report demonstrating—

(1) The progress made by the grantee in the most recent budget period, including progress based on the performance measurement requirements for the grant, if any;

(2) The effectiveness of the grant, including fulfilling the performance measurement requirements of the approved application, if any; and

(3) The effect of the project on the participants served by the project, if any.

(Authority: 20 U.S.C. 1221e-3 and 3474.)

12. Amend § 75.708 by:

A. Revising the section heading.

B. Revising paragraph (a).

C. Redesignating paragraph (b) as paragraph (e).

D. Adding new paragraphs (b), (c) and (d).

The revision and additions read as follows.

§ 75.708

Subgrants.

(a) A grantee may not make a subgrant under a program covered by this part unless authorized by statute or by paragraph (b) of this section.

(b) The Secretary may, through an announcement in the Federal Register, authorize subgrants when necessary to meet the purposes of a program. In this announcement, the Secretary will—

(1) Designate the types of entities, e.g., State educational agencies, local educational agencies, institutions of higher education, and nonprofit organizations, to which subgrants can be awarded; and

(2) Indicate whether subgrants can be made to entities identified in an approved application or, without regard to whether the entity is identified in an approved application, have to be selected through a competitive process set out in subgranting procedures established by the grantee.

(c) If authorized under paragraph (b) of this section, a subgrant is allowed if it will be used by that entity to directly carry out project activities described in that application.

(d) The grantee, in awarding subgrants under paragraph (b) of this section, must—

(1) Ensure that subgrants are awarded on the basis of an approved budget that is consistent with the grantee's approved application and all applicable Federal statutory, regulatory, and other requirements;

(2) Ensure that every subgrant includes any conditions required by Federal statute and executive orders and their implementing regulations; and

(3) Ensure that subgrantees are aware of requirements imposed upon them by Federal statute and regulation, including the Federal anti-discrimination laws enforced by the Department.

* * * * *

PART 77— DEFINITIONS THAT APPLY TO DEPARTMENT REGULATIONS

13. The authority citation for part 77 is revised to read as follows:

Authority: 20 U.S.C. 1221e-3 and 3474, unless otherwise noted.

14. Amend § 77.1(c) by adding the following definitions in alphabetical order:

§ 77.1

Definitions that apply to all Department programs.

* * * * *

(c) * * *

Ambitious means promoting continued, meaningful improvement for program participants or for other individuals or entities affected by the grant, or representing a significant advancement in the field of education research, practices, or methodologies. When used to describe a performance target, whether a performance target is ambitious depends upon the context of the relevant performance measure and the baseline for that measure.

* * * * *

Baseline means the starting point from which performance is measured and targets are set.

* * * * *

Evidence of promise means there is empirical evidence to support the theoretical linkage(s) between at least one critical component and at least one relevant outcome presented in the logic model for the proposed process, product, strategy, or practice. Specifically, evidence of promise means the conditions in paragraphs (a) and (b) of this section are met:

(i) There is at least one study that is a—

(A) Correlational study with statistical controls for selection bias;

(B) Quasi-experimental study that meets the What Works Clearinghouse Evidence Standards with reservations; ^[1] or

(C) Randomized controlled trial that meets the What Works Clearinghouse Evidence Standards with or without reservations.^[2]

(ii) The study referenced in paragraph (a) found a statistically significant or substantively important (defined as a difference of 0.25 standard deviations or larger), favorable association between at least one critical component and one relevant outcome presented in the logic model for the proposed process, product, strategy, or practice.

* * * * *

Large sample means an analytic sample of 350 or more students (or other single analysis units) who were randomly assigned to a treatment or control group or 50 or more groups (such as classrooms or schools) that contain 10 or more students (or other single analysis units) and that were randomly assigned to a treatment or control group.

* * * * *

Logic model (also referred to as theory of action) means a well-specified conceptual framework that identifies key components of the proposed process, product, strategy, or practice (i.e., the active “ingredients” that are hypothesized to be critical to achieving the relevant outcomes) and describes the relationships among the key components and outcomes, theoretically and operationally.

* * * * *

Moderate evidence of effectiveness means one of the following conditions is met:

(i) There is at least one study of the effectiveness of the process, product, strategy, or practice being proposed that meets the What Works Clearinghouse Evidence Standards without reservations,^[3] found a statistically significant favorable impact on a relevant outcome (with no statistically significant and overriding unfavorable impacts on that outcome for relevant populations in the study or in other studies of the intervention reviewed by and reported on by the What Works Clearinghouse), and includes a sample that overlaps with the populations or settings proposed to receive the process, product, strategy, or practice.

(ii) There is at least one study of the effectiveness of the process, product, strategy, or practice being proposed that meets the What Works Clearinghouse Evidence Standards with reservations,^[4] found a statistically significant favorable impact on a relevant outcome (with no statistically significant and overriding unfavorable impacts on that outcome for relevant populations in the study or in other studies of the intervention reviewed by and reported on by the What Works Clearinghouse), includes a sample that overlaps with the populations or settings proposed to receive the process, product, strategy, or practice, and includes a large sample and a multi-site sample (Note: multiple studies can cumulatively meet the large and multi-site sample requirements as long as each study meets the other requirements in this paragraph).

* * * * *

Multi-site sample means more than one site, where site can be defined as an LEA, locality, or State.

* * * * *

National level describes the level of scope or effectiveness of a process, product, strategy, or practice that is able to be effective in a wide variety of communities, including rural and urban areas, as well as with different groups (e.g., economically disadvantaged, racial and ethnic groups, migrant populations, individuals with disabilities, English learners, and individuals of each gender).

* * * * *

Performance measure means any quantitative indicator, statistic, or metric used to gauge program or project performance.

Performance target means a level of performance that an applicant would seek to meet during the course of a project or as a result of a project.

* * * * *

Quasi-experimental design study means a study using a design that attempts to approximate an experimental design by identifying a comparison group that is similar to the treatment group in important respects. These studies, depending on design and implementation, can meet What Works Clearinghouse Evidence Standards with reservations ^[5] (they cannot meet What Works Clearinghouse Evidence Standards without reservations).

Randomized controlled trial means a study that employs random assignment of, for example, students, teachers, classrooms, schools, or districts to receive the intervention being evaluated (the treatment group) or not to receive the intervention (the control group). The estimated effectiveness of the intervention is the difference between the average outcome for the treatment group and for the control group. These studies, depending on design and implementation, can meet What Works Clearinghouse Evidence Standards without reservations.^[6]

Regional level describes the level of scope or effectiveness of a process, product, strategy, or practice that is able to serve a variety of communities within a State or multiple States, including rural and urban areas, as well as with different groups (e.g., economically disadvantaged, racial and ethnic groups, migrant populations, individuals with disabilities, English learners, and individuals of each gender). For an LEA-based project, to be considered a regional-level project, a process, product, strategy, or practice must serve students in more than one LEA, unless the process, product, strategy, or practice is implemented in a State in which the State educational agency is the sole educational agency for all schools.

Relevant outcome means the student outcome(s) (or the ultimate outcome if not related to students) the proposed process, product, strategy, or practice is designed to improve; consistent with the specific goals of a program.

* * * * *

Strong evidence of effectiveness means one of the following conditions is met:

(i) There is at least one study of the effectiveness of the process, product, strategy, or practice being proposed that meets the What Works Clearinghouse Evidence Standards without reservations,^[7] found a statistically significant favorable impact on a relevant outcome (with no statistically significant and overriding unfavorable impacts on that outcome for relevant populations in the study or in other studies of the intervention reviewed by and reported on by the What Works Clearinghouse), includes a sample that overlaps with the populations and settings proposed to receive the process, product, strategy, or practice, and includes a large sample and a multi-site sample (Note: multiple studies can cumulatively meet the large and multi-site sample requirements as long as each study meets the other requirements in this paragraph).

(ii) There are at least two studies of the effectiveness of the process, product, strategy, or practice being proposed, each of which: Meets the What Works Clearinghouse Evidence Standards with reservations,^[8] found a statistically significant favorable impact on a relevant outcome (with no statistically significant and overriding unfavorable impacts on that outcome for relevant populations in the studies or in other studies of the intervention reviewed by and reported on by the What Works Clearinghouse), includes a sample that overlaps with the populations and settings proposed to receive the process, product, strategy, or practice, and includes a large sample and a multi-site sample.

Strong theory means a rationale for the proposed process, product, strategy, or practice that includes a logic model.

Footnotes

1.  What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), currently found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

2.  See definition for full description of the two possible conditions.

1.  What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

2.  What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

1.  What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

2.  What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

3.  What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

4.  What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http://ies.ed.gov/​ncee/​wwc/​DocumentSum.aspx?​sid=​19.

5.  What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http:// ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.

6.  What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http:// ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.

7.  What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http:// ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.

8.  What Works Clearinghouse Procedures and Standards Handbook (Version 2.1, September 2011), which can currently be found at the following link: http:// ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.