[Federal Register Volume 62, Number 221 (Monday, November 17, 1997)]
[Rules and Regulations]
[Pages 61428-61432]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 97-30051]
[[Page 61427]]
_______________________________________________________________________
Part II
Department of Education
_______________________________________________________________________
34 CFR Part 701
Office of Educational Research and Improvement (OERI) Conduct and
Activities Evaluation Standards; Designation of Exemplary and Promising
Programs; Final Rule
Federal Register / Vol. 62, No. 221 / Monday, November 17, 1997 /
Rules and Regulations
[[Page 61428]]
DEPARTMENT OF EDUCATION
34 CFR Part 701
RIN 1850-AA52
Standards for Conduct and Evaluation of Activities Carried out by
the Office of Educational Research and Improvement (OERI)--Designation
of Exemplary and Promising Programs
AGENCY: Department of Education.
ACTION: Final regulation.
-----------------------------------------------------------------------
SUMMARY: The Assistant Secretary for Educational Research and
Improvement (OERI) is establishing final regulations pursuant to the
``Educational Research, Development, Dissemination, and Improvement Act
of 1994.'' The regulations are intended to provide quality assurance
that programs designated by the Department of Education as either
exemplary or promising have met criteria that will allow educators,
professional organizations, and others to use these programs with
confidence.
DATES: These regulations take effect December 17, 1997. However,
affected parties do not have to comply with the information collection
requirement in Sec. 701.4 until the Department of Education publishes
in the Federal Register notification of the compliance date and the
control number assigned by the Office of Management and Budget (OMB) to
this information collection requirement. Publication of the control
number notifies the public that OMB has approved this information
collection requirement under the Paperwork Reduction Act of 1995.
FOR FURTHER INFORMATION CONTACT: Stephen O'Brien, U.S. Department of
Education, 555 New Jersey Avenue, NW, Room 502B, Washington, D.C.
Telephone: (202) 219-2141. Internet: (Steve__O'Brien@ed.gov).
Individuals who use a telecommunication device for the deaf (TDD) may
call the Federal Information Relay Service (FIRS) at 1-800-877-8339
between 8 a.m. and 8 p.m., Eastern time, Monday through Friday.
Individuals with disabilities may obtain this document in an
alternate format (e.g., Braille, large print, audiotape, or computer
diskette) on request to the person listed in the preceding paragraph.
SUPPLEMENTARY INFORMATION: On March 31, 1994, President Clinton signed
Public Law 103-227, which includes Title IX, the Educational Research,
Development, Dissemination, and Improvement Act of 1994 (the Act). The
Act restructured OERI and provided it with a broad mandate to conduct
an array of research, development, dissemination, and improvement
activities aimed at strengthening the education of all students.
The Act directed the Assistant Secretary to develop, in
consultation with the National Educational Research Policy and
Priorities Board (the Board), the highest standards of professional
excellence necessary to govern the conduct and evaluation of all
research, development, and dissemination activities carried out by the
OERI. The legislation requires that the standards be developed in three
phases.
In the first phase, standards were promulgated to establish the
peer review process and evaluation criteria to be used for reviewing
applications for grants and cooperative agreements and proposals for
contracts. The Department published final regulations setting out these
standards on September 14, 1995 (60 FR 47808). The regulations in this
announcement address the second phase of development by establishing
the criteria for panels to use in reviewing potentially exemplary and
promising educational programs. The Assistant Secretary will later
publish proposed regulations for phase three of the standards, which
will govern how OERI evaluates performance of its recipients of grants,
contracts, and cooperative agreements.
The OERI legislation requires that expert panels be established to
review educational programs submitted by individuals or organizations.
The legislation also provides that the Secretary may identify
educational programs for the panels to review. The statute requires the
panels to recommend to the Secretary those programs that should be
designated as exemplary or promising and disseminated through the
Department's National Education Dissemination System. The law requires
that each panel consist of appropriately qualified experts and
practitioners and requires the Secretary to develop standards that
describe the procedures the panels will use in reviewing the
educational programs. Section 941(a)(3) of the law broadly defines
educational programs to include educational polices, research findings,
practices, and products. Educational programs may range in size and
complexity from an individual instructional program--such as an
elementary school science program--to a comprehensive reform initiative
involving multiple goals and participants. Programs at all levels of
education--preschool, elementary, secondary, and postsecondary--are
eligible for consideration.
In determining whether an educational program should be recommended
as exemplary or promising, each panel is required by the Act to
consider: (a) Whether, based on empirical data, the program is
effective and should be designated as exemplary or (b) whether there is
sufficient evidence to demonstrate that the program shows promise for
improving student achievement and should be designated as promising.
The Act expressly states that a panel shall not eliminate a program
from consideration based on the lack of one type of supporting data
such as test scores.
The evaluation process set forth in the final regulations will
ensure that programs disseminated by the Department are high-quality,
research-based programs that have provided evidence indicating they
have improved teaching, learning, or both, or has demonstrated other
worthy educational performance outcomes. The Department's dissemination
system is designed to make information about these promising and
exemplary programs available to the public as quickly as possible. The
system will enable the Department to respond to all forms of requests
for information and assistance, and to support the applications of
research and best practice. The system will use electronic networking
and the capabilities of:
--National Research Institutes;
--Educational Resources Information Center (ERIC);
--Regional Educational Laboratories;
--Department-supported dissemination and technical assistance
providers;
--National Library of Education;
--Eisenhower Regional Consortia and Clearinghouse, and
--Other public and private nonprofit entities, including education
associations and networks.
Until recently, the Department validated exemplary programs through
its Program Effectiveness Panel (PEP) and disseminated them through the
National Diffusion Network (NDN). Since this program no longer exists,
with the adoption of these standards the Department will evaluate and
disseminate promising educational programs in addition to exemplary
programs. The Department will also work in partnership with
constituency groups who have expertise in the specific topic areas
represented by the expert panels to develop coordinated procedures to
maximize their involvement in this work.
On June 3, 1996, the Secretary published a notice of proposed
rulemaking (NPRM) for this part in the
[[Page 61429]]
Federal Register (61 FR 27990-27993). These final regulations contain
three major changes from the NPRM. These changes are fully explained in
the ``Analysis of Comments and Changes'' elsewhere in this preamble.
The changes pertain to the standing panel; the distinction between
``promising'' and ``exemplary''; and the factors listed under the
criteria expert panels will use to evaluate programs.
Analysis of Comments and Changes
In response to the Secretary's invitation in the NPRM, seven
parties submitted comments on the proposed regulations. This included
comments from individual members of two pilot panels (math/science and
gender equity) that were appointed by the Secretary to field test the
expert panel process. In addition to the public comment, comments from
the Board's Subcommittee on Standards are addressed as required by the
legislation. The full Board approved the final regulations at a meeting
on September 26, 1997. An analysis of the comments and of the changes
in the regulations since publication of the NPRM follows.
Major issues are grouped according to subject with appropriate
sections of the regulations referenced in parentheses. Technical and
other minor changes--and suggested changes the Secretary is not legally
authorized to make under the applicable statutory authority--are not
addressed.
Eligibility (Sec. 701.3)
Comments: One commenter asked for clarification on who is eligible
to submit educational programs for designation as promising or
exemplary. Specifically, this commenter asked whether federally-funded
entities, such as the Regional Laboratories, will be required to go
through this process; whether local agencies that receive Federal
funding through states, such as under Title I of the Elementary and
Secondary Education Act (ESEA), may submit programs on their own;
whether sponsors need to be invited to submit or may submit on their
own initiative; and whether for-profit entities may submit.
Discussion: The law provides that ``individuals'' or
``organizations'' may submit educational programs for review. Since the
law is silent on the specific nature of the organizations, the
Secretary believes that for-profit agencies would be eligible to submit
programs for review. With respect to the OERI-funded Regional
Educational Laboratories, the law provides that the Secretary may
identify those programs for panel review. In addition, the Secretary
believes that the Laboratories could submit one or more of their
programs on their own initiative. The question of whether local
agencies that receive Federal funding through a State or Federal
entity, such as under Title I of the ESEA, can submit on their own or
must go through their funding agency, will be addressed in
administrative guidance.
Changes: None.
Content of Submissions (Sec. 701.4)
Comments: Three commenters made suggestions about this section. Two
commenters believed that requiring funding and staffing information was
burdensome and not germane to the designation of a program as promising
or exemplary. One commenter believed that this section should require
program sponsors to submit specific materials related to content and
methods. Another commenter believed that this section should include
the requirement that the program include evidence of sustainability of
improvement with targeted student populations.
Discussion: The Secretary believes that funding and staffing
information should be included to help determine whether an educational
program should be recommended as either exemplary or promising. The
Secretary agrees that sponsors should be required to submit information
or materials specific to content and methods, as available and
appropriate. The Secretary believes that the evidence of sustainability
of student improvement should be evaluated by peer reviewers in
accordance with Sec. 701.22.
Changes: Section 701.4(b)(7) has been renumbered as
Sec. 701.4(b)(8) and a new Sec. 701.4(b)(7) has been added to include a
provision for specific materials relevant to content and methods.
Procedures for Submitting Educational Programs (New Sec. 701.5)
Comments: One commenter believed that the regulations should
contain more specificity about the procedure for submitting programs to
the expert panels. This commenter requested specifics on who receives
the submissions and whether they may be submitted at any time or only
on specific dates.
Discussion: The Secretary agrees that the general submission
procedures should be included in regulations. A sponsor seeking the
exemplary or promising designation for its educational program may
submit its program at any time for consideration to the Assistant
Secretary, who will assign the submitted program to the appropriate
panel for review. The individual expert panels will set appropriate
timelines for program submissions. In addition, the Assistant Secretary
will periodically establish and announce in the Federal Register
specific topic areas of high priority. Sponsors of educational programs
in these areas will be invited to submit them for consideration.
Changes: A new Sec. 701.5 has been added to include general
procedures for submitting educational programs for review by an expert
panel.
Establishment of Panels (Sec. 701.10)
Comments: The Board's Subcommittee on Standards recommended a
change to the expert panel system. The Subcommittee thought that the
structure of having members of the expert panels drawn from a separate
standing panel of educational experts was an unwieldy, overly-
complicated structure. The Board recommended that the expert panels be
formed separately from a standing panel, which would instead provide an
administrative oversight and monitoring function for the expert panels.
Discussion: The Secretary agrees that the expert panel should be
formed separately from a standing panel. The Secretary will determine
the feasibility of establishing a separate standing panel for the
oversight and monitoring functions referred to by the Board--functions
which are administrative in nature and could also be performed by OERI
staff. Elimination of a reference to a standing panel in the
regulations would not alter the composition and function of the expert
panels as outlined in the NPRM.
Changes: Section 701.10(a) has been removed, Sec. 701.10(b) has
been revised, Sec. 701.11 has been removed, Sec. 701.12(a) has been
revised, and Sec. 701.12 has also been renumbered as Sec. 701.11.
Panel Membership (Secs. 701.11 and 701.12)
Comments: One commenter observed that Secs. 701.11 and 701.12 in
the NPRM did not explicitly state that those serving on the panels
would represent both the community of practice and that of research.
One commenter believed that each panel should include one or more
members with evaluation expertise in order to help evaluate evidence of
effectiveness.
Discussion: The Secretary agrees with these comments.
Changes: A sentence has been added at the end of the renumbered
Sec. 701.11(a) (formerly Sec. 701.12(a)) stating that the membership of
the expert panels will represent both the community of practice and the
community of research. Additionally, Secs. 701.11(b)(3) and
[[Page 61430]]
701.11(b)(4) have been renumbered as Secs. 701.11(b)(4) and
701.11(b)(5), respectively; and a new provision for Sec. 701.11(b)(3)
has been added to include the selection of at least one individual with
expertise in evaluating educational programs.
Difference Between Promising and Exemplary Programs (Sec. 701.21)
Comments: Five comments were received on the distinction between
promising and exemplary programs. As proposed in Sec. 701.21, the
distinction was based upon the generalizability of the educational
programs. Promising programs had to meet each of the criteria of
educational effectiveness in Sec. 701.22 (success, quality, educational
significance, and usefulness to others) with respect to only one
``context or population.'' Exemplary programs had to meet each of the
criteria ``with respect to multiple contexts or multiple populations.''
Two commenters believed that the distinction should stay the way it
was in the NPRM, although one of those suggested some clarifying
language. However, three commenters questioned the distinction on the
basis that it was too narrowly and artificially drawn and did not
reflect the commonly understood meaning of the words ``promising'' and
``exemplary.'' In this regard, one commenter believed that promising
programs should not have to meet every criterion in Sec. 701.22 at the
same level as exemplary programs. Two commenters believed that
promising programs should have to meet the criteria at the same level
as exemplary, but that the evidence required of promising programs
should be less stringent and that exemplary programs should be held to
a higher standard of evidence.
Discussion: The Secretary agrees with those commenters who
questioned the proposed distinction and advised OERI to give a more
common sense meaning to the terms ``promising'' and ``exemplary.'' The
Secretary believes that the distinction between promising and exemplary
programs specified in legislation is sufficient to cover these
concerns. The Secretary relies upon the expert judgment of the expert
panel reviewers in determining the nature and weight of evidence
necessary to designate a program as either promising or exemplary, and
in applying the criteria listed in Sec. 701.22 in making this
determination.
Changes: A revision has been made to the distinction between
``promising'' and ``exemplary.''
Criteria (Sec. 701.22)
Comments: Five commenters provided comments on this section and
suggested revisions to either the wording of the criteria or to the
content of the factors listed under each criterion or both. These
comments included comments from one member of the math/science pilot
panel and three members of the gender equity pilot panel. Although the
math/science panel member did provide comments specific to the proposed
criteria and factors, the consensus of this panel was that the expert
panel process would be better served if each panel developed its own
factors specific to the content or discipline or both under review by
the individual panel. One commenter suggested that the word
``replicability'' would better capture the concept for the criterion
entitled ``usefulness to others.'' In addition, OERI's Board (The
Subcommittee on Standards) thought that the regulations should be as
simple as possible and should give the expert panels as much discretion
as possible in evaluating programs submitted for review.
Discussion: In addition to the math/science and gender equity
panels, the Secretary will establish pilot panels in technology and
early reading in the next year. The Secretary has determined that until
the work of all four pilot panels is concluded, the regulation should
retain only the four criteria outlined in the NPRM in Sec. 701.22 and
allow each panel the flexibility to establish its own individual
factors under each criterion that are specific to its content or
discipline. The fact that the comments from the public suggested
various changes to the factors underscores the desirability of this
approach. While the final regulations will therefore no longer require
the expert panels to apply the factors listed in the NPRM, the
Secretary encourages each panel to look at these factors as suggested
examples. The Secretary will review the factors developed by all of the
panels to see if the criteria set forth in the final regulations need
to be modified.
Changes: The factors specified under each of the four criteria have
been eliminated and the criterion, ``usefulness to others'' has been
changed to ``replicability.''
Assessment of Educational Impact
In the NPRM the Secretary requested comments on whether the
proposed regulations would require transmission of information that is
being gathered by or is available from any other agency or authority of
the United States.
Based on the response to the NPRM and on its own review, the
Department has determined that the regulations in this document do not
require transmission of information that is being gathered by or is
available from any other agency or authority of the United States.
Electronic Access to This Document
Anyone may view this document, as well as all other Department of
Education documents published in the Federal Register, in text or
portable document format (pdf) on the World Wide Web at either of the
following sites:
http://ocfo.ed.gov/fedreg.htm
http://www.ed.gov/news.html
To use the pdf you must have the Adobe Acrobat Reader Program with
Search, which is available free at either of the previous sites. If you
have questions about using the pdf, call the U.S. Government Printing
Office toll free at 1-888-293-6498.
Anyone may also view these documents in text copy only on an
electronic bulletin board of the Department. Telephone: (202) 219-1511
or, toll free, 1-800-222-4922. The documents are located under Option
G--Files/Announcements, Bulletins and Press Releases.
Note: The official version of this document is the document
published in the Federal Register.
List of Subjects in 34 CFR Part 701
Education, Educational research, Reporting and recordkeeping
requirements.
(Catalog of Federal Domestic Assistance Number does not apply)
Dated: November 11, 1997.
Ricky T. Takai,
Acting Assistant Secretary for Educational Research and Improvement.
The Secretary amends chapter VII of title 34 of the Code of Federal
Regulations by adding a new part 701 to read as follows:
PART 701--STANDARDS FOR CONDUCT AND EVALUATION OF ACTIVITIES
CARRIED OUT BY THE OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT
(OERI)--DESIGNATION OF EXEMPLARY AND PROMISING PROGRAMS
Subpart A--General
Sec.
701.1 What is the purpose of these standards?
701.2 What definitions apply?
701.3 Who is eligible to submit an educational program for review?
[[Page 61431]]
701.4 What must a program sponsor submit for review?
701.5 What are the procedures for submitting an educational program
for review by an expert panel?
Subpart B--Selection of Panel Members
701.10 How are panels established?
701.11 How is the membership of expert panels determined?
Subpart C--The Expert Panel Review Process
701.20 How does an expert panel evaluate programs?
701.21 What is the difference between an exemplary and a promising
program?
701.22 What criteria are used to evaluate programs for exemplary or
promising designation?
Authority: 20 U.S.C. 6011(i), unless otherwise noted.
Subpart A--General
Sec. 701.1 What is the purpose of these standards?
(a) The standards in this part implement section 941(d) of the
Educational Research, Development, Dissemination, and Improvement Act
of 1994.
(b) These standards are intended to provide quality assurance that
educational programs designated by the U.S. Department of Education as
either exemplary or promising have met criteria that will allow
educators, professional organizations, and others to use these programs
with confidence.
(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))
Sec. 701.2 What definitions apply?
The following definitions apply to this part:
Assistant Secretary means the Assistant Secretary for the Office of
Educational Research and Improvement.
Educational programs mean educational policies, research findings,
practices, and products.
Program sponsor means a party submitting an educational program for
designation by the Secretary as either promising or exemplary.
Secretary means the Secretary of the Department of Education or an
official or employee of the Department acting for the Secretary under a
delegation of authority.
(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))
Sec. 701.3 Who is eligible to submit an educational program for
review?
Any public or private agency, organization or institution, or an
individual may submit an educational program for review.
(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))
Sec. 701.4 What must a program sponsor submit for review?
(a) To have an educational program considered for designation as
exemplary or promising, a sponsor must submit to the Secretary a
description of the program, program materials, and a discussion of the
program that is responsive to the criteria in Sec. 701.22.
(b) Information submitted must include, to the extent relevant to
the particular program,--
(1) A program abstract of 250 words or less;
(2) A description of the salient features of the program;
(3) A description of the program's philosophy and history;
(4) Site information, including demographics;
(5) A description of evaluation results;
(6) Funding and staffing information;
(7) Specific materials relevant to content and methods, as
appropriate; and
(8) Organization name, address, telephone and fax numbers, e-mail
address (if available), and contact person.
(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))
Sec. 701.5 What are the procedures for submitting an educational
program for review by an expert panel?
(a) An applicant seeking the exemplary or promising designation for
its educational program may submit its program at any time for
consideration to the Assistant Secretary, who will assign the submitted
program to the appropriate expert panel for review.
(b) The Assistant Secretary will periodically establish and
announce in the Federal Register specific topic areas of high priority.
Sponsors of educational programs in these areas will be invited to
submit their programs for consideration.
(c) The individual expert panels will set appropriate timelines for
reviewing program submissions.
(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))
Subpart B--Selection of Panel Members
Sec. 701.10 How are panels established?
The Assistant Secretary selects individuals, based on their areas
of expertise, to serve on expert panels in specific topic areas for the
purpose of reviewing and evaluating educational programs and
recommending, to the Secretary, those programs that should be
designated as exemplary or promising.
(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))
Sec. 701.11 How is the membership of expert panels determined?
(a) For the review of each program or group of programs, the
Assistant Secretary establishes an expert panel. The membership of the
expert panels will represent both the community of practice and the
community of research.
(b) In establishing the membership of each expert panel, the
Assistant Secretary--
(1) Selects individuals who have in-depth knowledge of the subject
area or content of the program or group of programs to be evaluated;
(2) Selects at least one current teacher, principal, or other
school-based or community-based professional;
(3) Selects at least one individual with expertise in evaluating
educational programs;
(4) Ensures that no more than one-third of the panel members are
employees of the Federal Government; and
(5) Ensures that each panel member does not have a conflict of
interest, as determined in accordance with paragraph (c) of this
section, with respect to any educational program the panel member is
asked to review.
(c) Panel members are considered employees of the U.S. Department
for the purposes of conflicts of interest analysis and are subject to
the provisions of 18 U.S.C. 208, 5 CFR 2635.502, and the Department's
policies used to implement those provisions.
(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))
Subpart C--The Expert Panel Review Process
Sec. 701.20 How does an expert panel evaluate programs?
(a) Each panel member shall--
(1) Independently review each program based on the criteria in
Sec. 701.22;
(2) Provide written comments based on an analysis of the strengths
and weaknesses of the program according to the criteria;
(3) Participate in site visits or other verification activities, if
appropriate; and
(4) Participate in a meeting of the expert panel, if appropriate,
to discuss the reviews.
(b) A panel may not eliminate an educational program from
consideration based solely on the fact that the program does not have
one specific type of supporting data, such as test scores.
(c) Each expert panel shall make a recommendation to the Secretary
as to
[[Page 61432]]
whether the program is exemplary, promising, or neither.
(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))
Sec. 701.21 What is the difference between an exemplary and a
promising program?
(a) In determining whether an educational program should be
recommended as exemplary or promising, the panel shall consider--
(1) Whether, based on empirical data, the program is effective and
should be designated as exemplary; or
(2) Whether there is sufficient evidence to demonstrate that the
program shows promise for improving student achievement and should be
designated as promising.
(b) The Secretary relies upon the judgment and expertise of peer
reviewers, as established in Sec. 701.11, to determine the nature and
extent of evidence required to distinguish between promising and
exemplary programs and to apply the four criteria established in
Sec. 701.22, and their own individual factors under each criterion in
making this determination.
(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))
Sec. 701.22 What criteria are used to evaluate programs for exemplary
or promising designation?
The Secretary establishes the following evaluation criteria for
expert panels to use in determining whether an educational program
should be recommended as exemplary, promising, or neither:
(a) Evidence of success.
(b) Quality of the program.
(c) Educational significance.
(d) Replicability.
(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))
[FR Doc. 97-30051 Filed 11-14-97; 8:45 am]
BILLING CODE 4000-01-P
