2015-19617. Final Priority-Technical Assistance Center for Vocational Rehabilitation Agency Program Evaluation and Quality Assurance

AGENCY:

Office of Special Education and Rehabilitative Services, Department of Education.

ACTION:

Final priority.

SUMMARY:

The Assistant Secretary for Special Education and Rehabilitative Services announces a priority under the Experimental and Innovative Training program. The Assistant Secretary may use this priority for competitions in fiscal year (FY) 2015 and later years. We take this action to focus Federal financial assistance on an identified national need. We intend the priority to support a Technical Assistance Center for Vocational Rehabilitation Agency Program Evaluation and Quality Assurance (PEQA).

DATES:

This priority is effective September 10, 2015.

FOR FURTHER INFORMATION CONTACT:

Don Bunuan, U.S. Department of Education, 400 Maryland Avenue SW., Room 5046, Potomac Center Plaza (PCP), Washington, DC 20202-2800. Telephone: [phone] or by email: [e-mail].

If you use a telecommunications device for the deaf (TDD) or a text telephone (TTY), call the Federal Relay Service (FRS), toll free, at 1-800-877-8339.

SUPPLEMENTARY INFORMATION:

Purpose of Program: This program is designed to (a) develop new types of training programs for rehabilitation personnel and to demonstrate the effectiveness of these new types of training programs for rehabilitation personnel in providing rehabilitation services to individuals with disabilities; and (b) develop new and improved methods of training rehabilitation personnel, so that there may be a more effective delivery of rehabilitation services by State and other rehabilitation agencies.

Program Authority: 29 U.S.C. 772(a)(1).

Applicable Program Regulations: 34 CFR part 385 and 387.

We published a notice of proposed priority for this competition in the Federal Register on May 28, 2015 (80 FR 30399). That notice contained background information and our reasons for proposing the particular priority. There are differences between the proposed priority and the final priority, and we explain those differences in the Analysis of Comments and Changes section of this notice.

Public Comment: In response to our invitation in the notice of proposed priority, four parties submitted comments.

Generally, we do not address technical and other minor changes. In addition, we do not address comments that raise concerns not directly related to the proposed priority.

Analysis of the Comments and Changes: An analysis of the comments and of any changes in the priority since publication of the notice of proposed priority follows.

Comment: One commenter observed that the priority should provide for continuing personnel development for those who have completed the Basic Certification Program and approach the intermediate level of competency. The commenter recommended allowing those who have completed the Basic Certification Training to qualify as intermediate-level program evaluators in order to access the Special Topical Trainings. In addition, two commenters recommended adding a technical assistance (TA) component that addresses quality improvement in the work of all vocational rehabilitation (VR) personnel, not just the VR agency's program evaluators. The commenters noted that quality improvement is an issue relevant to work at all levels of a VR agency; therefore, other VR staff need to understand the principles of program evaluation, quality assurance, and continuous improvement.

Discussion: We agree that a wide array of State VR agency personnel could benefit from a greater understanding of program evaluation and quality assurance principles. However, the focus of this priority is to advance the knowledge and skills of VR program evaluation personnel through specialized professional education and training. The priority is not intended as a vehicle for providing technical assistance to a broader range of VR personnel on general program evaluation and quality assurance principles.

Thus, the Basic Certification Program described in this priority is designed as an intensive, structured training program to increase the numbers and qualifications of VR program evaluators. The Special Topical Trainings are targeted to more advanced program evaluators, and we want to ensure that those individuals have first priority in attending those sessions. However, if additional space in a Special Topical Training is available, we believe it would be an appropriate and efficient use of resources to open enrollment to individuals who have completed the Basic Certification Program, and then, if seats still remain, to other State VR agency personnel whose current work responsibilities are closely aligned with the specific topic area of the training.

Changes: We have inserted a new paragraph (b) in the Special Topical Training section of the priority that would allow the PEQA, after ensuring that intermediate-level program evaluators have been given priority to register for a specific training session, to open registration to individuals who have completed the Basic Certification Program, and then to other VR personnel whose current work responsibilities are closely aligned with the specific topic area of the training, if additional space in such training is available.

Comment: One commenter recommended that the center support, strengthen, and augment existing communities of practice that focus on program evaluation, rather than establish new communities of practice to perform these functions.

Discussion: We agree that creating new communities of practice is not always necessary. Coordinating with, and enhancing the efforts of, existing communities of practice focused on program evaluation could also be beneficial in sharing information, exchanging ideas, and accomplishing the activities in paragraph (a) of the Coordination Activities section of the priority.

Changes: The communities of practice requirement in paragraph (a) of Coordination Activities under the Project Activities section has been revised to also permit the PEQA to support, strengthen, and augment existing communities of practice, and to establish new communities of practice, as needed, to act as vehicles for communication, exchange of information among program evaluation professionals, and a forum for sharing the results of capstone projects that are in progress or have been completed.

Comment: Two commenters mentioned a preference for substituting the term “continuous improvement” for “quality assurance” throughout the priority. Commenters cited the extensive use of “continuous improvement” in the proposed regulations implementing the Workforce Innovation and Opportunity Act (WIOA) that were published in the Federal Register on April 16, 2015 (80 FR 21059).

Discussion: We recognize the significance of the term “continuous improvement” and its use throughout WIOA. However, we believe that “quality assurance” and “program evaluation,” as described in this priority, represent key elements of the overall process of “continuous improvement.”

Changes: We have revised the initial paragraph of the priority to emphasize that continuous improvement is the overall goal of program evaluation and quality assurance. However, we have retained the priority's focus on skill development in the area of program evaluation and quality assurance. We have also added footnotes referencing the terms “program evaluation” and “quality assurance” as these terms are used in the field in order to clarify the use of those terms.

Comment: One commenter expressed concern about the process by which information and resources are disseminated from the TA Center in a timely manner for use by State VR agencies.

Discussion: Consistent with the provisions in the “Coordination Activities” section of the priority, we agree that timely dissemination of information and resources for use by State VR agencies is important, and mechanisms to ensure the timely dissemination of such materials will be included in the cooperative agreement.

Changes: None.

Comment: One commenter requested that the new center provide TA to tribal VR programs funded through the Rehabilitation Services Administration (RSA), observing that this would be particularly beneficial since tribal VR programs have many of the same requirements to demonstrate continuous improvement as State VR agencies.

Discussion: This priority is intended to assist State VR agencies to build their capacity to meet the performance accountability demands of core programs under WIOA's workforce system. Specifically, this priority is designed to assist State VR agencies to implement high-quality program evaluation and quality assurance programs through the education and training of VR evaluation personnel. Other programs of the Department address these and other needs of tribal VR agencies. Amendments made by WIOA to section 121 of the Rehabilitation Act require RSA to reserve funds from the set-aside for the American Indian Vocational Rehabilitation Services (AIVRS) program under section 110(c) to provide training and TA to assist governing bodies of Indian tribes in developing, conducting, administering, and evaluating their AIVRS projects.

Changes: None.

Comment: Two commenters requested that grant funds under this priority be used to provide logistical and technical support for an existing annual conference focused on program evaluation. Both commenters indicated that an opportunity for in-person interaction and networking would benefit the field as well as support the efforts of objectives of the priority.

Discussion: Nothing in the priority precludes an applicant from proposing to provide logistical and technical support for an existing annual conference focused on program evaluation and quality assurance, as long as such a proposal is consistent with paragraph (a) of the Coordination Activities section of the proposed Center.

Changes: None.

Comment: Two commenters recommended that funding be provided for travel for the cohorts of participants in the Basic Certification Program.

Discussion: Nothing in the priority would preclude an applicant from proposing to use grant funds to support participant travel for the in-person component of the Basic Certification Program, consistent with 34 CFR 387.41.

Changes: None.

Comment: One commenter asked whether the trainings detailed under paragraphs (a) and (b) of the Special Topical Training section describe the same or different trainings.

Discussion: Paragraphs (a) and (b) refer to the same trainings. Paragraph (a) of the Special Topical Training section requires the Center to develop topical trainings, and paragraph (b) requires that those same trainings be conducted no fewer than four times a year.

Changes: None.

Comment: One commenter asked whether the Basic Certification Program is an academic or a professional certificate program.

Discussion: The project is required to develop a basic certification program. Nothing in the priority precludes an applicant from proposing a program that also provides academic credit to participants. However, we note that the priority requires that the Basic Certification Program be offered at no cost to participants. As such, we believe it is unlikely that a project will offer academic credit to all participants, though applicants, with support from an institution of higher education, are welcome to propose such arrangements.

Changes: None.

Final Priority

The purpose of this priority is to fund a cooperative agreement for a training and technical assistance center that will assist State vocational rehabilitation (VR) agencies to improve performance management by building their capacity to carry out high-quality program evaluations ^[1] and quality assurance ^[2] practices that promote continuous program improvement.

The Technical Assistance Center for Program Evaluation and Quality Assurance (PEQA) will assist State VR agencies in building capacity through professional education and training of VR evaluators. To this end, PEQA will:

(a) Provide educational opportunities for State VR staff from recognized experts in program evaluation and quality assurance;

(b) Develop interagency collaboration networks and work teams committed to the improvement of quality assurance systems and tools; and

(c) Deliver technical, professional, and continuing educational support to State VR program evaluators.

Project Activities

To meet the requirements of this priority, the PEQA must, at a minimum, conduct the following activities:

Basic Certification Program

(a) Develop a one-year certificate program in VR program evaluation that will result in increasing the numbers and qualifications of program evaluators in State VR agencies. At a minimum, this certificate program must:

(1) Be designed to develop key competencies necessary for successful implementation of program evaluation and quality assurance activities, including, but not limited to:

(i) Knowledge of the State-Federal VR program;

(ii) Data collection methodologies;

(iii) Data analysis and interpretation;

(iv) Making evaluative judgments and recommendations;

(v) Effective communication of results (including presentations, drafting reports, and building partnerships); and

(vi) Ethical practice.

(2) Be responsive to the prior knowledge and skills of participants;

(3) Incorporate adult learning principles and opportunities for practice into training;

(4) Be delivered through multiple modalities and in an accessible format;

(5) Assess, at regular intervals, the progress of training participants toward attainment of the key competencies; and

(6) Require the completion of a capstone project in order to successfully complete the program. The capstone project must:

(i) Be completed within one year of the completion of formal coursework for the certificate program;

(ii) Be conducted on a topic responsive to the needs of the State VR agency and agreed to by the PEQA, the participant, and the State VR agency; and

(iii) Be completed as part of the normal work duties of the participant in the State VR agency.

(7) Be provided at no cost to participants, excluding travel and per diem costs, which may be provided by the sponsoring agency.

(b) Provide training through the certificate program to a cohort of eight to ten working professionals in each year of the project.

(c) Select participants for the certificate program based, in part, on the considered recommendation of their employing State VR agencies.

Special Topical Training

(a) Develop a series of special training opportunities for intermediate-level program evaluators. These training opportunities must, at a minimum:

(1) Be designed to develop higher-level knowledge, skills, and abilities of program participants;

(2) Be focused on a range of topics determined by the PEQA with input from State VR agencies and other relevant groups or organizations;

(3) Provide opportunities for hands-on application of the competencies discussed in the trainings;

(4) Be of sufficient duration and intensity to ensure that participants obtain the competencies discussed in the trainings; and

(5) Assess the progress of program participants in attaining the competencies discussed in the trainings.

(b) If, after ensuring that intermediate-level program evaluators have priority in registering for Special Topical Training provided under paragraph (a), the PEQA determines that additional space is available, the Center may open registration to individuals who have completed the Basic Certification Program described in this priority. In addition, if additional space in such training opportunities is still available after intermediate-level program evaluators and individuals who have completed the Basic Certification Program have been allowed to register, the Center may open registration to State VR agency personnel whose current work responsibilities are closely aligned with the specific topic area of the particular training opportunity.

Note: For purposes of this priority, an “intermediate-level program evaluator” is a program evaluator working for a State VR agency with the knowledge, skills, and abilities typically expected of a professional who has been in such a position for at least five years.

(c) Conduct no fewer than four special training opportunities each year of the project.

Coordination Activities

(a) Support, strengthen, and augment existing communities of practice, and establish new communities of practice, as needed, to act as vehicles for communication, exchange of information among program evaluation professionals, and a forum for sharing the results of capstone projects that are in progress or have been completed. These communities of practice must be focused on challenges facing program evaluation professionals and the development of key competencies to address such challenges;

(b) Maintain a Web site that, at a minimum:

(1) Provides a central location for later reference and use of capstone projects, resources from special training opportunities, and other relevant materials; and

(2) Ensures peer-to-peer access between State VR project evaluation professionals;

(c) Communicate and coordinate, on an ongoing basis, with other relevant Department-funded projects and those supported by the Departments of Labor, Commerce, and Health and Human Services; and

(d) Maintain ongoing communication with the RSA project officer and other RSA staff as required.

Application Requirements.

To be funded under this priority, applicants must meet the application and administrative requirements in this priority. RSA encourages innovative approaches to meet these requirements, which are:

(a) Demonstrate, in the narrative section of the application under “Significance of the Project,” how the proposed project will—

(1) Address State VR agencies' capacity to conduct high quality program evaluation and data analysis activities. To address this requirement, the applicant must:

(i) Demonstrate knowledge of emerging and best practices in program evaluation and quality assurance;

(ii) Demonstrate knowledge of current State VR and other efforts designed to improve evaluation and performance management practices.

(2) Increase the number of program evaluators working in State VR agencies who have obtained a certificate in their field of work and the number and quality of program evaluation activities performed by State VR agencies.

(b) Demonstrate, in the narrative section of the application under “Quality of Project Services,” how the proposed project will—

(1) Achieve its goals, objectives, and intended outcomes. To meet this requirement, the applicant must provide—

(i) Measurable intended project outcomes;

(ii) A plan for how the proposed project will achieve its intended outcomes; and

(iii) A plan for communicating and coordinating with relevant training programs and communities of practice, State VR agencies, and other RSA partners.

(2) Use a conceptual framework to develop project plans and activities, describing any underlying concepts, assumptions, expectations, beliefs, or theories, as well as the presumed relationships or linkages among these variables, and any empirical support for this framework.

(3) Be based on current research and make use of evidence-based practices. To meet this requirement, the applicant must describe:

(i) How the current research about adult learning principles and implementation science will inform the proposed training; and

(ii) How the proposed project will incorporate current research and evidence-based practices in the development and delivery of its products and services.

(4) Develop products and provide services that are of high quality and sufficient intensity and duration to achieve the intended outcomes of the proposed project. To address this requirement, the applicant must describe—

(i) Its proposed curriculum for a certificate program for VR evaluation professionals;

(ii) Its proposed plan for recruiting and selecting trainees for the certification program;

(iii) Its proposed plan for collecting information on the impact of capstone projects;

(iv) Its proposed plan for identifying, selecting and addressing the special topical program evaluation and quality assurance related training needs of State VR agency staff;

(v) Its proposed plan for annual follow-up with participants in special training opportunities;

(5) Develop products and implement services to maximize the project's efficiency. To address this requirement, the applicant must describe—

(i) How the proposed project will use technology to achieve the intended project outcomes; and

(ii) With whom the proposed project will collaborate and the intended outcomes of this collaboration.

(c) Demonstrate, in the narrative section of the application under “Quality of the Evaluation Plan,” how the proposed project will—

(1) Measure and track the effectiveness of the training provided. To meet this requirement, the applicant must describe its proposed approach to—

(i) Collecting data on the effectiveness of training activities;

(ii) Analyzing and reporting data on the effectiveness of training, including any proposed standards or targets for determining effectiveness;

(2) Collect and analyze data on specific and measurable goals, objectives, and intended outcomes of the project, including measuring and tracking the effectiveness of the training provided. To address this requirement, the applicant must describe—

(i) Its proposed evaluation methodologies, including instruments, data collection methods, and analyses;

(ii) Its proposed standards or targets for determining effectiveness;

(iii) How it will use the evaluation results to examine the effectiveness of its implementation and its progress toward achieving the intended outcomes; and

(iv) How the methods of evaluation will produce quantitative and qualitative data that demonstrate whether the project and individual training activities achieved their intended outcomes.

(d) Demonstrate, in the narrative section of the application under “Adequacy of Project Resources,” how—

(1) The proposed project will encourage applications for employment from persons who are members of groups that have traditionally been underrepresented based on race, color, national origin, gender, age, or disability, as appropriate;

(2) The proposed key project personnel, consultants, and subcontractors have the qualifications and experience to achieve the project's intended outcomes;

(3) The applicant and any key partners have adequate resources to carry out the proposed activities; and

(4) The proposed costs are reasonable in relation to the anticipated results and benefits.

(e) Demonstrate, in the narrative section of the application under “Quality of the Management Plan,” how—

(1) The proposed management plan will ensure that the project's intended outcomes will be achieved on time and within budget. To address this requirement, the applicant must describe—

(i) Clearly defined responsibilities for key project personnel, consultants, and subcontractors, as applicable; and

(ii) Timelines and milestones for accomplishing the project tasks.

(2) Key project personnel and any consultants and subcontractors will be allocated to the project and how these allocations are appropriate and adequate to achieve the project's intended outcomes, including an assurance that such personnel will have adequate availability to ensure timely communications with stakeholders and RSA;

(3) The proposed management plan will ensure that the products and services provided are of high quality; and

(4) The proposed project will benefit from a diversity of perspectives, including those of State and local personnel, technical assistance providers, researchers, and policy makers, among others, in its development and operation.

Types of Priorities

When inviting applications for a competition using one or more priorities, we designate the type of each priority as absolute, competitive preference, or invitational through a notice in the Federal Register. The effect of each type of priority follows:

Absolute priority: Under an absolute priority, we consider only applications that meet the priority (34 CFR 75.105(c)(3)).

Competitive preference priority: Under a competitive preference priority, we give competitive preference to an application by (1) awarding additional points, depending on the extent to which the application meets the priority (34 CFR 75.105(c)(2)(i)); or (2) selecting an application that meets the priority over an application of comparable merit that does not meet the priority (34 CFR 75.105(c)(2)(ii)).

Invitational priority: Under an invitational priority, we are particularly interested in applications that meet the priority. However, we do not give an application that meets the priority a preference over other applications (34 CFR 75.105(c)(1)).

This notice does not preclude us from proposing additional priorities, requirements, definitions, or selection criteria, subject to meeting applicable rulemaking requirements.

Note: This notice does not solicit applications. In any year in which we choose to use this priority, we invite applications through a notice in the Federal Register.

Executive Orders 12866 and 13563

Regulatory Impact Analysis

Under Executive Order 12866, the Secretary must determine whether this regulatory action is “significant” and, therefore, subject to the requirements of the Executive order and subject to review by the Office of Management and Budget (OMB). Section 3(f) of Executive Order 12866 defines a “significant regulatory action” as an action likely to result in a rule that may—

(1) Have an annual effect on the economy of $100 million or more, or adversely affect a sector of the economy, productivity, competition, jobs, the environment, public health or safety, or State, local, or Tribal governments or communities in a material way (also referred to as an “economically significant” rule);

(2) Create serious inconsistency or otherwise interfere with an action taken or planned by another agency;

(3) Materially alter the budgetary impacts of entitlement grants, user fees, or loan programs or the rights and obligations of recipients thereof; or

(4) Raise novel legal or policy issues arising out of legal mandates, the President's priorities, or the principles stated in the Executive order.

This final regulatory action is not a significant regulatory action subject to review by OMB under section 3(f) of Executive Order 12866.

We have also reviewed this final regulatory action under Executive Order 13563, which supplements and explicitly reaffirms the principles, structures, and definitions governing regulatory review established in Executive Order 12866. To the extent permitted by law, Executive Order 13563 requires that an agency—

(1) Propose or adopt regulations only upon a reasoned determination that their benefits justify their costs (recognizing that some benefits and costs are difficult to quantify);

(2) Tailor its regulations to impose the least burden on society, consistent with obtaining regulatory objectives and taking into account—among other things and to the extent practicable—the costs of cumulative regulations;

(3) In choosing among alternative regulatory approaches, select those approaches that maximize net benefits (including potential economic, environmental, public health and safety, and other advantages; distributive impacts; and equity);

(4) To the extent feasible, specify performance objectives, rather than the behavior or manner of compliance a regulated entity must adopt; and

(5) Identify and assess available alternatives to direct regulation, including economic incentives—such as user fees or marketable permits—to encourage the desired behavior, or provide information that enables the public to make choices.

Executive Order 13563 also requires an agency “to use the best available techniques to quantify anticipated present and future benefits and costs as accurately as possible.” The Office of Information and Regulatory Affairs of OMB has emphasized that these techniques may include “identifying changing future compliance costs that might result from technological innovation or anticipated behavioral changes.”

We are issuing this final priority only on a reasoned determination that its benefits justify its costs. In choosing among alternative regulatory approaches, we selected those approaches that maximize net benefits. Based on the analysis that follows, the Department believes that this regulatory action is consistent with the principles in Executive Order 13563.

We also have determined that this regulatory action does not unduly interfere with State, local, and tribal governments in the exercise of their governmental functions.

In accordance with both Executive orders, the Department has assessed the potential costs and benefits, both quantitative and qualitative, of this regulatory action. The potential costs are those resulting from statutory requirements and those we have determined as necessary for administering the Department's programs and activities. The benefits of the Rehabilitation Training program have been well established over the years through the successful completion of similar projects funded for the purpose of improving the skills of State VR agency staff. The priority would specifically improve the skills of State VR agency evaluators. A project of this type will be particularly beneficial to State VR agencies in this era of increased emphasis on accountability and program results.

Intergovernmental Review: This program is subject to Executive Order 12372 and the regulations in 34 CFR part 79. One of the objectives of the Executive order is to foster an intergovernmental partnership and a strengthened federalism. The Executive order relies on processes developed by State and local governments for coordination and review of proposed Federal financial assistance.

This document provides early notification of our specific plans and actions for this program.

Accessible Format: Individuals with disabilities can obtain this document in an accessible format (e.g., braille, large print, audiotape, or compact disc) on request to the program contact person listed under FOR FURTHER INFORMATION CONTACT.

Electronic Access to This Document: The official version of this document is the document published in the Federal Register. Free Internet access to the official edition of the Federal Register and the Code of Federal Regulations is available via the Federal Digital System at: www.gpo.gov/​fdsys. At this site you can view this document, as well as all other documents of this Department published in the Federal Register, in text or Adobe Portable Document Format (PDF). To use PDF you must have Adobe Acrobat Reader, which is available free at the site.

You may also access documents of the Department published in the Federal Register by using the article search feature at: www.federalregister.gov. Specifically, through the advanced search feature at this site, you can limit your search to documents published by the Department.

Dated: August 5, 2015.

Michael K. Yudin,

Assistant Secretary for Special Education and Rehabilitative Services.

Footnotes

1.  “Program evaluation” is “the appropriate, timely, and systematic collection, analysis, and reporting of data to facilitate stakeholder judgement concerning program worth in regards to its design, demands, size and type of effect, match between effect and need, cost effectiveness, strength of casual connections and utility.” Leahy, M.J., Thielsen, V.A., Millington, M.J., Austin, B., & Fleming, A. (2009). Quality assurance and program evaluation: Terms, models, and applications. Journal of Rehabilitation Administration, 33(2), 69-82.

2.  “Quality assurance” is “a systematic process designed to identify, analyze, and eliminate variations in processes or outcomes.” Leahy, M.J., Thielsen, V.A., Millington, M.J., Austin, B., & Fleming, A. (2009). Quality assurance and program evaluation: Terms, models, and applications. Journal of Rehabilitation Administration, 33(2), 69-82.