[Federal Register Volume 60, Number 61 (Thursday, March 30, 1995)]
[Notices]
[Pages 16507-16515]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 95-7743]
-----------------------------------------------------------------------
DEPARTMENT OF LABOR
Evaluation of the Summer Youth Employment and Training Program
AGENCY: Office of the Secretary, Labor.
ACTION: Expedited review under the Paperwork Reduction Act.
-----------------------------------------------------------------------
SUMMARY: The Employment and Training Administration, Department of
Labor, in carrying out its responsibilities under the Paperwork
Reduction Act (44 U.S.C. Chapter 35, 5 CFR 1320 (53 FR 16618, May 10,
1988)), is submitting a study to examine the range of practices
currently being used in the Summer Youth Employment and Training
Administration (SYETP) to deliver educational services. It will assess
the quality of training and evaluate contributions to the educational
deficiencies of participants.
DATES: The Employment and Training Administration has requested an
expedited review of this submission under the Paperwork Reduction Act;
this Office of Management and Budget (OMB) review has been requested to
be completed by April 14, 1995.
FOR FURTHER INFORMATION CONTACT:
Comments and questions regarding the Evaluation of the SYETP should be
directed to Mr. Kenneth A. Mills, Departmental Clearance Officer,
Office of Information Resource Management Policy, U.S. Department of
Labor, 200 Constitution Avenue NW., Room N-1301, Washington, DC 20210,
(202) 219-5095.
Comments should also be sent to OMB, Office of Information and
Regulatory Affairs, Attn: OMB Desk Officer for ETA, NEOB Room 10235,
Washington, DC 20503, (202) 395-7316.
Any member of the public who wants to comment on the information
collection request which has been submitted to OMB should advise Mr.
Mills of this intent at the earliest possible date.
Average Burden Hours/Minutes Per Response: 113 minutes
Frequency of Response: One time
Number of Respondents: 9,115
Total Annual Burden Hours: 17,167 hours
Total Annual Responses: 9,115
Affected Public: Individuals or households; Non-for-profit
institutions; State, Local or Tribal Government
Respondents Obligation to Reply: Voluntary
Signed at Washington, DC, this 24th day of March 1995.
Kenneth A. Mills,
Departmental Clearance Officer.
I. Introduction
This document represents a request for approval of the data
collection protocols to be used in the Evaluation of the Summer Youth
Employment and Training Program, being conducted by Social Policy
Research Associates (SPR) and Brandeis University's Center for Human
Resources, under contract to the U.S. Department of Labor (DOL). The
study uses qualitative (case study) and quantitative data collection
and analysis methods to examine training practices being used in the
Summer Youth Employment and Training Program (SYETP), Title II-B of
JTPA. The Introduction to this document provides a brief overview of
the study and its purposes, and it discusses the data collection
procedures and analysis plans. Subsequent sections respond to the
Office of Management and Budget's (OMB) specific instructions for
justification and address issues related to the collection of
information using statistical methods.
Background
Funded under Title II-B of the Job Training Partnership Act, SYETP
has its origins in a thirty-year federal commitment to create summer
jobs for disadvantaged youth. However, developments in recent years
have as well affirmed an emphasis on providing educational services.
For example, amendments to Title II-B enacted in 1986 enumerated the
enhancement of basic educational skills and encouragement of school
completion as explicit goals of the program. Further, SDAs were
required to assess the reading and mathematics skill levels of SYETP
participants and to provide remedial and basic education services where
appropriate. Subsequent DOL issuances reinforced the educational
emphasis of the Summer Youth program and encouraged efforts to link
work and learning. [[Page 16508]]
Purposes of the Study
The changing focus of SYETP raises questions about the proper role
of the program's educational component and the feasibility of
diagnosing and meaningfully redressing the basic skills deficiencies of
large numbers of youth within the compressed time frame of the summer
program. Thus, the objectives of the evaluation are to examine the
range of practices currently being used to deliver educational
services, explain variation in service designs, assess the quality of
training being provided, and evaluate its ability to meet the needs of
participants and make significant contributions to their educational
deficiencies. Ultimately the study will enable DOL to gauge the
adequacy of services currently being provided, identify areas of
weakness and, conversely, service designs that appear especially
efficacious, and provide leadership and technical assistance to improve
training practices.
Conceptual Framework
Guiding the data collection and analysis efforts are a client-level
model of high quality educational services and a system-level model of
factors that determine training practices. The client-level model of
training quality, presented in Exhibit I, depicts how clients flow
through the SYETP program, the quality indicators for each type of
service that the program provides, and the intended consequences of
high-quality services for youth. Steps identified in this model are:
Recruitment, assessment, and service planning practices.
Quality indicators associated with this phase of service delivery
include whether programs have a clear strategy for which youth should
be targeted and effective procedures to recruit them, whether they
conduct a comprehensive assessment of youths' skills and interests, and
whether the assessment results are used to develop an individualized
service strategy tailored to the skills and interests of each
participant.
Providing effective educational services, either through
classroom or work-based instruction. Quality indicators for both
training content and instructional methods are identified in the
exhibit, including whether the training objectives are well-specified,
whether they promote the educational skills needed in the workplace,
whether training is provided in a functional context, whether
participants' progress is documented, whether there are ample
opportunities to learn, and whether the style of instruction promotes
active learning that is adaptive to the needs of individual
participants.
Providing linkages with continuing educational activities,
to sustain and build on learning gains.
In contrast to the client-level model, the system-level model,
shown in Exhibit II, is intended as a casual (rather than a temporal)
model and identifies factors that influence service delivery, including
those that facilitate or impede the development of high-quality
educational services. The far right box of this model contains the
elements of high-quality SYETP educational services that were described
in the client-level model. The exhibit schematically identifies aspects
of Federal and State policies and the local environment that can affect
an SDA's program design, and it shows how design decisions and
educational provider characteristics, in turn, affect the quality of
educational services provided. Specifically, it identifies:
Federal, State, and local influences on programs' designs,
including federal Title II-B policies, other Federal initiatives and
policies, State JTPA and educational policies, and characteristics of
local youth and of the local area.
SDA design factors, including program goals, target
groups, and service delivery arrangements.
Attributes of the service providers who deliver
educational services to participants, including the types of
institutions, their history, objectives, and funding sources.
Questions for the Evaluation
The preceding conceptual frameworks give rise to a number of
specific questions to be investigated in the project. These include
issues relating to the design of services at the SDA level, the design
of services at the level of the educational provider, and the quality
and impact of educational services.
The design of SYETP at the SDA level.
--What general objectives have SDAs established for their Title II-B
programs? What specific objectives (in terms of skills to be conveyed,
benchmarks to be achieved) have been established for the Title II-B
educational components?
--Do programs identify priority client groups? If so, what target
groups have they established? Who makes those decisions, and how and
why were they made?
--What types of providers are used by the SDA for educational
instruction? How were these providers selected and why were they
selected?
--How are other services, including supportive services and stipends,
used in the summer youth program? How are these services used to
support educational and other goals for the program?
--What linkages has the SDA established between its Titles II-B and II-
C programs?
--What ``front-end'' and ``back-end'' linkages has the SDA established
with public schools? Who instigated these linkages and who maintains
them? Are the linkages formal or informal?
--What role have federal and state policies and local influences played
in the SDA's design decisions? How have these policies been perceived
and implemented?
The design of SYETP at the provider level
--What types of organizations provide educational instruction? What
objectives have been established for their programs?
--Why did the provider decide to participate in the summer youth
program and how was it selected?
--What objectives has the provider established for its educational
program? What skills (e.g., basic skills, SCANS skills) is it
endeavoring to teach? Is it attempting to link learning and work?
--What service design is it using to meet these objectives? Who
developed the design and why? Was the design established explicitly for
the summer youth educational program?
--How are educational services sequenced? How was the curriculum
developed?
--How did it recruit and train its staff?
--How have the SDA's objectives for the summer youth educational
program been communicated to the service provider and how have they
been acted upon?
--What role has the SDA played in designing the provider's educational
services, including its content and method of delivery? How does the
SDA monitor the services that are being provided and how does it
suggest changes?
--How does the provider's design reflect other elements of the local
context, including the needs of the community, the characteristics of
youth in the area, and the characteristics of the school district?
The quality and impact of educational services
--What procedures are used by SDAs to recruit youth for the summer
program? Do recruitment methods correspond to their targeting goals?
--How is the participant's initial assessment conducted and how is
[[Page 16509]] subsequent progress assessed and documented?
--How are individual service plans developed? Are the service plans
truly individualized to the needs, skills, and interests of each
participant? Does the youth play an active role in formulating the
service plan? Are the resulting goals clear and ambitious (without
being unrealistic)?
--Does the educational instruction that is being provided have well-
specified objectives? Do the objectives indicate skills to be acquired
(rather than knowledge to be learned)?
--Does the instruction emphasize skills needed in the workplace? Are
the skills taught in a functional context?
--Does the instruction promote active learning and training for
transfer?
--Is instruction adaptive and provided by capable and caring adults who
view their role as a ``facilitator''?
--Have linkages been established to provide feedback to schools or
other programs serving the youth?
--What implications do service provider characteristics and design
decisions have for the quality of educational services?
--What implications does participation in SYETP, in general, have for
stabilizing or improving academic or other achievements, promoting
school completion, and increasing the motivation to learn?
--What implications do alternative designs for delivering educational
services (e.g., provider characteristics, the locus on instruction)
have for these same youth outcomes?
Study Design
To address the research questions identified above, this study uses
two evaluation components--a process study and a client-level study of
outcomes.
The Process Study
The process study uses a series of ``nested'' qualitative case
studies to examine the design and operation of SYETP services at 30
SDAs nationwide and approximately 3 educational providers at each of
these SDAs (up to 90 providers total). The data collection activities
will consist of a review of plans for the Summer Program as well as 4-
day on-site visits to each selected SDA and its associated providers,
during which time researchers will meet with SDA and provider
administrators and staff, classroom instructors, and worksite
supervisors, and they will observe educational instruction. While on
site, researchers also will conduct 1 focus group with approximately 5-
6 youth participants at each educational provider visited and review
case files for 2 youths at each provider. Follow-up telephone
discussions also will be conducted with youths selected for the case
file reviews and with their parents and school counselors to learn of
retrospective impressions of and satisfaction with the SYETP experience
and perceived impacts on subsequent achievements and behaviors in
school.
Selecting the Sample
As part of the study, samples are being drawn of 30 SDAs and up to
3 educational activities in each SDA (or 90 total). To ensure that the
resulting sample will be nationally representative, the 30 SDAs are
being selected using stratified random sampling. In selecting the
sample, all SDAs nationwide are being assigned to one of 4 strata. The
first 3 of these groups are defined according to the percent of their
Summer Youth participants who receive educational instruction, with the
first stratum consisting of those SDAs with percents between 1% and
41%, the second between 42% and 73%, and the third between 73% and
100%. These cutoffs were chosen so that approximately equal numbers of
youths receiving educational instruction are in each of the three
strata. The 4th stratum consists of those SDAs for whom information on
the number of participants in educational instruction is not available.
Within each strata, SDAs were sampled with the odds of selection
proportionate to the number of participants being served,\1\ so that
the resulting sample of SDAs is approximately self-weighting.
\1\Dollar allocations for the Summer Youth program were used in
the fourth stratum, because the number of participants receiving
educational instruction is not available for these SDAs.
---------------------------------------------------------------------------
Because this study is intended to describe and compare the
effectiveness of a wide variety of approaches to building the
educational skills of SYETP participants, educational providers are
selected within each of the 30 SDAs using purposive selection methods.
Specifically, all educational providers used by these SDAs are to be
categorized according to their:
Content emphasis (e.g., basic skills only; SCANS
foundation skills and/or competencies; or other academic subjects, such
as science, history, or art).
Locus of educational instruction (e.g., classroom-based,
work-based, or both).
Type of provider (e.g., SDA; secondary school, other
educational institution such as community college or technical college,
or other).
Targeted participants (e.g., 14-15 year olds, 16-18 year
olds, other target groups).
Providers are being selected to ensure the diversity of the sample
(both within the SDA and across all 30 SDAs) with respect to these
dimensions.
Data Collection
The field protocols, or topic guides, developed for this process
study are designed to guide the data collection activities. These
protocols will permit site visitors to tailor discussions and
observations on a standardized set of issues to the particular context
of each case study SDA and sampled educational activity. The following
topic guides have been developed and are submitted for OMB's review:
SDA Guide #1 includes the topics to be covered in
discussions with SDA policy, planning, and administrative staff,
including those relating to the goals, design, and management of the
Summer Program.
SDA Guide #2 includes the discussion topics to be used
with SDA staff responsible for direct operation or oversight of client
recruitment, assessment, service planning, and case management services
for Title II-B participants.
Program Guide #1 includes the topics to be covered in
discussions with administrators of the selected educational activity,
staff that participated in the planning and development of the detailed
curriculum, and supervisors responsible for hiring, training, and
overseeing instructors/work site supervisors involved in educational
activities.
Program Guide #2 includes the topics to be used in
discussions about the classroom-based learning approach with classroom
instructors or other staff whose primary responsibility is to support
learning in a classroom or individual study setting (e.g., tutors,
educational resource staff).
Program Guide #3 includes the topics to be used in
discussions with work project coordinators and worksite supervisors who
are involved in work-based learning. This guide includes topics for
projects using the 100% work-based learning approach as well as topics
for staff involved in work activities that are closely coordinated with
classroom-based learning.
Program Guide #4 is a guide for structured observations of
educational activities.
Program Guide #5 is a guide for structured review of
curriculum materials.
Client Guide #1 describes the topics to be addressed in
focus group [[Page 16510]] discussions with approximately 5 youth
participating in each selected educational activity.
Client Guide #2 will be used to extract relevant
information for the case history sample from the participants' written
case files at the SDA or provider.
Client Follow-Up Guide describes the topics to be
discussed with selected youth participants (i.e., those selected for
the case file review) several months after their Summer Program's
participation has ended.
Parent/Guardian Guide will guide the issues to be
addressed with these youths' parent or guardian during the follow-up
period.
Counselor Guide describes the topics to be addressed with
the youths' secondary school counselors during the school year
following the youths' summer participation.
Data Analysis
The analysis of the case studies will begin with a within-site
explanatory analysis. This task will consist of bringing to bear the
data that has been collected to arrive at a comprehensive picture of
the practices in each of the SDAs and service providers that were
selected for the study and how they have contributed to the needs of
the participants. A particular objective will be to uncover especially
innovative practices, with an eye to understanding how they were
implemented and what makes them work so well.
The next step will consist of cross-site comparisons to synthesize
the findings. This analysis will clarify further the unique procedures
that programs adopt to deliver high quality training in a variety of
environments and arrive at an understanding of commonalities and
differences between programs and how these are related to effective
practices.
Client-Level Study of Outcomes
In addition to collecting and analyzing information from the case
studies about program practices, this study also will gather and
analyze quantitative information for a sample of approximately 1,800
youths who participate in the Summer Program in 1994 and an additional
4,000 youths who participate in the 1995. By compiling and analyzing
information for a sample of participants on the services that were
received and the outcomes that were obtained, the study will draw
inferences regarding the relative efficacy of various service design
and delivery methods.
Selecting Participants for the Study
A key element of the overall research design is to tie the results
from the case study observations of classroom instruction and
appraisals of training quality to the analysis of participants'
outcomes. In this way, inferences can be drawn regarding the
relationship between the training practices observed in the field to
the consequences of those practices for the youth who receive them. For
this reason, participants selected for the client-level study of
outcomes will be those whose SYETP instruction was delivered by the
service providers whose practices were observed on site. Specifically,
participants will be selected in the following ways:
Preliminary information received from the 30 SDAs has led
us to determine that approximately 15 of them require pre-tests and
post-tests of basic skills for all participants receiving academic
instruction. All youth served in the summer of 1994 by the 3 selected
providers in these SDAs will be included in the study. This will yield
an expected sample of approximately 1,800 respondents.
The service providers visited for the case studies in the
30 SDAs included in the study will each be asked to administer a common
pre-test/post-test in the summer of 1995, as well as a brief instrument
measuring self-esteem. All youth served by these providers will be
included in the study. This will yield approximately an additional
3,600 respondents.
A randomly chosen sample of 400 youths not receiving
educational instruction also will be included in the study as a
comparison group.
Data Collection
The plan for the client-level study of outcomes is based on the
analysis of information for the sampled participants drawn from a
variety of sources and that, to a large degree, already exists. Thus,
the compilation of these data for analysis purposes entails data
gathering at least as much as new data collection. Specific data
sources to be used are these:
The SDA's MIS. Although the specific types of information
doubtless will vary from one SDA to the next, most SDAs' MIS will
include: participant's demographic and background characteristics
(e.g., race, school status, gender), barriers to employment (e.g.,
whether the youth is a limited-English speaker or has a disability),
and summary information about services received. SDAs will be requested
to transmit these data to the contractor electronically (e.g., on data
diskette).
SDA's Client Files. Those SDAs able to provide pre-test/
post-test scores for youth served in the summer of 1994 will forward
those scores to the contractor for data entry. All participating SDAs
will forward the hard-copy pre-test/post-tests and self-esteem surveys
of sampled participants served in 1995 to the contractor for scoring
and keypunching.
School Records. An important objective of the study is to
learn how participants (at least those who are students) fare in their
subsequent schooling. Outcomes of interest include measures of academic
achievement (e.g., grade-point-average), but also evidence of
behavioral problems (e.g., as evidenced by absenteeism, suspensions/
expulsions). Thus, school record information will be abstracted for
sampled summer 1995 youths who sign and have their parents/guardians
sign a consent form.
Exhibit 3 summarizes the variables to be measured from these
sources.
Data Analysis
A preliminary analysis will be conducted using SDA MIS data and
pre-test/post-test scores for the 1,800 youth who participated in the
summer program in 1994, selected as described above. A more
comprehensive analysis for a larger sample will be conducted when
school record data are collected, for youths who participated in the
summer of 1995. Additional outcomes to be examined with these data
include: self-esteem, school attendance, grade completion, grade-point
average, and absenteeism. Two types of analysis will be conducted:
Descriptive analyses, which will paint a picture of the
characteristics of persons receiving educational instruction in the
sampled programs, the types of services received, and the outcomes
obtained.
Explanatory analyses that will examine the efficacy of
alternative service designs and delivery mechanisms for subsequent
outcomes.
Reporting
The project's major deliverables include:
An Interim Report. This report will detail the results of
the process analysis, describing results from the case studies
regarding how services are designed and delivered. It also will include
the preliminary results from the study of outcomes based on the data
collected for youth who participated in the summer of
1994. [[Page 16511]]
A Technical Assistance Guide (TAG). The TAG will be a
practitioner's guide describing effective practices in the delivery of
educational services, focusing especially on how educational
instruction can be delivered in a functional, work-related context.
A Final Report. This report will represent a summation of
the study's findings and recommendations. As such, it will include the
content of the Interim Report, combined with the comprehensive results
of the study of outcomes.
II. Supporting Statement
A. Justification
1. Circumstances Making the Data Collection Necessary
The Department of Labor (DOL) is considering ways of improving the
educational component of the JTPA Title II-B Summer Youth Employment
and Training program (SYETP), in keeping with Secretary Reich's ``First
Jobs/New Jobs/Better Jobs'' initiative. Its objectives for SYETP are to
improve the program's effectiveness in assisting young people acquire
strong workplace foundation skills (including basic skills, thinking
skills, and interpersonal skills) and gain an appreciation of the
inextricable connection between learning and success in the workplace.
As part of its effort to foster program improvements, DOL needs to
obtain a thorough understanding of educational services currently being
provided to summer youth participants--including how participants are
assessed, the curriculum being used, and how the educational and work
components of SYETP are integrated--and identify particularly
efficacious practices.
As part of its response to Executive Order No. 12862 requiring all
Federal agencies to develop customer service standards, DOL also needs
to know participants' views about the services they received in SYETP,
including their service needs and how well the program responded to
those needs. This information is critical to implementing changes that
can improve program responsiveness.
2. Use of Information and Consequences if Not Collected
The information being collected in this study will be used to
address these objectives:
1. Describe variation in the design of SYETP educational services
across service delivery areas (SDAs) and their service providers, with
respect to general goals and objectives they have established for the
program, their targeting decisions, assessment procedures, specific
skills being taught, the locus of instruction, linkages between work
and learning, and linkages with public schools and year-around Title
II-C JTPA services.
2. Describe variation in the quality of educational services,
including whether assessments are comprehensive, whether service
strategies are individualized to the needs and interests of
participants, whether the participants are actively involved in
formulating the service plan, whether educational instruction has well-
specified objectives relating to skills to be acquired, whether skills
are taught in a functional context and emphasize skills needed in the
workplace, and whether instruction is adaptive.
3. Identify factors that explain variation in how educational
services are being designed and delivered, such as federal policies,
opportunities for technical assistance and training, state-level
partnerships between JTPA and the school system, and other local
influences.
4. Document consequences of participation in SYETP educational
services, especially high quality services, for participants' skill
levels, subsequent academic achievement, and school attendance and
performance.
5. Document participants' satisfaction with the program, including
their assessment of the helpfulness of the services they received.
If this information is not collected, DOL will not have the
information it needs to evaluate how educational services are being
delivered or their effectiveness, and thus it will not have the
necessary foundation for implementing program improvements.
3. Considerations to Reduce Burden
The data collection activities have been designed to minimize the
burden on respondents in four major ways. First, pre-existing
information will be utilized wherever possible to minimize the need for
new data collection. These pre-existing sources will include SDAs'
plans for their Title II-B programs, RFPs and contracts written by SDAs
to secure the services of the direct providers of educational
instruction, data collected as part of last summer's study of SYETP,
test scores, information from school records, and existing MIS data
compiled by SDAs about their participants' characteristics, services,
and outcomes. These data sources can be forwarded to the contractor
with minimal burden to SDA or school or provider staff and to program
participants. Where data abstraction requires hand-coding (e.g., from
school records), abstractors will be compensated by the contractor.
Second, where feasible (and at least with respect to the MIS data),
information will be transferred to the contractor electronically (i.e.,
via modem or data diskette), greatly facilitating the data transmission
process.
Third, only data of direct relevance to the goals of the study will
be collected.
Fourth, much time on site will be devoted to the unobtrusive
observation of educational instruction and the review of written
documents and participants' case files, and this too should be
minimally burdensome to SDA and service provider staff or participants.
4. Efforts to Identify Duplication
A study conducted of SYETP during the summer of 1993 included 50
on-site visits and a mail survey of all SDAs (1205-0327, expired 12/
93). However, this data collection focused on general operational
issues, did not entail on-site observations of classroom instruction to
characterize its quality and did not attempt to study youth at any
point beyond their period of participation.
Additional information available about SYETP comes from the SDAS's
plans for their summer's activities. However, these documents provide
no information about how or how well the plans are implemented, nor do
they allow an assessment of the instruction's quality or effectiveness,
nor do they speak to the participants' satisfaction with the services
they received.
Finally, states are required to submit annual reports providing
aggregate counts of participants served and their characteristics
(1205-0200, expires 7/97). However, these simple summary reports are
useful for little more than identifying the numbers of persons of
different ages and education levels who were served.
5. Why Similar Information Cannot be Used
Information from the sources described above will be used to the
fullest extent possible in the study being planned. Indeed, these data
provide a strong foundation to support the study by providing essential
background and other information. However, DOL has concluded, on the
basis of the effort to identify duplication, that these pre-existing
sources are not adequate to characterize the quality or education
services, support an analysis of the factors associated with high
quality services, describe the consequences of participation in SYETP
for subsequent achievements, or document participants' satisfaction
with the program. Nor are they adequate, consequently, to support DOL's
efforts to foster program improvements.
6. Burden on Small Businesses
Some activities associated with this study will involve the
collection of data from the administrators or staff of organizations
providing educational instruction as part of SYETP, and some of these
entities may be small businesses. However, as described under #3,
``Considerations to Reduce Burden,'' only information of direct
relevance to the study's objectives will be collected while on site.
Secondly, much on-site data collection to be conducted at service
providers will involve the unobtrusive observation of classroom
instruction and the review of client case files, and it will thus
entail minimal burden on the providers' administrators or staff.
Finally, as part of the agreement allowing them to deliver services
under JTPA, providers acknowledge DOL's right to evaluate and/or
monitor their activities and services.
7. Consequences of Less Frequent Data Collection
The data collection activities associated with this study will be
conducted one time only.
8. Collection Inconsistent With 5 CFR 1320.6
Data collection will be consistent with 5 CFR 1320.6.
9. Efforts to Consult With Persons Outside the Agency
Responsibility for devising and carrying out the data collection
rests with DOL's contractor, Social Policy Research Associates (SPR),
and its subcontractor, Brandeis University's Center for Human
Resources. Key personnel associated with these institutions are
nationally known experts in evaluation research and have in-depth
knowledge of employment and training programs in general and Summer
Youth programs in particular.
Additionally, the study team has enlisted the aid of additional
experts, who are serving as consultants on the project. Their advice
was solicited regarding the usefulness of the data elements to be
collected, the feasibility of the data collection plan, and the clarity
of instructions. These consultants are:
Ms. Nancy Bross, Public Policy Support, 1377 McLendon Ave., N.E.,
Atlanta, Georgia 30307, (404) 581-9895
Ms. Lee Bruno, Consultant, 3106 Old Largo Road, Upper Marlborough,
Maryland 20772, (301) 627-1415
Ms. Janice Hendrix, North Central Indiana PIC, 36 West Fifth St., Suite
102-B, Peru, Indiana 46970, (317) 473-5571
Mr. Gill Ritt, Career Resource Associates, 2932 Sumac Drive, Atlanta,
Georgia 30360, (404) 698-8427
Mr. Kip Stottlemyer, Consultant, 1408 Milestone Drive, Collierville,
Tennessee 38017, (901) 854-1438
In addition, all the protocols guiding the conversations with key
respondents have been pre-tested on not more than 9 respondents, and
modifications to the protocols were made on this basis where it seemed
appropriate.
10. Assurances of Confidentiality
The information to be collected will be held strictly confidential
and will be used for research purposes only. To ensure confidentiality,
DOL will require that the study team take the following measures:
Access to the data will be limited to the contractor's
project team members only.
Reports to DOL will focus on describing and analyzing the
range of service designs and training practices that were observed and
will not associate a design or process with any specific SDA or service
provider, except by way of providing an example of exemplary practices,
and then only with the SDA's approval.
Reports to DOL that contain individual vignettes based on
the experiences of participants will not contain individual names or
any other identifying information.
The contractor's project team members will be trained in
the confidentiality requirements and cautioned to use the data for
research purposes only.
11. Justification of Questions of a Sensitive Nature
Two sources of data are potentially sensitive. First, pre-tests and
post-tests will be administered to youth included in the study. Second,
school record information will be abstracted for those youth in the
sample who participated in the summer program of 1995 and who return to
school in the fall. These data elements are imperative to examine
learning gains for those who receive educational services and to
examine if SYETP participation is associated with improved school
performance.
However, for the most part these data elements do not represent new
data collection activities. JTPA currently requires that all summer
youth be administered a test of basic skills to determine their need
for basic skills remediation, and many SDAs also administer post-tests
to document learning gains.
Similarly, information about school performance will be abstracted
from existing student files. Moreover, youths and their parents/
guardians will be asked to sign a consent form before the abstraction
will be conducted. This form will outline the objectives of the study
and ask the youth and his/her parent to allow access to student records
for purposes of the evaluation. It will be explained that participation
in the study is completely voluntary and that a refusal to participate
will not jeopardize the youth's receiving SYETP services.
12. Cost to the Federal Government and to Respondents
The total estimate cost to the federal government for the
collection and analysis of these data is $849,543. Because the study
will be conducted over 3 years,\2\ the average per annum cost is
approximately $283,000. This amount includes the costs of designing the
field protocols, performing the on-site visits and telephone follow-up,
recording observations from the site visits, collecting the client-
level data for the study of outcomes, analyzing the data, and preparing
two reports on the results (i.e., an Interim Report and a Final Report)
and a Technical Assistance Guide (to disseminate information on
effective practices). The method used to derive this figure entailed a
quantification of hours of effort involved by each study team member
and included expenses for materials and services (e.g., photocopying
expenses and expenses involved in binding the report).
The costs to respondents result only from the time spent answering
the questions. Estimates of the time to respond are presented
below. [[Page 16512]]
\2\A separate PWR package will be submitted for any burden
associated with follow-up work done after the first year.
---------------------------------------------------------------------------
13. Estimate of Burden
Below is the estimate of the respondent burden. Time estimates are
based on the pretest of the instruments (for the topic guides to be
used in the process study) or from the use of the instruments in
previous studies (for the pre-test/post-test and self-esteem
[[Page 16513]] scales). It is anticipated that these will be conducted
within the next year and the burden hours represent the first year
burden claimed. If follow-up activities extend beyond one year, an
Inventory Correction Worksheet will be submitted.
------------------------------------------------------------------------
Number of Minutes per
Instruments respondents respondent Total hours
------------------------------------------------------------------------
SDA Guides for Discussions With:
1. Policy, Planning, and
Administrator Staff......... 60 60 60
2. Recruitment, Service
Planning, and Case
Management Staff............ 60 45 45
Program Guides for Discussions
With:
1. Program Administrators.... 90 90 135
2. Classroom Instructors..... 85 20 28
3. Work Project Coodinators.. 10 20 3
Client Guides for:
1. Focus Group with
Participants................ 450 15 112
2. Participants, at Follow-up 120 15 30
Guide for Discussions with
Parents......................... 120 5 10
Guide for Discussions with
Regular School Counselors....... 120 15 30
Pre-test and Post-test of
Participant's Basic Skills
(Workplace Literacy Test)....... 4,000 240 16,000
Self-Esteem Instrument (Rosenberg
Self-Esteem Scale).............. 4,000 10 667
------------------------------------------------------------------------
Additional data collection to be used in the project represents the
abstraction or review of existing information. There are no respondents
for these guides and thus they entail minimal burden on SDA or provider
personnel or program participants beyond copying documents or data
files and shipping them to the contractor. Where hand-extraction of
information is required (e.g., from student records, abstractors (SDA
or service provider personnel) will be compensated.
14. Reason for Change in Burden
This is a new collection as reported in ETA's ICB (Information
Collection Budget). The first year's burden of 17,120 hours is being
submitted now. An Inventory Correction Worksheet will be submitted for
any follow--up activities in the out years.
15. Plans for Statistical Analysis
Data to be collected for this project for the process study
generally will not be analyzed using qualitative research methods,
findings will be detailed in a narrative, and their implications for
improving program quality will be detailed.
Data collected for the study of outcomes will be analyzed using
statistical methods to address these research issues:
What are the characteristics of persons receiving
educational services in the Summer Youth program? Are educational
services targeted to those who have a greater need for remediation?
What types of educational services were provided?
Specifically, in what subject areas (e.g., math, reading, other
academic subjects, SCANS skills) was instruction provided? With what
intensity?
How does the intensity and nature of the training received
relate to outcomes, including learning gains, school attendance rates,
grades in school, rates of absenteeism, and suspensions and expulsions?
How do the outcomes for youth who received educational instruction
compare to those in the compoarison group?
Data for this component of the project will be compiled in
various phases:
Phase I: Collect MIS and pre-test/post-test data for
sample members who participated in the summer program in 1994.
Collection of this information will occur during the winter of 1995.
Phase II: Collect MIS, pre-test/post-test, and self-esteem
data for sample members who participated in the summer program in 1995.
Collection of this information will occur during the winter of 1996.
Phase III: Collect school record information. These data
will be collected during the summer of 1996, after the conclusions of
the 1995-96 school year, so that school outcomes measured for those who
participated in the 1995 summer program will reflect a full school
year.
Methods to be used in analyzing these data will include univariate
and multivariate statistics. Specifically, univariate distributions
will be calculated to describe the characteristics of participants,
their services, and their outcomes. Cross-tabulations will be used to
examine the relationship between variables. Multivariate analyses,
primarily regression analysis, will be used to examine how various
participant characteristics and measures of services received relate to
outcomes.
The project's major deliverables include:
An Interim Report. This report will detail the results of
the process analysis, describing results from the case studies
regarding how services are designed and delivered. It also will include
the preliminary results from the study of outcomes based on the data
collected for youth who participated in the summer of 1994. This report
will be completed at the end of the Summer of 1995.
A Technical Assistance Guide (TAG). The TAG will be a
practitioner's guide describing effective practices in the delivery of
educational services, focusing especially on how educational
instruction can be delivered in a functional, work-related context.
The TAG will be prepared in the Spring of 1996.
A Final Report. This report will represent a summation of
the study's findings and recommendations. As such, it will include the
content of the Interim Report, combined with the comprehensive results
of the study of outcomes. This report will be prepared in the Spring of
1997.
B. Collection of Information Employing Statistical Methods
This process study utilizes qualitative case study data collection
and analysis methods. In terms of identifying appropriate respondents
in each local site and analyzing case study data, qualitative rather
than statistical methods will be used. Discussions of estimation
procedures and degree of accuracy (power analysis) in generalizing
sample findings to the universe of all potential respondents are not
applicable to the process study, because findings will not be expressed
in quantitative terms.
The study of outcomes will employ statistical methods, however.
These [[Page 16514]] methods are described in the rest of this section.
1. Potential Respondent Universe and Sampling Methods
Approximately 625,000 youths can be expected to participate in the
Title II-B program during each of the summers of 1994 and 1995, if
current levels of funding are maintained. Of these, about 40%, or
250,000, will be receiving educational services. These youth are served
by the nation's approximately 640 service delivery areas (SDAs).
The youths to be included in the study will be served by
approximately 90 service providers used for educational instruction in
30 SDAs that were selected for examination in the process study. To
ensure that the sample of SDAs is nationally representative, the 30
SDAs are selected using stratified random sampling. In selecting the
sample, all SDAs nationwide are assigned to one of 4 strata. The first
3 of these groups are defined according to the percent of their Summer
Youth participants that receive educational instruction, with the first
stratum consisting of those SDAs with percents between 1 % and 41%, the
second between 42% and 73%, and the third between 74% and 100%. These
cutoffs were chosen so that approximately equal numbers of youths
receiving educational instruction are in each of these three strata.
The 4th stratum consists of those SDAs for whom information of the
number of participants in educational instruction is not available.
Approximately an equal number of SDAs were drawn from each stratum
and, within each stratum, SDAs were sampled with the odds of selection
proportionate to the number of participants being served,\3\ so that
the resulting sample would be approximately self-weighting. The number
of SDAs in each stratum and the number selected for the study are shown
below.
\3\Dollar allocations for the Summer Youth program were used in
the fourth stratum, because the number of participants receiving
educational instruction is not available for these USAs.
------------------------------------------------------------------------
Number
Percent of the SDA's youths receiving educational Total of SDAs
instruction number in the
of SDAs sample
------------------------------------------------------------------------
Low: 1% to 41%........................................ 273 8
Med. 42% to 73%....................................... 132 8
High: 74% to 100%..................................... 101 8
Information missing................................... 117 7
------------------------------------------------------------------------
Because this study is intended to describe and compare the
effectiveness of a wide variety of approaches to building the
educational skills of SYETP participants, educational providers are
selected within each of the 30 SDAs using purposive selection methods.
Specifically, all educational providers used by these SDAs are to be
categorized according to their:
Content emphasis (e.g., basic skills only; SCANS
foundation skills and/or competencies; or other academic subjects, such
as science, history, or art).
Locus of educational instruction (e.g., classroom-based,
work-based, or both).
Type of provider (e.g., SDA; secondary school, other
educational institution such as community college or technical college,
or other).
Targeted participants (e.g., 14-15 year olds, 16-18 year
olds, other target groups).
Providers are being selected to ensure the diversity of the sample
(both within the SDA and across all 30 SDAs) with respect to these
dimensions.
Because most providers serve fairly few youths, all youths served
by the selected providers during the summer of 1995 will generally be
selected for the study of outcomes. However, for large providers (those
serving more than approximately 70 participants), youths will be
selected who attended classes served by the instructors who were
observed by the site visitors, the 400 youth to be selected for the
comparison group will be selected randomly, approximately 15 from each
SDA.
Following the above procedures, an approximately equal number of
youths will be selected from each SDA. MIS data for each of these
youth, should be available without exception, as will pre-test and
post-test scores and the measure of self-esteem. Similarly, because
SDAs typically require access to school records as part of the
assessment process, we anticipate that high percentages of sampled
youths and their parents/guardians will sign the consent forms allowing
the researchers' access to this information. At least an 80% rate of
cooperation is anticipated.
2. Procedures for the Collection of Information
Sample Selection. As discussed above, the sample has been drawn in
a two-stage process. First, a sample of SDAs and their providers was
chosen, and next participants served by these providers are selected,
along with a randomly chosen sample of participants for the comparison
group.
Degree of Accuracy. To meet DOL's objectives for the survey, the
sample size must be sufficient to allow reliable estimation of
relatively small differences in outcomes across various service
strategies. Let us suppose that the outcome variable is a percentage
(e.g., the percentage of participants who complete their next grade
level), that the average of the outcome is 50%, that we can explain 25%
of the outcome's variation with all predictor variables combined, and
that 10% of the variance in the educational components can be explained
by other control variables. Under these circumstances, the sample size
to be used for this study would be able to detect an approximately 3
percentage points difference in outcomes across a dichotomous measure
of service (e.g., instruction is provided in a functional context or
not). Note that these are generally fairly conservative assumptions.
For example, sample size requirements would be less stringent if the
average of the outcome were either higher or lower than 50%.
Estimation Procedures. As described in Section A, Item 15, several
estimation techniques will be used. First, means and univariate
distributions will be calculated to describe the sample. Second, t-
tests of means (for outcomes measured on a continuous scale) and chi-
square tests (for categorical variables) will be calculated to
determine whether outcomes vary significantly for participants with
different characteristics (e.g., age) or who received different
services. Third, multivariate analysis methods will be used, with
various measures of outcomes as the dependent variable. Independent
variables will include participant characteristics (e.g., age, gender,
pre-test scores) and measures of the types of services received.
Ordinary least squares (OLS) will be used as the estimation
technique for most of these multivariate models, because of its
desirable properties. However, OLS is inefficient when the dependent
variable is categorical (e.g., whether the next school grade was
completed). In these cases, logit analysis will be used.
3. Methods to Maximize Response Rates
Most data to be used for the client study represent pre-existing
records collected by SDAs and schools. For this reason, response rates
should be quite high for all components of the data collection.
Potentially, however, some participants or their parents may deny the
researchers access to school records. To minimize this possibility,
SDAs and their service providers will be contacted far in advance of
the start of the 1995 summer program, and their cooperation will be
enlisted. Thus, when youth are first enrolled in the program they can
be told immediately that they are being [[Page 16515]] asked to
participate in the study and the study's importance can be explained
carefully to them.
4. Tests of Procedures
The data collection for the client-level study involves no new
survey or other instruments. Therefore, no test of procedures is deemed
necessary.
5. Contractor and Individuals Consulted
The Department of Labor has contracted with Social Policy Research
Associates (SPR) to design, conduct, and analyze the study of outcomes.
Key personnel at SPR at Dr. Ronald D'Amico, Dr. Katherine Dickinson,
and Mr. Richard West. They may be contacted at: Social Policy Research
Associates, 200 Middlefield Road Suite 100, Menlo Park, CA 94025. Their
phone is (415) 617-8625.
[FR Doc. 95-7743 Filed 3-29-95; 8:45 am]
BILLING CODE 4510-22-M
