2012-2521. Notice of Random Assignment Study To Evaluate Workforce Investment Act Adult and Dislocated Worker Programs; Final Notice

AGENCY:

Employment and Training Administration (ETA), Labor.

ACTION:

Notice.

SUMMARY:

The Employment and Training Administration (ETA) of the Department of Labor (DOL or the Department) will conduct an evaluation to provide rigorous, nationally-representative estimates of the net impacts of intensive services and training provided under the Workforce Investment Act (WIA) Adult and Dislocated Worker Programs. The Department has determined that it is in the public interest to use a random assignment impact methodology for the study. This methodology will provide ETA with estimates of the net impacts of WIA intensive services and training that are offered during the evaluation study period. On July 21, 2011 (76 FR 43729-43731), the Department solicited comments concerning the Department's plan to use random assignment methodology in carrying out the study. This notice is to provide the Department's response to the comments received.

FOR FURTHER INFORMATION CONTACT:

Eileen Pederson, U.S. Department of Labor, Employment and Training Administration, Office of Policy Development and Research, 200 Constitution Avenue NW., Frances Perkins Bldg., Room N-5641, Washington, DC, 20210. Telephone: [phone] (this is not a toll-free number) or email: [e-mail]. Individuals with hearing or speech impairments may access the telephone number above via TTY by calling the toll-free Federal Information Relay Service at 1-[phone] (TTY/TDD).

SUPPLEMENTARY INFORMATION:

I. Background

On July 21, 2011, the Department announced its plans to conduct an evaluation of the net impacts of intensive services and training provided under WIA (Pub. L. 105-220) Adult and Dislocated Worker Programs. To obtain rigorous, nationally representative estimates of WIA's effectiveness for adults and dislocated workers, the Department determined that it would use random assignment impact methodology for the evaluation.

The design of the study was described as follows: The evaluation will be done in approximately 30 randomly selected LWIAs. WIA applicants in the selected LWIAs who are eligible for intensive services would be randomly assigned to one of three groups. The three research groups to which they would be assigned are: (1) The full-WIA group—adults and dislocated workers in this group can receive any WIA services and training for which they are eligible, (2) the core-and-intensive group—adults and dislocated workers in this group can receive any WIA services for which they are eligible but no training, and (3) the core-only group—adults and dislocated workers in this group can receive only WIA core services but no intensive services or training.

In the LWIAs selected for the study, all applicants for intensive services and/or training will be asked to participate in the study during the 12-18 month study enrollment period. They will be informed of the evaluation, provided an opportunity to ask questions or seek clarification of their role and responsibilities should they agree to participate, and then required to give their consent to participate. Applicants who do not consent to participate in the study will not be randomly assigned to one of the study groups but will be allowed to receive core services only. The participant enrollment period will range between 12 and 18 months in each LWIA.

To protect the rights and welfare of WIA program applicants who agree to participate in the evaluation, the evaluation team, led by researchers from Mathematica and its subcontractor MDRC, submitted the WIA Adult and Dislocated Worker Programs evaluation design to MDRC's Institutional Review Board (IRB) for concurrence. An IRB is a committee specifically responsible for protecting the rights and welfare of humans involved in biomedical and behavioral research. On June 17, 2010, MDRC's IRB determined this study to be of no more than minimal risk and approved it.

The Department requested comments concerning its intent to carry out the random assignment study described above. The Department asked for comments focused on whether there is a methodology that would yield as credible and reliable impacts of the WIA programs as random assignment, but avoids adverse effect on the study participants. The Department also welcomed comments that suggest ways to more effectively minimize any adverse impact on the study participants who participate in the study described above.

II. Discussion of Comments Received

The Department received comments from four sources in response to the notice. The comments were received from two workforce departments, one advocacy group, and one private citizen. The Department's responses to the comments are provided below.

Comment: Two commenters asked about how other sources of funding for services would be accounted for in the study. One of these commenters asked whether the core-only group would have access to other partner services and, if so, the commenter suggested that the study take it into account through the follow-up survey. The other commenter was concerned that the study would not capture the nature of the “WIA system” in which individuals are referred to partner services.

Response: Training and employment services can be funded by many sources other than WIA (such as via Pell grants or State grants) and, as a commenter pointed out, staff from the One-Stop Career Centers may refer customers to services funded by these other sources. Accordingly, we will estimate the impact of WIA-funded services in communities as they exist, which includes services from other providers, rather than estimate the impact of WIA-funded services in a vacuum. Hence, all customers in the study (including members of the core-only group) will have access to services provided by other community service providers. After the 12-18 month study enrollment period concludes, we will conduct two follow-up surveys of study participants, one at 15 months after enrollment and one at 30 months after. The surveys will collect detailed information on services received by study participants. The LWIA counselors will be trained not to refer anyone to services to which they would not have referred them in the absence of the study. The study research questions address the impact of the additional receipt of WIA-funded intensive services and training above the other services in the community that customers may access. Through our analysis of the study, we will document the WIA system that is in place in each of the participating LWIAs. Through on-site visits, researchers will document the availability of non-WIA training and employment services provided by partner agencies.

Comment: One commenter suggested that because of variation in how LWIAs label services as core, intensive, or training, the study should focus on the impact of specific services with similar intensity rather than the relative effectiveness of level of service as authorized by WIA.

Response: As the commenter correctly points out, the activities labeled as core, intensive, and training vary by LWIA. However, consistent with the commenter's suggestion, for purposes of this study, during the analysis of outcomes and qualitative data, we will develop a standard definition of services so that the definition of “core services,” “intensive services,” and “training” are similar across all study sites, irrespective of how each LWIA individually defines its services.

Comment: One commenter asked how the study would estimate the effectiveness of training when customers are randomly assigned to a group that is eligible for training in general—which can vary tremendously across and within LWIAs—rather than a specific type of training. Two commenters noted the importance of taking into account the fact that different study participants in the full-WIA group may receive different forms of training for different industries and occupations.

Response: Customers will be randomly assigned to three groups: (1) The core-only group, in which customers can only receive core services; (2) the core-and-intensive services group, in which customers can receive core or intensive services, but not training; and (3) the full-WIA group, in which customers can receive core, intensive, and training services. We will estimate the impact of training by comparing customers in the full-WIA group with customers in the intensive-services group. We can account statistically for customers in the full-WIA group who do not receive training. To do so, we will use (1) information from the study registration forms on counselors' projections made prior to random assignment on the likelihood customers will receive WIA training services and (2) standard instrumental variables methods that assume zero impacts for those in the full-WIA group who do not receive WIA intensive or training services.

We agree with the commenters that it is important to take into account different types of training that may be received by participants. Our follow-up survey will collect detailed information about the type of training received, including the occupation targeted by the training, the length of the training, any credentials received as a result of the training, and the type of training provider. Through qualitative data collection, the researchers also will document the types of training available through the local workforce investment area and larger community.

Comment: One commenter asked whether the study enrollment period would be long enough to capture the impacts of training.

Response: The enrollment period refers to the period during which customers will be enrolled into the study and randomly assigned. Participants in the study will be surveyed twice, once at 15 months after random assignment and once at 30 months after random assignment. The 30-month follow-up period is long enough to capture the impacts of training, including quite long-term training.

Comment: One commenter was concerned about 50 percent of the participants being placed in a control group. The commenter was also concerned about the time and effort taken by LWIA staff to determine eligibility for training and then after that, the time a customer is denied access to WIA-funded training because of the study.

Response: Only about six percent of all study participants will be placed in one of the two restrictive-service groups (that is, the core- or intensive-services groups). All study participants will be allowed to receive core services or other services available in the community. This percentage limits services to the smallest number of customers while still meeting requirements for a valid, rigorous evaluation of the impact of intensive services and training. Random assignment of customers will be determined right after they have been determined eligible for intensive services, and the time it takes for customers to be determined eligible for training will take place well after they know the results of random assignment. Customers randomly assigned to one of the restrictive service groups will be eligible to reapply for all WIA intensive services and training 15 months after random assignment into the study.

Comment: The Department received three comments that questioned the timing of the evaluation at a time of high unemployment. These commenters had two concerns: (1) that it is wrong to deny customers services in time of great need, and (2) that services are less effective when there is high unemployment.

Response: While demand for intensive services and training increases as unemployment increases, there has not been an increase in funding for these services. Hence, because of funding limitations, not all customers who apply for these services can access them even in the absence of the study. During the study enrollment period, random assignment is also a fair way to allocate the resources that are available. The LWIAs participating in the study will be able to serve the same number of people who were able to access services and training in the absence of the study. Rather than the One-Stop allocating resources on a first-come, first-served basis, random assignment will give everyone who enrolls during the entire study enrollment period an equal chance of receiving intensive services and training.

Whether intensive services and training are more or less effective when unemployment is high is unknown. Arguments can be made that training will be more effective in these periods because it provides workers with skills for which there is demand, or connections with employers that the average person may not have; arguments can also be made that training is ineffective in periods of high unemployment because there are no jobs in the areas in which customers are trained. However, a study of the impacts of training in Europe finds that the magnitude of the impacts is higher in periods of high unemployment (Lechner and Wunsch, 2006, IZA discussion paper number 2355). The Department believes that the public workforce investment system must prove its worth under all economic conditions, including during times of economic challenges and high unemployment, since Federal funding for these programs is not—currently—predicated on the country's or area's employment situation.

Comment: Two comments mentioned the additional work the study will require of local staff. One commenter suggested that participating sites should receive monetary compensation for participating in the study.

Response: We recognize that the study requires additional work of local staff in the selected LWIAs and, therefore, participating LWIAs are receiving compensation for extra costs incurred due to the study. In addition, evaluation staff will work with staff in the selected LWIAs to minimize the effect that study procedures may have on each area's ongoing procedures.

Comment: One commenter suggested that there should be adjustments for performance measures for participating sites.

Response: The restricted service groups are so small that the evaluation is unlikely to affect performance measures for participating sites.

The Department appreciates the comments received in response to the request for public comment. All the comments gave useful information and provided suggestions which we had already incorporated into the study's design. The responses provided above outline the specific aspects of the evaluation methodology that address each comment.

Conclusion: Accordingly, the Department has determined that it is in the public interest to use a random assignment methodology for the study since this methodology will provide the most reliable estimates of the net impacts of WIA intensive services and training.

Signed: at Washington, DC, this 25th day of January 2012.

Jane Oates,

Assistant Secretary for Employment and Training.