2015-04097. Notice of Intent To Seek Approval To Establish an Information Collection

AGENCY:

National Science Foundation.

ACTION:

Notice and request for comments.

SUMMARY:

Under the Paperwork Reduction Act of 1995, Public Law 104-13 (44 U.S.C. 3501 et seq.), and as part of its continuing effort to reduce paperwork and respondent burden, the National Science Foundation (NSF) is inviting the general public and other Federal agencies to comment on this proposed information collection.

Comments: Comments are invited on: (a) Whether the proposed collection of information is necessary for the proper performance of the functions of the Foundation, including whether the information will have practical utility; (b) the accuracy of the Foundation's estimate of the burden of the proposed collection of information; (c) ways to enhance the quality, utility, and clarity of the information to be collected; and (d) ways to minimize the burden of the collection of information on those who are to respond, including through the use of automated collection techniques or other forms of information technology.

DATES:

Written comments on this notice must be received by April 28, 2015, to be assured consideration. Comments received after that date will be considered to the extent practicable. Send comments to address below.

FOR FURTHER INFORMATION CONTACT:

Ms. Suzanne H. Plimpton, Reports Clearance Officer, National Science Foundation, 4201 Wilson Boulevard, Suite 1265, Arlington, Virginia 22230; telephone [phone]; or send email to [e-mail]. Individuals who use a telecommunications device for the deaf (TDD) may call the Federal Information Relay Service (FIRS) at 1-800-877-8339, which is accessible 24 hours a day, 7 days a week, 365 days a year (including federal holidays).

SUPPLEMENTARY INFORMATION:

Title Of Collection: Generic Clearance of Survey Improvement Projects From the National Science Foundation.

OMB Number: 3145—NEW.

Expiration Date of Approval: Not applicable.

Type of Request: Intent to seek approval to establish a generic clearance for survey improvement projects for the National Science Foundation.

Abstract: The National Science Foundation (NSF) requests that the Office of Management and Budget (OMB) grant a generic clearance that will allow NSF to rigorously develop, test, and evaluate its survey instruments and methodologies. NSF has a mandate to “provide a central clearinghouse for the collection, interpretation, and analysis of data on scientific and engineering resources and to provide a source of information for policy formulation by other agencies of the Federal Government.” This request is part of an ongoing initiative to improve NSF surveys as recommended by both its own guidelines and those of OMB.^[1]

In the last decade, state-of-the art data collection and analysis methods have been increasingly instituted by NSF and other federal agencies, and are now routinely used to improve the quality and timeliness of data and analyses. These new methods or techniques many times help reduce respondents' cognitive workload and burden. The purpose of this generic clearance is to allow NSF to continue to adopt and use these methods or techniques to improve its current data collections on science, engineering, and technology inputs, outputs and outcomes. They will be used to improve the content of existing surveys, to aid in the development of new data collections to capture changes in the U.S. science and engineering (S&E) enterprise, and to fill gaps in coverage of the S&E enterprise in the existing NSF portfolio.

Following standard OMB requirements, NSF will submit to OMB an individual request for each survey improvement project it undertakes under this generic clearance. NSF will request OMB approval in advance and provide OMB with a copy of the questionnaire (if one is used) and materials describing the project.

NSF envisions using a variety of survey improvement techniques, as appropriate to the individual projects, such as focus groups, cognitive and usability laboratory and field techniques, exploratory interviews, behavior coding, respondent debriefing, pilot studies, pretests and split-panel tests. NSF has used such techniques in previous activities conducted under generic clearances granted to individual divisions.

a. Focus Groups. A qualitative methodology that brings together a small number of relatively homogenous subjects to discuss pre-identified topics. A protocol containing questions or topics focused on a particular issue or issues is used to guide these sessions, and is administered by a trained facilitator. Focus groups are useful for exploring and identifying issues with either respondents or stakeholders. Focus groups are a good choice during the development of a survey or survey topic, when a pre-existing questionnaire or survey questions on the topic do not yet exist. NSF has used focus groups for several projects under the Science Resources Statistics generic clearance (OMB Control Number 3145-0174) to assist with redesign of surveys when it became evident that the content of a survey was outdated and did not reflect current issues or the context that respondents were facing.

b. Cognitive and Usability Laboratory and Field Techniques. A qualitative methodology that refers to a set of tools employed to study and identify errors that are introduced during the survey process. These techniques are generally conducted by a researcher with an individual respondent, though observers may sometimes be present. Cognitive techniques are generally used to understand the question-response process, whereas usability is generally used to understand respondent reactions to the features of an electronic survey instrument, for instance, its display and navigation. In concurrent interviews, respondents are asked to think aloud as they actually answer the survey. In retrospective interviews, respondents answer the survey as they would normally, then `think aloud' afterwards. Other techniques, which are described in the literature and which will be employed as appropriate include: Follow-up probing, memory cue tasks, paraphrasing, confidence rating, response latency measurements, free and dimensional sort classification tasks, and vignette classifications. The objective of all of these techniques is to aid in the development of surveys that work with respondents' thought processes, thus reducing response error and burden. These techniques are generally very useful for studying and revising a pre-existing questionnaire. NSF has used cognitive and usability testing in previous generic clearance projects (OMB Control Numbers 3145-0157 and 3145-0174) to improve existing survey items, to develop and refine new content on existing surveys, and to explore content for new surveys.

c. Exploratory Interviews. A technique where interviews are conducted with individuals to gather information about a topical area. These may be used in the very early stages of developing a new survey. They may cover discussions related to administrative records, subject matter, definitions, etc. Exploratory interviews may also be used to investigate whether there are sufficient issues related to an existing data collection to consider a redesign. NSF has used such interviews extensively in recordkeeping studies with respondents to several of its establishment surveys to determine both what types of records institutions keep (and therefore what types of information they can supply), as well as where and in what format such records are kept.

d. Respondent Debriefing. A technique in which individuals are queried about how they have responded to a particular survey, question, or series of questions. The purpose of the debriefing is to determine if the original survey questions are understood as intended, to learn about respondents' form filling behavior and recordkeeping systems, or to elicit respondents' satisfaction with the survey. This information can then be used (especially if it is triangulated with other information) to improve the survey. This technique can be used as a qualitative or quantitative measurement, depending on how it is administered. This technique has been employed in NSF generic clearance projects (OMB Control Number 3145-0174) to identify potential problems with existing survey items both quantitatively (response behavior study, or RBS, using web survey questions with respondents to the Survey of Graduate Students and Post-doctorates in Science and Engineering, or GSS) and qualitatively (interviews using semi-structured protocols with Higher Education R&D Survey respondents).

e. Pilot Studies/Pretests. These methodologies are used to test a preliminary version of the data collection instrument, as was done with the Early Career Doctorate Project. Pretests are used to gather data to refine questionnaire items and scales and assess reliability, validity, or other survey measurement issues. Pilot studies are also used to test aspects of implementation procedures. The sample may be purposive in nature, or limited to particular groups for whom the information is most needed. Alternatively, small samples can be selected to statistically represent at least some aspect of the survey population.

f. Split Panel Tests. A technique for controlled experimental testing of alternatives. Thus, they allow one to choose from among competing questions, questionnaires, definitions, error messages, surveys, or survey improvement methodologies with greater confidence than other methods alone. Split panel tests conducted during the actual fielding of the survey are superior in that they support both internal validity (controlled comparisons of variables under investigation) and external validity (represent the population under study). Nearly any of the previously mentioned survey improvement methods can be strengthened when teamed with this method.

g. Behavior Coding. A quantitative technique in which a standard set of codes is systematically applied to respondent/interviewer interactions in interviewer-administered surveys or respondent/questionnaire interactions in self-administered surveys. Though this technique can quantifiably identify problems with the wording of questions, it does not necessarily illuminate the underlying causes.

Use of the Information: The information obtained from these efforts will be used to develop new NSF surveys and improve current ones. These surveys will generally be used to monitor outputs and outcomes of NSF funding over time (particularly data that is not being collected in annual and final reports), and manage and improve programs. Data collected through survey questionnaires can be used in program evaluation studies and can be matched to administrative data to understand NSF's portfolio of investments. Specifically, the information from the survey questionnaire improvement projects will be used to reduce respondent burden and to improve the quality of the data collected in these surveys. These objectives are met when respondents are presented with plain, coherent, and unambiguous questionnaires asking for data compatible with respondents' memory and/or current reporting and recordkeeping practices. The purpose of the survey improvement projects will be to ensure that NSF surveys are continuously attempting to meet these standards of excellence. Improved NSF surveys will help policy makers make decisions on R&D funding, graduate education, scientific and technical workforce, innovation, as well as contribute to increased agency efficiency and reduced survey costs. In addition, methodological findings have broader implications for survey research and may be presented in technical papers at conferences or published in the proceedings of conferences or in journals.

Estimate of Burden: NSF estimates that a total reporting burden of 171,000 hours over the three years of the requested generic clearance is possible from working to evaluate/improve existing surveys and to develop new ones. This includes both the burden placed on respondents participating in each activity as well as burden imposed on potential respondents during screening activities. Table 1 provides a list of potential improvement projects for which generic clearance activities might be conducted, along with estimates of the number of respondents and burden hours that might be involved in each.

Table 1—Potential Improvement Projects

Improvement project type	Number of respondents 2	Hours
Cognitive Testing	5,000	15,000
Focus Groups	5,000	10,000
Card Sorting	5,000	5,000
Interviews	5,000	5,000
Panelist Survey	7,000	12,000
Past Awardee Survey	9,000	14,000
Usability Testing	5,000	10,000
Additional surveys not specified	35,000	100,000
Total	76,000	171,000

Respondents: The respondents are PIs, program coordinators, or participants in NSF activities.

Estimates of Annualized Cost to Respondents for the Hour Burdens

The cost to respondents generated by the list of potential projects is estimated to be $3,205,680 over the three years of the clearance. No one year's cost would exceed $3,205,680. In other words, if all work were done in one year, costs in that one year would be $3,205,680 and the costs in each of the other 2 years would be zero. As in previous requests for generic clearance authority, the total cost was estimated by summing all the hours that might be used on all projects over the three years (76,000) and multiplying that figure by the hourly wage ($42.18) of the level of employee who typically answers NSF questionnaires or attends NSF workshops. This wage amount is the May 2011 national cross-industry estimate of the mean hourly wage for a financial analyst, or Job Category 13-2051, by the Bureau of Statistics. http://www.bls.gov/​oes/​#data. The total hours are based on similar NSF projects over the past few years.

There are no capital, startup, operation or maintenance costs to the respondents. The costs generated by future data collections will be described in the clearance request for each specific data collection. NSF does not anticipate any capital, startup, operation, or maintenance costs for future surveys.

Dated: February 23, 2015.

Suzanne H. Plimpton,

Reports Clearance Officer, National Science Foundation.

Footnotes

1.  NSF Information Quality Guidelines are available on http://www.nsf.gov/​policies/​infoqual.jsp. OMB Information Quality Guidelines are available on http://www.whitehouse.gov/​omb/​inforeg/​infopoltech.html. OMB standards and guidelines for statistical surveys are available on http://www.whitehouse.gov/​omb/​inforeg/​statpolicy/​standards_​stat_​surveys.pdf.

2.  Number of respondents listed for any individual survey may represent several methodological improvement projects.