2010-31881. Assessment Technology Standards Request for Information (RFI)

AGENCY:

Office of Innovation and Improvement, U.S. Department of Education.

ACTION:

Notice of request for information to gather technical expertise pertaining to assessment technology standards.

SUMMARY:

The purpose of this RFI is to collect information relating to assessment technology standards. Toward that end, we are posing a series of questions to which we invite interested members of the public to respond. The Department anticipates making use of this information in the following ways. First of all, we expect to use this information to help determine the appropriate interoperability standards for assessments and related work developed under the Race to the Top Assessment (RTTA) program. Secondly, we expect to use this information to help us develop related standards-based programs. For example, we might, in the future, offer additional grants, contracts, or awards and some of those offerings may include similar interoperability requirements. This RFI may be used to help set the interoperability requirements for those offerings as well as the existing RTTA program.

Under the RTTA program, the Department requires grantees to develop assessments that (see http://www2.ed.gov/​programs/​racetothetop-assessment/​executive-summary.pdf,, p. 78):

“5. Maximize the interoperability of assessments across technology platforms and the ability for States to switch their assessments from one technology platform to another by—

(a) Developing all assessment items to an industry-recognized open-licensed interoperability standard that is approved by the Department during the grant period, without non-standard extensions or additions; and

(b) Producing all student-level data in a manner consistent with an industry-recognized open-licensed interoperability standard that is approved by the Department during the grant period.”

DATES:

Written submissions must be received by the Department on or before 5 p.m., Washington, DC time, on January 17, 2011.

ADDRESSES:

We encourage submissions by e-mail using the following address: [e-mail]. You must include the term “Assessment RFI response” in the subject line of your e-mail. If you prefer to send your input by mail or hand delivery, address it to Steve Midgley, Office of Educational Technology, Attention: Assessment RFI, U.S. Department of Education, 400 Maryland Avenue, SW., Room 7E202, Washington, DC 20202-0001.

FOR FURTHER INFORMATION CONTACT:

Steve Midgley, U.S. Department of Education, 400 Maryland Avenue, SW., Room 7E202, Washington, DC 20202-0001 by phone at 202-453-6381 or e-mail at [e-mail].

If you use a telecommunications device for the deaf (TDD), call the Federal Relay Service (FRS), toll free, at 1-800-877-8339.

SUPPLEMENTARY INFORMATION:

1. Introduction

The Department is seeking information on technology standards that may be applied to the management and delivery of education-related assessments, as well as those that may be applied to the capture and reporting of assessment results within distributed online learning environments (i.e. learning environments with components managed by more than one organization). THIS IS A REQUEST FOR INFORMATION (RFI) ONLY. This document uses the term “technology standards” to refer to assessment technology standards, specifications, technical approaches and implementations, and any other functional or formal descriptions of technical functionality. (Note: This document refers to curricular or content standards specifically as “curricular standards.”) Information about non-assessment technology standards and related issues may be relevant and included in responses, but this RFI is specifically inquiring into technology standards related to assessments of learning. For the purpose of this RFI, the Department does not distinguish between technology specifications and technology standards produced by consortia, other groups, or nationally or internationally recognized technology standards development organizations.

This RFI is issued solely for information and planning purposes and does not constitute a Request for Proposals (RFP) or a promise to issue an RFP or notice inviting applications (NIA). This request for information does not commit the Department to contract for any supply or service whatsoever. Further, the Department is not at this time seeking proposals and will not accept unsolicited proposals. Responders are advised that the Department will not pay for any information or administrative costs that a person or entity may incur in responding to this RFI. All costs associated with responding to this RFI will be solely at the interested party's expense. Not responding to this RFI will not preclude individuals or organizations from applying under future contract or grant competition. If the Department issues an RFP or NIA, it will be posted on the Federal Business Opportunities (https://www.fbo.gov/​) Web site (in the case of contracts) or the Federal Register (http://www.gpoaccess.gov/​fr/​) Web site (in the case of grants, or other awards). It is the responsibility of the potential offerors to monitor these sites to determine whether the Department issues an RFP or NIA after considering the information received in response to this RFI. Any company or industry proprietary information contained in responses should be clearly marked as such, by paragraph, such that publicly releasable information and proprietary information are clearly distinguished. Any clearly marked proprietary information received in response to this request will be properly protected from unauthorized disclosure. The Department will not use proprietary information submitted from any one source to establish the capability and requirements for any future acquisition or grant competition so as not to inadvertently restrict competition. The Department may publicly release or use any or all materials submitted which are not so marked.

The documents and information submitted in response to this RFI become the property of the U.S. Government and will not be returned.

2. Background

The Department is investigating open technology standards and specifications to support the interoperable delivery (that is, delivery in a way that allows effective use across multiple systems or components) of State- or locally selected content and assessments for purposes of education and training when conducted via online learning platforms. As a part of this effort, the Department is investigating the availability and current practice of open technology standards and innovative technologies to support management, delivery, and exchange of assessment content, and the capture and reporting of assessment results.

Existing technologies may serve as the basis for the creation of new open technology standards and specifications, if implementation details related to these technologies can be disclosed and provided without restriction for technical standardization or use. We expect that applicable open technology standards and specifications will be combined with other technology standards, current or to be developed, providing the assessment capabilities for online learning platforms that will support the next generation of technology for learning content. Therefore, this RFI seeks information on a range of solutions and approaches to standardization of assessment via technology, including deployment, collection and reporting solutions, techniques, and technology standards.

It is possible that RTTA grantees will be able to use one or more existing technology standards, or it may be that additional development work will be required to obtain sufficiently complete technology standards for the program. It is also possible that one or more existing technology standards are suitable but are not licensed in a way that will permit free and open use by the public. Through this RFI, the Department seeks to uncover and gather information on how to resolve as many of these issues as possible.

The Department may engage in additional work to address these issues at the conclusion of its analysis of the responses to this RFI.

There are numerous efforts underway across the Department that can benefit from assessment technology standardization of assessment content, results, and reporting interoperability. For example, the Department is providing significant funding for the development of “next- generation” assessment systems via the RTTA program (see http://edocket.access.gpo.gov/​2010/​pdf/​2010-8176.pdf;​; http://www2.ed.gov/​programs/​racetothetop-assessment/​index.html). In order to promote technological innovation and market competition, the Department has specified that all assessment content developed under this program be developed using an “industry recognized open-licensed interoperability standard” that is approved by the Department. The assessment content developed under the program must also be made freely available to any State, technology platform provider, or others that request it for purposes of administering assessments (consistent with test security and protection requirements). Moreover, the standards and technology for controlling sensitive data (assessment results and related information) must also maintain the privacy of any individually identifiable information while permitting secure interchange among authorized systems. The Department intends that these requirements, taken as a whole, give States the flexibility to switch from one technology platform to another, allowing multiple providers to compete for States' business and for States to make better decisions about cost and value. Use of technology standards that meet these requirements will help ensure that public investments in assessment instruments and related technology can be used in the education sector as broadly as possible and, at the same time, contribute to a competitive and innovative market place.

Through this notice, the Department solicits advice, technical information, additional questions (that is, questions in addition to those put forward later in this notice), and other input as to how the Department can select the best available technology standard(s) for the RTTA program, as well as general information related to assessment technology standards and technology and policy.

3. Context for Responses

3.1 The primary intent of this RFI is to explore existing, in-process, or planned open technology standards, specifications, and technology products that support the management, delivery, and exchange of assessment content and the capture and exchange of assessment results. While the focus of this RFI is assessment technology standards, the Department recognizes that assessment generally occurs within the context of broader learning activities (whether online or offline) and, therefore, does not wish to restrict the range of responses to assessment-only approaches. The Department, therefore, also welcomes responses that address broader technology standards or approaches that are relevant to the handling of assessment management, delivery, or reporting. As mentioned earlier, the Department has required RTTA grantees to adopt a technical standard (or standards) that permit interoperability of the assessments and technology developed by that program. To help focus our consideration of the comments provided in the response to this RFI, we have developed several questions regarding the development of assessment technology standard(s) and their application to the RTTA program. Because these questions are only a guide to help us better understand the issues related to the development of interoperable technology standards for assessments, respondents do not have to respond to any specific question. Commenters responding to this RFI may provide comments in a format that is convenient to them.

3.2 Questions About Assessment Technology Standards

General and Market Questions

3.2.1 Current Landscape. What are the dominant or significant assessment technology standards and platforms (including technologies and approaches for assessment management, delivery, reporting, or other assessment interoperability capabilities)? What is the approximate market penetration of the major, widely adopted solutions? To what degree is there significant regional, educational sub-sector, or international diversity or commonality regarding the adoption of various technology standards and capabilities, if any?

3.2.2 Timelines. Approximately how long would it take for technology standards setting and adoption processes to obtain a technology standard that meets many or all of the features or requirements described in this RFI? What are the significant factors that would affect the length of that timeline, and how can the impact of those factors be mitigated? More specifically, would the acquisition of existing intellectual property (IP), reduction or simplification of specific requirements, or other strategies reduce the time required to develop these technology standards and processes?

3.2.3 Process. What process or processes are appropriate for the adoption, modification, or design of the most effective technology standard in a manner that would answer many or all of the questions in this RFI? We are interested in learning the extent to which the uses of one or another process would affect the timeline required to develop the technology standards.

3.2.4 Intellectual Property. What are the potential benefits and costs to the Federal Government, States, and other end-users of different IP restrictions or permissions that could be applied to technology standards and specifications? Which types of licensed or open IP (e.g., all rights reserved, MIT Open License, or Gnu Public License) should be considered as a government technology standard? How should openness relating to the IP of technology standards be defined and categorized (e.g., Open Source Initiative-compatible license, free to use but not modify, non-commercial use only, or proprietary)

3.2.4.1 Existing Intellectual Property. What are the IP licenses and policies of existing assessment technology standards, specifications, and development and maintenance policies? Are the documents, processes, and procedures related to these IP licenses and policies publicly available, and how could the Department obtain them?

3.2.5 Customizing. Can assessment tools developed under existing technology standards be customized, adapted, or enhanced for the use of specific communities of learning without conflicting with the technology standard under which a particular assessment tool was developed? Which technology standards provide the greatest flexibility in permitting adaption or other enhancement to meet the needs of different educational communities? What specific provisions in existing technology standards would tend to limit flexibility to adapt or enhance assessment tools? How easy would it be to amend existing technology standards to offer more flexibility to adapt and enhance assessment tools to meet the needs of various communities? Do final technology standards publications include flexible IP rights that enable and permit such customizations? What are the risks and the benefits of permitting such customization within technology standards? When would it make sense to prevent or to enable customization?

3.2.6 Conformance and Testing. Do existing technology standards or technologies include specifications or testing procedures that can be used to verify that a new product, such as an assessment tool, meets the technology standards under which it was developed? What specifications or testing procedures exist for this purpose, e.g., software testing suites, detailed specification descriptions, or other verification methods? Are these verification procedures included in the costs of the technology standards, or provided on a free or fee-basis, or provided on some combination of bases?

3.2.7 Best Practices. What are best practices related to the design and use of assessment interoperability technology standards? Where have these best practices been adopted, and what are the general lessons learned from those adoptions? How might such best practices be effectively used in the future?

Technological Questions Regarding Assessment Technology Standards

3.2.8 Interoperable Assessment Instruments. What techniques, such as educational markup or assessment markup languages (see also http://en.wikipedia.org/​wiki/​Markup_​language), exist to describe, package, exchange, and deliver interoperable assessments? How do technology standards include assessments in packaged or structured formats? How can technology standards enable interoperable use with resources for learning content? How can technology standards permit assessment instruments and items to be exchanged between and used by different assessment technology systems?

3.2.9 Assessment Protection. For this RFI, “Assessment Protection” means keeping assessment instruments and items sufficiently controlled to ensure that their application yields valid results. (See also paragraph below, “Results Validity.”) When assessment instruments or content are re-used or shared across organizations or publicly, are there capabilities or strategies in the technology standards to assist in item or instrument protection? What mechanisms or processes exist to ensure that assessment results are accurate and free from tampering? Do examples exist of public or semi-public assessment repositories that can provide valid tests or assessments while still sharing assessment items broadly?

3.2.10 Security and Access. In what ways do technology standards provide for core security issues, such as access logging, encryption, access levels, and inter-system single-sign-on capabilities (i.e., one login for systems managed by different organizations)?

3.2.11 Results Validity. For this RFI, “Results Validity” means protecting the statistical validity and reliability of assessment instruments and items. How can interoperable instruments be managed to ensure they are administered in a way that ensures valid results? Are solutions regarding assurance or management of validity appropriate for inclusion in technology standards, or should they be addressed by the communities that would use the technology standards to develop specific assessments?

3.2.12 Results Capture. How can technology standards accurately link individual learners, their assessment results, the systems where they take their assessments, and the systems where they view their results? How do technology standards accurately make these linkages when assessments, content, and other data reside across numerous, distinct learning and curriculum management systems, sometimes maintained by different organizations?

3.2.13 Results Privacy. How do technology standards enable assessment results for individual learners to be kept private, especially as assessments results are transferred across numerous, distinct learning systems? How can such results best be shared securely over a distributed set of systems managed by independent organizations that are authorized to receive the data, while still maintaining privacy from unauthorized access?

3.2.14 Anonymization. Do technology standards or technologies permit or enable anonymization of assessment results for research or data exchange and reporting? How do various technology standards accomplish these tasks? For example, where a number of students take a test, can their answers be anonymized (through aggregation or other techniques) and shared with researchers to examine factors related to the assessment (e.g., instructional inputs, curriculum, materials, validity of the instrument itself) without revealing the identity of the learners? Is this an area where technology standards can help?

3.2.15 Scoring and Analysis of Results. How can technology standards be used for the scoring, capture, recording, analysis or evaluation of assessment results?

3.2.15.1 Results Aggregation and Reporting. How can technology standards enable assessment results to be aggregated into statistical or other groupings? How can technology standards provide capabilities for results (aggregated or raw) to be reported across multiple technology systems? For example, if a learner takes an assessment in one system, but the results are to be displayed in another, how do technology standards address transferring results across those systems? How do technology standards address aggregation of results for a number of learners who are assessed in one system and whose results are displayed in yet another technology system? Can anonymization controls be included with aggregation and reporting solutions to ensure individual data privacy and protection (see also 3.2.14 above).

3.2.16 Sequencing. How do technology standards enable assessment items stored within an assessment instrument to be sequenced for appropriate administration, when the assessment consists of more than a single linear sequence of items? For example, how do technology standards address computer-adaptive assessments? How are the logic rules that define such sequencing embedded within a technology standard?

3.2.17 Computer-Driven scoring. How do technology standards permit, enable, or limit the ability to integrate computer-driven scoring systems, in particular those using “artificial intelligence,” Bayesian analysis, or other techniques beyond traditional bubble-fill scoring?

3.2.18 Formative, Interim, and Summative Assessments. What technology and technology standards exist that support formative, interim, and summative assessments? What technology standards support non-traditional assessment methods, such as evidence, competency, and observation-based models?

3.2.19 Learning and Training. What applications or technology standards exist that can apply assessment results to support learning and training? Are there technology standards or applications that support more than one of the following: Early learning, elementary/secondary education, postsecondary education, job training, corporate training, and military training?

3.2.20 Repositories. What technology standards-based assessment instruments, questions, or item banks (or repositories and learning management systems) are used to manage and deliver assessments?

3.2.21 Content Lifecycle. How can technology standards be employed to support an assessment content lifecycle (creation, storage, edit, deletion, versioning, etc.)?

3.2.22 Interfaces and Services. What interoperability specifications for application program interfaces (APIs) or Web services interfaces to assessment management, delivery and tracking systems have been developed? How are they organized? What are the best practices related to their design and usage? How broadly have they been adopted, and what are the lessons learned from those who have designed or implemented them?

3.2.23 Internal Transparency and Ease of Use. Are there technology standards and communication protocol implementations that are “human readable?” What are the benefits and risks of “human readable” technology standards? Some technology standards are not comprehensible without tools to unpack, decode, or otherwise interpret the implementation data resulting from use of the technology standard. Other technology standards, such as HTML, RTF and XML, are largely readable by a reasonably sophisticated technical user. RESTful-designed Web services are often specifically intended to be readable by, and even intuitive to, such users as well. We ask commenters to consider the extent to which various technology standards possess native “human readability” and comprehensibility.

3.2.24 Discovery and Search. How is the discovery of items or instruments (or other elements) handled within a technology standard or technology? For example, are there search APIs that are provided to permit a search? How are metadata exposed for discovery by search engines or others?

3.2.25 Metadata. What kinds of metadata about assessments (i.e., information describing assessments) are permitted to be stored within technology standards or technologies? How do technology standards accommodate structured data (such as new State curriculum standards) that were not anticipated when the technology standard was designed? How are metadata describing unstructured (such as free-text input) and semi-structured data incorporated within assessment technology standards?

3.2.26 Recommendation, Rating, and Review. Do technology standards or technologies permit rating, review, or recommendations to be incorporated within an item, instrument, or other element? If so, in what ways? How are conflicting ratings handled? Do technology standards or technologies permit “reviews of reviews” (e.g., “thumbs up/down” or “Rate this review 1-5”)? Is the rating or review system centralized, or are multiple analyses of the rating data permitted by distributed participants?

3.2.27 Content and Media Diversity. What types of diverse content types and forms of assessment content exist that extend beyond traditional paper-based assessments translated to an electronic delivery medium? We are interested in learning more about electronic delivery and interaction media, such as performance-based assessments, games, virtual worlds, mobile devices, and simulations.

3.2.28 Accessibility. How do technology standards ensure that the platforms are accessible to all persons with disabilities? How can technology standards ensure the availability of accommodations based on the individual needs of persons with disabilities? What factors are important to consider so that accessibility capabilities can be included within an interoperable technology standard, both for end-users, as well as operators, teachers, and other administrators? How are issues related to Universal Design for Learning (UDL) relevant to standards for accessible use? How can technology standards provide for, improve, or enhance Section 504 and 508 of the Rehabilitation Act compliance for assessment technology?

3.2.29 English Learners. How do technology standards ensure that assessment platforms support the assessment, reporting of results, and other capabilities related to the assessment of English learners?

Questions about process and IP for technology standards development include:

3.2.30 Transparency. How do the organizations that develop assessment technology standards approach development and maintenance activities? Is it common for such work to be performed in an unrestricted or open public forum? Are there examples of organizations conducting technology standards development through private (e.g., membership-driven) activities? Are the final work products produced through standards-development activities made publicly available in a timely manner? If not, when or for how long is it necessary to keep these products private? What circumstances require, justify, or benefit from protecting trade secrets or intellectual property?

3.2.31 Participation. Does the development of assessment technology standards depend on membership fees from individuals and organizations who wish to contribute to development and maintenance activities? Are there requirements for “balance” within membership across different constituencies? What are the cost and structure of such memberships? Are there viable alternative methods for generating revenue necessary to conduct the work? What are the most realistic and useful ways to generate participation, fund work, and ensure public access to a technology standards-setting process?

3.2.32 Availability. What are the costs associated with final publication of technology standards, and with all supporting materials for those standards, and can these assessment products be made available at nominal or no cost to users? Do technology standards require restrictions for use or application, including limitations on derivation, resale, or other restrictions? Is it appropriate to obtain patent, copyright, or trademark protections for assessment technology standards? Are the publications for technology standards and materials provided in a machine-readable, well-defined form? Are there restrictions or limitations on any future application of the publications and materials after initial release? Are developer-assistance materials (e.g., Document Type Definitions, test harnesses, code libraries, reference implementations) also made available free under an open-license? In what circumstances should technology standards-setting organizations retain rights or control, or impose restrictions on the use of publications, derivations, and resale or developer-assistance technologies, as opposed to open-licensing everything? When should materials be made freely available (that is, at no cost to the consumer) while still retaining most or all copyright license rights?

3.2.33 Derivation. For technology standards, do copyright licenses for publications and all supporting materials and software licenses for software artifacts permit the unrestricted creation and dissemination of derivative works (a.k.a. “open licensed”)? Do such open licenses contain restrictions that require publication and dissemination of such works in a manner consistent with the openness criteria described by, for example, a GNU Public License (a.k.a. “viral licensed”) or an MIT Public License (a.k.a. “academic licensed”)? Are there policies or license restrictions on derivative works intended to prevent re-packaging, re-sale, or modifications without re-publication for assessment technology standards?

3.2.34 Licensing Descriptions (for materials contained within the standard, not for the standard's licensing itself). How do technology standards address licensing terms for assessment resources described within the technology standard? Are there successful technology standards or approaches for describing a wide variety of license types, including traditional per-use licensing, Web-fulfillment, free (but licensed), open (but licensed, including commercial or non-commercial use permitted), and public domain status. Are there other resource licensing issues that should be addressed within a technology standard as a best practice?

Accessible Format: Individuals with disabilities can obtain this document in an accessible format (e.g., Braille, large print, or audiotape) on request to the program contact person listed under FOR FURTHER INFORMATION CONTACT.

Electronic Access to This Document: You can view this document, as well as all other documents of this Department published in the Federal Register, in text or Adobe Portable Document Format (PDF) on the Internet at the following site: http://www.ed.gov/​news/​fedregister.

To use PDF you must have Adobe Acrobat Reader, which is available free at this site. If you have questions about using PDF, call the U.S. Government Printing Office (GPO), toll free, at 1-888-293-6498; or in the Washington, DC, area at [phone].

Note:

The official version of this document is the document published in the Federal Register. Free Internet access to the official edition of the Federal Register and the Code of Federal Regulations is available on GPO Access at: http://www.gpoaccess.gov/​nara/​index.html.

Program Authority: 20 U.S.C. 6771.

Dated: December 15, 2010.

James Shelton, III,

Assistant Deputy Secretary for Innovation and Improvement.