2014-00404. Notice of Request for Information (RFI)

SUMMARY:

The Office of Science and Technology Policy requests public comments to inform its policy development related to high-impact learning technologies. This Request for Information offers the opportunity for interested individuals and organizations to identify public and private actions that have the potential to accelerate the development, rigorous evaluation, and widespread adoption of high-impact learning technologies. The focus of this RFI is on the design and implementation of “pull mechanisms” for technologies that significantly improve a given learning outcome. Comments must be received by 11:59 p.m. on March 7, 2014, to be considered. In your comments, please reference the question to which you are responding.

DATES:

Comments must be received by 11:59 p.m. on March 7, 2014, to be considered.

ADDRESSES:

Respondents are encouraged to submit their comments through one of the following methods. Email is the preferred method of submission. Please do not include in your comments information of a confidential nature, such as sensitive personal information or proprietary information. Responses to this notice are not offers and cannot be accepted by the Federal Government to form a binding contract or issue a grant. Information obtained as a result of this notice may be used by the Federal Government for program planning on a non-attribution basis. Please be aware that your comments may be posted online.

• Email: [e-mail]. Email submissions will receive an electronic confirmation acknowledging receipt of your response, but will not receive individualized feedback on any suggestions.
• Postal Mail: Office of Science and Technology Policy, Attn: Cristin Dorgelo, 1650 Pennsylvania Avenue NW., Washington, DC 20504. Submissions by postal mail must be received by the deadline, and should allow sufficient time for security processing.
• Fax: [phone]

SUPPLEMENTARY INFORMATION:

This Request for Information (RFI) offers the oppm1unity for interested individuals and organizations to identify public and private actions that have the potential to accelerate the development, rigorous evaluation, and widespread adoption of high-impact learning technologies. The focus of this RFI is on the design and implementation of “pull mechanisms” for technologies that significantly improve a given learning outcome. Pull mechanisms increase the incentives to develop specific products or services by committing to reward success. Examples of pull mechanisms include incentive prizes, Advance Market Commitments, milestone payments, “pay for success” bonds, and purchasing consm1ia. The public input provided through this notice will inform the deliberations of the Office of Science and Technology Policy (OSTP).

Background

OSTP is interested in identifying policies and serving as a catalyst for public-private pat1nerships that have the potential to accelerate the development, rigorous evaluation, and widespread adoption of high-impact learning technologies. For example, imagine if learners in the United States had access to technologies that:

• Dramatically reduced the large and persistent gap in vocabulary size between children from wealthy and poor households.
• Allowed middle and high school students to outperform their international peers in math and science.
• Enabled English-language learners that are reading at several grade levels below average to catch up after only a year.
• Gave non-college bound students an industry skills ce1tification or set of cognitive skills (e.g. literacy, numeracy, ability to understand and apply chmis, graphs and diagrams) that are a ticket to a middle-class job, increasing their employability and their incomes by $10,000-$20,000 or more in less than a year.
• Doubled the percentage of community college students that pass remedial math, which is currently only 30 percent.
• Successfully delivered a “growth mindset” intervention to teachers and students.
• Were as effective as a personal tutor, were as engaging as the best video game, and improved the more students used them.

Currently, there is a large gap between the relatively modest impact that technology has had on education, particularly in K-12, and the transformative impact that it has had in many aspects of our economic and social life. For example, businesses are using information and communications technologies to dramatically increase productivity, tap the expe1iise of their employees, slash the time needed to develop new products, tailor products and services to meet the needs of individual consumers, orchestrate global networks of suppliers, derive insights from huge volumes of transactional data, and improve their products and services by conducting rapid, low-cost experiments.

Education, particularly K-12 education, remains relatively untouched by advances in our understanding of how people learn, how to design instruction that incorporates those insights, and the explosion in information technologies such as low-cost smartphones and tablets, cloud computing, broadband networks, speech recognition and speech synthesis, predictive analytics, data mining, machine learning, intelligent tutors, simulations, games, computer-suppmied collaborative work, and many other technologies. That is why President Obama has proposed ConnectED, a new initiative to connect 99 percent of America's students to the Internet through high-speed broadband and high-speed wireless within 5 years.

Learning technologies will be much more effective if they informed by “learning science”—advances in disciplines in fields such as neuroscience, cognitive science, educational psychology, and discipline-based education research that shed light on how people learn. This research can provide actionable insights on issues such as student motivation, the circumstances under which prior knowledge helps or hmis learning, how students can organize knowledge in rich and meaningful ways, and the ways in which students can progress from novice to expeti in a given domain.

There are a number of reasons for the gap between the potential of learning science and technology and the cunent state-of-the-practice:

• The United States is investing 0.1 percent of K-12 expenditures on R&D, compared to 2 percent in mature industries and 18.7 percent in the pharmaceutical industry. This extremely low level of investment in educational R&D has clearly limited the pace of innovation.
• Entrepreneurs seeking to develop and market new products to the K-12 market face a number of challenges, including low per-pupil expenditures on software, lengthy adoption cycles, and a highly fragmented market. This in turn limits the amount that companies can spend on research and product development.
• It is difficult for companies to make authoritative claims about the impact of their products on learning outcomes assessed through rigorous third-party validation, which limits the premium that school districts and other consumers of learning technology are willing to pay for high-quality, effective products.

This suggests that an effective national strategy for increasing the impact of learning science and technology should address both the “supply” and “demand” for advanced learning technologies.

To increase the “supply” of learning technology, the Federal government and philanthropists could increase funding for research and development and support training grants and scholarships in relevant disciplines such as educational psychology, cognitive science, instructional design, artificial intelligence, etc. The National Science Foundation is funding a program called “Cyberlearning Transforming Education” and the Depmiment of Defense is supporting research in advanced training technologies. The President FY14 Budget request includes funding for a “DARPA for Education” (ARPA-ED).

The Power

However, there has been little discussion of the potential of what economists call “pull mechanisms” to accelerate the development, evaluation, and adoption of high-impact learning technologies.

As economists have recently noted, governments and other funders can suppmt innovation using “push” programs (e.g. funding grants and contracts to universities and companies, providing tax incentives for R&D, or supporting government laboratories) and “pull” mechanisms that “increase the rewards for developing specific products by committing to reward success.” Push programs pay for research inputs; pull mechanisms pay for research outcomes.

“Pull mechanisms” have been used successfully in the field of global health. In December 2010, children in developing countries began receiving a vaccine that will prevent deaths from “pneumococcal” diseases including pneumonia, meningitis, and sepsis. Nearly one million young children die every year from pneumococcal infections, with 90 percent of these deaths occurring in developing countries.

The development of this vaccine was accelerated by a $1.5 billion “Advance Market Commitment” backed by five governments and a private foundation. Pharmaceutical companies that have agreed to provide the vaccine at $3.50 per dose to low-income countries for the next 10 years will receive additional payments from the $1.5 billion in donor commitments. The AMC increased the size and predictability of the market for pneumococcal vaccines, which increased the willingness of companies to invest in high-volume production of these vaccines for developing country markets. Expe1ts predict that this AMC will save 7 million lives over the next twenty years.

Non-binding commitments to purchase products can also provide market pull, if there is both a clearly defined performance specification and a strong expression of interest from potential buyers. For example, in June 2013, the U.S. Department of Energy put together a coalition of the Federal government and over 200 major commercial building pmtners that issued a challenge to U.S. manufacturers: “If you can build wireless sub-meters that cost less than $100 apiece and enable us to identify opportunities to save money by saving energy, we will buy them.” At least 18 manufacturers agreed to take up the challenge. In 2011, the Department of Energy put together a similar and successful challenge for energy-efficient and cost-effective commercial air conditioners, with the first manufacturer meeting the challenge in May 2012.

In addition, Federal agencies have offered almost 300 incentive prizes on Challenge.gov, providing opportunities for citizen solvers to offer novel solutions to tough problems, while minimizing risk to Federal agencies by only paying for success. More information about pull mechanisms can be found in this supplemental information document.

OSTP is interested in stimulating a conversation about how pull mechanisms could be used to accelerate the development, evaluation, and adoption of learning technologies. Some of the advantages of pull mechanisms are that a funder can (a) pay only for success; (b) set a goal without having to choose in advance which team or approach is most likely to be successful; and (c) increase the number and intellectual diversity of the teams that are working to solve a particular problem. Although there a variety of different types of pull mechanisms, they generally require establishing a clear goal and an agreed-upon set of metrics for evaluating progress towards that goal. If education is going to benefit from increased use of pull mechanisms, policy-makers and stakeholders have to identify some specific challenges that are important and measurable, and where it is plausible that learning technology can help improve student outcomes.

Using Pull Mechanisms for Learning Technologies

Pull mechanisms can be used for social interventions that do not use technology. For example, the first “social impact bond” is being used by the United Kingdom to reduce recidivism among 3,000 prisoners. The United Kingdom's Depa11ment for International Development (DfiD) is supporting a “Results-Based Aid” approach to improving education in Ethiopia. Under this pilot, DfiD will make grant payments to the education ministry for the increase in the number of students above a baseline that sits for or passes the national grade 10 exam. There will be additional payments for students in the poorest regions, and for girls compared to boys.

It may also make sense to experiment with pull mechanisms to accelerate the development and rigorous evaluation of learning technologies. Some of the potential advantages of learning technologies include:

• Low marginal cost: The marginal cost of making software or digital content and services available to more students is very low, although the fixed cost of R&D and rigorous evaluation may be high. This is why IT stmtups are able to grow rapidly—the cost of serving tens or hundreds of millions of customers does not increase arithmetically with the number of customers.
• Ability to maintain high levels of “time on task”: For example, good game developers can keep users riveted for hours at a time. They can create experiences that are intrinsically motivating, and that offer an increasingly difficult set of challenges that keep users in the “sweet spot” between being bored and frustrated.
• Continuous improvement: The productivity of most public sector services is flat or negative. Researchers and entrepreneurs have ideas for developing online services that get better the more people use them by (a) conducting many low-cost experiments to discover what works; and (b) collect, analyze and act on the data that can be generated online.
• Learning anytime, anywhere: Mobile devices allow individuals to access digital content at a time, place, and pace that is convenient for them. This might be particularly impmiant for an adult who is trying to upgrade their skills while balancing the competing demands of work and family.
• Digital tutors: Research suggests that the average student tutored one-on-one using “mastery learning” techniques (students are helped to master each concept before proceeding to a more advanced learning task) performed better than 98 percent of the students that learn the same material using conventional instructional methods. Projects funded by DARPA and the Office of Naval Research suggest that it may be possible to develop “digital tutors” that model the one-on-one interaction between a world-class subject matter expeti and a student. A pilot suppmied by the Veteran's Administration is allowing unemployed veterans that use the digital tutor for 6 months to get IT jobs that pay $40,000 to $80,000.
• Personalization: Researchers and firms are developing software and online services that are personalized to the needs, background, interests and skill levels of individuals.
• Interactive simulations that enable “learning by doing”: Researchers have developed simulations in areas such as physics, chemistry, biology, emih science, and math. For example, an “Energy Skate Park” simulation allows students to explore energy conservation with multiple different variables (shape of the track, starting height and speed of the skater, mass of the skater, and friction). Students can quicldy repeat experiments and rapidly explore the effect of many different parameters.
• Embedded assessment: Technology can help provide continuous assessment of a given set of knowledge, skills and abilities if the designers know (a) what behaviors would constitute evidence that a student has mastered a given competency; and (b) which tasks can elicit those behaviors.

Questions

To stimulate a national conversation on whether and how pull mechanisms might be used to accelerate the development of high-impact learning technologies, OSTP seeks public comment on the questions listed below:

(1) What learning outcomes would be good candidates for the focus of a pull mechanism to catalyze the creation and use of new learning technology? These outcomes could be relevant to early childhood education, K-20, life-long learning, workforce readiness and skills, etc.

(2) How are these learning outcomes currently measured and assessed?

(3) What information exists about current U.S. performance relative to this learning outcome? What information exists about the presence (currently available or potential given current trends or breakthroughs) or absence of effective interventions (technology-based, offline, or hybrid) to improve this learning outcome?

(4) Why would a pull mechanism in this area accelerate innovation in learning technology?

(5) What role might different stakeholders (e.g. Federal agencies, state and local educational agencies, foundations, researchers, practitioners, companies, investors, or non-profit organizations) play in designing, funding, and implementing a pull mechanism for learning technology? What role would your organization be willing to play?

(6) What changes in public policy would facilitate experimentation with pull mechanisms at different levels of government?

Response to this RFI is voluntary. Responders are free to address any or all the above items, as well as provide additional information that they think is relevant to accelerating the development, rigorous evaluation and widespread adoption of high-impact learning technologies. Please note that the U.S. Government will not pay for response preparation or for the use of any information contained in the response.

Ted Waelder,

Deputy Chief of Staff and Assistant Director.

Supplementary Information: Overview of Pull Mechanisms

Incentive prizes are one type of “pull mechanism”—results-based market incentives designed to overcome market failures and catalyze itmovation. Experts often make a distinction between “recognition” prizes that honor past achievements and “inducement” or “incentive” prizes that encourage participants in the competition to achieve a particular goal. In a 2009 repot1, McKinsey identified six prize archetypes that provide a useful framework for identifying types of prizes that can best achieve different types of goals:

• Exemplar Prizes that define excellence within an area.
• Point Solution Prizes that aim to spur development of solutions for a pmiicular well-defined problem. Solutions can include software applications, algorithms, predictive models, ideas, business plans, policy proposals, designs, or prototypes.
• Market Stimulation Prizes that try to establish the viability of a market to address a potential market failure, mobilize additional human talent and financial capital to jumpstati the development of a new industry, or change public perceptions about what is possible.
• Exposition Prizes that are designed to highlight a broad range of promising ideas and practices, attract attention, and mobilize capital to further develop the winning innovations.
• Participation Prizes that create value during and after the competition- not through conferral of the prize award itself but through their role in encouraging contestants to change their behavior or develop new skills that may have beneficial effects during and beyond the competition.
• Network Prizes that build networks and strengthen communities by organizing winners into new problem-solving communities that can deliver more impact than individual effmis.

Other types of pull mechanisms include:

• Advance Market Commitments: Binding commitments to purchase, or to subsidize purchase, of a ce1iain volume of a product at a fixed prize, if the product meets pre defined performance characteristics (pneumococcal vaccine and Department of Energy examples discussed above).
• Buyer's Consortia: Cooperative agreements between purchasers of products that leverage the combined buying power of those purchasers to drive down the price of products, such as a buyer's consmiium set up for Maine school districts to purchase specialized software and specific assistive technology devices.
• Pay-for-Success Bonds: Under a Pay for Success bond, also known as a social impact bond, the financing organization and the Federal, state, or local government enter into a contract that specifies the population to be served, the outcomes to be achieved, the measurement methodology to be used, and the schedule of payments to be made. The financing organization works with philanthropic and other investors to invest in innovative, data-driven service providers that can achieve results. One example of a pay for-success bond program is an initiative in New York targeted at reducing recidivism in adult males.
• Milestone-based Payments: Payment terms in a standard grant or contract in which the payment for each performance milestone established in the statement of work is not made until the milestone is proven to have been achieved. One example of this approach has been successfully demonstrated in NASA 's Commercial Orbital Transportation Services (COTS) program.
• Priority Review Vouchers: An accelerated regulatory review offered to products that meet certain performance or cost criteria, such as the FDA Innovation Pathway and USPTO 's Patents for Humanity.
• Patent Buyout: An offer to buy out the patent rights to a product that meets specified performance conditions at a set price (price for patent usually marked up over market value; followed by placing of the patent into the public domain to encourage competition for commercialization of the product). One example is the purchase of the patent for the Daguerreotype process by the French government in 1839.