2013-28198. Announcement of Requirements and Registration for the Predict the Influenza Season Challenge

AGENCY:

Centers for Disease Control and Prevention (CDC), Department of Health and Human Services (HHS).

Award Approving Official: Thomas R. Frieden, MD, MPH, Director, Centers for Disease Control and Prevention, and Administrator, Agency for Toxic Substances and Disease Registry.

ACTION:

Notice.

SUMMARY:

The Centers for Disease Control and Prevention (CDC) located within the Department of Health and Human Services (HHS) announces the launch of the Predict the Influenza Season Challenge.

Each year annual epidemics of influenza occur in the United States. However, variations in the timing and intensity of the season occur each year. Early insights into the timing of the beginning, the peak, and the intensity of the influenza season would be very useful in planning vaccination campaigns, targeting resources and therefore reducing costs associated with influenza prevention and control, and communicating prevention messages to the public. Mathematical and statistical models can be useful in predicting the timing and impact of the influenza season, but no models published to date have successfully predicted key influenza season milestones with sufficient accuracy.

The Influenza Division, National Center for Immunization and Respiratory Diseases, CDC would like to invest in innovation through the research and development of mathematical and statistical models that use digital surveillance data (e.g. Twitter, internet search data, web surveys, etc.) to predict the timing, peak, and intensity of the upcoming influenza season. By hosting this challenge, the Influenza Division is able to garner the technical innovation required to accurately forecast the influenza season for less resources than would be required through more traditional mechanisms. Furthermore, this challenge will allow the Influenza Division to foster competition and receive and evaluate multiple influenza season forecasts from a number of scientists using a variety of different methodologies, which would not be possible if a single entity were funded. Accurate influenza forecasts have the potential to reduce long-term costs by more efficiently utilizing resources that are available to track influenza and implement control measures during the influenza season.

The challenge could be solved by applying any mathematical, statistical, or other approach to predictive modeling. This challenge will provide the Influenza Division with methods that advance the science of prediction modeling, enhance the understanding of influenza modeling and the use of digital data for influenza surveillance, and improve the implementation of prevention and control measures for seasonal influenza.

The historical national surveillance data that could be used to enable training and correlation model development by competitors are available at http://gis.cdc.gov/​grasp/​fluview/​fluportaldashboard.html and are updated every Friday at noon. The competitors' predictions and methodology describing their models will comprise their initial challenge entry. Eight subsequent bi-weekly submissions of predictions based on the submitted methodology are also required. Predictions must be national in scope but may also include HHS region predictions; all predictions must be scaled to ILINet. Competitors' models will be evaluated based on the methodology and how well the predictions match the 2013-14 influenza season as measured by the U.S. Outpatient Influenza-like Illness Surveillance Network (ILINet: http://www.cdc.gov/​flu/​weekly/​overview.htm#Outpatient).

The competition will award a $75,000 prize and singular recognition to the person or team that most closely predicts the influenza season.

DATES:

Registration opens on November 23 and closes December 2, 2013. Register by sending email to [e-mail] and include your first and last name, your address, and your phone number. Please see “Registration Process for Participants” below for additional details.

The Competition Submission Period is from December 1, 2013-March 27, 2014. The first submission of the prediction must be received by December 2, 2013. Subsequent submissions will be required biweekly until the close of the submission period and must be received by 11:59 p.m. Eastern on December 19, 2013; January 2, 2014; January 16, 2014; January 30, 2014; February 13, 2014; February 27, 2014; March 13, 2014; and March 27, 2014.

Judging will take place between March 28, 2014 and May 30, 2014. Winners will be announced by June 20, 2014.

FOR FURTHER INFORMATION CONTACT:

Matthew Biggerstaff, MPH, Centers for Disease Control and Prevention, 1600 Clifton Rd NE MS A32; Atlanta, GA 30333, Phone: [phone], Email: [e-mail].

SUPPLEMENTARY INFORMATION:

Subject of Challenge Competition: Entrants of the Predict the Influenza Season Challenge are asked to predict the beginning, the peak, and the intensity of the 2013-2014 influenza season at the national level and at each or any Health and Human Services (HHS) region level(s) in the United States by developing mathematical and statistical models that utilize digital surveillance data (e.g. Twitter data, mining internet search term data, internet-based surveys). If there are questions about the eligibility of different data sources, please contact Matthew Biggerstaff at 404-639-3747 or [e-mail].

Eligibility Rules for Participating in the Competition:

To be eligible to win a prize under this challenge, an individual or entity—

(1) Must be at least 18 years old;

(2) Shall have registered to participate in the competition under the rules promulgated by HHS/CDC;

(3) Shall have complied with all the requirements under this section;

(4) In the case of a private entity, shall be incorporated in and maintain a primary place of business in the United States, and in the case of an individual, whether participating singly or in a group, shall be a citizen or permanent resident of the United States; and

(5) May not be a Federal entity or Federal employee acting within the scope of their employment.

(6) Shall not be an HHS employee working on their applications or submissions during assigned duty hours.

(7) Shall not be an employee of or contractor at HHS/CDC.

(8) Federal grantees may not use Federal funds to develop COMPETES Act challenge applications unless consistent with the purpose of their grant award.

(9) Federal contractors may not use Federal funds from a contract to develop COMPETES Act challenge applications or to fund efforts in support of a COMPETES Act challenge submission.

An individual or entity shall not be deemed ineligible because the individual or entity used Federal facilities or consulted with Federal employees during a competition if the facilities and employees are made available to all individuals and entities participating in the competition on an equal basis.

By participating in this challenge, an individual or organization agrees to assume any and all risks related to participating in the challenge. Individuals or organizations also agree to waive claims against the Federal Government and its related entities, except in the case of willful misconduct, when participating in the challenge, including claims for injury; death; damage; or loss of property, money, or profits, and including those risks caused by negligence or other causes.

By participating in this challenge, individuals or organizations agree to protect the Federal Government against third party claims for damages arising from or related to challenge activities.

Individuals or organizations are not required to hold liability insurance related to participation in this challenge.

Registration Process for Participants: To register, please send an email to [e-mail] and include your first and last name, your address, and your phone number. The subject line of the email submission must be labeled “Predict the Flu Challenge [Competitors Name] [Date of Submission]”. Additionally, this email should include the following statements: “By participating in this competition, I agree to assume any and all risks and waive claims against the Federal Government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from my participation in this prize contest, whether the injury, death, damage, or loss arises through negligence or otherwise. Additionally, I agree to indemnify the Federal Government against third party claims for damages arising from or related to competition activities.” Registration opens on November 23, 2013, and closes 11:59 p.m. EST on December 2, 2013.

The Competition Submission Period is from December 1, 2013-March 27, 2014. The first submission of the prediction must be received by December 2, 2013 and include a narrative describing the methodology of the prediction model and the results of that model (figures, tables, or narratives) using digital surveillance data (e.g. Twitter data, mining internet search term data, internet-based surveys) that predicts the beginning, the peak, and the intensity of the 2013-2014 influenza season at the national level in the United States. The methodology and data source(s) submitted to CDC must match the actual methods utilized by the competitors when making their influenza season predictions. In addition to the national-level predictions, competitors may also submit predictions of the beginning, the peak, and the intensity of the 2013-2014 influenza seasons for any of the 10 HHS regions.

Subsequent submissions will be required biweekly until the close of the submission period and must include the updated results of the prediction model (figures, tables, or narratives) at the national level in the United States using digital surveillance data; predictions for any of the 10 HHS regions will also be accepted, and submitting predictions for the 10 regions can potentially add to the final scores competitors. Subsequent submissions that include the updated predictions of the beginning, peak, and the intensity of the 2013-2014 influenza season must be received by 11:59 p.m. Eastern on December 19, 2013; January 2, 2014; January 16, 2014; January 30, 2014; February 13, 2014; February 27, 2014; March 13, 2014; and March 27, 2014. A total of 9 submissions must be received over the course of the contest to be eligible for the contest prize; late submissions will not be considered.

Contest Prize: For the contest, one $75,000 prize will be awarded. The winning competitor will be recognized on the CDC influenza Web page.

Payment of the Prize: Prizes awarded under this competition will be paid by electronic funds transfer and may be subject to Federal income taxes. HHS will comply with the Internal Revenue Service withholding and reporting requirements, where applicable.

Basis Upon Which Winner Will Be Selected: A total of 9 submissions must be received over the course of the contest to be eligible for judging; late submissions will not be considered. The Competitors' model methodology and nine biweekly predictions will comprise their challenge entry. The selection of the winner for this challenge will be based on an evaluation of the methodology used to make the prediction and the accuracy of the prediction compared to the U.S. Outpatient Influenza-like Illness Surveillance Network (ILINet: http://www.cdc.gov/​flu/​weekly/​) at the national level and any of the included 10 HHS regions. National and HHS-level predictions must be scaled to ILINet.

Submissions will be judged by a panel of reviewers that may include CDC staff, public health officials, and/or academics from noncompeting colleges or universities. Judges will score submissions on a scale of 0 to 150 using the six criteria identified below. Based on the average score for each submission, the top submissions will be recommended to a Lead Judge in CDC's Influenza Division. The Lead Judge will assess the top submissions using the criteria below and select the first place entry. Judging will take place between March 28, 2014 and May 30, 2014.

i. Methodology (25)

• Are key materials to support the correct interpretation of the predictions by the judges, such as concepts, sources, and methods, provided as part of the submission?
• Are the predictions scaled to ILINet? Are the data and results presented clearly, allowing the judges who may not be experts in mathematical modeling to evaluate the model and its outputs?
• What are the data sources used to make the prediction? Is the data source representative? If not, which groups are underrepresented in the data? Are there any impacts caused by these exclusions?
• Is there a clear description of how the search terms were selected (if relevant)?
• Are there any serious accuracy or methodological problems with the prediction approach?
• Are measures provided that give an indication of how certain contestants are about their predictions? (e.g. confidence intervals, probability of the prediction occurring)?

ii. Predicting the Start of the Influenza Season (10)

• Does the model accurately predict the start of the influenza season at the national level and any of the included 10 HHS regions? The start of the season will be defined as the week when the percentage of visits reported through ILINet crosses the baseline value for three consecutive weeks. ILINet baseline values for the United States and the 10 HHS regions for the 2013-2014 influenza season are available at http://www.cdc.gov/​flu/​weekly/​overview.htm#Outpatient.

iii. Predicting the Peak Week of the Influenza Season (10)

• Does the model accurately predict the peak week of ILINet at the national level and any of the included 10 HHS regions? The peak week will be defined as the surveillance week that the ILINet percentage is the highest for the 2013-14 influenza season in the United States and the 10 separate HHS regions.

iv. Predicting the Intensity of the Influenza Season (25)

• Does the model predict the intensity of the influenza season at the national and any of the included HHS regional levels? The intensity will be defined as the number of weeks that ILINet remains above baseline and the highest numeric value that the ILINet percentage reaches in the United States and the 10 separate HHS regions.

v. Timeliness and Reliability of the Predictions (20)

• How many weeks before each prediction milestone (the start and the peak week) was the most accurate prediction made?
• Did the prediction of each milestone vary widely between the Competitors' submissions?

vi. Geography (10)

• How comprehensively are the geographic regions of the United States represented in the source data?

vii. Optional HHS Regional Predictions (Up to 50 Bonus Points)

• HHS regional predictions will be evaluated separately using the six criteria identified above. Each submitted regional prediction will add between 0 and 5 points to the competitor's final score, depending on the evaluation of the accuracy, timeliness, and reliability of the prediction.

Additional Information: The historical national surveillance data that could be used to enable training and correlation model development by competitors are available at http://gis.cdc.gov/​grasp/​fluview/​fluportaldashboard.html and are updated every Friday at noon.

The Competitors will not have to transfer their exclusive intellectual property rights to the CDC. Instead, the Competitors will grant to CDC non-exclusive license to practice their solutions.

Compliance With Rules and Contacting Contest Winners: Finalists and the contest winners must comply with all terms and conditions of these official rules, and winning is contingent upon fulfilling all requirements herein. The initial finalists will be notified by email, telephone, or mail after the date of the judging. Awards may be subject to Federal income taxes, and the Department of Health and Human Services will comply with the Internal Revenue Service withholding and reporting requirements, where applicable.

Privacy: If contestants choose to provide the CDC with personal information by registering or filling out the submission form through the Challenge.gov Web site, that information is used to respond to contestants in matters regarding their submission, announcements of entrants, finalists, and winners of the contest. Information is not collected for commercial marketing. Registering through the Challenge.gov Web site is not required, however. Registrants may submit an email to [e-mail] as noted in “Registration Process for Participants.” Winners are permitted to cite that they won this contest.

General Conditions: The CDC reserves the right to cancel, suspend, and/or modify the contest, or any part of it, for any reason, at CDC's sole discretion.

Participation in this contest constitutes a contestants' full and unconditional agreement to abide by the contest's official rules found at www.Challenge.gov.

Authority: 15 U.S.C. 3719.

Dated: November 19, 2013.

Stacey Hoffman,

Acting Director, Division of Executive Secretariat, Centers for Disease Control and Prevention.