2016-30593. Announcement of Requirements and Registration for a Prize Competition Titled: Sub-Seasonal Climate Forecast Rodeo

AGENCY:

Bureau of Reclamation, Interior.

ACTION:

Notice.

SUMMARY:

The Bureau of Reclamation is announcing the following prize competition, Sub-Seasonal Climate Forecast Rodeo. This Challenge seeks to improve on existing sub-seasonal forecasts and asks Solvers (i.e., competitors in the Challenge) to develop systems that perform demonstratively better than existing baseline forecasts for temperature and precipitation over a 15-42-day time frame. Solvers will have approximately 3 months to develop their system, at which point they are asked to provide forecasts every 2 weeks over a 13-month period, with the first month being a “pre-season” to become familiar with the submission and evaluation processes.

DATES:

Listed below are the specific dates pertaining to this prize competition. Please note that times in meteorology are based upon a world-wide 24-hour clock called Zulu time (Z). Additional detail on Zulu time is available in the SUPPLEMENTARY INFORMATION section of this notice.

1. First forecast due on or before 0Z (Zulu) March 21, 2017 (pre-season), and 0Z April 18, 2017 (actual competition).

2. Final forecast and hind-cast due on or before 0Z April 3, 2018.

3. Final submission due on or before 11:59 p.m. (U.S. Eastern Time) May 3, 2018.

4. Judging period ends on August 3, 2018.

5. Winners announced on or before September 5, 2018.

ADDRESSES:

The Sub-Seasonal Climate Forecast Rodeo Prize Competition will be posted on the following crowd-sourcing platforms where Solvers can register for this prize competition:

1. The Water Pavilion located at the InnoCentive Challenge Center: www.innocentive.com/​water-pavilion/​.

2. U.S. Federal Government Challenge Platform: www.Challenge.gov. InnoCentive, Inc. is administering this challenge under a challenge support services contract with the Bureau of Reclamation. Challenge.gov will re-direct the Solver community to the InnoCentive Challenge Center as the administrator for this prize competition. Additional details for this prize competition, including background information, templates, and the Challenge Agreement specific for this prize competition, can be accessed through either of these prize competition web addresses. The Challenge Agreement contains more details of the prize competition rules and terms that Solvers must agree with to be eligible to compete.

Information pertaining to this competition will be posted to the Bureau of Reclamation's current prize competitions Web page at www.usbr.gov/​research/​challenges/​current/​.

FOR FURTHER INFORMATION CONTACT:

Challenge Manager: Dr. David Raff, Science Advisor, Bureau of Reclamation, [phone], [e-mail]; Ken Nowak [phone], [e-mail].

SUPPLEMENTARY INFORMATION:

The Bureau of Reclamation (Reclamation) is announcing the following prize competition in compliance with 15 U.S.C. 3719, Prize Competitions. The intent is to spur innovation toward improved forecasts of temperature and precipitation using a real-time competition and cash prizes as incentives.

Prize Competition Summary: Improved sub-seasonal forecasts for weather and climate conditions (lead-times ranging from 15 to 45 days and beyond) would allow water managers to better prepare for shifts in hydrologic regimes such as the onset of drought or occurrence of wet weather extremes. The challenge with sub-seasonal weather and climate forecasting is that it encompasses the time frame where initial state or condition information, such as coupled land-atmosphere processes becomes less important, and slowly varying long term states, such as sea surface temperature, soil moisture, and snowpack, become more important to predictions. In addition, the relative importance of the initial state or condition, versus the longer term state, depends on the lead time, region of interest, and time of year.

Accurate sub-seasonal weather forecasting has proven to be particularly difficult to accomplish but is of great interest to water managers tasked with predicting sub-seasonal streamflow and water supply. Sub-seasonal forecasting, which spans approximately 15 to 45 days in the future, is difficult because it bridges short-term forecasting, where initial conditions primarily determine upcoming weather, and long-term forecasting in which slowly varying factors become more important.

This Challenge is asking Solvers to develop systems that improve upon existing sub-seasonal temperature and precipitation forecasts. Solvers are not required to develop entire systems from scratch. Methods could include, but are not limited to, approaches for improving the accuracy of existing sub-seasonal forecasts, techniques that leverage climate teleconnections, or statistical models. This Challenge will be active for approximately 17 months, starting with a 3-month development period followed by a 1-month “pre-season”, and a 12-month competition period. Following the competition period, Solvers will have 1 month to prepare final submissions. During the competition period, Solvers will be required to upload sub-seasonal forecasts every 2 weeks. An online leaderboard hosted by the National Integrated Drought Information System at drought.gov will track and display Solvers' performance for the duration of the competition period. Please note that InnoCentive usernames will be shared with the National Oceanic and Atmospheric Administration (NOAA) as part of the forecast evaluation process and leaderboard tracking. At the conclusion of the competition period, a final submission is required.

Prizes total $800,000. Four categories are defined by two forecast outlook periods and two forecast variables (temperature and precipitation). In each category, prizes for eligible solvers are as follows:

1st place—$100,000

2nd place—$50,000

3rd place—$25,000

In addition, one $25,000 prize per category may be awarded to an eligible solver based solely on hind-cast performance, submission of which is a requirement to be eligible for the above listed prizes.

Final submissions to the Challenge should include the following:

1. The detailed description of the proposed Solution addressing the specific Technical Requirements that are presented in the Detailed Description of the Challenge. This description should be accompanied by a well-articulated rationale for the proposed Solution.

2. Locally executable application and corresponding source code to implement the proposed Solution that are documented.

3. Proof-of-concept data obtained as outlined in the Detailed Description of the Challenge.

The Challenge award is contingent upon theoretical evaluation and operational validation of the submitted Solutions by the Seeker (Reclamation).

To receive an award, the Solvers will grant to the Seeker a non-exclusive license to practice their solutions and make the solution available as open source software licensable under either Berkeley Software Distribution or General Public License Open Source license.

Technical Requirements. Water managers and other users have indicated that they want the existing forecasts to be better rather than wanting the development of new forecast systems. Therefore, this competition seeks new and novel forecast method(s) to improve the forecasting of precipitation and temperature that can be incorporated into existing frameworks relied upon for water management. Solvers may leverage existing forecasts or ensembles in their solution, but must be able to demonstrate appreciable value added by the solution relative to any input or foundational framework. Specifically, the competition desires solutions that can outperform current operational forecasts (including forecasts from operational centers outside the U.S.) and a damped persistence forecast at a 1x1 degree gridded resolution for the western United States at two forecast outlooks: 15-28 days (weeks 3-4) and 29-42 days (weeks 5-6) for temperature and precipitation. Overall skill- spatially and across a range of weather/climate conditions- is most important. The ability to skillfully forecast extreme conditions is also very desirable. Any system that meets these criteria is sought.

Evaluation criteria. Forecast skill will be evaluated for temperature and precipitation separately since the drivers responsible for prediction of these variables are different and the subsequent skill level is also expected to be different. Moreover, the 15-28-day and 29-42-day periods will be evaluated individually for similar reasons. Winning forecasts must outperform NOAA's sub-seasonal modeling system, the Climate Forecast System Version 2 (CFSv2) and damped persistence forecasts (see definitions below). Specifically, skill will be evaluated individually for temperature and precipitation for weeks 3-4 and weeks 5-6 as the highest skill over the competition's identified geographic area, averaged over the entire competition time period. To be prize eligible, Solvers must also demonstrate historical skill of statistical significance that is equal to or greater than that of the CFSv2 through submission of a hind-cast analysis described below.

Definitions.

• Anomaly is defined as the difference between a given value and climatology for a specific location (grid cell by grid cell) and time.
• Climatology is defined as average temperature and precipitation for a specific 2-week period as computed from the Climate Prediction Center's daily unified gauge data set for precipitation at ftp://ftp.cpc.ncep.noaa.gov/precip/CPC_UNI_PRCP/GAUGE_GLB/, and the Climate Prediction Center's global gridded temperature data set over the 30 year period of 1981-2010 at ftp://ftp.cpc.ncep.noaa.gov/precip/wd52ws/global_temp/.
• A damped persistence forecast will be represented using seasonally developed regression coefficients based on the historical climatology period of 1981-2010 that relate observations of the past 2 weeks to the forecast outlook periods on a grid cell by grid cell basis.
• Skill is defined as spatial anomaly correlation, averaged over time. Final competition standings will be the average spatial anomaly correlation of all 26 forecasts issued for the western U.S. geographic area, as defined by the forecast submission template.

The CFSv2 forecasts at https://www.ncdc.noaa.gov/​data-access/​model-data/​model-datasets/​climate-forecast-system-version2-cfsv2 will be combined into a forecast baseline over the same time period as Solver submissions using an ensemble mean of the 32 ensemble-member forecasts leading up to each competition submission. The CFSv2 is run out to 45 days, 4 times daily, with four initial conditions per run for a total of 16 forecasts per day. For example, for a forecast due March 1st, the 16 CFSv2 ensemble-member forecasts issued on February 27 and the 16 CFSv2 ensemble-member forecasts issued on February 28 will be averaged together for each grid and over the two forecast-periods to become the forecast baseline for the March 1st due date. Further, the ensemble mean will be bias corrected using the method employed by NOAA's Climate Prediction Center in developing their operational forecasts. Specifically, this process involves comparing the CFSv2 reforecasts for the period of 1999-2010 with observed data to establish and correct daily bias on a grid cell by grid cell basis. This is done for temperature and precipitation at both forecast outlook periods.

Forecast submission requirements and instructions. Over the course of this Challenge, Solvers will be required to submit 2 pre-season forecasts and 26 forecasts during the year-long competition, for each of the four categories detailed above and must meet the following requirements:

1. Beginning at the start of the competition period and every 2 weeks thereafter (see below for required submission deadlines) for the duration of the competition, Solvers will submit 1x1 degree gridded forecasts corresponding to western U.S. competition geographic area as defined by the forecast template available on Reclamation's current competitions Web page listed in the ADDRESSES section of this notice for:

a. Average temperature (degrees C) for days 15-28.

b. Cumulative precipitation (millimeters) for days 15-28.

c. Average temperature (degrees C) for days 29-42.

d. Cumulative precipitation (millimeters) for days 29-42.

2. To be eligible for an award, Solvers must submit all 26 forecasts on time. A grace of two missed forecast per category will be allowed. For the purpose of computing skill, climatology will be substituted for a missed forecast. More than two missed forecast in any category will result in disqualification.

3. Forecast submissions will be made through a portal hosted by NOAA and must utilize the supplied template, available on Reclamation's current prize competitions Web page.

4. The first month of the 13-month competition period (i.e. the first two forecasts) will be considered a `pre-season' for Solvers to become familiar with the submission process and will not count toward their standing in the competition or against the two missed forecast grace described above.

Forecast submission deadlines. The forecast competition will run for 1 year following the `pre-season' with new forecast submissions required every 2 weeks. All forecasts need to be received by 0Z on the day of the required forecast submission. All aspects of meteorology are based upon a world-wide 24-hour clock called Zulu time (Z), more commonly called Coordinated Universal Time (UTC). More information on this topic is available on the National Weather Service Web site at http://www.srh.noaa.gov/​jetstream/​synoptic/​time.html, including conversion to common U.S. time zones. Specific forecast submission deadlines, including dates and times, will be available on Reclamation's current prize competitions Web page listed in the ADDRESSES section of this notice, the first of which is due approximately 3 months following the announcement of this challenge.

In addition to requirements listed above for the forecast submissions, proposed solutions must meet the following Technical Requirements:

1. For first, second or third place, systems must outperform, based on the formula described above, cumulatively over the competition time period, the CFSv2 baseline and damped persistence forecasts for at least one category: Week 3-4 temperature, week 5-6 temperature, week 3-4 precipitation, or week 5-6 precipitation.

2. Must demonstrate historical skill of statistical significance that is equal to or greater than that of the CFSv2 through submission of a hind-cast analysis. The CFSv2 reforecasts at http://nomads.ncdc.noaa.gov/​thredds/​catalog/​cfsr-hpr-ts45/​catalog.html will provide the baseline for the hind-cast, in the same way that the CFSv2 forecasts provide a baseline for the real-time competition. Two notable distinctions between the CFSv2 forecasts and reforecasts are (1) the reforecast baseline will be based on a mean of 8 ensemble members (only one set of initial conditions were used, thereby producing 4 reforecasts per day) and (2) bias correction to the reforecast will be developed and applied for each reforecast year independently, so as to not leverage knowledge of that year's reforecast performance toward bias correcting itself. Note that skill and prize eligibility will be evaluated for each category individually—i.e. to be prize eligible in a particular category, Solvers need only outperform the CFSv2 in that category. To do this, Solvers will submit, hind-casts for the four categories, issued every 2 weeks, for the period of 1999-2010, no later than the Challenge final forecast due date. The hind-cast should be performed as a “leave one out cross validation.” This is accomplished by removing one year of observed historical data from the period of 1999-2010, calibrating the model based on the remaining years of data, and forecasting that year. Hind-casts are to be issued at the month/day combinations specified in the forecast submission deadlines referenced above. Given the dates of the competition, each hind-cast will span 2 calendar years, thereby resulting in 11 1-year hind-cast periods for the “leave one out cross validation.” This process should be repeated until forecasts have been issued for all 11 years. Solvers will submit one historical hind-cast per category for the 11 years. For more information on this technique, see the discussion on cross-validation by Clarke et al. in Principles and Theory for Data Mining and Machine Learning (2009).

3. Solvers may utilize any available sub-seasonal forecast as the starting point for their system, but must demonstratively improve upon that forecast.

4. Must be written in C++, R, python, Fortran, or other widely recognized programming languages and be licensable under either Berkeley Software Distribution or General Public License Open Source license.

5. Solvers with winning solutions are required to provide all code, data, and other components of their forecast system necessary to run the system and reproduce the forecasts issued in the competition. Failure of code to reproduce performance during the competition or for the hind-cast may result in disqualification. Furthermore, the Solvers may be required to iterate with competition judges to ensure documentation is sufficient.

Project Deliverables: In addition to the hind-casts, 2 pre-season forecast submissions, and 26 forecast submissions during the year-long competition, Solvers must submit a final proposed Solution by the Challenge deadline. The submitted proposal should include the following:

1. Detailed description of a sub-seasonal forecasting system that meets the Technical Requirements listed in the Detailed Description and Requirements section of the Challenge.

2. A well-reasoned rationale supporting the methodology of the proposed system and addressing each of the Technical Requirements described in the Detailed Description.

3. Locally executable application and corresponding documented source code implementing the proposed Solution.

The Seeker may wish to partner with the Solver at the conclusion of the Challenge. Solver should describe their expertise and include a statement indicating their interest in this opportunity.

The proposal should not include any personal identifying information (name, username, company, address, phone, email, personal Web site, resume, etc.) or any information the Solvers may consider as their Intellectual Property they do not want to share.

Judging: An online leaderboard hosted by the National Integrated Drought Information System will track and display Solvers' performance for the duration of the competition period. The Challenge award is contingent upon theoretical evaluation and operational validation of the submitted Solutions by the Seeker. If multiple proposals meet all the Solution Requirements, the Seeker reserves the right to award only the top three solutions per category which they believe are of sound technical foundation. After the Challenge final submission deadline, submissions will be identified as potentially prize eligible, as determined by the quantitative forecast evaluation performed by NOAA and described above. Of those potentially prize eligible solutions, the Judging Panel will evaluate each with respect to the Solution Requirements and make a decision on winning solution(s). The Judging Panel may be composed of Federal and/or Non Federal scientists, engineers, and other technical experts, including subject matter experts from the listed collaborators for this Challenge. All Solvers that submit a proposal will be notified on the status of their submissions. Decisions by the Seeker cannot be contested.

Eligibility Rules: To be able to win a prize under this competition, an individual or entity must:

1. Agree to the rules of the competition (15 U.S.C. 3719(g)(1));

2. Be an entity that is incorporated in and maintains a primary place of business in the United States, or (b) in the case of an individual, a citizen or permanent resident of the United States (15 U.S.C. 3719(g)(3)).

However, submissions can be entertained from all Solvers regardless of whether they are U.S. citizens/entities. Meritorious submissions from non-eligible persons and entities, if any, will be recognized in publications issued by the Seeker announcing the results of the competition, such as press releases. Non-U.S citizens/permanent residents or non-U.S entities can also be included on U.S. teams. However, prizes—whether monetary or otherwise—will only be awarded to eligible persons and entities under the authority of the America COMPETES Reauthorization Act of 2010 (15 U.S.C. 3719).

3. Not be a Federal entity or Federal employee acting within the scope of their employment (15 U.S.C. 3719(g)(4)). A Federal entity is defined by 5 U.S.C. Appendix 8G with a list of current Federal entities periodically posted on the Federal Register.

4. Assume risks and waive claims against the Federal Government and its related entities (15 U.S.C. 3719(i)(1)(B)); and,

5. Not use Federal facilities, or consult with Federal employees during the competition unless the facilities and employees are made available to all individuals and entities participating in the competition on an equitable basis.

The following individuals or entities are not eligible regardless of whether they meet the criteria set forth above:

1. Any individual or organization who employs an evaluator on the Judging Panel or otherwise has a material business relationship or affiliation with any Judge.

2. Any individual who is a member of any Judge's immediate family or household.

3. The Seeker, participating organizations, and any advertising agency, contractor or other individual or organization involved with the design, production, promotion, execution, or distribution of the prize competition; and all employees, and all members of the immediate family or household of any such individual or organization.

4. Any individual or entity that uses Federal funds to develop the proposed solution now or any time in the past, unless such use is consistent with the grant award, or other applicable Federal funds awarding document. Note: Individuals or entities that have been funded by the Federal Government in the past to work within the technical domain of the competition are eligible provided their specific submission was not developed by them with Federal funds. Submissions that propose to improve or adapt existing federally funded technologies for the solution sought in this prize competition are also eligible. Individuals are also encouraged to consult with their employer Ethics Officer for additional guidance and considerations.

Consultation: Reclamation and collaborator scientists, engineers, and technical specialists were consulted in identifying and selecting the topic of this prize competition. Direct and indirect input from various stakeholders and the broader water resources community of practice were also considered.

Public Disclosure: InnoCentive, Inc. is administering this challenge under a challenge support services contract with Reclamation. Participation is conditioned on providing the data required on InnoCentive's online registration form. Personal data will be processed in accordance with InnoCentive's Privacy Policy which can be located at http://www.innocentive.com/​privacy.php. Before including your address, phone number, email address, or other personal identifying information in your proposal, you should be aware that the Seeker is under no obligation to withhold such information from public disclosure, and it may be made publicly available at any time. Neither InnoCentive nor the Seeker is responsible for human error, theft, destruction, or damage to proposed solutions, or other factors beyond its reasonable control.

Liability and Indemnification: By participating in this Challenge, each Solver agrees to assume any and all risks and waive claims against the federal government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from participation in this Challenge, whether the injury, death, damage, or loss arises through negligence or otherwise. By participating in this Challenge, each Solver agrees to indemnify the federal government against third party claims for damages arising from or related to Challenge activities

No Insurance Required: Based on the subject matter of the Challenge, the type of work that it will possibly require, as well as an analysis of the likelihood of any claims for death, bodily injury, or property damage, or loss potentially resulting from competition participation, Solvers are not required to obtain liability insurance or demonstrate financial responsibility in order to participate in this Challenge.

Dated: December 7, 2016.

Levi Brekke,

Acting Science Advisor.