03-31574. Pipeline Safety: Potential Service Disruptions in Supervisory Control and Data Acquisition Systems

AGENCY:

Research and Special Programs Administration (RSPA), DOT.

ACTION:

Notice; issuance of advisory bulletin.

SUMMARY:

RSPA's Office of Pipeline Safety (RSPA/OPS) is issuing this advisory notice to owners and operators of gas and hazardous liquid pipelines who use Supervisory Control and Data Acquisition (SCADA) systems. Pipeline owners and operators should establish thorough testing regimes when they design and implement modifications and enhancements of their SCADA systems. Owners and operators should consider using off-line or developmental workstations to test changes, then deploy the changes on-line under close monitoring at times when few operational changes are expected on the pipeline. Applying these techniques will help ensure that changes in the SCADA system environment do not have an unexpected effect on pipeline operations.

FOR FURTHER INFORMATION CONTACT:

Richard Huriaux, [phone]; or by e-mail, [e-mail]. This document can be viewed at the RSPA/OPS home page at http://ops.dot.gov. General information about the RSPA/OPS programs can be obtained by accessing RSPA's home page at http://rspa.dot.gov.

I. Advisory Bulletin (ADB-03-09)

To: Owners and Operators of Gas and Hazardous Liquid Pipeline Systems Who Use SCADA Systems.

Subject: Potential Service Disruptions in SCADA Systems.

Purpose: To inform pipeline owners and operators of the potential for service disruptions in SCADA systems caused by maintenance or enhancements of SCADA system configuration and other critical databases, and the possibility of those disruptions leading to or aggravating pipeline releases.

Advisory: Each pipeline owner or operator should review their procedures for the upgrading, configuring, maintaining, and enhancing its SCADA system. If not well thought out and thoroughly tested, such changes could cause inadvertent service disruptions in the SCADA system. Resulting conditions could may impede controllers responsible for operating the pipeline from promptly recognizing and reacting to abnormal conditions, and could potentially impact the controllers' abilities to restore normal operations. Owners and operators should ensure that SCADA system modifications do not degrade overall SCADA performance to an unacceptable level. To further reduce the potential effect of service disruptions, responsible personnel should coordinate significant and non-routine SCADA modifications to occur at times when no significant changes to pipeline operations are anticipated.

It is good practice for owners and operators of pipeline systems to periodically review their SCADA system configurations, operating procedures, and performance measurements to ensure that the SCADA computer servers are functioning as intended. Owners and operators should consider using off-line or development workstations/servers to help ensure that impending changes are tested as thoroughly as possible before moving the changes into production. Although off-line or development workstations can be valuable, they may not fully represent timing, load and other factors that will be present in the production environment. System modifications should be implemented via structured and managed processes to reduce the likelihood of unforeseen problems. Such controlled processes are especially important if an owner or operator makes changes directly in the on-line environment.

In addition, owners or operators should periodically confirm that associated design and maintenance personnel, whether employees, contractors, or third-party providers, are adequately skilled to perform SCADA system modifications without causing undesirable consequences. These same personnel should be cognizant of the critical system attributes that should be monitored during the testing phase of implementation.

SUPPLEMENTARY INFORMATION:

II. Background

This advisory bulletin responds to National Transportation Safety Board (NTSB) Recommendation P-02-05, which suggested that RSPA/OPS: “[i]ssue an advisory bulletin to all pipeline owners and operators who use supervisory control and data acquisition (SCADA) systems advising them to implement an off-line workstation that can be used to modify their SCADA system database or to perform developmental and testing work independent of their on-line systems. Advise owners and operators to use the off-line system before any modifications are implemented to ensure that those modifications are error-free and that they create no ancillary problems for controllers responsible for operating the pipeline.”

During an earlier investigation of a pipeline incident, RSPA/OPS inspectors identified inadequate SCADA performance as an operational safety concern, and published advisory bulletin ADB-99-03 on July 16, 1999 (64 FR 38501). That advisory identified eroding SCADA performance as a contributing factor to the accident.

Through subsequent analysis, it has become apparent that SCADA performance in general can be adversely impacted by system configuration changes, upgrades, or modifications to critical databases. There are several ways that pipeline owners and operators can reduce the risk of such conditions:

(1) Ensure that personnel assigned to these duties are adequately skilled in the maintenance and upgrading of the SCADA system configuration and critical databases.

(2) Know what critical metrics can be monitored that provide thorough and representative measures of system performance during testing and after the changes are implemented.

(3) Consider making the changes first on an isolated, off-line, or development workstation or processor, to test the effect of the changes prior to moving the work into the production environment.

(4) Recognize that the use of off-line or development workstations/servers to test impending changes can be valuable, but probably does not fully represent timing, load, and other factors present in the production environment.

(5) Know the limits and bounds of the testing regime, so that adequate and targeted vigilance may be applied during final testing and after initial implementation into the production environment.

(6) Coordinate significant and non-routine SCADA system modifications with pipeline controller operating personnel, so that revisions are implemented and tested at times when no significant changes to pipeline operations are anticipated.

Although NTSB Recommendation P-02-05 called only for an advisory bulletin, RSPA/OPS has taken additional actions to improve SCADA and controller operations and our inspection process. RSPA/OPS has initiated a study on the safety evaluation of pipeline SCADA technology. In early 2004, RSPA/OPS will revise its SCADA inspection protocols. Later in 2004, RSPA/OPS will begin development of a new, multi-tiered approach to inspection of SCADA systems.

RSPA/OPS has also initiated a study of Controller Certification in compliance with Section 13(b) of the Pipeline Safety Improvement Act of 2002. Section 13(b) of the Pipeline Safety Improvement Act of 2002 (PSIA), directs the Secretary of Transportation to develop tests and other requirements for certifying the qualifications of individuals who operate computer-based systems for controlling the operations of pipelines. The RSPA/OPS project team is evaluating current operator personnel qualification practices for pipeline controllers in collaboration with a study team sponsored by the gas and hazardous liquid industry. RSPA/OPS will develop an approach to certification programs and will undertake pilot testing. Through research and pilot program evaluations, RSPA/OPS will determine the best combination of prescriptive and performance-based requirements that should be considered as certification criteria for pipeline controllers.

Issued in Washington, DC on December 17, 2003.

Stacey L. Gerard,

Associate Administrator for Pipeline Safety.