2012-29362. Pipeline Safety: Using Meaningful Metrics in Conducting Integrity Management Program Evaluations

AGENCY:

Pipeline and Hazardous Materials Safety Administration (PHMSA), DOT.

ACTION:

Notice; Issuance of Advisory Bulletin.

SUMMARY:

PHMSA is issuing an Advisory Bulletin to remind operators of gas transmission and hazardous liquid pipeline facilities of their responsibilities, under Federal integrity management regulations, to perform evaluations of their integrity management programs using meaningful performance metrics.

FOR FURTHER INFORMATION CONTACT:

Alan Mayberry by phone at 202-366-5124 or by email at [e-mail]. All materials in this docket may be accessed electronically at http://www.regulations.gov. General information about the PHMSA Office of Pipeline Safety (OPS) can be obtained by accessing OPS's Internet home page at http://www.phmsa.dot.gov/​pipeline.

SUPPLEMENTARY INFORMATION:

I. Background

PHMSA's integrity management regulations require operators to establish processes to evaluate the effectiveness of their integrity management programs. Program evaluation is one of the key required program elements as established in the integrity management rules. For hazardous liquid pipelines, §§ 195.452(f)(7) and 195.452(k) require methods to measure program effectiveness:

§ 195.452(f) What are the elements of an integrity management program? An integrity management program begins with the initial framework. An operator must continually change the program to reflect operating experience, conclusions drawn from results of the integrity assessments, other maintenance and surveillance data, and evaluation of consequences of a failure on the high consequence area. An operator must include, at minimum, each of the following elements in its written integrity management program:

* * * * *

(7) Methods to measure the program's effectiveness (see paragraph (k) of this section);

§ 195.452(k) What methods to measure program effectiveness must be used? An operator's program must include methods to measure whether the program is effective in assessing and evaluating the integrity of each pipeline segment and in protecting the high consequence areas. (See Appendix C of this part for guidance on methods that can be used to evaluate a program's effectiveness.)

Appendix C provides more specific guidance on establishing performance measures, including the need to select measures based on the understanding and analysis of integrity threats to each pipeline segment. Appendix C also describes three general types of metrics that an integrity management program should have:

• Activity Measures that monitor the surveillance and preventive activities that are in place to control risk. These measures indicate how well an operator is implementing the elements of its integrity management program.
• Deterioration Measures that monitor operational and maintenance trends to indicate if the program is successful or weakening, or if the desired outcome is being achieved or not, despite the risk control activities in place.
• Failure Measures that reflect whether the program is effective in achieving the objective of improving integrity. These are typically lagging indicators that measure the number of releases, the volume spilled, percent recovered, etc.

Section 13 “Program Evaluation” of API Standard 1160, Managing Integrity for Hazardous Liquid Pipelines also provides additional guidance on the program evaluation process in which these measures are used to improve performance.

For gas transmission pipelines, §§ 192.911(i) and 192.945 define the requirements for establishing performance metrics and evaluating integrity management program performance.

§ 192.911

What are the elements of an integrity management program?

An operator's initial integrity management program begins with a framework (see § 192.907) and evolves into a more detailed and comprehensive integrity management program as information is gained and incorporated into the program. An operator must make continual improvements to its program. The initial program framework and subsequent program must, at minimum, contain the following elements. (When indicated, refer to ASME/ANSI B31.8S incorporated by reference, see § 192.7) for more detailed information on the listed element.)

* * * * *

(i) A performance plan as outlined in ASME/ANSI B31.8S, section 9 that includes performance measures meeting the requirements of § 192.945.

§ 192.945

What methods must an operator use to measure program effectiveness?

(a) General. An operator must include in its integrity management program methods to measure whether the program is effective in assessing and evaluating the integrity of each covered pipeline segment and in protecting the high consequence areas. These measures must include the four overall performance measures specified in ASME/ANSI B31.8S (incorporated by reference, see § 192.7 of this part), section 9.4, and the specific measures for each identified threat specified in ASME/ANSI B31.8S, Appendix A. An operator must submit the four overall performance measures as part of the annual report required by § 191.17 of this subchapter.

(b) External Corrosion Direct Assessment (ECDA). In addition to the general requirements for performance measures in paragraph (a) of this section, an operator using direct assessment to assess an external corrosion threat must define and monitor measures to determine the effectiveness of the ECDA process. These measures must meet the requirements of § 192.925.

The gas transmission requirements invoke ASME B31.8S-2004, Managing System Integrity of Gas Pipelines. Section 9 of this standard provides guidance on the selection of performance measures. It describes three categories of measures that are directly analogous to those noted above in Appendix C of Part 195. These are:

• Process or Activity Measures used to evaluate preventive and mitigation activities. These determine how well an operator is implementing the various elements of its integrity management program.
• Operational Measures, which include operational and maintenance trends that measure how well the system is responding to the integrity management program.
• Direct Integrity Measures, which include leaks, ruptures, injuries, and fatalities.

Furthermore, the hazardous liquid and gas transmission integrity management rules also require that operators retain adequate records to support integrity management program decisions and activities. These include the information that supports the selection of performance metrics, the performance metric data and trends, and the decisions that are based in whole or in part on these metrics. Specifically, the hazardous liquid integrity management program requirements are:

§ 195.452(l) What records must be kept? (1) An operator must maintain for review during an inspection:

* * * * *

(ii) Documents to support the decisions and analyses, including any modifications, justifications, variances, deviations and determinations made, and actions taken, to implement and evaluate each element of the integrity management program listed in paragraph (f) of this section.

(2) See Appendix C of this part for examples of records an operator would be required to keep.

Appendix C further states:

§ 195.452 Appendix C. VI. Examples of types of records an operator must maintain.

* * * * *

(22) methods used to measure the program's effectiveness.

The comparable gas transmission integrity management program requirements are:

§ 192.947

What records must be kept?

An operator must maintain, for the useful life of the pipeline, records that demonstrate compliance with the requirements of this subpart. At minimum, an operator must maintain the following records for review during an inspection.

* * * * *

(d) Documents to support any decision, analysis, and process developed and used to implement and evaluate each element of the baseline assessment plan and integrity management program. Documents include those developed and used in support of any identification, calculation, amendment, modification, justification, deviation and determination made, and any action taken, to implement and evaluate any of the program elements;

PHMSA's inspection protocols currently address the need to examine operator compliance with these requirements.

In its report on the September 9, 2010, gas pipeline accident in San Bruno, California, the National Transportation Safety Board (NTSB) identified concerns with Pacific Gas and Electric Company's (PG&E) self-assessments of its integrity management program. NTSB concluded that the company's self-assessments were “superficial and resulted in no improvements to the integrity management program.” As a result, NTSB recommended that PG&E:

Assess every aspect of your integrity management program, paying particular attention to the areas identified in this investigation, and implement a revised program that includes, at a minimum,

* * * * *

(4) an improved self-assessment that adequately measures whether the program is effectively assessing and evaluating the integrity of each covered pipeline segment. (Recommendation P-11-29)

In this same investigation, NTSB raised some concerns with PHMSA's oversight of performance-based safety programs such as integrity management. NTSB concluded that greater focus is needed on how performance-based safety systems are implemented, executed and evaluated, and whether problem areas are being detected and corrected. Critical to this overall process is the selection of meaningful metrics by operators that allow them to quantify, understand, and improve their own performance.

Following its investigation, NTSB issued two related recommendations for enhancing PHMSA's oversight of operator programs to assess the effectiveness of PHMSA's programs using performance metrics. These recommendations are:

Revise your integrity management inspection protocol to:

(1) incorporate a review of meaningful metrics;

(2) require auditors to verify that the operator has a procedure in place for ensuring the completeness and accuracy of underlying information;

(3) require auditors to review all integrity management performance measures reported to the Pipeline and Hazardous Materials Safety Administration and compare the leak, failure, and incident measures to the operator's risk model; and

(4) require setting performance goals for pipeline operators at each audit and follow up on those goals at subsequent audits. (Recommendation P-11-18)

(1) Develop and implement standards for integrity management and other performance-based safety programs that require operators of all types of pipeline systems to regularly assess the effectiveness of their programs using clear and meaningful metrics and to identify and then correct deficiencies; and (2) make those metrics available in a centralized database. (Recommendation P-11-19)

These recommendations reinforce the importance of a rigorous evaluation of a company's integrity management program in improving performance. Through this Advisory Bulletin, PHMSA is reminding operators of the importance of these regulation-required program elements. Operators should review their current programs for evaluating integrity management program effectiveness and the performance metrics used in these programs to be sure they provide a current and accurate representation of integrity management program performance. Further, operators should ensure that program improvements and corrective actions identified by these evaluations are implemented in a timely manner.

As a result of NTSB's recommendations, PHMSA is initiating efforts to strengthen its protocols and oversight of these key integrity management program elements. Beginning immediately, PHMSA's inspections will emphasize reviewing operator methods for integrity management program evaluation as required by § 192.945 and § 195.452(k) for gas transmission and hazardous liquid pipelines, respectively. PHMSA will evaluate specific metrics operators use to assess program effectiveness and how those metrics are used in a process of continuous improvement. PHMSA will also confirm that operators are maintaining adequate records of their program effectiveness evaluations and their performance metrics data, as well as the activities and decisions associated with all required integrity management program elements. Our inspectors will check to confirm that information and data gaps are aggressively being addressed and that assumptions are appropriately based on location-specific data.

II. Advisory Bulletin (ADB-20l2-10)

To: Owners and Operators of Hazardous Liquid and Gas Transmission Pipeline Systems

Subject: Using Meaningful Metrics in Conducting Integrity Management Program Evaluations

Advisory: To further enhance PHMSA's safety efforts and as an initial step in addressing NTSB Recommendations P-11-18 and P-11-19, PHMSA is issuing this Advisory Bulletin concerning operator integrity management program evaluation using meaningful metrics.

A critical program element of an operator's integrity management program is the systematic, rigorous evaluation of the program's effectiveness using clear and meaningful metrics. When executed diligently, this self-evaluation process will lead to more robust and effective integrity management programs and improve overall safety performance. This process is critical to achieving a mature integrity management program and a culture of continuous improvement. Program evaluation is a required integrity management program element as established in §§ 192.911(i) and 195.452(k) for gas transmission and hazardous liquid pipelines, respectively. In light of NTSB's findings following the San Bruno gas transmission incident, PHMSA is reminding operators about the importance of these requirements.

Operators are advised to critically review their processes and methods for evaluating integrity management program performance and take action to strengthen these processes where warranted. An effective operator performance evaluation process is expected to have the following characteristics:

• A well-defined description of the scope, objectives, and frequency of program evaluations.
• The use of periodic self-assessments, internal or external audits, management reviews, performance metrics analysis, benchmarking against other operators, or other self-critical evaluations to assess program effectiveness.
• Clear performance goals and objectives to measure the effectiveness of key integrity activities.
• Clear assignment of responsibility for implementing required actions.
• Review and follow-up of program evaluation results, findings, and recommendations, etc., by appropriate company managers.

Operators are also advised that a clear and meaningful set of performance metrics is essential to program effectiveness. An effective program for measuring integrity management program effectiveness should have the following characteristics:

• A description of the type of performance measures to be used, along with the data sources, data validation and quality assurance activities, the frequency of data collection, and any normalization factors.
• A means to update the performance measures (if needed) to assure they are providing useful information about the effectiveness of integrity management program activities.
• The use of performance metrics data to check and calibrate the operator's risk analysis tools to assure these best represent the performance of the operator's specific assets.

The performance metrics that are required to be reported to PHMSA annually, such as the number of miles of pipeline assessed, number of anomalies found requiring repair or mitigation, etc., are a small subset of the overall suite of metrics used by an operator to evaluate its program. A much larger set of operator-specific metrics to be used internally is needed to effectively evaluate an integrity management program performance. Metrics should be developed for each of the following:

• Overall program effectiveness indicated by the number of releases, number of injuries or fatalities, volume released, etc.
• Specific threats that include both leading and lagging indicators for the important integrity threats on an operator's systems. These include:

○ Activity Measures that monitor the surveillance and preventive activities that are in place to control risk.

○ Deterioration Measures that monitor operational and maintenance trends to indicate if the program is successful or weakening despite the risk control activities in place. (Also identified as Operational Measures in ASME B31.8S.)

○ Failure Measures that reflect whether the program is effective in achieving the objective of improving integrity. (Also identified as Direct Integrity Measures in ASME B31.8S)

• Metrics that measure and provide insights into how well an operator's processes associated with the various integrity management program elements are performing. Examples of such processes would include integrity assessment, risk analysis, the identification of preventive and mitigative measures, etc.

While operator-level rollups of metrics are useful for small operators, a robust program for large operators should also include metrics at a more granular level. The metrics should enable operators to drill down to understand the performance of specific systems or segments within systems. This is particularly important for the threat-specific metrics mentioned previously.

Finally, as required by §§ 195.452(l) and 192.947, operators must keep records supporting the decisions, analyses, and processes developed and used in their evaluation of integrity management program effectiveness. These records should include those justifying the selection of performance metrics, the performance metric data and trends, and how these metrics are used to improve the integrity management program. Operators should also be diligently working to eliminate information and data gaps throughout their entire integrity management program.

Issued in Washington, DC, on November 29, 2012.

Jeffrey D. Wiese,

Associate Administrator for Pipeline Safety.