GAO quant jocks missed the real safety benefit of SMS Compliance

compliance-v-enforcement
Share this article: FacebooktwitterlinkedinFacebooktwitterlinkedin

 

The GAO has reviewed the FAA’s Compliance Philosophy. Its quantification predilection drove it to measure enforcement and informal action numbers- before and after. A better metric would have been to examine the safety efficacy specific solutions adopted through number-driven SMS risk analyses.

 

GAO Seeks Better Oversight of FAA’s Compliance Approach

While the FAA’s shift to a compliance-oriented approach rather than enforcement stance has generally received positive reviews, a government watchdog recommends that the agency step up oversight of the the compliance program and better evaluate its effectiveness.

In a recent report to Congress, the Government Accountability Office (GAO) said the FAA lacks controls to ensure the program works as intended. The FAA has not appointed a central authority to serve as a lead oversight of the program agency-wide, the GAO said, nor does any office regularly collect and monitor data on the program’s use.

levels of FAA action

“FAA lacks information to help it holistically manage the day-to-day functioning of its compliance and enforcement efforts, as well as the means to communicate information to Congress and industry. Second, FAA has not evaluated this new approach to enforcement to determine whether the approach has met its goals,” the GAO said. “Without such an evaluation, FAA cannot know the true effect of the compliance program.”

The agency implemented the compliance program in 2015, moving from an enforcement-oriented philosophy to one that strives to resolve unintentional compliance issues through education and collaborative remediation. The FAA still will use enforcement tools when such efforts fail, or the case of intentional non-compliance.

The greatest effect of the approach has centered on the number of administrative actions the agency has taken. Legal actions have declined slightly, but since 2012 the number of administrative actions has dropped from 6,000 annually to 2,000 per year now. The greatest declines began after compliance enforcement took effect, the GAO said.

compliance-across-offices_

“FAA views the compliance program as a necessary step to evolve the oversight of aviation safety,” the GAO noted, adding that agency officials have credited the program with helping to contribute to safety improvements. The officials believe the program has given the agency a “wider view” into regulated entities because it encourages transparency and information sharing. Further, FAA officials said the program has provided them with an “expanded ability to make aviation safer,” the GAO added. 

Headquarters and officials in field offices share that view, and local FAA officials claim it has improved relationships. They provided examples where entities no longer felt a need to hide things. The GAO said field officers reported that “regulated entities are more willing to work with FAA to focus on the root cause of a violation or event and work to address it.”

Similarly, industry associations gave feedback expressing the belief the program has improved compliance safety and transparency. Further, they said the program allows the FAA to used tailored approaches to address issues.

However, the watchdog expressed concern that no specific office or entity is tasked with overseeing the program and believes the FAA should put such oversight in place. “Based on our review of FAA documents and our prior review of FAA’s enforcement policy, FAA’s oversight used to be more centralized in the Enforcement Division, where all administrative and legal enforcement actions were tracked in a single database,” the agency said. “Now, as FAA emphasizes compliance actions over enforcement actions, tracking of actions is less centralized and occurs more in individual program offices.”

Without such centralized oversight, “no one is looking across program offices to identify or share any best practices or lessons learned,” the GAO said. “Individual offices have information that might benefit other offices. Further, officials from one FAA field office told us that the compliance program could be improved by better sharing information on its use across the agency.”

The watchdog added that FAA should collect and analyze data to monitor use across all program offices and evaluate the program to ensure its effectiveness.


Actions Needed to Evaluate Changes to FAA’s Enforcement Policy on Safety Standards

Heather Krause , GAO Director, Physical Infrastructure Issues,

Heather Krause[1], GAO Director, Physical Infrastructure Issues, signed a 47 page report on the FAA’ Enforcement Policy and as usual, the document is supported by many statistical analyses of the agency’s enforcement actions since its 2005 implementation of its Safety Management System and Compliance initiative.

compliance-v-enforcement

With standard GAO precision, Ms. Krause’s paper quantifies trends in FAA formal enforcement actions, administrative actions, and compliance resolutions.

 

 

 

 

 

At p.16 of the paper this conclusion is made:

Our analysis of FAA data found that the total number of enforcement actions, counted by the fiscal year the actions were closed, decreased across the agency under the Compliance Program. At the same time, the number of compliance actions closed for three selected program offices has increased since the start of the Compliance Program (see fig. 4 THE COVER PICTURE of this post). According to FAA officials, this shift aligns with what FAA officials expected would happen under the Compliance Program.

Oddly enough, the stated purpose of the SMS and Compliance Initiative (Order 8900.1, Flight Standards Information Management System, Volume 14, Compliance and Enforcement; FAA Order 8000.373, Federal Aviation Administration Compliance Philosophy, and FAA Order 2150.3B, FAA Compliance and Enforcement Program) is to remove the threat of civil penalties (an ineffective means of encourage safety enhancement?). Instead, relying on communications, collaboration and development of proactive solutions, the Compliance Program strives to reduce risks posed by certificate safety risks identified by the data collected on a SPECIFIC CARRIER’S OPERATIONS.

The GAO seems to infer from its conclusions that there is no proof that Compliance is working, yet the report (p.29-30) includes this quote:

FAA officials stated that the Compliance Program has contributed to safety improvements since it began in 2015. According to FAA officials, the agency now has a wider view into regulated entities since the Compliance Program encourages transparency and information sharing between FAA and regulated entities. As a result, they said FAA has an expanded ability to make aviation safer. FAA officials also pointed to a continued decrease in fatalities and fatal accidents in the aviation industry. Although FAA officials acknowledged that the Compliance Program is not solely responsible for this decrease, they believe it plays a large role. FAA officials also stated that they see regulated entities participating more in identifying the root cause of a violation and then taking action to address the issue under the Compliance Program.

Officials from most industry associations and organizations we interviewed also expressed support or positive feedback for the Compliance Program. Officials from 8 of the 13 industry associations and organizations we interviewed said they believe the Compliance Program has been effective in improving compliance with safety standards and improving transparency with industry. For example, officials from one industry association said the Compliance Program allows FAA to address violations using tailored approaches that address the underlying issues and aim to prevent any repeat issues. Officials from some regulated entities we interviewed also told us there is more collaboration with FAA under the Compliance Program, since entities know they may not be fined or face an enforcement action.

Other examples of third party support of the safety benefits of the Compliance Program:

Strong Evidence That SMS/ASIAS/CAST/Compliance Is Working

Flight Safety Foundation Praises FAA’s & CASA’s New Safety Compliance Approach; Impact?

First, no one has claimed that the SMS/ Compliance policy would eradicate errors. The literature on this approach makes it clear that the goal is to reduce risk. All committed to safety culture seek on a daily basis ZERO errors, but all acknowledge that humans and machines are not perfect.

Second, the auditors should also note that the origins of the alleged problems at Allegiant Air, American and Southwest may predate their implementation of SMS. It is also possible, maybe even likely, that these carriers have not fully inculcated the culture which is needed to make SMS truly functional. There may be no causal relationship between SMS/Compliance and the problems at issue.

Third, while a review may find that the SMS preventative data function may have missed the relevant precursors to the fault. It may be tempting to categorically condemn this innovative, but highly regarded regulatory regimen.

  • Glacier-like process of the issuance of regulations – the cumulative impact of the basic requirements of the Administrative Procedure Act plus multiple collateral statutes (like a Small Business Impact Statement) require significant time to draft and review a proposal. Then reviews, both within the FAA (policy, economics, legal) and above it (DOT, OMB), cause additional  major delays. The gestation period from initial conceptualization to final implementation can consume as much as five to ten years. It is difficult, if not impossible, for the FAA to keep up with industry changes, the identification of new technical problems and new technology.
  • For twenty or more years, the budget and staffing of the FAA have been reduced. Whether it is true or not, the internal perception is that the FAA’s capacity to regulate, following the modus operandi of the past, is less than its existing safety agenda. The assignment of inspectors, for example, to the P&W powerplants of one airline is no longer possible. Congress has sent the message that the FAA must do more with less.
  • Consistency – the pattern of independent, inconsistent application of policy in the field was found by the OIG, GAOindustry and even an FAA ARC. That equates to different interpretations being applied to an airline depending on what ASI explained her/his view of the FARs as well as among airlines, on a macro level. This lack of consistency has subjected the Flight Standards organization to substantial criticism, to which senior management agreed.
  • The press and the NTSB frequently criticized the FAA that it reacted to accidents; that their revisions to rules made it a tombstone organization. Upon reflection, FAA senior management decided that it needed to be more proactive in its approach. An initiative began to find techniques in which the regulator could “get out in front” of safety risks.
  • ICAO and SMS—this UN organization decided that all civil aviation authorities should implement this state-of-the-art safety approach. Its basis is the analysis of meta data bases from shared airline information , extrapolating significant trends, assessing solutions, cooperative/consensus processes and quick implementation of preventative solutions. The cooperation element of SMS, which includes unions, the FAA and management, requires an open, trusting environment.
  • The experience with SMS caused the FAA to realize that a punitive/enforcement tactic and a cooperative compliance approach were antithetical; so, the Administrator issued a new Compliance Policy.

 

The GAO has focused on the easy to track numbers—cases opened and closed. The report failed to assess the real benefit of Compliance as mentioned in response to an auditor’s question:

Program allows FAA to address violations using tailored approaches that address the underlying issues and aim to prevent any repeat issues.”

The true brilliance of the SMS/Compliance approach is that risks are addressed BASED ON THE STRENGTHS AND WEAKNESSES of the specific carrier. FAR 121 and related rules are intended (for example)

to cover a massive airline with a global route structure and a small carrier,

to provide training rules which exact the same regimen to a pilot cadre of experienced, well educated captains and to a brand new airline;

to a maintenance organization full of labor strife and poor management and to an airline with an incredible safety culture…

The old regulatory regime was based on one size fits all.

The development of these specific solutions creates a new regulatory approach, labeled “individuation”:

SMS Future INDIVIDUATION And The RRTF’s SLUDGE Removal

individuation

To measure the effectiveness of the SMS Compliance Program– as opposed to the Enforcement and Informal Action actionsGAO should have assessed the risk reductions which represent the targeted outcome of this ICAO inspired, state-of-the-art, world recognized safety regimen– specific solution to higher risks. It is not entirely inappropriate to cite the macro safety trends mentioned above

The GAO report recommends that some FAA-wide coordination in counting the compliance actions. A better idea might be to have all of the principals involved in SMS come together and have them share some new, innovative solution designed to address  a specific problem.

violation to solution



 

 

 

 

 

[1] Heather Krause is a Director in GAO’s Physical Infrastructure team. She is one of GAO’s experts on the safety and operation of the national airspace system, including the integration of drones and commercial spaceflight into the national airspace.

Within GAO, Heather has worked in the Physical Infrastructure and Strategic Issues mission teams and previously led GAO’s duplication and cost savings work.

Heather joined GAO in 2003. She holds a bachelor of arts in political science from the University of Minnesota Duluth and a master’s degree in Public Policy from the University of Minnesota.

 

Share this article: FacebooktwitterlinkedinFacebooktwitterlinkedin

Be the first to comment on "GAO quant jocks missed the real safety benefit of SMS Compliance"

Leave a comment

Your email address will not be published.