FAA meltdown injected uncertainty, including for military pilots
The Washington Post January 14, 2023
After two Federal Aviation Administration contractors damaged an agency database, oddities started appearing in the technological tools that pilots — including those in the U.S. military — use to fly around the country safely, according to federal documents and interviews.
One of the systems, the Defense Internet NOTAM Service (DINS), usually includes a stream of FAA alerts on potential flight hazards. But by early Wednesday morning, those FAA Notices to Air Missions started appearing to military pilots in duplicate — or not at all. The ability for military users to retrieve the FAA warnings via the system became “impaired and unreliable,” according to an FAA technical bulletin obtained by The Washington Post.
Air Force pilots wound up calling on their own to “fill in the blanks” with information from areas they were flying into, said Rose Riley, an Air Force spokeswoman. “If they didn’t have everything they needed, they would call Air Traffic Control or they would call installation to installation,” she said. The military’s own NOTAMs system remained fully functional, she added. The issue was the missing FAA information.
Military pilots are authorized to fly in such circumstances, but it was far from the norm, Riley said.
“In talking to our operations team, it’s the first time they could recall that the FAA NOTAMs stopped completely for a period of time,” she said.
The FAA’s meltdown cascaded through the nation’s airspace, with the agency’s technical bulletin pointing to 19 government websites and applications potentially affected. It said processing of information about NOTAMs, flight plans, weather and some areas with flight limitations had been affected. The incident appears to have been the result of a mistake rather than anything malicious, according to a person familiar with the review into the outage, who added that information was still being verified.
The disruption delayed almost 11,000 flights, led to cancellations of more than 1,300 more and raised questions about the reliability of the sometimes-antiquated computer systems meant to keep the nation’s skies safe for travelers. It also underscored the FAA’s continued struggles to fix known problems.
In a bipartisan letter Friday to Transportation Secretary Pete Buttigieg, Reps. Sam Graves (R-Mo.) and Rick Larsen (D-Wash.) the leaders of the House Transportation Committee, pointed to a line in the department’s most recent budget request that described the NOTAMs system as relying on “failing vintage hardware” and seeking $30 million to fund upgrades.
“This shows the FAA was well aware of the issues facing the NOTAM system,” the lawmakers wrote in the letter, which was signed by 122 House members. “The failure to improve legacy systems is unacceptable, and the American people expect and deserve better.”
The Transportation Department didn’t immediately respond to a request for comment Friday evening.
The NOTAMs system distributes safety messages about abnormal conditions that pilots might encounter. Parts of the system that failed are 30 years old.
It began to break down Tuesday afternoon, and the FAA initially tried switching to a backup system, but the situation continued to deteriorate into the evening. The agency issued a bulletin warning that the system was down and continued to work on a fix. But by Wednesday morning, officials had determined they needed to reset the system.
The FAA ordered a rare nationwide ground stop shortly after 7 a.m. Wednesday, all but bringing commercial air travel to a standstill for about 90 minutes, wreaking havoc on the nation’s aviation system shortly after a technical failure at Southwest Airlines ahead of the Christmas rush left travelers stranded.
The FAA technical bulletin describes details of what it refers to as the “NAIMES Service Interruption,” using the acronym for National Airspace Aeronautical Information Enterprise System.
The bulletin said problems stemmed from “an observed production database failure.” One consequence of the breakdown was that all NOTAMs put in the system, for about a 24-hour period starting Tuesday afternoon, “have been removed from the system and must be reissued.”
The bulletin said there was outreach to entities who submit NOTAMs — including airports, air traffic control workers and others — encouraging them to “reissue all applicable NOTAMs for their facility and to review all NOTAMs issued in the last 24 hours to ensure NOTAM data accuracy.”
The FAA did not say if those with crucial information had reentered the relevant hazards or whether risks had been introduced into the system.
“Pilots can contact airport and military airfield towers and other FAA air traffic en route centers for NOTAM information if necessary,” the FAA said in a statement.
The bulletin also said there had been “intermittent delay in response” with Federal NOTAM System applications through at least 11 a.m. Thursday. While it said the system was back up and running, some challenges continued due to “high system load.”
Peter Aiken, an associate professor of information systems at Virginia Commonwealth University, said the fact that FAA personnel could damage a file and take down the nation’s aviation system points to “a weakness in the understanding of data issues in general.”
“Databases, just like automobiles, need regular maintenance,” Aiken said. “You do need an experienced mechanic to do that maintenance.”
Aiken said the FAA’s reliance on outside contractors can create communications challenges between government and outside employees responsible for systems upkeep, and noted that federal agencies such as the FAA are struggling to upgrade their legacy systems.
The FAA said its review of what went wrong is ongoing. The agency is also in the midst of a years-long effort to modernize the NOTAMs system.
Ed Bastian, chief executive of Delta Air Lines, said Friday that Wednesday’s breakdown is further evidence of what happens when the industry grows but investments in the systems designed to manage that growth are inconsistent.
“It’s very clear there has to be a call to action among our political leaders in Congress and at the White House to fund and properly provide the FAA with the resources they need to do their job,” Bastian said during a call with analysts. “This is a crystal clear example of the challenges the FAA has faced.”
Failure to invest, particularly as the national airspace becomes more congested, could lead to more delays and disruption, he said, hampering the industry’s ability to grow.
Meanwhile, scrutiny on the technological mishaps of another major domestic air carrier continued.
In a letter Friday to Bob Jordan, Southwest’s chief executive, more than a dozen Democratic lawmakers, including Sens Edward Markey (Mass.) and Richard Blumenthal (Conn.) pressed for answers on how the carrier would prevent a similar meltdown in the future.
“Although winter storm Elliott disrupted flights across the country, every other airline operating in the United States managed to return to a regular flight schedule shortly thereafter - except Southwest,” the lawmakers wrote. “Southwest must take all necessary steps to ensure that this debacle never happens again.”
The senators also questioned Jordan about the company’s decision last month to resume paying dividends to stockholders. As a condition of accepting federal dollars to keep workers on the job, airlines were barred from raising executive compensation, repurchasing stock or issuing dividends.
That prohibition expired in September, and last month Southwest was the first carrier to announce it would resume paying quarterly dividends. Southwest is expected to issue a $428 million dividend to shareholders at the end of January.