Boeing had 49 gaps in testing for its astronaut capsule before failed flight, independent review finds
By CHABELI CARRAZANA | Orlando Sentinel | Published: March 7, 2020
ORLANDO, Fla. (Tribune News Service) — An independent review of the decisions that led to a failed test of Boeing's Starliner astronaut capsule found systematic and widespread missteps in the legacy company's testing procedures and software development, prompting more NASA involvement in the agency's commercial human spaceflight program.
NASA has declared Boeing's mission a "high visibility close call" mishap, a low-level but fairly rare procedural decision that kicks off a process to ensure that the same problems are not prevalent elsewhere in the organization.
The review is now calling for 61 corrective actions to be undertaken by both Boeing and NASA in the coming months to stopper numerous shortfalls that led to an ill-fated mission for Starliner when it took off from the Space Coast in December.
At the conclusion of the process, which is expected to take several months, NASA and Boeing will announce whether they will require Boeing to repeat the December test, or whether the company will be cleared to proceed with a test mission carrying astronauts onboard.
Starliner's uncrewed demonstration mission late last year experienced three major issues midflight. Starliner first failed to perform a critical maneuver because the timer in the spacecraft was running 11 hours ahead, causing it to go into the wrong orbit. When teams tried to correct the problem manually, communications issues potentially caused by cell phone towers in the area blocked signals from reaching Starliner. Then, as teams hunted for other issues, they discovered another code error that could have caused the capsule to collide with its service module.
That problem was resolved before Starliner landed back in the New Mexico desert on Dec. 22, finishing an abbreviated mission that failed to see the capsule dock with the International Space Station.
"I think we can all agree that this was a close call," said Douglas Loverro, associate administrator for human exploration and operations at NASA. "We could have lost a spacecraft twice during this mission."
The reasons for the issues are grounded in software problems that went undetected by both Boeing and NASA.
Last week, Boeing admitted it failed to perform a full end-to-end integrated test with United Launch Alliance's Atlas V rocket, the vehicle that carries Starliner into space. Had Boeing completed the 25-hour-plus test in full, testing the two systems through every part of the launch, it would have caught the timing issue later experienced in flight.
Boeing also used a faulty emulator, which mimics hardware during testing, instead of the real hardware to test, causing it to miss the code issue that could have led to a collision.
In all, there were 49 gaps in testing, the company said during a media call on Friday.
"That doesn't correspond to 49 errors, but that is 49 gaps in testing that now our team will be able to go, perform that testing and identify whether there's any additional corrupt code. And if so, we'll be able to fix it," said John Mulholland, Boeing's vice president and program manager for Commercial Crew.
For example, there may have been four different ways the software could have run, and teams didn't test all four versions, Loverro said. That's fairly standard, but "we clearly recognize that we do need to go and test every logical condition," he said.
Too much decision-making authority was also given to Boeing's software board to approve changes, instead of bringing all major changes before an overall engineering review board.
"In any engineering organization that should have been brought up all the way to make sure that we coordinated that with them before those processes were approved," Loverro said.
NASA will also be adding a software lead to oversee Boeing's processes moving forward, as well as adding more software experts from the space agency to work with the company, said Kathy Lueders, the program manager for NASA's Commercial Crew Program. NASA plans to beef up Boeing's software and engineering review boards and provide independent software verification support.
Some of the recommendations will likely have a spillover effect to other programs. Lueders said NASA is also reevaluating the end-to-end testing conducted by SpaceX, the other contractor on the Commercial Crew Program that is but months away from its first crewed mission. NASA doesn't foresee the changes impacting SpaceX's schedule.
Boeing was initially awarded $4.2 billion as part of the Commercial Crew Program, while SpaceX got $2.6 billion to develop its capsule, Crew Dragon. SpaceX's test flight in March 2019 was successful without any major issues. A piloted mission could come as soon as the spring.
(c)2020 The Orlando Sentinel (Orlando, Fla.)
Visit The Orlando Sentinel at www.OrlandoSentinel.com
Distributed by Tribune Content Agency, LLC.