Hackers' monthslong head start hamstrings probe of US breach
By JORDAN ROBERTSON, KARTIKAY MEHROTRA AND WILLIAM TURTON | Bloomberg News | Published: December 19, 2020
WASHINGTON (Tribune News Service) — A vast number of investigations are underway inside federal agencies and private-sector companies to determine the extent of a broad cyber-attack by suspected Russian hackers, an effort that will likely spill well into the Biden administration before anyone can determine the full extent of the damage.
With the number of potential hacking victims in the thousands, a major challenge for investigators is determining which ones were the focus of a more targeted attack. For those that were actually hacked, figuring out what the attackers did while in their networks will be much more difficult. According to researchers and people familiar with the investigation so far, that access was in some cases lengthy and unfettered, carried out by hackers with the ability to cleverly masquerade as IT professionals who had legitimate reason to be poking around networks linking thousands of workstations.
The inquiry is complicated by numerous factors. The hackers were extraordinarily skilled and careful to cover their tracks. They roamed free in some networks in the U.S. government, and elsewhere, for as long as nine months. The attackers' method for infiltrating networks provided them with essentially an unfettered ability to do as they please.
"This is going to be a long ride," said Dmitri Alperovitch, the co-founder and former chief technology officer of the cybersecurity company Crowdstrike Inc. and now chairman of the Silverado Policy Accelerator. "We may never know the full scope of what happened here."
Investigators determined that the hackers inserted malicious code into software updates for a ubiquitous program used by IT administrators, affecting as many as 18,000 organizations directly. That's the number of times the maker of the software, Austin, Texas-based SolarWinds Corp., said customers accessed the tampered update.
A representative for SolarWinds didn't respond to requests for comment. The company acknowledged the complexity of the attack in a financial filing, saying it would take time to complete its own investigation.
The hard part isn't necessarily figuring out who those 18,000 software users are. Simply having downloaded the tampered versions of the software means an organization got infected. This assessment can take minutes.
The challenge is what comes next.
Cybersecurity company FireEye Inc.'s investigation into a breach of its own network led to the discovery of the broader attack. FireEye and Microsoft Corp., which aided in the investigation, have released extensive documentation detailing the specific files and internet addresses that organizations should look for in their network records to determine if the attackers entered through the door they opened. But that only tells an organization that they have a bigger problem.
The functionality of SolarWinds's Orion software complicates the process of trying to determine what — if anything — the hackers did on the network.
Orion is a software program used by IT administrators to monitor the health of their networks and push out software updates to computers. This is done in an automated way, eliminating the need for technicians to go machine by machine installing updates. As such, the Orion software acts as a sort of command center for an organization's IT department and interacts with other machines on the network. It uses the same permissions to change things as high-level administrators, according to the SolarWinds's website and a government contractor who has sold Orion software to many U.S. agencies. He requested anonymity to protect business relationships.
The network access gives the attacker the opportunity to steal the credentials of authorized users. For investigators to understand what the hackers may have accomplished, they have to first determine which footsteps to follow. At large organizations, sifting through network records from thousands of users can be an enormous challenge.
That has created a nightmare scenario for investigators and is one reason why the inquiries may take weeks, if not months, to unravel.
"The challenge of doing the forensics is you're going to be looking at logs of events with senior IT or global admin credentials, and then you have to figure out which ones are legit and which ones are attacker-related," said Eric Friedberg, co-president of cyber consultancy Stroz Friedberg.
Another complication is the skill of the hackers themselves, with U.S. officials noting their "sophistication and complex tradecraft" and private cybersecurity experts involved in unpacking the breaches describing them as among the most advanced they've seen. One described the manual process of hunting for evidence as "hell."
The probes are complex because investigators need to review a "massive" amount of data from historical network records including every interaction that infected SolarWinds's servers had with other machines on the network, according to Frank Downs, a former offensive analyst at the U.S. National Security Agency who is now director of proactive services at the cybersecurity company, BlueVoyant.
Investigators are not just looking for malware, which can be detected using automated tools in many cases, according to an intelligence community contractor working on multiple investigations related to the incident. Rather, the fear is the attackers could have made small changes to firewalls, network switches or other sensitive equipment that they could use to access networks in the future. Finding those changes may require manually reviewing those machines, the person said.
In some cases, according to Microsoft, hackers gained access to the systems at victim companies that manage user authentication. This allowed them to impersonate any user or account, including system administrators – the very people using the SolarWinds program for legitimate purposes. With that level of access, discerning credible actions from malicious ones is a meticulous task. Hackers can take any number of actions to hide their tracks, including creating accounts for themselves with total access.
The attack breached the computer networks of the departments of Homeland Security, Treasury, Commerce and State, according to people familiar with the matter. The Department of Energy said it was breached but was contained to business networks and didn't affect national security functions. FireEye says it is aware of "dozens" of victims that were targeted by these follow-on attacks.
It's not yet known what companies the hackers infiltrated.
The cybersecurity research company BitSight Technologies analyzed 260,000 organizations across 24 sectors to determine the prevalence of SolarWinds's Orion software, the tool used to infect victims. They discovered at least 14% of Fortune 1,000 companies use Orion. The software is most popular among technology, government and health care entities. Their findings don't indicate breaches but may help pinpoint sectors worthy of further investigation.
©2020 Bloomberg L.P.
Distributed by Tribune Content Agency, LLC