More on the SolarWinds Breach

The New York Times has more details.

About 18,000 private and government users downloaded a Russian tainted software update –­ a Trojan horse of sorts ­– that gave its hackers a foothold into victims’ systems, according to SolarWinds, the company whose software was compromised.

Among those who use SolarWinds software are the Centers for Disease Control and Prevention, the State Department, the Justice Department, parts of the Pentagon and a number of utility companies. While the presence of the software is not by itself evidence that each network was compromised and information was stolen, investigators spent Monday trying to understand the extent of the damage in what could be a significant loss of American data to a foreign attacker.

It’s unlikely that the SVR (a successor to the KGB) penetrated all of those networks. But it is likely that they penetrated many of the important ones. And that they have buried themselves into those networks, giving them persistent access even if this vulnerability is patched. This is a massive intelligence coup for the Russians and failure for the Americans, even if no classified networks were touched.

Meanwhile, CISA has directed everyone to remove SolarWinds from their networks. This is (1) too late to matter, and (2) likely to take many months to complete. Probably the right answer, though.

This is almost too stupid to believe:

In one previously unreported issue, multiple criminals have offered to sell access to SolarWinds’ computers through underground forums, according to two researchers who separately had access to those forums.

One of those offering claimed access over the Exploit forum in 2017 was known as “fxmsp” and is wanted by the FBI “for involvement in several high-profile incidents,” said Mark Arena, chief executive of cybercrime intelligence firm Intel471. Arena informed his company’s clients, which include U.S. law enforcement agencies.

Security researcher Vinoth Kumar told Reuters that, last year, he alerted the company that anyone could access SolarWinds’ update server by using the password “solarwinds123”

“This could have been done by any attacker, easily,” Kumar said.

Neither the password nor the stolen access is considered the most likely source of the current intrusion, researchers said.

That last sentence is important, yes. But the sloppy security practice is likely not an isolated incident, and speaks to the overall lack of security culture at the company.

And I noticed that SolarWinds has removed its customer page, presumably as part of its damage control efforts. I quoted from it. Did anyone save a copy?

EDITED TO ADD: Both the Wayback Machine and Brian Krebs have saved the SolarWinds customer page.

Another Massive Russian Hack of US Government Networks

Another Massive Russian Hack of US Government Networks

The press is reporting a massive hack of US government networks by sophisticated Russian hackers.

Officials said a hunt was on to determine if other parts of the government had been affected by what looked to be one of the most sophisticated, and perhaps among the largest, attacks on federal systems in the past five years. Several said national security-related agencies were also targeted, though it was not clear whether the systems contained highly classified material.


The motive for the attack on the agency and the Treasury Department remains elusive, two people familiar with the matter said. One government official said it was too soon to tell how damaging the attacks were and how much material was lost, but according to several corporate officials, the attacks had been underway as early as this spring, meaning they continued undetected through months of the pandemic and the election season.

The attack vector seems to be a malicious update in SolarWinds’ “Orion” IT monitoring platform, which is widely used in the US government (and elsewhere).

SolarWinds’ comprehensive products and services are used by more than 300,000 customers worldwide, including military, Fortune 500 companies, government agencies, and education institutions. Our customer list includes:

  • More than 425 of the US Fortune 500
  • All ten of the top ten US telecommunications companies
  • All five branches of the US Military
  • The US Pentagon, State Department, NASA, NSA, Postal Service, NOAA, Department of Justice, and the Office of the President of the United States
  • All five of the top five US accounting firms
  • Hundreds of universities and colleges worldwide

I’m sure more details will become public over the next several weeks.

Sidebar photo of Bruce Schneier by Joe MacInnis.

FireEye Hacked

FireEye Hacked

FireEye was hacked by — they believe — “a nation with top-tier offensive capabilities”:

During our investigation to date, we have found that the attacker targeted and accessed certain Red Team assessment tools that we use to test our customers’ security. These tools mimic the behavior of many cyber threat actors and enable FireEye to provide essential diagnostic security services to our customers. None of the tools contain zero-day exploits. Consistent with our goal to protect the community, we are proactively releasing methods and means to detect the use of our stolen Red Team tools.

We are not sure if the attacker intends to use our Red Team tools or to publicly disclose them. Nevertheless, out of an abundance of caution, we have developed more than 300 countermeasures for our customers, and the community at large, to use in order to minimize the potential impact of the theft of these tools.

We have seen no evidence to date that any attacker has used the stolen Red Team tools. We, as well as others in the security community, will continue to monitor for any such activity. At this time, we want to ensure that the entire security community is both aware and protected against the attempted use of these Red Team tools. Specifically, here is what we are doing:

  • We have prepared countermeasures that can detect or block the use of our stolen Red Team tools.
  • We have implemented countermeasures into our security products.
  • We are sharing these countermeasures with our colleagues in the security community so that they can update their security tools.
  • We are making the countermeasures publicly available on our GitHub.
  • We will continue to share and refine any additional mitigations for the Red Team tools as they become available, both publicly and directly with our security partners.

Consistent with a nation-state cyber-espionage effort, the attacker primarily sought information related to certain government customers. While the attacker was able to access some of our internal systems, at this point in our investigation, we have seen no evidence that the attacker exfiltrated data from our primary systems that store customer information from our incident response or consulting engagements, or the metadata collected by our products in our dynamic threat intelligence systems. If we discover that customer information was taken, we will contact them directly.

From the New York Times:

The hack was the biggest known theft of cybersecurity tools since those of the National Security Agency were purloined in 2016 by a still-unidentified group that calls itself the ShadowBrokers. That group dumped the N.S.A.’s hacking tools online over several months, handing nation-states and hackers the “keys to the digital kingdom,” as one former N.S.A. operator put it. North Korea and Russia ultimately used the N.S.A.’s stolen weaponry in destructive attacks on government agencies, hospitals and the world’s biggest conglomerates ­- at a cost of more than $10 billion.

The N.S.A.’s tools were most likely more useful than FireEye’s since the U.S. government builds purpose-made digital weapons. FireEye’s Red Team tools are essentially built from malware that the company has seen used in a wide range of attacks.

Russia is presumed to be the attacker.

Reuters article. Boing Boing post. Slashdot thread.

Russian media group Rambler attempting to hold Nginx hostage

Stock photo of empty jail cell.

Enlarge / This listing image is slightly hyperbolic—Nginx co-founders Sosoev and Konovalov didn’t do time in jail, they were “just” detained and interrogated at gunpoint in their homes at 7am local time.

Maxim Konovalov and Igor Sysoev—founders and creators of the popular Web server software Nginx—were arrested, detained, and interrogated last Thursday. Sysoev’s former employer, Rambler—Russia’s third-largest Internet company, which occupies a roughly similar position in Russian-language Internet to Yahoo or AOL at their height in the English-speaking world—alleged that it owned the rights to Nginx’s source code, due to Sysoev having originally developed it while an employee at Rambler.

In an interview with—a news site focusing on Russian and former Soviet Union reporting—founder Konovalov decried Rambler’s move as “a typical racket, simple as that,” and he went on to state that no attempt had been made to negotiate with or even notify him or Sysoev before the raid happened. Their first indication of a problem came with the police raids which detained the two, seized IT equipment from them, and interrogated them early that morning. Konovalov described the raid as “professional and polite, if you exclude the fact that special forces agents were standing around with automatic weapons… then there were interrogations. Generally speaking, the questions weren’t particularly interesting or pleasant.”

Konovalov characterized the move as a money-grabbing shakedown from the current leadership at Rambler, inspired by Nginx’s $670 million acquisition by American tech giant F5 Networks approximately six months earlier.

He told Meduza:

Nginx was officially registered in 2011, and it’s now 2019, and in all this time Rambler never raised any issues… there was the deal with F5, the big money became palpable, and then we see the desire to grab a piece of it for themselves. It’s a typical racket. Simple as that.

Konovalov and Sysoev were not even certain what criminal charges were filed against them. But earlier today, Rambler requested the Russian courts to drop the criminal charges and instead turned to civil litigation. This follows Konovalov’s earlier prediction that the criminal charges were merely being used as an excuse to go on a fishing expedition for leverage to use in a civil case. Rambler further claimed it was cutting ties with the “Lynwood” law firm which had filed criminal charges; but this seems likely to be a move for show only, since Lynwood Investments is tied to Alexander Mamut—a Russian billionaire who is co-owner of Rambler itself.

A simple cash grab?

Although Nginx co-founder Konovalov characterizes the move by Rambler as a simple cash grab inspired by Nginx’s $670 million acquisition, the potential ramifications are far wider-reaching than ~42 billion rubles in cold hard cash. A successful, retroactive acquisition of the rights to Nginx would not just give Rambler access to that cash—it would also provide the ability to declare the entire open source license of the Nginx platform invalid.

This would, in turn, open up effectively the entire developed world’s tech industry to shakedowns for licensing fees—both for continued operation, and in theory, retroactively for more than a decade of “unlicensed” usage.

Since the Nginx license was a weak, permissive license—largely akin to the BSD license, requiring nothing but acknowledgement of the original copyright notice in source code and documentation—Nginx has not just proliferated directly as a Web server used on general purpose computers but also as a key embedded component of many other solutions. For instance, Symantec’s Blue Coat appliances, Sophos’ Email Appliances, and Netflix’s Open Connect Appliances all depend on Nginx.

Moving back to “simple” software deployments, UK Internet services company Netcraft lists Nginx as the single-most common Internet-facing Web server on the planet in its Q3 2019 Web server survey, with more than 31 percent of all sites surveyed detected as Nginx. Filtering to only “active” sites seemingly reduces Nginx to the second-most common server, with Apache at 30 percent and Nginx at 20 percent. But this conveniently ignores a whopping 37 percent of “other” results, representing Web servers locked down in production too tightly to be easily classified. Many of those “other” servers will also be Nginx or Nginx derivatives.

As of December 2019, Nginx is even more popular than Apache. Netcraft confirms it.

Enlarge / As of December 2019, Nginx is even more popular than Apache. Netcraft confirms it.

If Russian courts were to grant a civil victory to Rambler and award it ownership of the rights to Nginx, the sweeping impact on the entire global technical industry is difficult even to estimate. A simple self-hosted blog might be able to swap out Nginx for Apache in a few hours. A more complex and heavily optimized site, designed to field a lot of traffic, might get back on its feet nearly as quickly but operate at reduced capacity for a week.

Meanwhile, the industry giants which depend on Nginx include Facebook, Netflix, and WordPress. Add in Cloudflare‘s Content Distribution Network and DDoS protection service, and it becomes easier to discuss what portion of the Internet wouldn’t stop working without Nginx than which ones would.

It seems difficult to believe that this fact is lost on the Rambler executives who initiated this grab. But it also seems difficult to believe that the rest of the world would tolerate it and honor a Russian-court decision with such far-ranging effects. Adding to the already ham-handed obviousness of the grab—which comes more than a decade after Nginx established itself as both a service company and a significant part of the global Internet infrastructure—Igor Ashmanov, a Rambler chief executive from the time Sysoev worked at the company, declared on Facebook that “developing software wasn’t part of [Sysoev’s] job description at all,” and “Rambler [probably can’t] come up with a single piece of paper, never mind a non-existent task to develop a web server.”

This author believes that it would be difficult to find a court outside Russia’s direct control that would issue injunctions based on such a decision which would necessarily bind the entire visible Internet from operation. As dark as politics has become, I believe sanctioning corruption this immediately and obviously visible and damaging to both tech industry giants and everyday citizens—No cat memes today? No pictures of each others’ lunches? Sacrilege!—would represent immediate political suicide no elected official would likely believe they could ignore.

New clues show how Russia’s grid hackers aimed for physical destruction

Transmission lines.

Joshua Lott/Bloomberg via Getty Images

For nearly three years, the December 2016 cyberattack on the Ukrainian power grid has presented a menacing puzzle. Two days before Christmas that year, Russian hackers planted a unique specimen of malware in the network of Ukraine’s national grid operator, Ukrenergo. Just before midnight, they used it to open every circuit breaker in a transmission station north of Kyiv. The result was one of the most dramatic attacks in Russia’s years-long cyberwar against its western neighbor, an unprecedented, automated blackout across a broad swath of Ukraine’s capital.

But an hour later, Ukrenergo’s operators were able to simply switch the power back on again. Which raised the question: Why would Russia’s hackers build a sophisticated cyberweapon and plant it in the heart of a nation’s power grid only to trigger a one-hour blackout?

A new theory offers a potential answer. Researchers at the industrial-control system cybersecurity firm Dragos have reconstructed a timeline of the 2016 blackout attack [PDF] based on a reexamination of the malware’s code and network logs pulled from Ukrenergo’s systems. They say that hackers intended not merely to cause a short-lived disruption of the Ukrainian grid but to inflict lasting damage that could have led to power outages for weeks or even months. That distinction would make the blackout malware one of only three pieces of code ever spotted in the wild aimed at not just disrupting physical equipment but destroying it, as Stuxnet did in Iran in 2009 and 2010 and as the malware Triton was designed to do in a Saudi Arabian oil refinery in 2017.

In an insidious twist in the Ukrenergo case, Russia’s hackers apparently intended to trigger that destruction not at the time of the blackout itself but when grid operators turned the power back on, using the utility’s own recovery efforts against them.

“While this ended up being a direct disruptive event, the tools deployed and the sequence in which they were used strongly indicate that the attacker was looking to do more than turn the lights off for a few hours,” says Joe Slowik, a Dragos analyst who formerly led the Computer Security and Incident Response Team at the Department of Energy’s Los Alamos National Laboratory. “They were trying to create conditions that would cause physical damage to the transmission station that was targeted.”

Setting a trap

The Ukraine-targeted blackout malware, known alternately as Industroyer or Crash Override, grabbed the attention of the cybersecurity community when the Slovakian cybersecurity firm ESET first revealed it in June 2017. It featured a unique ability to directly interact with an electric utility’s equipment, including features that could send automated, rapid-fire commands in four different protocols used in various power utilities to open their circuit breakers and trigger mass power outages.

But the new Dragos findings relate instead to an often-forgotten component of the 2016 malware, described in ESET’s original analysis [PDF] but not fully understood at the time. That obscure component of the malware, ESET pointed out, looked like it was designed to take advantage of a known vulnerability in a piece of Siemens equipment known as a Siprotec protective relay. Protective relays act as electric grid fail-safes, monitoring for dangerous power frequencies or levels of current in electric equipment, relaying that information to operators and automatically opening circuit breakers if they detect dangerous conditions that could damage transformers, melt power lines, or in rare cases even electrocute workers. A security flaw in Siemens protective relays—for which the company had released a software fix in 2015 but which remained unpatched in many utilities—meant that any hackers who could send a single data packet to that device could essentially put it in a sleep state intended for firmware updates, rendering it useless until manually rebooted.

In 2017, ESET had noted the disturbing implications of that malware component; it hinted that Industroyer’s creators might be bent on physical damage. But it was far from clear how the Siprotec-hacking feature could have actually caused more lasting damage. After all, the hackers had merely turned off the power at Ukrenergo, not caused the sort of dangerous power surge that disabling a protective relay might exacerbate.

The Dragos analysis may provide that missing piece of the Ukrenergo puzzle. The company says it obtained the Ukrainian utility’s network logs from a government entity—it declined to name which one—and for the first time was able to reconstruct the order of the hackers’ operations. First, the attackers opened every circuit breaker in the transmission station, triggering the power outage. An hour later, they launched a wiper component that disabled the transmission station’s computers, preventing the utility’s staff from monitoring any of the station’s digital systems. Only then did the attackers use the malware’s Siprotec hacking feature against four of the station’s protective relays, intending to silently disable those fail-safe devices with almost no way for the utility’s operators to detect the missing safeguards.1

The intention, Dragos analysts now believe, was for the Ukrenergo engineers to respond to the blackout by hurriedly re-energizing the station’s equipment. By doing so manually, without the protective relay fail-safes, they could have triggered a dangerous overload of current in a transformer or power line. The potentially catastrophic damage would have caused far longer disruptions to the plant’s energy transmission than mere hours. It could also have harmed utility workers.

That plan ultimately failed. For reasons Dragos can’t quite explain—likely a networking configuration mistake the hackers made—the malicious data packets intended for Ukrenergo’s protective relays were sent to the wrong IP addresses. The Ukrenergo operators may have turned the power back on faster than the hackers expected, outracing the protective relay sabotage. And even if the Siprotec attacks had hit their marks, backup protective relays in the station might have prevented a disaster—though Dragos’s analysts say that without a full picture of Ukrenergo’s safety systems, they can’t entirely game out the potential consequences.

But Dragos Director of Threat Intelligence Sergio Caltagirone argues that regardless, the sequence of events represents a disturbing tactic that wasn’t recognized at the time. The hackers predicted the power utility operator’s reaction and tried to use it to amplify the cyberattack’s damage. “Their fingers are not over the button,” Caltagirone says of the blackout hackers. “They’ve pre-engineered attacks that harm the facility in a destructive and potentially life-threatening way when you respond to the incident. It’s the response that ultimately harms you.”

Appetite for destruction

The specter of physical destruction attacks on electric utilities has haunted grid cybersecurity engineers for more than a decade, since Idaho National Labs demonstrated in 2007 that it was possible to destroy a massive, 27-ton diesel generator simply by sending digital commands to the protective relay connected to it. The engineer who led those tests, Mike Assante, told WIRED in 2017 that the presence of a protective relay attack in the Ukrenergo malware, though not yet fully understood at the time, hinted that those destructive attacks might finally be becoming a reality. “This is definitely a big deal,” warned Assante, who passed away earlier this year. “If you ever see a transformer fire, they’re massive. Big black smoke that all of a sudden turns into a fireball.”

If the new Dragos theory of the 2016 blackout holds true, it would make the incident only one of three times when in-the-wild malware has been designed to trigger destructive physical sabotage. The first was Stuxnet, the US and Israeli malware that destroyed a thousand Iranian nuclear enrichment centrifuges roughly a decade ago. And then a year after the Ukrainian blackout, in late 2017, another piece of malware known as Triton or Trisis, discovered in the network of Saudi oil refinery Petro Rabigh, was revealed to have sabotaged so-called safety-instrumented systems, the devices that monitor for dangerous conditions in industrial facilities. That last cyberattack, since linked to Moscow’s Central Scientific Research Institute of Chemistry and Mechanics, merely shut down the Saudi plant. But it could have led to far worse outcomes, including deadly accidents like an explosion or gas leak.

What worries Caltagirone the most is how much time has passed since those events and what the world’s industrial-control-system hackers might have developed over those three years. “Between this and Trisis, we now have two data points showing a pretty significant disregard for human life,” Caltagirone says. “But it’s what we’re not seeing that’s the most dangerous thing out there.”