NSA

VMware Flaw a Vector in SolarWinds Breach?

U.S. government cybersecurity agencies warned this week that the attackers behind the widespread hacking spree stemming from the compromise at network software firm SolarWinds used weaknesses in other, non-SolarWinds products to attack high-value targets. According to sources, among those was a flaw in software virtualization platform VMware, which the U.S. National Security Agency (NSA) warned on Dec. 7 was being used by Russian hackers to impersonate authorized users on victim networks.

On Dec. 7, 2020, the NSA said “Russian state-sponsored malicious cyber actors are exploiting a vulnerability in VMware Access and VMware Identity Manager products, allowing the actors access to protected data and abusing federated authentication.”

VMware released a software update to plug the security hole (CVE-2020-4006) on Dec. 3, and said it learned about the flaw from the NSA.

The NSA advisory (PDF) came less than 24 hours before cyber incident response firm FireEye said it discovered attackers had broken into its networks and stolen more than 300 proprietary software tools the company developed to help customers secure their networks.

On Dec. 13, FireEye disclosed that the incident was the result of the SolarWinds compromise, which involved malicious code being surreptitiously inserted into updates shipped by SolarWinds for users of its Orion network management software as far back as March 2020.

In its advisory on the VMware vulnerability, the NSA urged patching it “as soon as possible,” specifically encouraging the National Security System, Department of Defense, and defense contractors to make doing so a high priority.

The NSA said that in order to exploit this particular flaw, hackers would already need to have access to a vulnerable VMware device’s management interface — i.e., they would need to be on the target’s internal network (provided the vulnerable VMware interface was not accessible from the Internet). However, the SolarWinds compromise would have provided that internal access nicely.

In response to questions from KrebsOnSecurity, VMware said it has “received no notification or indication that the CVE 2020-4006 was used in conjunction with the SolarWinds supply chain compromise.”

VMware added that while some of its own networks used the vulnerable SolarWinds Orion software, an investigation has so far revealed no evidence of exploitation.

“While we have identified limited instances of the vulnerable SolarWinds Orion software in our environment, our own internal investigation has not revealed any indication of exploitation,” the company said in a statement. “This has also been confirmed by SolarWinds own investigations to date.”

On Dec. 17, DHS’s Cybersecurity and Infrastructure Security Agency (CISA) released a sobering alert on the SolarWinds attack, noting that CISA had evidence of additional access vectors other than the SolarWinds Orion platform.

CISA’s advisory specifically noted that “one of the principal ways the adversary is accomplishing this objective is by compromising the Security Assertion Markup Language (SAML) signing certificate using their escalated Active Directory privileges. Once this is accomplished, the adversary creates unauthorized but valid tokens and presents them to services that trust SAML tokens from the environment. These tokens can then be used to access resources in hosted environments, such as email, for data exfiltration via authorized application programming interfaces (APIs).”

Indeed, the NSA’s Dec. 7 advisory said the hacking activity it saw involving the VMware vulnerability “led to the installation of a web shell and follow-on malicious activity where credentials in the form of SAML authentication assertions were generated and sent to Microsoft Active Directory Federation Services (ADFS), which in turn granted the actors access to protected data.”

Also on Dec. 17, the NSA released a far more detailed advisory explaining how it has seen the VMware vulnerability being used to forge SAML tokens, this time specifically referencing the SolarWinds compromise.

Asked about the potential connection, the NSA said only that “if malicious cyber actors gain initial access to networks through the SolarWinds compromise, the TTPs [tactics, techniques and procedures] noted in our December 17 advisory may be used to forge credentials and maintain persistent access.”

“Our guidance in this advisory helps detect and mitigate against this, no matter the initial access method,” the NSA said.

CISA’s analysis suggested the crooks behind the SolarWinds intrusion were heavily focused on impersonating trusted personnel on targeted networks, and that they’d devised clever ways to bypass multi-factor authentication (MFA) systems protecting networks they targeted.

The bulletin references research released earlier this week by security firm Volexity, which described encountering the same attackers using a novel technique to bypass MFA protections provided by Duo for Microsoft Outlook Web App (OWA) users.

Duo’s parent Cisco Systems Inc. responded that the attack described by Volexity didn’t target any specific vulnerability in its products. As Ars Technica explained, the bypass involving Duo’s protections could have just as easily involved any of Duo’s competitors.

“MFA threat modeling generally doesn’t include a complete system compromise of an OWA server,” Ars’ Dan Goodin wrote. “The level of access the hacker achieved was enough to neuter just about any defense.”

Several media outlets, including The New York Times and The Washington Post, have cited anonymous government sources saying the group behind the SolarWinds hacks was known as APT29 or “Cozy Bear,” an advanced threat group believed to be part of the Russian Federal Security Service (FSB).

SolarWinds has said almost 18,000 customers may have received the backdoored Orion software updates. So far, only a handful of customers targeted by the suspected Russian hackers behind the SolarWinds compromise have been made public — including the U.S. Commerce, Energy and Treasury departments, and the DHS.

No doubt we will hear about new victims in the public and private sector in the coming days and weeks. In the meantime, thousands of organizations are facing incredibly costly, disruptive and time-intensive work in determining whether they were compromised and if so what to do about it.

The CISA advisory notes the attackers behind the SolarWinds compromises targeted key personnel at victim firms — including cyber incident response staff, and IT email accounts. The warning suggests organizations that suspect they were victims should assume their email communications and internal network traffic are compromised, and rely upon or build out-of-band systems for discussing internally how they will proceed to clean up the mess.

“If the adversary has compromised administrative level credentials in an environment—or if organizations identify SAML abuse in the environment, simply mitigating individual issues, systems, servers, or specific user accounts will likely not lead to the adversary’s removal from the network,” CISA warned. “In such cases, organizations should consider the entire identity trust store as compromised. In the event of a total identity compromise, a full reconstitution of identity and trust services is required to successfully remediate. In this reconstitution, it bears repeating that this threat actor is among the most capable, and in many cases, a full rebuild of the environment is the safest action.”

NSA on Authentication Hacks (Related to SolarWinds Breach)

The NSA has published an advisory outlining how “malicious cyber actors” are “are manipulating trust in federated authentication environments to access protected data in the cloud.” This is related to the SolarWinds hack I have previously written about, and represents one of the techniques the SVR is using once it has gained access to target networks.

From the summary:

Malicious cyberactors are abusing trust in federated authentication environments to access protected data. The exploitation occurs after the actors have gained initial access to a victim’s on-premises network. The actors leverage privileged access in the on-premises environment to subvert the mechanisms that the organization uses to grant access to cloud and on-premises resources and/or to compromise administrator credentials with the ability to manage cloud resources. The actors demonstrate two sets of tactics, techniques,and procedures (TTP) for gaining access to the victim network’s cloud resources, often with a particular focus on organizational email.

In the first TTP, the actors compromise on-premises components of a federated SSO infrastructure and steal the credential or private key that is used to sign Security Assertion Markup Language (SAML) tokens(TA0006, T1552, T1552.004). Using the private keys, the actors then forge trusted authentication tokens to access cloud resources. A recent NSA Cybersecurity Advisory warned of actors exploiting a vulnerability in VMware Access and VMware Identity Manager that allowed them to perform this TTP and abuse federated SSO infrastructure.While that example of this TTP may have previously been attributed to nation-state actors, a wealth of actors could be leveraging this TTP for their objectives. This SAML forgery technique has been known and used by cyber actors since at least 2017.

In a variation of the first TTP, if the malicious cyber actors are unable to obtain anon-premises signing key, they would attempt to gain sufficient administrative privileges within the cloud tenant to add a malicious certificate trust relationship for forging SAML tokens.

In the second TTP, the actors leverage a compromised global administrator account to assign credentials to cloud application service principals (identities for cloud applications that allow the applications to be invoked to access other cloud resources). The actors then invoke the application’s credentials for automated access to cloud resources (often email in particular) that would otherwise be difficult for the actors to access or would more easily be noticed as suspicious (T1114, T1114.002).

This is an ongoing story, and I expect to see a lot more about TTP — nice acronym there — in coming weeks.

Related: Tom Bossert has a scathing op-ed on the breach. Jack Goldsmith’s essay is worth reading. So is Nick Weaver’s.

Michael Ellis as NSA General Counsel

Over at Lawfare, Susan Hennessey has an excellent primer on how Trump loyalist Michael Ellis got to be the NSA General Counsel, over the objections of NSA Director Paul Nakasone, and what Biden can and should do about it.

While important details remain unclear, media accounts include numerous indications of irregularity in the process by which Ellis was selected for the job, including interference by the White House. At a minimum, the evidence of possible violations of civil service rules demand immediate investigation by Congress and the inspectors general of the Department of Defense and the NSA.

The moment also poses a test for President-elect Biden’s transition, which must address the delicate balance between remedying improper politicization of the intelligence community, defending career roles against impermissible burrowing, and restoring civil service rules that prohibit both partisan favoritism and retribution. The Biden team needs to set a marker now, to clarify the situation to the public and to enable a new Pentagon general counsel to proceed with credibility and independence in investigating and potentially taking remedial action upon assuming office.

The NSA general counsel is not a Senate-confirmed role. Unlike the general counsels of the CIA, Pentagon and Office of the Director of National Intelligence (ODNI), all of which require confirmation, the NSA’s general counsel is a senior career position whose occupant is formally selected by and reports to the general counsel of the Department of Defense. It’s an odd setup — ­and one that obscures certain realities, like the fact that the NSA general counsel in practice reports to the NSA director. This structure is the source of a perennial legislative fight. Every few years, Congress proposes laws to impose a confirmation requirement as more appropriately befits an essential administration role, and every few years, the executive branch opposes those efforts as dangerously politicizing what should be a nonpolitical job.

While a lack of Senate confirmation reduces some accountability and legislative screening, this career selection process has the benefit of being designed to eliminate political interference and to ensure the most qualified candidate is hired. The system includes a complex set of rules governing a selection board that interviews candidates, certifies qualifications and makes recommendations guided by a set of independent merit-based principles. The Pentagon general counsel has the final call in making a selection. For example, if the panel has ranked a first-choice candidate, the general counsel is empowered to choose one of the others.

Ryan Goodman has a similar article at Just Security.

The NSA is Refusing to Disclose its Policy on Backdooring Commercial Products

Senator Ron Wyden asked, and the NSA didn’t answer:

The NSA has long sought agreements with technology companies under which they would build special access for the spy agency into their products, according to disclosures by former NSA contractor Edward Snowden and reporting by Reuters and others.

These so-called back doors enable the NSA and other agencies to scan large amounts of traffic without a warrant. Agency advocates say the practice has eased collection of vital intelligence in other countries, including interception of terrorist communications.

The agency developed new rules for such practices after the Snowden leaks in order to reduce the chances of exposure and compromise, three former intelligence officials told Reuters. But aides to Senator Ron Wyden, a leading Democrat on the Senate Intelligence Committee, say the NSA has stonewalled on providing even the gist of the new guidelines.

[…]

The agency declined to say how it had updated its policies on obtaining special access to commercial products. NSA officials said the agency has been rebuilding trust with the private sector through such measures as offering warnings about software flaws.

“At NSA, it’s common practice to constantly assess processes to identify and determine best practices,” said Anne Neuberger, who heads NSA’s year-old Cybersecurity Directorate. “We don’t share specific processes and procedures.”

Three former senior intelligence agency figures told Reuters that the NSA now requires that before a back door is sought, the agency must weigh the potential fallout and arrange for some kind of warning if the back door gets discovered and manipulated by adversaries.

The article goes on to talk about Juniper Networks equipment, which had the NSA-created DUAL_EC PRNG backdoor in its products. That backdoor was taken advantage of by an unnamed foreign adversary.

Juniper Networks got into hot water over Dual EC two years later. At the end of 2015, the maker of internet switches disclosed that it had detected malicious code in some firewall products. Researchers later determined that hackers had turned the firewalls into their own spy tool here by altering Juniper’s version of Dual EC.

Juniper said little about the incident. But the company acknowledged to security researcher Andy Isaacson in 2016 that it had installed Dual EC as part of a “customer requirement,” according to a previously undisclosed contemporaneous message seen by Reuters. Isaacson and other researchers believe that customer was a U.S. government agency, since only the U.S. is known to have insisted on Dual EC elsewhere.

Juniper has never identified the customer, and declined to comment for this story.

Likewise, the company never identified the hackers. But two people familiar with the case told Reuters that investigators concluded the Chinese government was behind it. They declined to detail the evidence they used.

Okay, lots of unsubstantiated claims and innuendo here. And Neuberger is right; the NSA shouldn’t share specific processes and procedures. But as long as this is a democratic country, the NSA has an obligation to disclose its general processes and procedures so we all know what they’re doing in our name. And if it’s still putting surveillance ahead of security.

NSA Advisory on Chinese Government Hacking

NSA Advisory on Chinese Government Hacking

The NSA released an advisory listing the top twenty-five known vulnerabilities currently being exploited by Chinese nation-state attackers.

This advisory provides Common Vulnerabilities and Exposures (CVEs) known to be recently leveraged, or scanned-for, by Chinese state-sponsored cyber actors to enable successful hacking operations against a multitude of victim networks. Most of the vulnerabilities listed below can be exploited to gain initial access to victim networks using products that are directly accessible from the Internet and act as gateways to internal networks. The majority of the products are either for remote access (T1133) or for external web services (T1190), and should be prioritized for immediate patching.

Sidebar photo of Bruce Schneier by Joe MacInnis.

Google Responds to Warrants for “About” Searches

One of the things we learned from the Snowden documents is that the NSA conducts “about” searches. That is, searches based on activities and not identifiers. A normal search would be on a name, or IP address, or phone number. An about search would something like “show me anyone that has used this particular name in a communications,” or “show me anyone who was at this particular location within this time frame.” These searches are legal when conducted for the purpose of foreign surveillance, but the worry about using them domestically is that they are unconstitutionally broad. After all, the only way to know who said a particular name is to know what everyone said, and the only way to know who was at a particular location is to know where everyone was. The very nature of these searches requires mass surveillance.

The FBI does not conduct mass surveillance. But many US corporations do, as a normal part of their business model. And the FBI uses that surveillance infrastructure to conduct its own about searches. Here’s an arson case where the FBI asked Google who searched for a particular street address:

Homeland Security special agent Sylvette Reynoso testified that her team began by asking Google to produce a list of public IP addresses used to google the home of the victim in the run-up to the arson. The Chocolate Factory [Google] complied with the warrant, and gave the investigators the list. As Reynoso put it:

On June 15, 2020, the Honorable Ramon E. Reyes, Jr., United States Magistrate Judge for the Eastern District of New York, authorized a search warrant to Google for users who had searched the address of the Residence close in time to the arson.

The records indicated two IPv6 addresses had been used to search for the address three times: one the day before the SUV was set on fire, and the other two about an hour before the attack. The IPv6 addresses were traced to Verizon Wireless, which told the investigators that the addresses were in use by an account belonging to Williams.

Google’s response is that this is rare:

While word of these sort of requests for the identities of people making specific searches will raise the eyebrows of privacy-conscious users, Google told The Register the warrants are a very rare occurrence, and its team fights overly broad or vague requests.

“We vigorously protect the privacy of our users while supporting the important work of law enforcement,” Google’s director of law enforcement and information security Richard Salgado told us. “We require a warrant and push to narrow the scope of these particular demands when overly broad, including by objecting in court when appropriate.

“These data demands represent less than one per cent of total warrants and a small fraction of the overall legal demands for user data that we currently receive.”

Here’s another example of what seems to be about data leading to a false arrest.

According to the lawsuit, police investigating the murder knew months before they arrested Molina that the location data obtained from Google often showed him in two places at once, and that he was not the only person who drove the Honda registered under his name.

Avondale police knew almost two months before they arrested Molina that another man ­ his stepfather ­ sometimes drove Molina’s white Honda. On October 25, 2018, police obtained records showing that Molina’s Honda had been impounded earlier that year after Molina’s stepfather was caught driving the car without a license.

Data obtained by Avondale police from Google did show that a device logged into Molina’s Google account was in the area at the time of Knight’s murder. Yet on a different date, the location data from Google also showed that Molina was at a retirement community in Scottsdale (where his mother worked) while debit card records showed that Molina had made a purchase at a Walmart across town at the exact same time.

Molina’s attorneys argue that this and other instances like it should have made it clear to Avondale police that Google’s account-location data is not always reliable in determining the actual location of a person.

“About” searches might be rare, but that doesn’t make them a good idea. We have knowingly and willingly built the architecture of a police state, just so companies can show us ads. (And it is increasingly apparent that the advertising-supported Internet is heading for a crash.)

Former NSA Director Keith Alexander Joins Amazon’s Board of Directors

About Bruce Schneier

I am a public-interest technologist, working at the intersection of security, technology, and people. I’ve been writing about security issues on my blog since 2004, and in my monthly newsletter since 1998. I’m a fellow and lecturer at Harvard’s Kennedy School and a board member of EFF. This personal website expresses the opinions of neither of those organizations.

Cryptic Rumblings Ahead of First 2020 Patch Tuesday

Sources tell KrebsOnSecurity that Microsoft Corp. is slated to release a software update on Tuesday to fix an extraordinarily serious security vulnerability in a core cryptographic component present in all versions of Windows. Those sources say Microsoft has quietly shipped a patch for the bug to branches of the U.S. military and to other high-value customers/targets that manage key Internet infrastructure, and that those organizations have been asked to sign agreements preventing them from disclosing details of the flaw prior to Jan. 14, the first Patch Tuesday of 2020.

According to sources, the vulnerability in question resides in a Windows component known as crypt32.dll, a Windows module that Microsoft says handles “certificate and cryptographic messaging functions in the CryptoAPI.” The Microsoft CryptoAPI provides services that enable developers to secure Windows-based applications using cryptography, and includes functionality for encrypting and decrypting data using digital certificates.

A critical vulnerability in this Windows component could have wide-ranging security implications for a number of important Windows functions, including authentication on Windows desktops and servers, the protection of sensitive data handled by Microsoft’s Internet Explorer/Edge browsers, as well as a number of third-party applications and tools.

Equally concerning, a flaw in crypt32.dll might also be abused to spoof the digital signature tied to a specific piece of software. Such a weakness could be exploited by attackers to make malware appear to be a benign program that was produced and signed by a legitimate software company.

This component was introduced into Windows more than 20 years ago — back in Windows NT 4.0. Consequently, all versions of Windows are likely affected (including Windows XP, which is no longer being supported with patches from Microsoft).

Microsoft has not yet responded to requests for comment. However, KrebsOnSecurity has heard rumblings from several sources over the past 48 hours that this Patch Tuesday (tomorrow) will include a doozy of an update that will need to be addressed immediately by all organizations running Windows.

Update 7:49 p.m. ET: Microsoft responded, saying that it does not discuss the details of reported vulnerabilities before an update is available. The company also said it does “not release production-ready updates ahead of regular Update Tuesday schedule. “Through our Security Update Validation Program (SUVP), we release advance versions of our updates for the purpose of validation and interoperability testing in lab environments,” Microsoft said in a written statement. “Participants in this program are contractually disallowed from applying the fix to any system outside of this purpose and may not apply it to production infrastructure.”

Original story:

Will Dormann, a security researcher who authors many of the vulnerability reports for the CERT Coordination Center (CERT-CC), tweeted today that “people should perhaps pay very close attention to installing tomorrow’s Microsoft Patch Tuesday updates in a timely manner. Even more so than others. I don’t know…just call it a hunch?” Dormann declined to elaborate on that teaser.

It could be that the timing and topic here (cryptography) is nothing more than a coincidence, but KrebsOnSecurity today received a heads up from the U.S. National Security Agency (NSA) stating that NSA’s Director of Cybersecurity Anne Neuberger is slated to host a call on Jan. 14 with the news media that “will provide advanced notification of a current NSA cybersecurity issue.”

The NSA’s public affairs folks did not respond to requests for more information on the nature or purpose of the discussion. The invitation from the agency said only that the call “reflects NSA’s efforts to enhance dialogue with industry partners regarding its work in the cybersecurity domain.”

Stay tuned for tomorrow’s coverage of Patch Tuesday and possibly more information on this particular vulnerability.