Ars Technica

VMware Flaw a Vector in SolarWinds Breach?

U.S. government cybersecurity agencies warned this week that the attackers behind the widespread hacking spree stemming from the compromise at network software firm SolarWinds used weaknesses in other, non-SolarWinds products to attack high-value targets. According to sources, among those was a flaw in software virtualization platform VMware, which the U.S. National Security Agency (NSA) warned on Dec. 7 was being used by Russian hackers to impersonate authorized users on victim networks.

On Dec. 7, 2020, the NSA said “Russian state-sponsored malicious cyber actors are exploiting a vulnerability in VMware Access and VMware Identity Manager products, allowing the actors access to protected data and abusing federated authentication.”

VMware released a software update to plug the security hole (CVE-2020-4006) on Dec. 3, and said it learned about the flaw from the NSA.

The NSA advisory (PDF) came less than 24 hours before cyber incident response firm FireEye said it discovered attackers had broken into its networks and stolen more than 300 proprietary software tools the company developed to help customers secure their networks.

On Dec. 13, FireEye disclosed that the incident was the result of the SolarWinds compromise, which involved malicious code being surreptitiously inserted into updates shipped by SolarWinds for users of its Orion network management software as far back as March 2020.

In its advisory on the VMware vulnerability, the NSA urged patching it “as soon as possible,” specifically encouraging the National Security System, Department of Defense, and defense contractors to make doing so a high priority.

The NSA said that in order to exploit this particular flaw, hackers would already need to have access to a vulnerable VMware device’s management interface — i.e., they would need to be on the target’s internal network (provided the vulnerable VMware interface was not accessible from the Internet). However, the SolarWinds compromise would have provided that internal access nicely.

In response to questions from KrebsOnSecurity, VMware said it has “received no notification or indication that the CVE 2020-4006 was used in conjunction with the SolarWinds supply chain compromise.”

VMware added that while some of its own networks used the vulnerable SolarWinds Orion software, an investigation has so far revealed no evidence of exploitation.

“While we have identified limited instances of the vulnerable SolarWinds Orion software in our environment, our own internal investigation has not revealed any indication of exploitation,” the company said in a statement. “This has also been confirmed by SolarWinds own investigations to date.”

On Dec. 17, DHS’s Cybersecurity and Infrastructure Security Agency (CISA) released a sobering alert on the SolarWinds attack, noting that CISA had evidence of additional access vectors other than the SolarWinds Orion platform.

CISA’s advisory specifically noted that “one of the principal ways the adversary is accomplishing this objective is by compromising the Security Assertion Markup Language (SAML) signing certificate using their escalated Active Directory privileges. Once this is accomplished, the adversary creates unauthorized but valid tokens and presents them to services that trust SAML tokens from the environment. These tokens can then be used to access resources in hosted environments, such as email, for data exfiltration via authorized application programming interfaces (APIs).”

Indeed, the NSA’s Dec. 7 advisory said the hacking activity it saw involving the VMware vulnerability “led to the installation of a web shell and follow-on malicious activity where credentials in the form of SAML authentication assertions were generated and sent to Microsoft Active Directory Federation Services (ADFS), which in turn granted the actors access to protected data.”

Also on Dec. 17, the NSA released a far more detailed advisory explaining how it has seen the VMware vulnerability being used to forge SAML tokens, this time specifically referencing the SolarWinds compromise.

Asked about the potential connection, the NSA said only that “if malicious cyber actors gain initial access to networks through the SolarWinds compromise, the TTPs [tactics, techniques and procedures] noted in our December 17 advisory may be used to forge credentials and maintain persistent access.”

“Our guidance in this advisory helps detect and mitigate against this, no matter the initial access method,” the NSA said.

CISA’s analysis suggested the crooks behind the SolarWinds intrusion were heavily focused on impersonating trusted personnel on targeted networks, and that they’d devised clever ways to bypass multi-factor authentication (MFA) systems protecting networks they targeted.

The bulletin references research released earlier this week by security firm Volexity, which described encountering the same attackers using a novel technique to bypass MFA protections provided by Duo for Microsoft Outlook Web App (OWA) users.

Duo’s parent Cisco Systems Inc. responded that the attack described by Volexity didn’t target any specific vulnerability in its products. As Ars Technica explained, the bypass involving Duo’s protections could have just as easily involved any of Duo’s competitors.

“MFA threat modeling generally doesn’t include a complete system compromise of an OWA server,” Ars’ Dan Goodin wrote. “The level of access the hacker achieved was enough to neuter just about any defense.”

Several media outlets, including The New York Times and The Washington Post, have cited anonymous government sources saying the group behind the SolarWinds hacks was known as APT29 or “Cozy Bear,” an advanced threat group believed to be part of the Russian Federal Security Service (FSB).

SolarWinds has said almost 18,000 customers may have received the backdoored Orion software updates. So far, only a handful of customers targeted by the suspected Russian hackers behind the SolarWinds compromise have been made public — including the U.S. Commerce, Energy and Treasury departments, and the DHS.

No doubt we will hear about new victims in the public and private sector in the coming days and weeks. In the meantime, thousands of organizations are facing incredibly costly, disruptive and time-intensive work in determining whether they were compromised and if so what to do about it.

The CISA advisory notes the attackers behind the SolarWinds compromises targeted key personnel at victim firms — including cyber incident response staff, and IT email accounts. The warning suggests organizations that suspect they were victims should assume their email communications and internal network traffic are compromised, and rely upon or build out-of-band systems for discussing internally how they will proceed to clean up the mess.

“If the adversary has compromised administrative level credentials in an environment—or if organizations identify SAML abuse in the environment, simply mitigating individual issues, systems, servers, or specific user accounts will likely not lead to the adversary’s removal from the network,” CISA warned. “In such cases, organizations should consider the entire identity trust store as compromised. In the event of a total identity compromise, a full reconstitution of identity and trust services is required to successfully remediate. In this reconstitution, it bears repeating that this threat actor is among the most capable, and in many cases, a full rebuild of the environment is the safest action.”

Iowa Prosecutors Drop Charges Against Men Hired to Test Their Security

On Sept. 11, 2019, two security experts at a company that had been hired by the state of Iowa to test the physical and network security of its judicial system were arrested while probing the security of an Iowa county courthouse, jailed in orange jumpsuits, charged with burglary, and held on $100,000 bail. On Thursday Jan. 30, prosecutors in Iowa announced they had dropped the criminal charges. The news came while KrebsOnSecurity was conducting a video interview with the two accused (featured below).

The courthouse in Dallas County, Iowa. Image: Wikipedia.

Gary DeMercurio, 43 of Seattle, and Justin Wynn, 29 of Naples, Fla., are both professional penetration testers employed by Coalfire Labs, a security firm based in Westminster, Colo. Iowa’s State Court Administration had hired the company to test the security of its judicial buildings.

Under the terms of their contract (PDF), DeMercurio and Wynn were permitted to impersonate staff and contractors, provide false pretenses to gain physical access to facilities, “tailgate” employees into buildings, and access restricted areas of those facilities. The contract said the men could not attempt to subvert alarm systems, force-open doors, or access areas that require protective equipment.

When the duo’s early-morning Sept. 11 test of the security at the courthouse in Dallas County, Iowa set off an audible security alarm, they followed procedure and waited on-site for the police. DeMercurio and Wynn said when the county’s sheriff deputies arrived on the scene just a few minutes later, they told the officers who they were and why they were there, and that they’d obtained entry to the premises via an unlocked door.

“They said they found a courthouse door unlocked, so they closed it from the outside and let it lock,” Dan Goodin of Ars Technica wrote of the ordeal in November. “Then they slipped a plastic cutting board through a crack in the door and manipulated its locking mechanism. (Pentesters frequently use makeshift or self-created tools in their craft to flip latches, trigger motion-detected mechanisms, and test other security systems.) The deputies seemed impressed.”

To assuage concerns they might be burglars, DeMercurio and Wynn produced an authorization letter detailing the job they’d been hired to do and listing the names and mobile phone numbers of Iowa state employees who could verify their story.

After contacting some of the court officials listed in the letter, the deputies seemed satisfied that the men weren’t thieves. That is, until Dallas County Sheriff Chad Leonard showed up.

“The pentesters had already said they used a tool to open the front door,” Goodin recounted. “Leonard took that to mean the men had violated the restriction against forcing doors open. Leonard also said the men attempted to turn off the alarm—something Coalfire officials vehemently deny. In Leonard’s mind that was a second violation. Another reason for doubt: one of the people listed as a contact on the get-out-of-jail-free letter didn’t answer the deputies’ calls, while another said he didn’t believe the men had permission to conduct physical intrusions.”

DeMercurio and Wynn were arrested, jailed, and held for nearly 24 hours before being released on a $100,000 bail. Initially they were charged with felony third-degree burglary and possessing burglary tools, although those charges were later downgraded to misdemeanor trespass.

What initially seemed to Coalfire as a momentary lapse of judgment by Iowa authorities quickly morphed into the surreal when state lawmakers held hearings questioning why and how someone in the state’s employ could have so recklessly endangered the safety and security of its citizens.

DeMercurio and Wynn, minus the orange jumpsuits.

Judicial Branch officials in Dallas County said in response to this grilling that they didn’t expect Coalfire’s physical penetration testing to be conducted outside of business hours. State Sen. Amy Sinclair was quoted as telling her colleagues that “the hiring of an outside company to break into the courthouses in September created ‘significant danger, not only to the contractors, but to local law enforcement, and members of the public.’”

“Essentially a branch of government has contracted with a company to commit crimes, and that’s very troubling,” lamented Iowa state Sen. Zach Whiting. “I want to find out who needs to be held accountable for this and how we can do that.”

Those strong words clashed with a joint statement released Thursday by Coalfire and Dallas County Attorney Charles Sinnard:

“Ultimately, the long-term interests of justice and protection of the public are not best served by continued prosecution of the trespass charges,” the statement reads. “Those interests are best served by all the parties working together to ensure that there is clear communication on the actions to be taken to secure the sensitive information maintained by the judicial branch, without endangering the life or property of the citizens of Iowa, law enforcement or the persons carrying out the testing.

Matthew Linholm, an attorney representing DeMercurio and Wynn in the case, said the justice system ceases to serve its crucial function and loses credibility when criminal accusations are used to advance personal or political agendas.

“Such a practice endangers the effective administration of justice and our confidence in the criminal justice system,” Linholm told The Des Moines Register, which broke the news of the dropped charges.

While the case against Coalfire’s employees has rallied many in the cybersecurity community around the accused, not everyone sees this dispute in black-and-white. Chris Nickerson, a digital intrusion specialist and founder of LARES Consulting, said in a Twitter post Thursday that “when a company puts us in harm’s way due to their poor planning, failed sales education, inadequate project management and deplorable contract management…We shouldn’t celebrate them. We should hold them accountable.”

Asked to elaborate, Nickerson referred to a recent podcast which touched on the arrests.

“The things that concern me about this situation are more of the pieces of safety that exist across how the industry instruments doing these types of engagements,” Nickerson said. “They seem very, very reasonable and obvious once they become obvious but until then they’re completely foreign to people.”

“It’s really on the owners of the organization to educate the customer of those potential pitfalls,” Nickerson continued. “Because there isn’t a good standard. We haven’t all gotten together and institutionalized the knowledge that we have in our heads and dump it down to paper so that someone who is new to the field being tasked with this can go through and say, ‘Hey, did you ask them if the city versus the state versus the building owner and the real estate people…are all of these people in lock step?’”

Coalfire CEO Tom McAndrew seemed to address this point in our interview Thursday, saying there were two unique aspects of this particular engagement. First, although the client in this case said they did not want Coalfire to make local law enforcement aware of the ongoing engagement prior to testing the physical security of the site, it was clear after the fact that state officials never did that on their own.

More importantly, McAndrew said, there was ambiguity around who actually owned the buildings that they were hired to test.

“If you’re doing a test for the state and you walk into the building and it’s the courthouse and you’re doing a test for the court system, you’d think that they would have jurisdiction or own it, and that turned out not to be the case in this scenario because there’s some things the state owns and some things the county owns, and that was something we weren’t aware of as we did some of this work,” he said. “We didn’t understand the nuances.”

Asked what Coalfire has learned from this ordeal, McAndrew said his company is likely to insist that local, state and even federal law enforcement be informed in advance of any penetration tests, at least as far as those engagements relate to public entities.

“When we look at the contracts and we look at who’s authorized to do what…typically, if a [chief security officer] says test these IP addresses, we would say okay that’s enough,” he said. “But we’re questioning from a legal perspective at what point does that need to have legal counsel review.”

McAndrew said it’s probably time for experts from various corners of the pen testing community to collaborate in documenting best practices that might help others avoid a repeat of the scenario in Dallas County.

“There’s no standard in the industry,” he said. “When it comes to these sorts of issues in red teaming — the legal challenges and the contracts — there’s really nothing out there. There are some things that can’t be undone. There’s the mugshots that are out there forever, but even as we get the charges dropped, these are permanently going to be in the federal database. This is a permanent thing that will reside with them and there’s no legal way we’re aware of to get these charges removed from the federal database.”

McAndrew said while he remains frustrated that it took so long to resolve this dispute, he doesn’t believe anyone involved acted with malicious intent.

“I don’t think there were any bad people,” he said. “Everyone was trying to do the right things — from law enforcement to the sheriff to the judges to the county — they all had the right intentions. But they didn’t necessarily all have the right information, and possibly people made decisions at levels they weren’t really authorized to do. Normally that’s not really our call, but I think people need to be thinking about that.”

Subscribe to Ars and get a YubiKey 5

The Ars Technica logo has been photoshopped onto a stock photo of dials and knobs.

Enlarge (credit: Aurich Lawson / Getty)

If you’re in the United States, chances are that you’re looking forward to a long weekend with family, friends, food, and football in some order. As we pause for Thanksgiving, we at Ars are thankful for the smartest and most insightful audience on the Internet. (Seriously, we’ve seen the comments at other sites…) To show our appreciation, we’re teaming up with YubiKey once again.

Since launching Ars Pro last year, we’ve seen one request above all from readers and subscribers: a YubiKey two-factor authentication device that supports USB-C. We’ve heard your feedback loud and clear, and today we’re proud to announce that new Ars Pro++ subscribers can choose from either the YubiKey 5c or the YubiKey 5 NFC.

As you might have guessed from the name, the YubiKey 5c plugs into the USB-C ports that are ubiquitous on newer hardware. The $45 YubiKey 5 NFC works with NFC-enabled iOS, Android, and Windows 10 devices, allowing you to authenticate just by tapping the YubiKey against your device. And if your laptop or PC is rocking old-school USB-A ports, the YubiKey 5 NFC will slide right in.

Read 4 remaining paragraphs | Comments