Neil Schwartzman

Sendgrid Under Siege from Hacked Accounts

Email service provider Sendgrid is grappling with an unusually large number of customer accounts whose passwords have been cracked, sold to spammers, and abused for sending phishing and email malware attacks. Sendgrid’s parent company Twilio says it is working on a plan to require multi-factor authentication for all of its customers, but that solution may not come fast enough for organizations having trouble dealing with the fallout in the meantime.

Image: Wikipedia

Many companies use Sendgrid to communicate with their customers via email, or else pay marketing firms to do that on their behalf using Sendgrid’s systems. Sendgrid takes steps to validate that new customers are legitimate businesses, and that emails sent through its platform carry the proper digital signatures that other companies can use to validate that the messages have been authorized by its customers.

But this also means when a Sendgrid customer account gets hacked and used to send malware or phishing scams, the threat is particularly acute because a large number of organizations allow email from Sendgrid’s systems to sail through their spam-filtering systems.

To make matters worse, links included in emails sent through Sendgrid are obfuscated (mainly for tracking deliverability and other metrics), so it is not immediately clear to recipients where on the Internet they will be taken when they click.

Dealing with compromised customer accounts is a constant challenge for any organization doing business online today, and certainly Sendgrid is not the only email marketing platform dealing with this problem. But according to multiple emails from readers, recent threads on several anti-spam discussion lists, and interviews with people in the anti-spam community, over the past few months there has been a marked increase in malicious, phishous and outright spammy email being blasted out via Sendgrid’s servers.

Rob McEwen is CEO of Invaluement.com, an anti-spam firm whose data on junk email trends are used to improve the spam-blocking technologies deployed by several Fortune 100 companies. McEwen said no other email service provider has come close to generating the volume of spam that’s been emanating from Sendgrid accounts lately.

“As far as the nasty criminal phishes and viruses, I think there’s not even a close second in terms of how bad it’s been with Sendgrid over the past few months,” he said.

Trying to filter out bad emails coming from a major email provider that so many legitimate companies rely upon to reach their customers can be a dicey business. If you filter the emails too aggressively you end up with an unacceptable number of “false positives,” i.e., benign or even desirable emails that get flagged as spam and sent to the junk folder or blocked altogether.

But McEwen said the incidence of malicious spam coming from Sendgrid has gotten so bad that he recently launched a new anti-spam block list specifically to filter out email from Sendgrid accounts that have been known to be blasting large volumes of junk or malicious email.

“Before I implemented this in my own filtering system a week ago, I was getting three to four phone calls or stern emails a week from angry customers wondering why these malicious emails were getting through to their inboxes,” McEwen said. “And I just am not seeing anything this egregious in terms of viruses and spams from the other email service providers.”

In an interview with KrebsOnSecurity, Sendgrid parent firm Twilio acknowledged the company had recently seen an increase in compromised customer accounts being abused for spam. While Sendgrid does allow customers to use multi-factor authentication (also known as two-factor authentication or 2FA), this protection is not mandatory.

But Twilio Chief Security Officer Steve Pugh said the company is working on changes that would require customers to use some form of 2FA in addition to usernames and passwords.

“Twilio believes that requiring 2FA for customer accounts is the right thing to do, and we’re working towards that end,” Pugh said. “2FA has proven to be a powerful tool in securing communications channels. This is part of the reason we acquired Authy and created a line of account security products and services. Twilio, like other platforms, is forming a plan on how to better secure our customers’ accounts through native technologies such as Authy and additional account level controls to mitigate known attack vectors.”

Requiring customers to use some form of 2FA would go a long way toward neutralizing the underground market for compromised Sendgrid accounts, which are sold by a variety of cybercriminals who specialize in gaining access to accounts by targeting users who re-use the same passwords across multiple websites.

One such individual, who goes by the handle “Kromatix” on several forums, is currently selling access to more than 400 compromised Sendgrid user accounts. The pricing attached to each account is based on volume of email it can send in a given month. Accounts that can send up to 40,000 emails a month go for $15, whereas those capable of blasting 10 million missives a month sell for $400.

“I have a large supply of cracked Sendgrid accounts that can be used to generate an API key which you can then plug into your mailer of choice and send massive amounts of emails with ensured delivery,” Kromatix wrote in an Aug. 23 sales thread. “Sendgrid servers maintain a very good reputation with [email service providers] so your content becomes much more likely to get into the inbox so long as your setup is correct.”

Neil Schwartzman, executive director of the anti-spam group CAUCE, said Sendgrid’s 2FA plans are long overdue, noting that the company bought Authy back in 2015.

Single-factor authentication for a company like this in 2020 is just ludicrous given the potential damage and malicious content we’re seeing,” Schwartzman said.

“I understand that it’s a task to invoke 2FA, and given the volume of customers Sendgrid has that’s something to consider because there’s going to be a lot of customer overhead involved,” he continued. “But it’s not like your bank, social media account, email and plenty of other places online don’t already insist on it.”

Schwartzman said if Twilio doesn’t act quickly enough to fix the problem on its end, the major email providers of the world (think Google, Microsoft and Apple) — and their various machine-learning anti-spam algorithms — may do it for them.

“There is a tipping point after which receiving firms start to lose patience and start to more aggressively filter this stuff,” he said. “If seeing a Sendgrid email according to machine learning becomes a sign of abuse, trust me the machines will make the decisions even if the people don’t.”

Sipping from the Coronavirus Domain Firehose

Security experts are poring over thousands of new Coronavirus-themed domain names registered each day, but this often manual effort struggles to keep pace with the flood of domains invoking the virus to promote malware and phishing sites, as well as non-existent healthcare products and charities. As a result, domain name registrars are under increasing pressure to do more to combat scams and misinformation during the COVID-19 pandemic.

By most measures, the volume of new domain registrations that include the words “Coronavirus” or “Covid” has closely tracked the spread of the deadly virus. The Cyber Threat Coalition (CTC), a group of several thousand security experts volunteering their time to fight COVID-related criminal activity online, recently published data showing the rapid rise in new domains began in the last week of February, around the same time the Centers for Disease Control began publicly warning that a severe global pandemic was probably inevitable.

The total number of domains registered per day that contain a COVID-19 related term, according to DomainTools. The red line indicates the count of domains that DomainTools determined are “likely malicious.” The blue line refers to domains that are likely benign.

“Since March 20th, the number of risky domains registered per day has been decreasing, with a notable spike around March 30th,” wrote John Conwell, principal data scientist at DomainTools [an advertiser on this site]. “Interestingly, legitimate organizations creating domains in response to the COVID-19 crisis were several weeks behind the curve from threat actors trying to take advantage of this situation. This is a pattern DomainTools hasn’t seen before in other crises.”

Security vendor Sophos looked at telemetry from customer endpoints to illustrate the number of new COVID-related domains that actually received traffic of late. As the company noted, one challenge in identifying potentially malicious domains is that many of them can sit dormant for days or weeks before being used for anything.

Data from security vendor Sophos, published by the Cyber Threat Coalition, shows the number of Coronavirus or COVID-19 themed domains registered per week that received traffic.

“We can see a rapid and dramatic increase of visits to potentially malicious domains exploiting the Coronavirus pandemic week over week, beginning in late February,” wrote Sophos’ Rich Harang. “Even though still a minority of cyber threats use the pandemic as a lure, some of these new domains will eventually be used for malicious purposes.”

CTC spokesman Nick Espinosa said the first spike in visits was on February 25, when group members saw about 4,000 visits to the sites they were tracking.

“The following two weeks starting on March 9 saw rapid growth, and from March 23 onwards we’re seeing between 75,000 to 130,000 visits per weekday, and about 40,000 on the weekends,” Espinosa said. “Looking at the data collected, the pattern of visits are highest on Monday and Friday, and the lowest visit count is on the weekend. Our data shows that there were virtually no customer hits on COVID-related domains prior to February 23.”

Milwaukee-based Hold Security has been publishing daily and weekly lists of all COVID-19 related domain registrations (without any scoring assigned). Here’s a graph KrebsOnSecurity put together based on that data set, which also shows a massive spike in new domain registrations in the third week of March, trailing off considerably over the past couple of weeks.

Data: Hold Security.

Not everyone is convinced we’re measuring the right things, or that the current measurements are accurate. Neil Schwartzman, executive director of the anti-spam group CAUCE, said he believes DomainTool’s estimates on the percentage of new COVID/Coronavirus-themed domains that are malicious are too high, and that many are likely benign and registered by well-meaning people seeking to share news or their own thoughts about the outbreak.

“But there’s the rub,” he said. “Bad guys get to hide amidst the good really effectively, so each one needs to be reviewed on its own. And that’s a substantial amount of work.”

At the same time, Schwartzman said, focusing purely on domains may obscure the true size and scope of the overall threat. That’s because scammers very often will establish multiple subdomains for each domain, meaning that a single COVID-related new domain registration could eventually be tied to a number of different scammy or malicious sites.

Subdomains can not only make phishing domains appear more legitimate, but they also tend to lengthen the domain so that key parts of it get pushed off the URL bar in mobile browsers.

To that end, he said, it makes perhaps the most sense to focus on new domain registrations that have encryption certificates tied to them, since the issuance of an SSL certificate for a domain is usually a sign that it is about to be put to use. As noted in previous stories here, roughly 75 percent of all phishing sites now have the padlock (start with “https://”), mainly because the major Web browsers display security alerts on sites that don’t.

Schwartzman said more domain registrars should follow the example of Los Angeles-based Namecheap Inc., which last month pledged to stop accepting the automated registration of website names that include words or phrases tied to the COVID-19 pandemic. Since then, a handful of other registrars have said they plan to manually review all such registrations going forward.

The Internet Corporation for Assigned Names and Numbers (ICANN), the organization that oversees the registrar industry, recently sent a letter urging registrars to be more proactive, but stopped short of mandating any specific actions.

Schwartzman called ICANN’s response “weak tea.”

“It’s absolutely ludicrous that ICANN hasn’t stepped up, and they will bear significant responsibility for any deaths that may happen as a result of all this,” Schwartzman said. “This is a CYA response at best, and dictates to no one that they should do anything.”

Michael Daniel, president of the Cyber Threat Alliance — a cybersecurity industry group that’s also been working to fight COVID-19 related fraud — agreed, saying more pressure needs to be applied to the registrar community.

“It’s really hard to do anything about this unless the registrars step up and do something on their own,” Daniel said. “It’s either that or the government gets involved. That doesn’t mean some [registrars] aren’t doing what they can, but in general what the industry is doing is nowhere near as fast as the bad guys are generating these domains.”

The U.S. government may well soon get more involved. Earlier this week, Senators Cory Booker (D-N.J.), Maggie Hassan (D-N.H.) and Mazie K. Hirono (D-Hawaii) sent letters to eight domain name company leaders, demanding to know what they were doing to combat the threat of malicious domains, and urging them to do more.

“As cybercriminals and other malevolent actors seek to take advantage of the Coronavirus pandemic, it is critical that domain name registrars like yours (1) exercise diligence and ensure that only legitimate organizations can register Coronavirus-related domain names and domain names referencing online communications platforms; (2) act quickly to suspend, cancel, or terminate registrations for domains that are involved in unlawful or harmful activity; and (3) cooperate with law enforcement to help bring to justice cybercriminals profiting from the Coronavirus pandemic,” the senators wrote.