Federal Communications Commission

Who’s Behind Monday’s 14-State 911 Outage?

Emergency 911 systems were down for more than an hour on Monday in towns and cities across 14 U.S. states. The outages led many news outlets to speculate the problem was related to Microsoft‘s Azure web services platform, which also was struggling with a widespread outage at the time. However, multiple sources tell KrebsOnSecurity the 911 issues stemmed from some kind of technical snafu involving Intrado and Lumen, two companies that together handle 911 calls for a broad swath of the United States.

Image: West.com

On the afternoon of Monday, Sept. 28, several states including Arizona, California, Colorado, Delaware, Florida, Illinois, Indiana, Minnesota, Nevada, North Carolina, North Dakota, Ohio, Pennsylvania and Washington reported 911 outages in various cities and localities.

Multiple news reports suggested the outages might have been related to an ongoing service disruption at Microsoft. But a spokesperson for the software giant told KrebsOnSecurity, “we’ve seen no indication that the multi-state 911 outage was a result of yesterday’s Azure service disruption.”

Inquiries made with emergency dispatch centers at several of the towns and cities hit by the 911 outage pointed to a different source: Omaha, Neb.-based Intrado — until last year known as West Safety Communications — a provider of 911 and emergency communications infrastructure, systems and services to telecommunications companies and public safety agencies throughout the country.

Intrado did not respond to multiple requests for comment. But according to officials in Henderson County, NC, which experienced its own 911 failures yesterday, Intrado said the outage was the result of a problem with an unspecified service provider.

“On September 28, 2020, at 4:30pm MT, our 911 Service Provider observed conditions internal to their network that resulted in impacts to 911 call delivery,” reads a statement Intrado provided to county officials. “The impact was mitigated, and service was restored and confirmed to be functional by 5:47PM MT.  Our service provider is currently working to determine root cause.”

The service provider referenced in Intrado’s statement appears to be Lumen, a communications firm and 911 provider that until very recently was known as CenturyLink Inc. A look at the company’s status page indicates multiple Lumen systems experienced total or partial service disruptions on Monday, including its private and internal cloud networks and its control systems network.

Lumen’s status page indicates the company’s private and internal cloud and control system networks had outages or service disruptions on Monday.

In a statement provided to KrebsOnSecurity, Lumen blamed the issue on Intrado.

“At approximately 4:30 p.m. MT, some Lumen customers were affected by a vendor partner event that impacted 911 services in AZ, CO, NC, ND, MN, SD, and UT,” the statement reads. “Service was restored in less than an hour and all 911 traffic is routing properly at this time. The vendor partner is in the process of investigating the event.”

It may be no accident that both of these companies are now operating under new names, as this would hardly be the first time a problem between the two of them has disrupted 911 access for a large number of Americans.

In 2019, Intrado/West and CenturyLink agreed to pay $575,000 to settle an investigation by the Federal Communications Commission (FCC) into an Aug. 2018 outage that lasted 65 minutes. The FCC found that incident was the result of a West Safety technician bungling a configuration change to the company’s 911 routing network.

On April 6, 2014, some 11 million people across the United States were disconnected from 911 services for eight hours thanks to an “entirely preventable” software error tied to Intrado’s systems. The incident affected 81 call dispatch centers, rendering emergency services inoperable in all of Washington and parts of North Carolina, South Carolina, Pennsylvania, California, Minnesota and Florida.

According to a 2014 Washington Post story about a subsequent investigation and report released by the FCC, that issue involved a problem with the way Intrado’s automated system assigns a unique identifying code to each incoming call before passing it on to the appropriate “public safety answering point,” or PSAP.

“On April 9, the software responsible for assigning the codes maxed out at a pre-set limit,” The Post explained. “The counter literally stopped counting at 40 million calls. As a result, the routing system stopped accepting new calls, leading to a bottleneck and a series of cascading failures elsewhere in the 911 infrastructure.”

Compounding the length of the 2014 outage, the FCC found, was that the Intrado server responsible for categorizing and keeping track of service interruptions classified them as “low level” incidents that were never flagged for manual review by human beings.

The FCC ultimately fined Intrado and CenturyLink $17.4 million for the multi-state 2014 outage. An FCC spokesperson declined to comment on Monday’s outage, but said the agency was investigating the incident.

FCC Proposes to Fine Wireless Carriers $200M for Selling Customer Location Data

The U.S. Federal Communications Commission (FCC) today proposed fines of more than $200 million against the nation’s four largest wireless carriers for selling access to their customers’ location information without taking adequate precautions to prevent unauthorized access to that data. While the fines would be among the largest the FCC has ever levied, critics say the penalties don’t go far enough to deter wireless carriers from continuing to sell customer location data.

The FCC proposed fining T-Mobile $91 million; AT&T faces more than $57 million in fines; Verizon is looking at more than $48 million in penalties; and the FCC said Sprint should pay more than $12 million.

An FCC statement (PDF) said “the size of the proposed fines for the four wireless carriers differs based on the length of time each carrier apparently continued to sell access to its customer location information without reasonable safeguards and the number of entities to which each carrier continued to sell such access.”

The fines are only “proposed” at this point because the carriers still have an opportunity to respond to the commission and contest the figures. The Wall Street Journal first reported earlier this week that the FCC was considering the fines.

The commission said it took action in response to a May 2018 story broken by The New York Times, which exposed how a company called Securus Technologies had been selling location data on customers of virtually any major mobile provider to law enforcement officials.

That same month, KrebsOnSecurity broke the news that LocationSmart — a data aggregation firm working with the major wireless carriers — had a free, unsecured demo of its service online that anyone could abuse to find the near-exact location of virtually any mobile phone in North America.

In response, the carriers promised to “wind down” location data sharing agreements with third-party companies. But in 2019, Joseph Cox at Vice.com showed that little had changed, detailing how he was able to locate a test phone after paying $300 to a bounty hunter who simply bought the data through a little-known third-party service.

Gigi Sohn is a fellow at the Georgetown Law Institute for Technology Law and Policy and a former senior adviser to former FCC Chair Tom Wheeler in 2015. Sohn said this debacle underscores the importance of having strong consumer privacy protections.

“The importance of having rules that protect consumers before they are harmed cannot be overstated,” Sohn said. “In 2016, the Wheeler FCC adopted rules that would have prevented most mobile phone users from suffering this gross violation of privacy and security. But [FCC] Chairman Pai and his friends in Congress eliminated those rules, because allegedly the burden on mobile wireless providers and their fixed broadband brethren would be too great. Clearly, they did not think for one minute about the harm that could befall consumers in the absence of strong privacy protections.”

Sen. Ron Wyden (D-Ore.), a longtime critic of the FCC’s inaction on wireless location data sharing, likewise called for more string consumer privacy laws, calling the proposed punishment “comically inadequate fines that won’t stop phone companies from abusing Americans’ privacy the next time they can make a quick buck.”

“Time and again, from Facebook to Equifax, massive companies take reckless disregard for Americans’ personal information, knowing they can write off comparatively tiny fines as the cost of doing business,” Wyden said in a written statement. “The only way to truly protect Americans’ personal information is to pass strong privacy legislation like my Mind Your Own Business Act [PDF] to put teeth into privacy laws and hold CEOs personally responsible for lying about protecting Americans’ privacy.”