Who’s Behind Monday’s 14-State 911 Outage?

Emergency 911 systems were down for more than an hour on Monday in towns and cities across 14 U.S. states. The outages led many news outlets to speculate the problem was related to Microsoft‘s Azure web services platform, which also was struggling with a widespread outage at the time. However, multiple sources tell KrebsOnSecurity the 911 issues stemmed from some kind of technical snafu involving Intrado and Lumen, two companies that together handle 911 calls for a broad swath of the United States.

Image: West.com

On the afternoon of Monday, Sept. 28, several states including Arizona, California, Colorado, Delaware, Florida, Illinois, Indiana, Minnesota, Nevada, North Carolina, North Dakota, Ohio, Pennsylvania and Washington reported 911 outages in various cities and localities.

Multiple news reports suggested the outages might have been related to an ongoing service disruption at Microsoft. But a spokesperson for the software giant told KrebsOnSecurity, “we’ve seen no indication that the multi-state 911 outage was a result of yesterday’s Azure service disruption.”

Inquiries made with emergency dispatch centers at several of the towns and cities hit by the 911 outage pointed to a different source: Omaha, Neb.-based Intrado — until last year known as West Safety Communications — a provider of 911 and emergency communications infrastructure, systems and services to telecommunications companies and public safety agencies throughout the country.

Intrado did not respond to multiple requests for comment. But according to officials in Henderson County, NC, which experienced its own 911 failures yesterday, Intrado said the outage was the result of a problem with an unspecified service provider.

“On September 28, 2020, at 4:30pm MT, our 911 Service Provider observed conditions internal to their network that resulted in impacts to 911 call delivery,” reads a statement Intrado provided to county officials. “The impact was mitigated, and service was restored and confirmed to be functional by 5:47PM MT.  Our service provider is currently working to determine root cause.”

The service provider referenced in Intrado’s statement appears to be Lumen, a communications firm and 911 provider that until very recently was known as CenturyLink Inc. A look at the company’s status page indicates multiple Lumen systems experienced total or partial service disruptions on Monday, including its private and internal cloud networks and its control systems network.

Lumen’s status page indicates the company’s private and internal cloud and control system networks had outages or service disruptions on Monday.

In a statement provided to KrebsOnSecurity, Lumen blamed the issue on Intrado.

“At approximately 4:30 p.m. MT, some Lumen customers were affected by a vendor partner event that impacted 911 services in AZ, CO, NC, ND, MN, SD, and UT,” the statement reads. “Service was restored in less than an hour and all 911 traffic is routing properly at this time. The vendor partner is in the process of investigating the event.”

It may be no accident that both of these companies are now operating under new names, as this would hardly be the first time a problem between the two of them has disrupted 911 access for a large number of Americans.

In 2019, Intrado/West and CenturyLink agreed to pay $575,000 to settle an investigation by the Federal Communications Commission (FCC) into an Aug. 2018 outage that lasted 65 minutes. The FCC found that incident was the result of a West Safety technician bungling a configuration change to the company’s 911 routing network.

On April 6, 2014, some 11 million people across the United States were disconnected from 911 services for eight hours thanks to an “entirely preventable” software error tied to Intrado’s systems. The incident affected 81 call dispatch centers, rendering emergency services inoperable in all of Washington and parts of North Carolina, South Carolina, Pennsylvania, California, Minnesota and Florida.

According to a 2014 Washington Post story about a subsequent investigation and report released by the FCC, that issue involved a problem with the way Intrado’s automated system assigns a unique identifying code to each incoming call before passing it on to the appropriate “public safety answering point,” or PSAP.

“On April 9, the software responsible for assigning the codes maxed out at a pre-set limit,” The Post explained. “The counter literally stopped counting at 40 million calls. As a result, the routing system stopped accepting new calls, leading to a bottleneck and a series of cascading failures elsewhere in the 911 infrastructure.”

Compounding the length of the 2014 outage, the FCC found, was that the Intrado server responsible for categorizing and keeping track of service interruptions classified them as “low level” incidents that were never flagged for manual review by human beings.

The FCC ultimately fined Intrado and CenturyLink $17.4 million for the multi-state 2014 outage. An FCC spokesperson declined to comment on Monday’s outage, but said the agency was investigating the incident.

Lawsuit forces CenturyLink to stop charging “Internet Cost Recovery Fee”

A white truck with a CenturyLink logo is parked next to a building.

Enlarge / A CenturyLink repair truck in Estes Park, Colorado, in 2018.

CenturyLink has agreed to pay a $6.1 million penalty after Washington state regulators found that the company failed to disclose fees that raised actual prices well above the advertised rates. CenturyLink must also stop charging a so-called “Internet Cost Recovery Fee” in the state, although customers may end up paying the fee until their contracts expire unless they take action to switch plans.

“CenturyLink deceived consumers by telling them they would pay one price and then charging them more,” Washington Attorney General Bob Ferguson said in an announcement yesterday. “Companies must clearly disclose all added fees and charges to Washingtonians.”

Ferguson encouraged Washington residents “who believe they have received bills that include undisclosed fees to file a complaint” with the state.

Ferguson’s office said it began investigating CenturyLink in 2016 “after receiving complaints from consumers that their actual bills were more than the advertised price, or the price that they were promised by sales representatives.”

Here’s what Ferguson’s office found:

There were three main fees CenturyLink did not disclose: a broadcast fee of $2.49 per month, a sports fee of $2.49 per month, and CenturyLink’s “Internet Cost Recovery Fee,” ranging from $0.99 to $1.99 per month.

CenturyLink charged its Internet Cost Recovery Fee to 650,000 Washingtonians. Of those, another 60,000 were also charged the broadcast and sports fees. These fees alone added up to $7 per month to a television subscriber’s bill—$84 per year.

The investigation found that CenturyLink did not adequately disclose additional taxes and fees for its cable, Internet and telephone services.

CenturyLink admitted no wrongdoing but agreed to a financial settlement and changes in business practices as part of a consent decree filed in King County Superior Court on Monday. The attorney general’s office detailed its allegations in a lawsuit filed the same day.

Internet Cost Recovery Fee

The attorney general’s office said that “CenturyLink is required to… stop charging its Internet Cost Recovery Fee” in Washington state. CenturyLink says the fee “helps defray costs associated with building and maintaining CenturyLink’s High-Speed Internet broadband network, as well as the costs of expanding network capacity to support the continued increase in customers’ average broadband consumption.” In other words, the fee covers the company’s normal costs of doing business but is excluded from advertised rates in order to make CenturyLink’s service sound cheaper than it really is. CenturyLink has been charging $1.99 for the Internet Cost Recovery Fee in Washington and continues to charge an Internet Cost Recovery Fee of $3.99 per Internet connection in other states.

There are circumstances in which CenturyLink can continue charging the fee in the Evergreen State until the end of customers’ contracts, but the company must give everyone a chance to opt out of the fee. It would be easier for customers if CenturyLink simply had to eliminate the fee immediately for everyone, regardless of their contract status, but the settlement between Washington and CenturyLink isn’t that simple.

“Within 90 days of the effective date of this consent decree, CenturyLink shall not charge any new Washington consumers for any Internet Cost Recovery Fee or Broadband Cost Recovery Fee,” the consent decree said. Also within 90 days, CenturyLink must notify current Washington customers that they can cancel service without paying an early termination fee or switch to another plan that doesn’t include the Internet Cost Recovery Fee.

If customers who are under contract do not cancel service or switch plans within 30 days of receiving that notice, “these consumers will continue to be charged the Internet or Broadband Cost Recovery Fee through the expiration of their Internet service plans,” the consent decree said.

To settle the case, CenturyLink must also “disclose the actual price of its services, including charges and fees” in sales materials and advertising, provide order confirmations with complete bill summaries to customers “within three days after consumers order services from CenturyLink,” and “honor any and all incentives and discounts promised to consumers.” Like the promise to stop charging the Internet Cost Recovery Fee, these requirements only apply in Washington state.

To ensure that CenturyLink lives up to its commitment, the company must retain all sales call recordings and correspondence related to sales calls and submit compliance reports to the attorney general’s office after one year and again after three years.