AT&T explains why it blocked Cloudflare DNS: It was just an accident

AT&T has been blocking the new Cloudflare DNS service, but AT&T says the blocking was unintentional and that it will fix the problem soon.

The blocking is affecting AT&T home Internet customers who use an AT&T gateway. Cloudflare unveiled its DNS service on April 1, and users in DSLReports forum threads almost immediately started complaining that they couldn't access it. One thread began on April 1, within hours of Cloudflare's announcement.

"I am now unable to reach 1.1.1.1 and 1.0.0.1, which is unfortunate because those are the DNS servers I use," one user wrote.

Other AT&T Internet customers say they were able to use Cloudflare's DNS temporarily, and then they received a firmware update that blocked Cloudflare DNS. Some users said the problem occurred after a firmware update to AT&T's Arris BGW210-700 gateway.

"I have a BGW210-700 from AT&T. I was using 1.1.1.1 and 1.0.0.1 without issues until this morning," one user wrote on April 10. "From the logs, it seems AT&T pushed a new firmware down to the device and restarted it remotely. Now, I cannot reach 1.1.1.1 or 1.0.0.1."

"1.1.1.1 was working for me on AT&T after Cloudflare released 1.1.1.1, then shortly after that it ceased working," another AT&T customer wrote this week. "Maybe the firmware update has a bug, but it's very suspiciously timed." In reply to that comment, another user said that "it worked for a day or so and then stopped."

Controversy continued to build this week when Reddit and Hacker News threads pointed to the original complaints and described ongoing problems. On Wednesday, Cloudflare CEO Matthew Prince criticized AT&T and seemed to indicate that he thought the blocking is intentional. AT&T "appear[s] to be actively locking down the past and breaking Internet standards in the process," Prince wrote in a tweet.

Once upon a time @ATTcares used to promise they'd enable the future, so disappointing they now appear to be actively locking down the past and breaking Internet standards in the process. https://t.co/LPPDDtXETs

— Matthew Prince (@eastdakota) May 2, 2018

“Unintentional IP address conflict”

When contacted by Ars, Prince said Cloudflare was still trying to figure out what happened and that he hoped it was just a mistake. Shortly after, an AT&T spokesperson told Ars that the blocking was an accident.

"With the recent launch of Cloudflare's 1.1.1.1 DNS service, we have discovered an unintentional gateway IP address conflict with 1 of their 4 useable IPs and are working to resolve the issue," AT&T told Ars yesterday.

AT&T also told us that most of its customers should be able to access Cloudflare DNS using the alternate 1.0.0.1 address. AT&T didn't say when it will roll out a fix.

Some users confirmed that they could use Cloudflare's 1.0.0.1 address even though 1.1.1.1 wasn't working for them.

Upon hearing AT&T's statement, Prince told Ars that "my hunch is it was unintentional" and that he is glad AT&T is working to resolve it. AT&T didn't tell us how many of its customers were affected, and Prince said he didn't know how many people had the problem.

Cloudflare chose 1.1.1.1 because it wanted a memorable address.

The problem reportedly affects multiple AT&T gateways. One customer ran a traceroute on April 1 and found that AT&T's Arris 5268AC gateway "has been assigned 1.1.1.1 on an internal interface."

Some Cisco equipment apparently does the same; another person writing in a DSLReports forum pointed to years-old support threads showing that Cisco gear was using 1.1.1.1. Such equipment "uses 1.1.1.1 as a virtual IP to redirect to when the device needs to be set up for the first time, or uses it as a captive portal to authenticate guest Wi-Fi, such as in hotels and restaurants and such," the person wrote.

Recent articles about Cloudflare DNS at TechTarget and ZDNet point to several other examples of non-AT&T equipment and services using 1.1.1.1 as well. Chief scientist Geoff Huston of APNIC Labs, which partners with Cloudflare on DNS, called this practice "squatting on somebody else's address," according to ZDNet.

AT&T’s controversial history

There haven't been any recent reports of AT&T blocking other major DNS services.

Although there's reason to think the blocking wasn't intentional, AT&T's public stances on net neutrality and privacy helped make people suspicious about the company's motives. AT&T sued the Federal Communications Commission in 2015 in order to eliminate net neutrality rules that forbid ISPs from blocking or throttling websites and online services.

While AT&T lost that lawsuit, its lobbying helped convince the FCC to ditch the net neutrality rules after Republicans took over the commission majority last year. AT&T claimed during its anti-net neutrality lobbying campaign that it never blocked third-party applications, even though AT&T did block FaceTime on its cellular network in 2012 when users tried to access the application from certain data plans.

Cloudflare pitches 1.1.1.1 as a privacy tool that can help deter ISPs from monitoring one's Internet usage. AT&T lobbied against broadband privacy rules last year, and the company used to charge fiber Internet customers extra for privacy. AT&T fiber customers who did not opt in to a traffic scanning system that analyzed Internet usage in order to deliver personalized ads had to pay at least $29 more per month than customers who consented to the scanning.

AT&T ended the controversial traffic scanning program in September 2016, but it says that it still wants the "flexibility" to expand advertising-focused business models to compete against Facebook, Amazon, and Google.

One AT&T user who couldn't connect to 1.1.1.1 or 1.0.0.1 wrote on April 5 that it "Makes you wonder why AT&T would be continuing to roll this [firmware] out knowing they are blocking DNS servers. I wonder if it's on purpose due to the added privacy offered by 1.1.1.1?"

Other people suspected it was just a mistake.

"This is almost certainly just there to block people who mistakenly paste in an example configuration somewhere," a Hacker News poster speculated. "Also, why on earth would AT&T block 1.1.1.1 and not Google DNS and OpenDNS?"