
T-Mobile, Verizon and AT&T phone calls are failing across the US - keyi
https://www.theverge.com/2020/6/15/21292024/t-mobile-verizon-att-calls-failing-down-phone-networks-us-carriers
======
mlacks
Looks to be a tech issue related to the Sprint merger.

That is the current suggestion: (reposted from elsewhere in this thread:
[https://puck.nether.net/pipermail/outages/2020-June/013124.h...](https://puck.nether.net/pipermail/outages/2020-June/013124.html)

This is because of an update in the systems from the Sprint side to help merge
with T Mobile. That happened from 12am-6am EST and Sprint's systems clashed
with T Mobile's. I'm glad I have off today.

~~~
Dirlewanger
And Verizon's reason?

~~~
kelnos
Do we have confirmation that Verizon is actually having issues, or are people
on Verizon trying to call T-Mobile customers, failing, and are blaming their
own carrier, Verizon?

~~~
sjg007
I have a Tmobile MVNO and I can call Verizon users but not vice versa. I can
also not call any Tmobile number. My wife has Verizon and can't call me but
can call others except for Tmobile #s.

~~~
leesalminen
It sounds like the parent knows how to troubleshoot accurately :).

------
blhack
What could cause all major cell phone providers to experience outages that
began at the same time in the same day?

A solar flare seems unlikely, since WiFi does not seem to be effected (for me
at least).

Routing issue maybe?

One possibility: it’s all just T-Mobile, but people are bad at reporting. For
instance: if a non technical person tries to call a t-mobile number from their
Verizon phone, and experiences an “outage”, they may report this as a problem
with their own phone rather than understanding that it was on the receiving
end (and knowing to try to call other carriers).

This could also explain problems with Facebook messenger (if the T-Mobile
outage is effecting data). It’s not a problem with messenger etc. it’s just
that messenger is the service that some T-Mobile customers are trying to use
and having it not work. So they report it as a problem with the black box they
know as their phone.

That or we are getting hacked. And if that’s the case then buckle in and
charge up your ham radios.

~~~
adrianpike
Easy: someone fat-fingered SS7 the same way someone fat-fingered BGP.

[https://en.wikipedia.org/wiki/Signalling_System_No._7](https://en.wikipedia.org/wiki/Signalling_System_No._7)
[https://blog.cloudflare.com/how-verizon-and-a-bgp-
optimizer-...](https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-
knocked-large-parts-of-the-internet-offline-today/)

~~~
bogomipz
No it's not the same at all. A carrier's SS7 configuration does not get
propagated out to other carriers.

~~~
jcrawfordor
Quite possible though is a significant disruption of one of the many services
like LNP which are implemented via SS7 and shared by all carriers - there is
indeed a complex set of inter-dependencies between carriers that are all
somewhat generically referred to as part of the SS7 system, several of which
are required to successfully set up a call or deliver a text message.

~~~
bogomipz
All the major carriers maintain their own local LNP database. The only updates
that would be distributed via a third party(NPAC) would be recently ported
land lines.

------
adamfeldman
Update #2 from T-Mobile's President of Technology (3:02 PM PT):

"Teams continue to work as quickly as possible to fix the voice & messaging
problems some are seeing. Data services are now available & some calls are
completing. Alternate services like WhatsApp, Signal, iMessage, Facetime etc.
are available. Thanks for your patience."

[https://twitter.com/NevilleRay/status/1272650750665953280](https://twitter.com/NevilleRay/status/1272650750665953280)

~~~
nabakin
Signal! I'm glad to see that one in there and ahead of iMessage no less!

~~~
notRobot
The apps are listed in reverse alphabetical order ;P

------
ankurkwv
[https://downdetector.com/](https://downdetector.com/) looks pretty wild right
now.

~~~
Shank
This is a good case of causation and correlation being conflated. Of course
T-Mobile users will report problems with YouTube even if it’s just their
carrier. Any localized ISP problem will show up in outage reports for top
websites because that’s what everyone uses.

~~~
sjg007
I mean, the thing is.. they can all get to downdetector.. so :)

------
LeoPanthera
Twitter moved really fast on this one, and not in a good way. 4chan created a
meme map declaring that the USA was under a DDOS cyberattack, and all of
Twitter apparently immediately decided that this was the truth - including
some US senators.

(It does not appear to be a DDOS attack.)

~~~
drunkcatsdgaf
youre doing the same thing. 4chan didnt create a "meme", the map youre seeing
is from digitalattackmap.

~~~
wybiral
How do people get digitalattack map to show today? I can only see yesterday on
there and some people are posting images that say the 15th.

Either way, it's being taken out of context. There are always DDoS attacks
like that. The past couple of months has had constant traffic on their
timeline.

~~~
drunkcatsdgaf
some other maps are showing the same stuff, but DDOS maps are inconsistent due
to how the data is gathered. None of them are truely a panopticon.

I have my own theory that these sites are actually the ones astroturfing this
stuff when services go down.

------
jedberg
It's a good thing there are only three wireless providers so this was easy to
track down. /s

~~~
dehrmann
There were going to be three one way or another; Sprint wasn't a viable
carrier on its own.

------
DebtDeflation
I'm on T-Mobile and can confirm that I have been unable to make or receive
calls for the past hour. Also, my home Comcast internet seems to be operating
at dial-up speed and with insane latency. Something big is going on.

~~~
ngngngng
How about texts? My wife and I couldn't send or receive texts for a while
earlier today.

~~~
edoceo
Three hour delay on texts from Twilio to T-Mobile. No voice connect from
Twilio to Verizon, delayed and too.

------
koolba
Fun fact: If your 2FA is based on SMS then you can’t log in until this is
resolved.

~~~
partiallypro
Off topic, but I had 2FA setup on my phone for various things, and had it
stolen overseas. I was able to lock the phone and erase it...but I was stuck
overseas without being able to access anything or reset it. There has to be a
better way.

~~~
0xdeadb00f
TOTP with password protected key backups in the cloud?

(Bonus: recovery keys you keep offline, in your wallet or something)

~~~
partiallypro
I did have some keys backed-up in the Cloud. One issue is that because I was
overseas a lot of security protocols locked me out of my accounts -despite- my
keys and required an SMS message first...which of course I didn't have a phone
with my SIM, so it was impossible to do.

------
JCBird1012
There were early reports that Level 3 was having some routing issues earlier
today, but it's been confirmed that they're not having any problems whatsoever
(for anyone who may have been suspecting that)

[https://twitter.com/zackwhittaker/status/1272648878639255553...](https://twitter.com/zackwhittaker/status/1272648878639255553?s=20)

------
PascLeRasc
Oh, so that's why no one at my work can get on our VPN now. We need to do away
with SMS 2FA immediately.

~~~
vmception
Why are _you_ using SMS 2FA?

~~~
PascLeRasc
I use 1Password 2FA everywhere I can and choose services specifically for that
reason. I just switched cell providers today because Verizon didn’t offer it.
I don’t have influence over my employers IT though.

------
hnrodey
T-Mobile user here, had to restart my device (iPhone) in order to make calls.
The data network seems stable as I was able to FaceTime another person via the
T-Mobile data network.

~~~
tzs
Same here. Just read about this outage, confirmed I could not place calls.
Restarted my iPhone, and now I can place calls.

------
thebean11
Oh interesting, I couldn't get a verification code from Chase earlier today.
Just tried sending a text to myself; I can send but not receive texts. That's
pretty wild, it's been like that for >4 hours for me it seems.

~~~
tzs
I tried logging in to a couple financial sites that should have sent me text
codes, but they did not even try. They just accepted my name/password and went
to my account page.

I tried from a Firefox private window, my work Firefox profile (which has
never went to those sites), a Chrome incognito window, and a Chrome instance
being run under Selenium, so make sure I had no cookies from a previous visit
to allow bypassing the text code.

So now I'm a bit confused. I hope the sites aren't doing something stupid,
like associating successful text code verification with the IP address, and
caching that on their end.

Maybe they've temporarily turned off requiring text codes to avoid locking
people out during this outage?

------
zarmin
Whatever is the source, I really hope it's not malicious. The US is
vulnerable.

~~~
DebtDeflation
Uh oh.

[https://www.digitalattackmap.com/](https://www.digitalattackmap.com/)

US and Brazil both under a massive DDOS attack right now.

~~~
kelnos
If you run it back, it looks like that's been the case for at least a couple
weeks now. I guess that's unrelated to what's going on today?

------
loughnane
My wife and I have T-Mobile and so couldn’t call each other... but we could
over signal.

Not surprising, but still useful.

EDIT: if all the other comments are correct and this is _really_ widespread
then I am a little bit surprised signal is working well

------
krustyburger
A friend’s phone has been non-functional for hours now. With the size of the
T-Mobile user base, this outage almost certainly has already had real human
costs in the form of an inability to summon emergency services.

~~~
zamadatix
Depending what is causing the outage 911 may still function. Or it may not.
Just spreading to not assume 911 is unavailable because normal calls don't
work/your plan expired/whatever else is going on as 911 gets a few special
technical perks that make it available more often than normal calls.

~~~
jaywalk
Exactly right. 911 calls are handled differently at almost every step of the
process, starting with the handset and carrying all the way through to the 911
call center.

~~~
dtmmax33
[https://www.clickorlando.com/news/local/2020/06/15/cell-
carr...](https://www.clickorlando.com/news/local/2020/06/15/cell-carrier-
outages-impacting-911-system-in-several-areas-in-central-florida/)

911 in Central Florida is having issues.

------
jeffbee
Can't call from landline to t-mobile iphone, just dead air, not even
busy/fast-busy/three-tone call fail.

~~~
Multicomp
Same. AT&T to T-Mobile line. Dead air, insta hang up.

Signal messages appear to be unaffected so far, so one more anecdote that data
works, but IDK if they were delivered to my group warning them about this via
cellular data or wifi.

------
serf
What's up with the networks that went down a few hours staggered from the
rest?

Frontier Networks, Playstation Network, and Spectrum all went down early this
morning; and I mean actually down. I had a Fingbox at my parents house tattle
on Frontier Networks as soon as the connection got less spotty, and then the
network dropped again.

The ISP/data failures are more interesting to me, to be honest, than the cell
networks. I wonder if they're related. The cell networks all seem to share a
lot of cascading failures, historically. It's intriguing that something is
happening to affect the rest of the world.

The cynic in me says that someone just re-routed the internet through whatever
new DPI-esque system is now in place, but I hope the reason for this mess is
more innocent.

------
soheil
Come to think of it seems like telecomm companies have been elevated to the
national security level of importance. I'm not sure if it has always been
true, but with more and more reliance on tech that depends on these companies
the danger is escalating. Does anyone know if these companies are vetted and
inspected on any regular basis by government agencies the same way airlines
for example are? A trivial mistake by one of these carriers can have dire
ramifications for the entire nation as it seems to have been the cause of the
current outage.

------
peter_d_sherman
What might be needed is something like voice-packet across voice-network
tracing/logging... sort of like, given a voice connection ID, tag packets
going through it in a certain way, when routers or forwarding equipment en
route see this tag, then send back an "I received the packet at <timestamp>"
message, or something like that.

If something like that already exists, then T-Mobile should know exactly where
the network is failing, and why.

If not, then maybe it, or something like it would be good to have...

I have learned from many years of software engineering that a working system
is NOT proof that the system will continue to work into the future, that good
logging/tracing of all system components is a necessity not a nicety, and that
if you can't tell exactly when/where/why/how your system failed within 5
minutes of looking at log files -- you need to revamp your logging system so
that that becomes possible.

If it takes T-Mobile more than 5 minutes to accurately diagnose the root cause
of this by looking at log files, then that is proof (to me, as an Engineer)
that their logging system ranges from suboptimal to non-existent, and that
fixing that so that this never happens again (via fixing their logging system)
-- should be their absolute first corporate priority -- after whatever is
causing this is fixed.

Just one man's opinion.

I will now entertain disagreements and flames...

------
jaden
Several other networks are showing up on downdetector.com too, including
Sprint, U.S. Cellular and Consumer Cellular.

------
glitcher
Can confirm in AZ. My wife and I are on T-Mobile and are unable to call each
other, with both of us at home.

~~~
atlgator
I know this is the digital age but you could just try speaking to her
directly. /s

~~~
glitcher
Then how would she ever find her phone? ;)

~~~
737maxtw
Can confirm, buying my significant other an iWatch cleaned up my call history
from that point forward considerably.

~~~
elliekelly
The Find My iPhone feature of the Apple Watch is the best thing ever. I don’t
know why Apple doesn’t advertise it.

------
jdofaz
I called my t-mobile number and got a busy signal, haven't heard that in a
while.

------
newman8r
Perhaps incidents like these will increase the popularity of amateur radio.
I'm curious if there's any studies on the potential impacts of the US carriers
begin down for a day or a week or more.

~~~
Multicomp
My hope is for someone to complete a smartphone-based ad hoc mesh network that
can use bluetooth, wifi to pass messages along, up to 6 hops or similar.

Currently, the closest thing I'm aware of for that is cwtch which is a
ricochet.im rewrite

~~~
elliekelly
I have no idea what it’s called but when I was in Cuba 5 or 6 years ago they
used something like this on their mobile phones. They basically had an
“internet” via bluetooth but without needing to connect to the “real” internet
which was very expensive and monitored by the government.

~~~
adrianpike
SNET, it seems. Awesome!
[https://en.wikipedia.org/wiki/Internet_in_Cuba#SNET](https://en.wikipedia.org/wiki/Internet_in_Cuba#SNET)

~~~
pubutil
Cuba is amazing at making the most of what they have. I hadn’t heard of SNET
before this, but I know they’ve kept classic cars running for decades out of
necessity

~~~
elliekelly
I am not exaggerating in the slightest when I say that Cuba must have the
highest concentration of “hackers” of any place in the world. We were on a bus
going to the far west end of Cuba (where not many people travel to) and it
broke down in the middle of no where and the bus driver managed to fix it with
some scotch tape and a twisted up plastic shopping bag.

I’ve been to 70 countries and the Cuban people are hands down the most
entrepreneurial of them all.

------
AznHisoka
DDOS is trending on Twitter now. Could it be an attack?

~~~
adrianpike
Unlikely that an attack would manifest itself in this way. I think it's an SS7
screwup since it seems that signaling is mostly what's gone pear shaped.

~~~
LinuxBender
SS7 would not affect website uptime. All the carrier websites are having
issues.

[edit] As others have pointed out, if b-number routing is failing, then their
support sites would be flooded with people checking their account. So maybe
the subscribers are in effect overloading the sites. That would make sense.

~~~
Rebelgecko
Couldn't that just be the result of tens of millions of people simultaneously
trying to contact support?

~~~
LinuxBender
Could be. With SS7 outages, first things you would see is b-number analysis
failing (call routing) and SMS text messages failing.

------
supernova87a
Just to be clear, this is only happening to POTS (?) dialing phone numbers, is
that right?

It shouldn't (or doesn't) affect Skype/Whatsapp/iMessage/Facetime/etc types of
data calls does it?

Or is it T-Mobile, Verizon, AT&T _subscribers_ and their phones, cellular
phones + data connections?

~~~
jabedude
> It shouldn't (or doesn't) affect Skype/Whatsapp/iMessage/Facetime/etc types
> of data calls does it?

Anecdotally, this is correct. I was unable to place a normal call today, but
was able to Facetime the other device.

------
SSLy
Facebook is experiencing issues regarding chat (including on Instagram) in
Europe too. Might be related?

------
coronadisaster
The NSA implemented a new system?

~~~
acruns
it would more likely be that one of their systems being down would cause this
;)

~~~
coronadisaster
either way, it is very suspicious

------
ohhhwell
Google Fi from iPhone to Android worked after a long delay. Failed the other
direction :|

------
mke
The worst part: we have to wait 5 years to hear about this on Darknet Diaries!

------
hindsightbias
The test before the storm

------
kossTKR
Why are so many people in this thread sure that this is "just Sprint merger
mishap" when i and many others have been unable to use Facebook Messenger and
Instagram chat in Europe?

What does those services have to do with US cellular networks?

~~~
mynameisvlad
Exactly, what _do_ those services have to do with US cellular networks?
Nothing.

Hence why nobody is talking about them in this thread.

~~~
kossTKR
I came here from the DDOS thread on the HN frontpage because Messenger was
down (and instagram DM's), which i haven't tried before here in Scandinavia in
my 10 years of FB. Same for my colleagues.

I see at least 3 other people mentioning it in this thread, and lots on Reddit
and on Twitter in connection to these articles.

That for me was at least was at little interesting when the overall theme of
this new "incident" has been "something large scale going on".

------
_sbrk
Ijit Pai is on the case. Everything will be better soon.

The current FCC leadership is about as inept as the Fed, except that they're
not ashamed to admit they're selling out to carriers.

------
blakeg
Major ddos attack

------
ourcat
DDOS Attack apparently.
[https://twitter.com/OutageAlert2020/status/12726387880119951...](https://twitter.com/OutageAlert2020/status/1272638788011995141)

~~~
ourcat
Confused why this got -4 on it?

Is linking to graphs about the issue frowned upon here?

When submitted, it looked quite relevant to the discussion within the minutes
beforehand.

~~~
Pwntastic
Likely the same reason that a few other comments were downvoted: there wasn't
an actual DDOS attack.

Additionally, your linked graph doesn't actually contain any context or
resources that could be used to verify it. It is in fact an uncredited photo
from [0] which was refuted in the children of a few other comments[1,2] that
were available at the time of your post

[0]: [https://www.digitalattackmap.com/](https://www.digitalattackmap.com/)

[1]:
[https://news.ycombinator.com/item?id=23533054](https://news.ycombinator.com/item?id=23533054)

[2]:
[https://news.ycombinator.com/item?id=23533510](https://news.ycombinator.com/item?id=23533510)

️

