
Dutch Telephone Outage Takes Out Nation's Emergency Number - perfunctory
https://www.nytimes.com/aponline/2019/06/24/world/europe/ap-eu-netherlands-phone-outage.html
======
jacquesm
It's a huge mess. It's been going on for _hours_ and there is still no status
report on how long it will take to fix this. Fire, Ambulance and police are
all affected, and half the regular phone network is also out so there are no
alternate paths available. Big fuckup, people will die because of this and the
post-mortem will be a very interesting read.

~~~
LeonM
What makes you think a telco will ever do a publicly accessible post-mortem
after an outage?

They will probably have to do one under government pressure, but it'll be so
heavily NDA'd that no one will ever read it.

~~~
Zenst
For many business customers they will be contractually obliged to provide a
root cause analysis (pretty standard for telcos).

So if they don't publicly out the report, then the odd's of it staying
contained out of public eye are pretty darn high.

The only area in which they could curtail it's release would be if some
terrorist type factor or related activity that they could class in that catch-
all definition. But even then, details will emerge and it would only delay its
publication to fulfill their contractual obligations (think mobile telcos and
large business that contract network capacity and links from them with lovely
contract SLA's).

~~~
dredmorbius
s/darn high/darn low/ ?

~~~
Zenst
Ah yes, my bad - meant that it would be highly unlikely to stay out of public
circulation.

~~~
rags2riches
High odds are for unlikely events. Low odds means something will probably
happen. People often use the expression the other way around.

------
Kim_Bruning
I had just finished work on an asterisk system when the issue hit. I was so
frustrated that I couldn't find the issue.

... Until I checked the news and found out that things weren't working for
anyone.

Quickly put the customer onto the secondary fallback line, and now they're
actually happy with me ;-)

~~~
jeffrallen
So it was YOUR fault!

~~~
DonHopkins
I was playing SimCity, extremely immersed in the game, and I'd just bulldozed
a road that had a power line going across it, then the power in the actual
house I was in suddenly went out! I was stunned, and regretted what I'd just
done, afraid it was my fault.

------
lowdose
In the first emergency alert a completely wrong alternative number being
broadcasted, plus a spelling error in the police station. After an hour the
first mistake was corrected. I assume the authorities are collectively hanging
in the curtains under the influence of a wildly spread panic attack.

~~~
himlion
Yeah, amazing how they manage to broadcast a wrong alternative emergency
number to millions of people.

~~~
lowdose
I would say a message template for the most common alerts would not be an
excessive luxury.

------
skrebbel
I just got an "NL Alert" about this, some kind of faux SMS sent straight to my
phone. With a WhatsApp number, an alternative phone number, and a Twitter
account. They seemed pretty well prepared. Emergency service via Twitter! Call
me impressed if they can pull that off.

~~~
dr_dshiv
It is the same kind of format as the Amber Alert things I used to get in the
states. How do they push that out? Based on location, cell tower usage, sim
card?

~~~
namibj
It's like wifi broadcasts. It transmits to all devices in cell. It might do
this for many cells and many operators/physical networks at the same time.

~~~
fsiefken
I didn't get them on my cellphone which was connected through the KPN mobile
network, appearently there is some provider dependency here

~~~
tjohns
Each provider uses a different cell tower (or at least their own antennas on a
shared mast), with their own independent radio frequencies.

"All devices in cell" is still accurate. Each cell belongs to a single
provider.

It's up to the network backend to make sure all providers get the emergency
message at the same time to forward.

~~~
jacquesm
> Each provider uses a different cell tower

This is incorrect. A lot of providers share infrastructure, there are even
providers without infrastructure.

~~~
tjohns
You are of course correct. But getting into the details behind MVNOs and
roaming is a much more complicated topic than I'd want to type into a small
comment box, and not really relevant to the issue at hand. :)

My point is just that CMAS-style cell broadcasts need to be repeated by each
cell site, and cell sites aren't _universally_ shared by all carriers.

~~~
jacquesm
It appears the problem here was at a higher level, the signalling fabric that
transports the data for call setup, teardown, roaming and SMS, and to
complicate matters even further apparently the 4G network is where the problem
originated but it had effects far outside that. That's also why the whole
thing appeared so patchy from outside.

------
Tharkun
This has happened several times over the past couple of years in Belgium as
well. Bugs and mishaps happen, and I'm sure the systems were designed to be
fault tolerant. But perhaps it's time to stop relying on POTS alone for
emergency numbers? Is there any particular reason we can't reach emergency
services over the internet? Even something as banal as Skype or Facetime would
make for a decent backup.

~~~
londons_explore
Emergency services are apparantly too important to allow something as
'unreliable' as skype/facetime.

I don't understand why regulators can't drop the reliability requirements and
understand that if skype, facetime, whatsapp, viber, twitter, etc. _all_ have
99.9% reliability, then the chances of them all being down at once is far
lower than your POTS line being down.

Just have a #911Emergency official hashtag and have a bunch of people reply to
it. That's the way the people want it.

~~~
FabHK
1\. POTS aims at "5 nines" reliability. [1]

2\. > the chances of them all being down at once is far lower than your POTS
line being down.

That depends _crucially_ on the correlation. If there are common fault modes
(such as: internet down), the probability might be way higher. (Remember, it
used to be common wisdom that it was possible that one or two home owners
won't be able to pay their mortgages, but the probability that many will be
unable to pay their mortgages is so low that all these MBS and CDO's can be
rated AAA...)

3\. EDIT to add: Of course, adding communication options can only increase the
probability that you can reach your emergency provider, but there's a reason
that Skype etc. used to come with these big disclaimers that they're no
replacement for POTS, namely the excellent reliability of POTS.

[1] "Although POTS provides limited features, low bandwidth, and no mobile
capabilities, it provides greater reliability than other telephony systems
(mobile phone, VoIP, etc.). Many telephone service providers attempt to
achieve dial-tone availability more than 99.999% of the time the telephone is
taken off-hook. This is an often cited benchmark in marketing and systems-
engineering comparisons, called the "five nines" reliability standard."

[https://en.wikipedia.org/wiki/Plain_old_telephone_service#Re...](https://en.wikipedia.org/wiki/Plain_old_telephone_service#Reliability)

~~~
walshemj
Better that Five nines - experiencing a major switch failure should be
something you experience once every two or three generations.

~~~
londons_explore
In the UK, BT phone lines go down (cut existing calls) for about 5 minutes
about once a week between 2am and 3am... I'm guessing some software update?

------
consp
The alternate path is whatsapp and twitter as mentioned in the alerts. Or ask
the person with t-mobile/vodafone connections to call the alternative number.
Though I do agree about the other parts.

~~~
jacquesm
The alerts have reached only a fraction of all subscribers. Interestingly,
some KPN customers can still call each other.

~~~
consp
My empirical study for this is as follows: I have 7 phones on and around my
desk, of which are 3 iOS devices (4s and newer), and 4 android (4.4 and up all
different manufacturers, sony, samsung, hmd, one plus brand) with different or
no operator sim installed and all but the 4s got it.

I guess it is cell related in that case.

edit: the 4s was subscribed to the kpn network. Others were not or I can no
longer check. Strange thing is I'm getting them first through my personal
phone's second sim (vodafone, primary is t-mobile).

~~~
jacquesm
Are those on KPN? If so then that is interesting, if not then that is pretty
much in line with the article. SMS and voice traffic _within_ the KPN network
is just fine, going outside or dialing in is not working and SMS traffic from
the outside in also does not work (so far).

No SIM is only interesting in sofar as those phones have attached themselves
to the KPN network, they could choose _any_ network for emergency services.

~~~
WildGreenLeave
I'm having an iPhone XS, emergency alerts enabled and I did not receive a
single message. I'm using KPN as my provider and am on the KPN network.

Interestingly all of my coworker and friends did receive a message, but as far
as I'm aware they are all on a different network.

I do not like that the supposed 'emergency' messages dit not receive my phone
at all. Not even delayed.

------
softgrow
I used to work on reliability of the emergency number in South Australia which
could stand: \- failure of reserved circuits from outlying exchanges \-
failure of trunk exchange \- failure of operators exchange \- evacuation of
operators \- failure of backup operator exchange \- evacuation of backup
operators \- failure of special circuits to emergency services \- evacuation
of emergency services operations rooms \- failure of backup lines to emergency
services \- emergency services advertised alternate numbers (spread across
multiple exchanges in some cases) etc etc

Can't work out why just the emergency service number failed not other things
as well. Bit surprised.

------
Zenst
I find it interesting that they published alternative emergency numbers upon
social media and more importantly, push notification via mobile phone
networks.

Why couldn't those mobile phone networks of just rerouted the emergency number
to the alternative number?

~~~
zandjager
Because not all the network was down/inaccessible; only KPN. So I'm guessing
the alternative numbers were on the still reachable networks.

~~~
Zenst
I was referring to the mobile networks able to do the broadcast and
questioning why they couldn't of done a number routing for their network to
rout all emergency calls to the alternative number. Which for their customer
permutations of use, would be far more efficient and indeed safer if you
factor in the delay in the user having to read an sms in the middle of a
crisis that dictated calling the emergency service in the first place.

~~~
Doxin
It wasn't just the emergency number that was borken. The entire KPN telephony
network was having issues. The alternative number could not be reached from
phones using the KPN network, so having emergency calls on those phones
redirect to the alternative wouldn't help much. Presumably the other networks
didn't put routing into place since that'd be a rather large configuration
change on short notice to a safety critical system.

~~~
Zenst
Again - the point I was making was. IF mobile networks were sending out a cell
broadcast of an alternative number to use upon `their` networks. THEN why
didn't they just do a routing change upon `their` networks. `their` networks
were working - hence they was able to send out a network broadcast. They
equally sent a new emergency number that was not upon KPN and as such worked.

As for large network configuration changes - pish, it really isn't that hard
and would literally be a simple update upon their network only that they could
just as easily change back. Not talking a global change update, just a per
network (mobile) that was working (or the ones that were and able to push out
a cell notification).

Hope that made it clearer for you.

------
NKosmatos
Searched for more technical details but couldn't find anything. Anyone knows
who is the main vendor of KPN? Huawei, Ericsson, NokiaSiemens, ZTE? I bet the
supplier is going to feel some 'heat' from this outage.

~~~
jandeboevrie
Huawei runs most of KPN. Bet you we'll never know the real cause just
management b _llsh_ t. The real reason probably is not spending money on the
right places (aka people, redundancy, protocols, upgrades, etc) but on CEO
lunches or stuff like that.

~~~
tinco
I thought it was ZTE, though all the new 5G stuff is going to be Huawei.

~~~
0x76
From personal experience most routers they send to their customers are indeed
rebranded ZTE routers.

This of course doesn't necessarily mean they use ZTE everywhere but can be an
indication.

------
cr1895
Here’s a live blog from the Dutch public broadcaster NOS (in Dutch):

[https://nos.nl/liveblog/2290490-112-storing-politie-geeft-
al...](https://nos.nl/liveblog/2290490-112-storing-politie-geeft-alternatief-
noodnummer-088-6628240.html)

------
tnolet
They should have used Docker in React and then this would have never happened.

~~~
majewsky
"React in Docker", that I can imagine. But "Docker in React"!? I demand to
know how this works.

------
jabbernotty
An alternative number has been spread through the national broadcasting
association (the publicly funded tv, radio and web news).

~~~
LeonM
Yep, I just received it on my phone, which wasn't fun because I was listening
to music, my phone blasted the emergency tone at full volume through my
earpods. It really hurt :(

Anyway, good to see that the emergency system works. I just received a second
message with a whatsapp number and twitter handle which you can use in case of
emergency.

~~~
DonHopkins
Oh shit -- it was a really loud jarring sound, it must have been painful. They
have recently started sending test phone alerts at the same time they perform
the regular first Monday of the month noon emergency alert siren tests in
Amsterdam. Maybe you should set an alarm just before then to make sure you're
not listening to music!

[https://www.government.nl/topics/counterterrorism-and-
nation...](https://www.government.nl/topics/counterterrorism-and-national-
security/question-and-answer/public-warning-sirens)

~~~
londons_explore
Or just turn emergency alerts off on your phone.

Average number of millimorts saved by those alerts I bet is really low.

~~~
LeonM
I don't want to turn it off, but it should obey the volume setting if
headphones are connected (your phone can detect that).

But I guess this is vendor specific, in my case it was a Samsung S9, maybe
other vendors implement it differently

------
LeonM
KPN now has a notice on the website frontpage that the outage is fixed and the
emergency number is reachable again.

------
kawfey
Any word if amateur radio operators are being put into service for this?

~~~
oneplane
That doesn't really exist much in The Netherlands. Probably because the
infrastructure doesn't really attract alternative means of communication.

------
XCSme
They sent like 5 emergency alerts on the phones, it was so confusing.

------
doctorRetro
Aside: Shout out to the telemarketer who pressured me to get a land line years
ago because they're the only guaranteed access to emergency numbers and "never
go down."

------
fbi-director
Situation has been resolved as of now.

------
huis
I'm not sure you can call this a big fuckup, we will have to wait for news
about the cause.

Yes it is bad, very bad. But I'm sure they got all the redundancy in place and
thought about so much more than we can imagine. But even then there might be
some extreme event that will throw 'soot in the food'.

I think it is just great how this event is handled. Almost everybody I know
knows the alternative alarm number. Police is in the streets so you can walk
to them. And as far as we know there are no looters taking their chances.

So yeah, bad, but we have to wait before we can mark it as big fuckup.

~~~
jacquesm
This event should not have happened in the first place _and_ it was
demonstrably mis-handled in several critical ways.

The fact that the police has responded in the proper way has nothing to do
with how KPN has handled this so far and it immediately puts in question the
retiring of the siren system used for emergencies, which for cost-cutting
reasons has been replaced by the national mobile phone alert system, which
apparently does not work when you really need it.

~~~
apexalpha
Wondering what you mean with: "demonstrably mis-handled in several critical
ways."

Other than the outage itself, not much has happened?

Also, what would a siren help in this regard? Can you play the new phone
number in morse on the airhorn siren?

~~~
jacquesm
> demonstrably mis-handled in several critical ways.

\- No fall-back system in place.

\- Alternative alert system did not work.

\- Alternative alert went out with the wrong information.

That seems plenty to me.

As far as the siren is concerned: the general alert system is meant for
serious threats to public safety, it does not require 'a new phone number in
morse', the output is a single bit: siren on: stay indoors, close doors and
windows, wait for all clear sign (three short bursts on the siren).

~~~
techcode
I'm not so sure about always "stay indoors" part - in particular in case the
dams leak and half of the country is about to be under water.

I guess depends on the height of your house - though in quite a few places
even rooftop might be below or close to sea level.

~~~
jacquesm
Those are pretty far down on the likelihood scale, though we have had
instances of that, especially near river dikes in the last couple of years. We
do not have a lot of dams here.

The more likely cause for such an alarm is chemical pollution from either an
accidental release or a fire. Their old use was for air raids but those are
even more unlikely (at the moment, of course this can change).

------
darkhorn
They should have used Erlang
[https://youtu.be/uKfKtXYLG78](https://youtu.be/uKfKtXYLG78)

I wonder what coused this issue? Dependency issue in Maven? May be they cannot
figure out what that Java error means?

------
m4lvin
Maybe related to the Cloudflare outage???
[https://news.ycombinator.com/item?id=20262214](https://news.ycombinator.com/item?id=20262214)

