
O2 'to seek millions' in damages over data outage - bauc
https://www.bbc.co.uk/news/business-46499366
======
tiemand
Ericsson said last week that "an initial root cause analysis" had indicated
that the "main issue was an expired certificate in the software versions
installed with these customers".

~~~
tialaramex
I suspect I'll never know, but I would be really interested to know what sort
of certificate. Was this part of some larger public system or was it a purely
internal PKI? If internal, did the certificates _do_ anything of value or did
they purely make the system more fragile because somebody thought it ought to
have certificates?

One of the arguments in favour of abusing the Web PKI to do other things is
that tooling for the Web PKI is in a relatively good place. And a lot of other
things that you might ideally hope exist actually DO exist for the Web PKI.
There's independent oversight, somebody is actually reading those yawn-making
audit reports, it isn't perfect but it's not just make believe either.

But on the other hand, the Web PKI is ours, and so if it suits us to change it
we don't really care that this is annoying for your payment card systems, jet
aeroplanes, nuclear submarines or whatever else you've duct-taped the Web PKI
into. This was fun to watch with SHA-1 for example and is currently causing
some fallout for names with underscores in (DNS can handle a name with an
underscore in it but they're prohibited for hostnames. Lots of people ignored
that, but that's their problem, not ours and they are not happy about that).

~~~
becauseiam
Ericsson have both their own PKI infrastructure[0], at least for software
integrity checking (however that certificate is valid since March this year,
and the CRL[1] it refers to is empty), in addition to using other certificate
authorities for everything such as the hosting of websites to internal
infrastructure[2]. I suspect it wasn't any of the above - rather it is the PKI
that is used in running the IPSec networking done between carrier's RANs and
other parts of core network[3], which is probably Ericsson's own internal CA.

[0] [https://www.ericsson.com/en/about-us/enterprise-
security/pki](https://www.ericsson.com/en/about-us/enterprise-security/pki)

[1]
[http://crl.ericsson.net/Ericsson_Software_Deliverable_Integr...](http://crl.ericsson.net/Ericsson_Software_Deliverable_Integrity_Protection_Root_CA_A1.crl)

[2] [https://crt.sh/?q=%.ericsson.net](https://crt.sh/?q=%.ericsson.net)

[3]
[https://en.wikipedia.org/wiki/System_Architecture_Evolution](https://en.wikipedia.org/wiki/System_Architecture_Evolution)

~~~
joecool1029
Further expanding. GPRS Tunneling Protocol (GTP)[1] is what gets used to
connected to the provider's data/voice network. This could be over any medium
(wifi, GSM, UMTS, or LTE). It's likely this was the cert protecting GTP-C's
ipsec tunnel[2] as without the ability to signal, pretty much everything on
the network goes down.

[1]
[https://en.wikipedia.org/wiki/GPRS_Tunnelling_Protocol](https://en.wikipedia.org/wiki/GPRS_Tunnelling_Protocol)

[2] [https://cyber-
defense.sans.org/resources/papers/gsec/securin...](https://cyber-
defense.sans.org/resources/papers/gsec/securing-gprs-network-infrastructure-
network-operator-039s-perspective-107183#page=14)

~~~
Rjevski
As far as I know GTP is only used for data sessions (PDP) contexts. It should
not affect handset registration and circuit-switched voice.

------
fogetti
At the same time Softbank's network was also down at the same time in Japan
for the same Ericsson issue. The scale is quite surprising.

~~~
kakkun
Softbank's press release [0] mentions that a total of 11 countries were
affected by the outage. I can't seem to find any information on what other
countries were affected though.

[0]
[https://www.softbank.jp/en/corp/group/sbm/news/press/2018/20...](https://www.softbank.jp/en/corp/group/sbm/news/press/2018/20181206_02/)

~~~
sondh
Vietnam's second largest mobile network operator Mobifone was affected for 3
hours[1]. Now we know 3: UK, Japan and Vietnam. I'm sure it's possible track
down the other 8.

[1] (link in Vietnamese) [https://congnghe.tuoitre.vn/ericsson-xin-loi-ve-su-
co-sap-nh...](https://congnghe.tuoitre.vn/ericsson-xin-loi-ve-su-co-sap-nhieu-
mang-di-dong-tren-toan-cau-20181207123405213.htm)

~~~
ksec
Wow, Hong Kong's Smartone uses Ericsson and wasn't effected. I am surprised at
how wide spread the problem is. How could Ericsson messed this up when the it
has the biggest chance to grab business from Huawei.

------
stingraycharles
This seems like a publicity stunt more than anything else. There's no way they
actually lost 100MM here, and doing this ensures the public sees Ericsson as
the real villain, and paints O2 as the victim.

It's a very clever strategy, nonetheless.

~~~
jw1224
Just some dirty back-of-the-napkin numbers, but theoretically, £100MM doesn't
seem unreasonable.

O2 have 25MM direct customers, plus another 7MM through reseller networks — so
that's 32MM people directly affected by the outage.

Presuming O2 receives an average of £30 per customer per month (which I think
is a reasonable estimate), that's £940MM/month, working out to £32MM per day
in a 30-day month.

O2 are refunding customers 2 days' worth of service for the outage, so that
would be £64MM in lost revenue.

Add on top of that brand and reputation damage — and other factors I've no
doubt forgotten to include — and £100MM doesn't seem unreasonable.

~~~
walshemj
Presumably there is a considerable amount for reputational damage.

~~~
jw1224
I hadn't considered it when I wrote my earlier comment, but I wouldn't be
surprised to see O2 receive their own reputational damage invoices from their
reseller networks (O2, Tesco, etc.) in relation to the problems.

