
BT fault hits broadband users and banks - 0xbadf00d
http://www.bbc.co.uk/news/technology-36844712
======
thomseddon
It's a power outage in Telecity Harbour Exchange, from the LINX mailing list
this morning:

08:12

\- We are currently experiencing an outage impacting some of our equipment
located at Telecity/Equinix Harbour Exchange in London. It is impacting both
the Juniper and Extreme LANs.

\- We are still investigating to get further information, but we have lost all
management access to the devices at the moment.

08:45

\- We have had confirmation from Telecity/Equinix that there was indeed a
partial loss of power impacting one of the power feeds.

\- Power failed at 07:55 BST and was restored at 08:17 BST for the majority of
our equipment.

\- Unfortunately one of our edge routers (edge4-tch) on the Juniper LAN is
still down at the moment.

09:33

\- Power to edge4-tch has also been restored now. The router has been back up
since 9:15 BST.

\- All member services should be restored.

\- We have an engineer on site and are working with Telecity/Equinix to
understand why we lost all power to that device.

\- We are also waiting for a full report from Telecity/Equinix about the cause
of the power failure.

\---

We don't peer with BT so ou traffic was OK but we did notice loss of service
from AWS to BTNet

~~~
mattlondon
Great update thanks.

Perhaps I am being naive, but I thought data centers had back-up power to
cover this exact problem? I.e. battery power until the generators kick in...
or is this how it used to be done and these days no one bothers and just
relies on fail-over to other sites????

~~~
pyvpx
despite best efforts, sometimes the generators don't kick in. sometimes not
all of them kick in. sometimes one ATS in one feed in one section that just
happens to house one half of an important switch chassis doesn't work as it
did during tests last quarter.

~~~
snowy
True. But Data centers usually have an A and B power systems independent of
each other for redundancy . Equipment is usually connected to both. The
failure of either should not cause an outage. (Unless some one is stupid
enough to connect core network equipment to only A or B power)

~~~
marcusr
Sometimes only A power or B power is delivered to the rack (otherwise you
could receive different phases).

Power failures also occur sometimes when the issue is on the wrong side of
where power is distributed - we've lost power to our servers when maintenance
has gone wrong for example.

~~~
snowy
From my point of view if you end up with mission critical equipment connected
only to A or B power then its engineered wrong.

~~~
edwhitesell
Very true, but sometimes this happens at fault of the facility, not the
affected company. I've been in a facility experiencing widespread power issues
because customers were incorrectly delivered A+A or B+B power, instead of A+B.

------
Fifer82
The best thing about being BT is that you own everything so you don't have to
justify anything to anyone. Since half of the lords have long term shares in
the monopoly the moment that it went private, then your back is covered as far
as law is concerned so long as you pay your dividends. You can just laugh as
the customer goes somewhere else, because you will still be picking up 75% of
the broadband bill.

I also love how BT pays out so much to shareholders, and the public purse
bankrolls the infrastructure, then everyone pays BT wholesale. What an
outstanding deal.

BT is corrupt and I would like to see it fragmented.

~~~
spacecowboy_lon
BT doesn't own Telecity I suspect that if they did it wouldn't have happened.

And BT pays out around 3% dividend to its owners which is not excessive a lot
of BT's detractors have an axe to grind (Sky, Murdoch and Virgin) and want to
cherry pick customers and to hell with the USO.

------
anexprogrammer
I find the Talktalk chairman's quoted comments hilarious, though I assume
they're not specifically in relation to this outage. If it were Talktalk's
network spending would drop about 90% and complaints rise a similar amount.
It's not like TT would be unaffected by a Telecity outage. I don;t think TT
ever cared what their customers think, just reducing prices and service to the
bone.

~~~
pjc50
OpenReach own _all_ the ADSL wiring - that's what TT have to buy from them.
Most consumer ISPs are quite 'virtual' and dependent on BT for actual
provisioning.

The manager of a tiny geek-friendly ISP, Andrews&Arnold, frequently complains
about this on his blog: [http://www.revk.uk/](http://www.revk.uk/)

~~~
anexprogrammer
Well they own exchange to home. Since unbundling it's a bit fuzzy as most of
them have their DSLAM in the exchange and pay rents for space in exchange and
run virtual over BT's backbone and so forth. But at least it freed up a lot
more competition. Hence Sky asking for Fibre cabinet unbundling recently.

If it was A&A or Zen's chairman quoted there'd have been no need for snark. ;)

------
shocks
Graphs here:
[http://www.ukinternetreport.co.uk/?p=d&id=215](http://www.ukinternetreport.co.uk/?p=d&id=215)

------
voltagex_
UKNOF's mailing list is private so it's hard to get a handle on what's going
on, but from what I've heard, a DC went down and BT's infrastructure hasn't
recovered.

~~~
forcer
There isn't any insightful discussion going on. Suprisingly. So you are not
missing out on any info.

------
chriscoffee
Apparently down to Telicity Group DC having power failure which BT heavily
rely on

~~~
dijit
Which DC, there are many.

Two London ones I can think of is north of the city (Oliver's yard) and
docklands.

~~~
ropiku
Arstechnica says Telehouse North Docklands
[http://arstechnica.co.uk/business/2016/07/bt-isps-
telehouse-...](http://arstechnica.co.uk/business/2016/07/bt-isps-telehouse-
north-major-outage/)

~~~
mrseb
We've updated the story - did some more investigation. Definitely looks like
it was Telecity.

------
forcer
We run one the most popular speed test site in the UK at
[http://www.broadbandspeedchecker.co.uk](http://www.broadbandspeedchecker.co.uk)
. We are getting 20x more people testing today than usual. Also, because Ookla
is down.

------
charlieegan3
It'd be nice if we got email notifications about this. I wasted time trying to
work out what was going on (and if the issue was in our local network).

~~~
bonaldi
You want to them to _email_ you to tell you your internet service is down? Er
...

~~~
Swizec
Most people have redundancy. Computer has wifi, phone has LTE. Usually
different providers.

I would love an email thst my internet service is down. Or text.

~~~
vidarh
Incidentally BT also has infrastructure or delivering SMS to landline voice
numbers via text to speech - or at least had as of a couple of years ago - so
as lon as it's the broadband service and not the end user lines that were down
they'd still be able to deliver it.

------
chrisstu
The infuriating this is that whenever this happens at BT, they say nothing
about the cause and nothing about how they will prevent recurrence. As a BT
Infinity customer, I've had a total loss of internet several times over the
last couple of years and never any feedback from BT afterwards.

~~~
tonylemesmer
They seem like one of those obnoxious organisations that think if they provide
any information then it will only invite a shitstorm. Unfortunately too many
people still chose them as a provider. I nearly. Nearly switched to them last
month because of their prices and BT Sport package. But fortunately I'm stuck
on a 12 month contract with IDNet. Shame.

~~~
radiowave
As a colleague of mine used to say when I worked there: "It's a good thing
we're not a communications company."

------
stevesimmons
My BT Infinity home broadband in south London is affected. 'mtr' shows any
traffic via the BT-owned IP address 31.55.187.177 gets 88% packet loss when it
hits that router. Web sites using other routes are ok.

------
jsingleton
And again the next day:
[https://news.ycombinator.com/item?id=12135794](https://news.ycombinator.com/item?id=12135794)

------
billpg
I noticed this morning that HTTPS was failing but HTTP was fine, but it "fixed
itself" after ten minutes. No idea if this was related.

~~~
tehbeard
Had issues yesterday where BT was complaining that my machine was using non BT
DNS (all our machines at home use google's 8.8.8.8 / 8.8.4.4)

Slightly concerning since we never enabled parental controls (Which the
message implied was the reason it wanted us to use BT DNS) and now hearing
that somehow only https traffic was affected by an issue.

~~~
redcalx
As an aside, I've found BT's DNS to be very flaky; sometimes down (or
unreachable at least) for many hours at a time. I switched to google's and
never had any problems since.

------
jayflux
Is there a reason there are so many Data Centres in Docklands? Is there
something about that area that makes it great for datacentres?

~~~
dx034
Most investment banks are based there (Canary Wharf), they pay a lot of money
for low latency.

------
CM30
Huh, so that's why my internet connection was timing out on a lot of sites
earlier today.

------
anon4711
Their customers should rightly be outaged.

------
8draco8
This is probably the reason why linux.dropbox.com was down for brief time
today

~~~
pbhjpbhj
Out of all the sites you could have mentioned why did you think that (the deb
repo for Dropbox's packages) was pertinent to this story?

~~~
8draco8
Because it was. This and ubuntu repo was not working on my PC during BT
outage, other sites worked. After BT sorted it out I didn't have any problems.
I asumme that it was the issue with some BT connection between me and dropbox
server

------
plq
Might be a better organized coup attempt from the remain supporters :)

~~~
J_Darnley
Middle and upper class folks cutting off their Netflix and Spotify
entertainment while also leaving them unable to tweet incessantly about it. I
don't think so.

------
jasonkester
Living in England, you really need a downtime strategy for your internet
connection. While it's seldom a complete country-wide thing like this one, you
can still expect your broadband to drop out for up to an hour at a time at
least once every day.

BT is just that incapable of providing DSL to residences.

I live in a tiny village in rural France half the time, and never really think
about the internet connection. Just like you never worry that water won't come
out of the kitchen sink when you turn it on.

But then I come back to England for the summer and there at the front of my
mind several times a day is the Internet connection. Why is it super slow all
of a sudden? Hey, it dropped again. I hope it comes back soon because I need
to check stuff in. I have a personal hotspot set up on my phone that gets used
within my house on a weekly basis. There's an OpenReach van parked out back a
couple times a year fixing yet another fault on our line.

Nowhere else I've lived has it been anywhere near this bad. I look forward to
the day where BT has some actual competition.

~~~
ChrisMeek
I've been with BT for a very long time (5+ years) and other than during
unseasonably violent storms this is the first time that I've suffered any kind
of outage at all. Not only that but I'm in the middle of nowhere and still get
fibre (to cab at least)

~~~
Nexxxeh
Then you're been very lucky.

[http://www.theregister.co.uk/2016/02/03/bt_blames_faulty_rou...](http://www.theregister.co.uk/2016/02/03/bt_blames_faulty_router_for_mass_internet_outage/)

[http://www.theregister.co.uk/2015/11/17/bt_outage_three_days...](http://www.theregister.co.uk/2015/11/17/bt_outage_three_days_fix/)

[http://www.theregister.co.uk/2015/07/29/bt_broadband_in_broa...](http://www.theregister.co.uk/2015/07/29/bt_broadband_in_broadbased_brownout_and_titsup_incidents/)

[http://www.theregister.co.uk/2014/12/12/bt_infinity_working_...](http://www.theregister.co.uk/2014/12/12/bt_infinity_working_to_fix_the_problem_after_three_days_of_outages/)

[http://www.theregister.co.uk/2014/06/30/bt_major_dns_outage/](http://www.theregister.co.uk/2014/06/30/bt_major_dns_outage/)

[http://www.theregister.co.uk/2014/02/25/bt_cable_theft/](http://www.theregister.co.uk/2014/02/25/bt_cable_theft/)

And that's in just over two years.

