
Finance world in the dark as Bloomberg terminals go offline - jacquesm
http://www.telegraph.co.uk/finance/newsbysector/mediatechnologyandtelecoms/digital-media/11544186/Finance-world-in-the-dark-as-Bloomberg-terminals-go-offline.html
======
PhantomGremlin
I don't understand how Bloomberg could be down worldwide.

Worldwide???

99% of what Bloomberg terminals are useful for can easily be replicated in
multiple data centers. Why not, at a minimum, three separate regions, e.g.
Asia, Europe, America?

These terminals lease for $24,000 a year. No discount for multiple terminals.
I wouldn't be surprised if a single big investment bank pays $50 million a
year to Bloomberg for its terminals. Michael Bloomberg is a billionaire.
Surely the company can throw a few crumbs into some redundant data centers and
availability zones.

And if it's down because of a botched worldwide software update, the people
responsible should be immediately walked out the door and told "you're too f
__*ing stupid to work here any more ".

IMO.

Sorry to be so dramatic. But the financial world depends on these terminals
(for better or for worse). They have a right to expect unrealistic
reliability.

~~~
quadhome
Bloomberg has just paid a lot for their experience of what mistakes to not
make in the future.

Walking them out the door would be a fantastic waste of money.

~~~
chrisbennet
That's the way a _smart_ business would think. I wonder how smart Bloomberg
is? It's not uncommon for the goals of employees ]assign blame, cover ass,
grab more power, short term profit] to be misaligned with the long term goals
of the business.

~~~
apaprocki
There is a lot of assumption going on (that a human messed up). Even if that
was the case I've never known it to be a place that would have knee jerk
reactions like that. The company is more interested in saying "failures will
happen, how do we prevent them from becoming major issues?".

~~~
chrisbennet
That sounds like a smart company.

------
7sigma
Former bloomberg employee here. While the bloomberg service usually runs very
well, its built mostley on spaghetti code that has been copy pasta'ed since
the 80s and 90s. It might have improved since I worked there, but there were
some serious deficiencies in development methodology.

~~~
apaprocki
I don't know when you worked there, but that is not how things are run.
Source: I work there.

~~~
raldi
Hi Andrew!

What I remember from my time there (2003-2007) was how quickly everyone was
able to ship code despite (or perhaps because of) no global style guide, unit
test mandates, or even rules about copy-pasting code. Each team was able to do
things the way they thought best, thanks to an incredibly resilient, flexible
production infrastructure that could tolerate, isolate, route around, and
quickly repair any newly-introduced bug.

The code path (Bloomberg four-letter function, to those familiar with the
service) running through the bug could nearly instantly, and if I'm not
mistaken, in a totally automated way, be flipped over to a hot backup instance
running the previous week's code, and a patch or rollback could be released to
the live code almost as soon as it was written.

Most tech companies see the problems caused by coding too fast and respond by
doing things to slow people down. Bloomberg cured the ills of excess
programming velocity by turning the speed up even further. Like Facebook's
"Move fast and break things", but with an extra, "...and then move even faster
to fix them", and in a way that (mostly) insulated customers from being
exposed to the breakage.

The fact that it's front-page news when there's a hiccup shows that it's a
man-bites-dog event. You'd be in trouble if it _wasn 't_ major news when there
was a service incident.

~~~
tomp
That's very interesting, and definitely an infrastructure many companies
should strive for and could learn from.

I'm wondering, however, not how problems were handled, but how were they
_detected_ \- code throwing exceptions is the easy case, the real problem is
code running almost correctly, but with slightly wrong outputs. I imagine
there could be many different "sensibility" checks implemented, (e.g. no
negative prices, no huge jumps price jumps), but in general it appears to be a
very interesting and complex problem.

------
ghshephard
I wonder how many people appreciate the Irony that twitter is now the
universal mechanism for people to report outages, and get updates on when
critical services are down/up.

~~~
jacquesm
All those that were early enough to see twitter go through their growing pains
probably.

------
jhull
Is there an alternative to Bloomberg? I haven't ever heard of traders using
anything else.

~~~
MDib
Depends on exactly what but FactSet. Have used both, FactSet is more user
friendly, Bloomberg is way faster once you get used to it.

------
PhantomGremlin
FYI another discussion going on. Parent link there is to an NY Times article.
[https://news.ycombinator.com/item?id=9393884](https://news.ycombinator.com/item?id=9393884)

------
whatok
IB still down for some people; think I heard 25% worldwide.

Just happy everything was up for me before getting to work.

------
chvid
It also looks like the markets are dropping sharply (SnP futures etc). I
wonder if there is a connection?

