
Bloomberg Terminals Suffer Widespread Failures - svtrent
http://www.nytimes.com/2015/04/18/business/dealbook/bloomberg-terminals-outage.html?_r=0
======
chollida1
The outage isn't that bad. I've talked to 5 other firms in Canada this morning
and haven't heard of a single one that is down.

Bloomberg's tickering plant is still working and their emsx(trading system) is
up and running, albeit a bit slow right now.

I think it was on a stackoverflow podcast that Joel or Jeff said that it makes
you wonder about people who complain about down time if they've ever built
something of their own.

Google's been down, ditto for facebook, twitter and Azure. If you build a big
service, something will eventually fail, Bloomberg is no different, if
anything they are much better than Reuters or CapIQ for availability.

I saw this quote from another comment here:

> but the widespread reliance on the $10k+ per year terminal in the financial
> services industry means that this thing is ripe for disruption.

I completely disagree, Bloomberg isn't some slow lethargic big company. They
iterate relentlessly, their help desk is the best I've ever had the pleasure
of dealing with.

If I've got a problem with the trading system, their first level help will get
me to a technical specialist in that area in under 30 seconds. If they can't
solve it then I'll talk to a developer in 30 minutes.

Bloomberg is the leader in teh financial space, not only because their tech is
the best in teh business, but because they listen to their customer better
than their competitors, they release more frequently, and when they want to
enter an area( say risk management) they go out and buy someone who is already
a leader in that field.

In short they never stop running, if anything they are increasing the distance
between them and their "peers".

As one example, they just posted a message to every Bloomberg terminal user
telling us that the issue was an internal networking issue that has been
solved.

 __EDIT __to answer this question:

> How do they protect the ticker tape thing from attacks, like say, a DDOS?

2 things:

1) This question really doesn't make sense. How do you DDOS a one way line?
They send you quotes for the tickers that you request. You can't really DDOS
them, unless you do something like request tickers over and over again in a
tight loop, in which case their api will rate limit you.

2) its a closed system. Only terminal users can access it and only through the
bloomberg terminal or their api, so they control all the access points.

~~~
hackuser
> Google's been down, ditto for facebook, twitter and Azure

The first three, at least, are free services and availability isn't a priority
for many of their customers. Bloomberg users have different requirements.

Boeing 747's are complex systems; it's not acceptable if one fails
occasionally.

~~~
cheriot
> Boeing 747's are complex systems; it's not acceptable if one fails
> occasionally.

Yet they have failed. Reality isn't always acceptable.

------
apaprocki
"Service has been fully restored. We experienced a combination of hardware and
software failures in the network, which caused an excessive volume of network
traffic. This led to customer disconnections as a result of the machines being
overwhelmed. We discovered the root cause quickly, isolated the faulty
hardware, and restarted the software. We are reviewing our multiple redundant
systems, which failed to prevent this disruption."

------
octo_t
[https://news.ycombinator.com/item?id=9393096](https://news.ycombinator.com/item?id=9393096)
previous discussion from earlier today.

------
PhantomGremlin
LOL. Maybe it was a can of coke???

    
    
       But reports from inside the company suggested
       that a spilled can of Coke in one of the server
       rooms had been responsible for knocking out
       systems across two continents.
    

[http://www.dailymail.co.uk/news/article-3043927/Did-spilt-
CO...](http://www.dailymail.co.uk/news/article-3043927/Did-spilt-COKE-cause-
City-computer-blackout-brought-world-s-financial-markets-standstill.html)

~~~
el_duderino
I honestly can't believe some (semi reputable) news sites are spewing this
bullshit.

Fuckin please.

------
akramhussein
Random anecdote: Was told by a trader a few years back that one trading desk
had both Bloomberg and Thomson Reuters systems next to one another and once
they were able to capitalise on Bloomberg being a few seconds faster to make a
net gain.

~~~
RayVR
This is really not possible. If there's any truth to this story at all you
have forgotten the real version.

------
mbrzusto
<onion> bloomberg network unable to report crash of major financial
information service </onion>

------
Scoundreller
Moderately off-topic (and don't say: "Today? Nowhere."):

If I wanted to play around with a Bloomberg for a few hours in North America,
just to see what the fuss is about, how could I go about doing this?

~~~
lmg643
New York Public library (business/finance library) on Madison Ave between
33-34th (NYC) has 2-3 terminals for open public use. Usually a short wait but
you can use them.

------
memossy
Bloomberg starts at $1,750 a month

At that level of cost there is an expectation of almost perfect up time and
multiple redundancy - the only reason people pay that much is they believe
they can get more value than that out of it..

~~~
hueving
Cost isn't necessarily tied to uptime expectations. It's tied to the value
people get from it. Bloomberg is so far ahead of it's competitors that this
minor event probably won't lose them any business.

------
bbgtoreuters
Decades of crusty code - C++ wrapped around C wrapped around Fortran, home
brewed versions of the STL, nih syndrome across the tooling. Actually
surprised outages not more common. Still, given their interview process is
basically parlour trick questions about pointers and char arrays, it is quite
telling!

~~~
bachback
Bloomberg started out as a network system independent of the Internet. Michael
Bloomberg was a network engineer. I'm not sure they ever really switched to
proper TCP/IP. This company sure looks like a dinosaur these days. However its
hard to imagine a company being able to replace even a chunk of their
infrastructure. They are tightly integrated into all institutions and there is
no other service which offers a complete coverage of all the world's financial
assets.

~~~
apaprocki
Yes, we use proper TCP/IP :)

~~~
kokey
The issue was probably something like a broadcast storm because of defective
fault detection on some redundant network kit. That won't stop people blaming
it on the fact that the stack isn't built on the latest hipster frameworks.

~~~
noir_lord
While I see your point do we have to throw the word "hipster" at everything
new?.

I'm not a huge fan of Javascript and it's proliferation of frameworks but some
of that stuff is objectively both good and useful.

~~~
santaclaus
Using the term 'hipster' to refer to the newest and shiniest web framework
always felt like a bit of a misnomer to me. A solid portion of hipsterdom is
rooted in recycling of the obscure and of the vintage (vinyl, the popularity
of the lumber sexual look, etc). It seems like the 'hipster' thing to do would
be to write a web framework in PL/I or cobol or something.

~~~
Aloha
New and obscure are all too frequently synonymous.

That said, there might be a market for a web framework in PL/I there is a
whole lot of IBM big iron out there :-P

------
anonu
Is this the first crack in the Hoover Dam that is Bloomberg Terminals?
Probably not - but the widespread reliance on the $10k+ per year terminal in
the financial services industry means that this thing is ripe for disruption.
The key isn't to build a terminal replacement though. fintech startups need to
chip away at little bits and pieces - and need to inter-operate as much as
possible. Only then will the bbg dominance be reined in.

~~~
anmilo
one of the reasons i'm generally Ok with Slack's valuation is that there is a
ton of room for a messaging service to offer a service that is secure and
auditable at the level that the SEC/FINRA and Banks require. Like the article
mentions, if you talk to the people using BBG, it's really the chat
functionality that provides the main value for literally everyone, at least in
my experience (former front office dev)

~~~
Karunamon
I'm unfamiliar with these terminals (have kind of always wanted to play with
one), but at first blush, a "secure and auditable" messaging system does not
seem that difficult to build.

Is it a legitimate compliance or practical implementation problem, or is it
just the network effect?

~~~
whatok
Network effect.

~~~
schintan
Its not just the network effect..Bloomberg's messaging system is tightly
integrated with the other functionality of the terminal..for example one can
execute a trade via a message or get live streaming quotes.

------
keithpeter
_“What I miss is the instant Bloomberg chats, which I rate higher than trading
or data feeds. The fact is, Bloomberg connects 100 percent of the Street, and
all that human intelligence is what makes markets hum.”_

I enjoyed the article and I have squirrelled this quote from a trader away for
future use - the need to meet and _exchange_ information is why we used to
have stock exchanges of course.

~~~
jsprogrammer
Except this information is apparently being exchanged in a closed, proprietary
market.

------
RockyMcNuts
Probably just a large sugary beverage having its revenge on Mike Bloomberg.

[http://www.businessinsider.com/reason-for-the-global-
bloombe...](http://www.businessinsider.com/reason-for-the-global-bloomberg-
outage-2015-4)

But if you want to cripple the financial system, Bloomberg's a good single
point of failure to attack. That and Seamless, which also had a hiccup today.

------
innguest
Amazing how propaganda machine NY Times is fast to cover this story.

That crash was totally not strategic at all. /s

~~~
ceejayoz
What weird conspiracy theory are you flogging? I can't even make sense of the
accusation here.

