

Trading Program Ran Amok, With No ‘Off’ Switch - mindblink
http://dealbook.nytimes.com/2012/08/03/trading-program-ran-amok-with-no-off-switch/

======
SagelyGuru
'Prediction is difficult, especially with regard to the future'.

It looks like they were using some new algorithm, which should have made them
a lot of money, had the market gone up after their massive purchases. In that
case, they would have pocketed fat bonuses and would not be on the news.

However, it has not happened, so the crying and the search for a scapegoat is
on. It sounds like the case of the banking business as usual: 'heads I win,
tails you lose'.

Ultimately, there is a really serious problem with the concept of limited
personal liability for companies engaging in speculation. It is an assymetric
arrangement, whereby the directors are entitled to the profits but are never
personally responsible for the losses. With such rules of the game, it is
advantageous to take crazy risks. Expect to see a lot more of this and many
more taxpayer funded bailouts.

~~~
yummyfajitas
Huh? Knight Capital lost a bunch of money, and will likely go bankrupt.
Essentially, Knight's software bug transferred a bunch of money from Knight to
everyone else. This poses minimal systematic risk to anyone else, and they
will almost certainly get no bailout.

The markets have already recovered. The S&P was down a little bit on thurs and
recovered by friday. Knight is down 60%.

<http://www.google.com/finance?q=INDEXSP%3A.INX%2C+NYSE%3AKCG>

This is ultimately a situation of the market being a robust and stable
dynamical system.

~~~
gkuan
Yes, no risk to others unless you happen to be one of their newly acquired
futures broker unit's customers with $411 million in deposits.
[http://www.reuters.com/article/2012/08/02/knightcapital-
regu...](http://www.reuters.com/article/2012/08/02/knightcapital-regulators-
idUSL2E8J26R720120802?type=companyNews)

~~~
yummyfajitas
Did you read the article you just linked to? Penson's customers have their
money in accounts completely segregated from Knight's electronic trading
division.

 _"It isn't like we found out that Knight was stealing money," Sommers [a CFTC
commissioner] said._

The CFTC is just watching carefully to make sure it stays that way.

~~~
gkuan
Moreover, I would suggest you read Johnson et al's research on mini-flash
crashes (<http://arxiv.org/pdf/1202.1448.pdf>) if you hold that the markets
are stable dynamical systems.

~~~
yummyfajitas
How does this research suggest the market is unstable? According to these
authors, the market was (in their view) dangerously perturbed 18,520 times,
more than once per day. In spite of that, it remained stable. The flash crash
took an afternoon to recover from. Knight took a day.

If you perturb a system over and over and each time it quickly swings back to
equilibrium, that's pretty strong evidence it is stable.

------
svdad
What I wonder, following this story this week, is how the software quality
controls at a place like Knight compare with those for life-critical systems
like those in, e.g., aviation.

On one hand, you'd think the QA in finance would be pretty solid, considering
that the survival of the company could be at stake (witness Knight). On the
other hand, I have a feeling that even there, people just don't take it that
seriously.

Would love to hear from anyone with more experience writing software for these
industries.

~~~
izaidi
Unlike high-frequency trading, aviation is highly regulated. In the United
States the FAA specifies pretty detailed development standards for avionics
software (e.g., DO-178B: <http://en.wikipedia.org/wiki/DO-178B>). We're
unlikely to see similarly strict requirements for financial software anytime
soon.

~~~
wpietri
Having talked with people who write life-critical code, the regulation isn't
really what makes it safe. Safety comes from good engineering.

The regulation just makes it much harder to bring an unsafe product to market,
and makes it clearer who to blame when people die.

~~~
wglb
But don't you think the existence of regulation influences the culture?

~~~
toomuchtodo
Penalties influence the culture. As we all know, the first lesson in economics
is that incentives matter.

Sometimes, the right people aren't being incentivized to do the right thing.

------
nmcfarl
They lost $440 million (and amount greater than their market cap), and
possibly the company, on what the world knows to be incompetence.

At some point if I couldn’t stop it - I’d be tempted to just kill the power to
the server rooms, all of them. There just has to be a way to cut your losses.

~~~
forgotusername
I'd love to know what qualifies you to throw a word like incompetence around
here. My best guess is the reason it took 45 minutes to shut it off was due to
a judgement call: burn through free cash, or take out all their customers too.
Bear in mind some of the largest retail brokerages in the world hang off
Knight.

Their primary functions are acting as an order destination and a market-maker,
for efficiency's sake an obvious conclusion would be that both functions are
combined in the same software (in a market where microseconds matter). So
given the choice of taking a cash hit (a potentially short term affair), or a
reputation hit (a much longer term and most likely fatal affair), it's
entirely possible Knight knowingly made the right decision.

It's worth note that the eventual deficit amounts to somewhere in the region
of one year's net income, hardly insurmountable (and how many investment
opportunities promise close to 100% return in a single year?).

Listening to the CEO on Bloomberg, it was clear that minimizing damage to
customers was their primary goal (he made this point several times in the 5
minute interview), and that he appeared comfortable with the outcome.

~~~
fauigerzigerk
$440 million is four times their 2011 net income. I doubt the CEO is
comfortable with the outcome of being on the brink of bankruptcy. He is trying
to arrange a fire sale of the company as we speak.

But I think you are right that they tried to avoid an outage. The
incompetence, if any, is that they apparently did not know how much money they
were losing and still kept the system going. It wasn't a caclulated risk but
rather an incaculable one.

I imagine it's not easy to know how much you're losing at any moment in time.
They certainly knew they were building huge positions, but knowing how much
they were going to lose on those positions requires an estimate of the price
at which the positions can be closed (or a hedge).

What I cannot imagine is that it is common practice to leave this kind of
decision to an individual's judgement call. There have to be rules for a
situation like this. And there's only one sensible rule for a rogue algo
racking up unknowable losses. Kill it and deal with the consequences later.
Anything else is negligent.

~~~
forgotusername
They had >= $300m free cash as of June, that's how I arrived at the $100mish
deficit.

------
bagosm
So, some of the owners were looking for a way out, and magically this thing
broke loose and started giving away (basically) free money to undisclosed
receipients. In the meantime all the technicians were fast asleep and couldnt
kick the machines down or something, while they were losing milions of dollars
per minute. This article is a completely honest recap by completely honest
people, about completely honest traders/bankers (bankers are not people).

Edit: on a COMPLETELY unrelated note, trading firms/banks are known to
actively pursue the extraction of money from their clients with bogus
trades/advice [http://www.nytimes.com/2012/03/14/opinion/why-i-am-
leaving-g...](http://www.nytimes.com/2012/03/14/opinion/why-i-am-leaving-
goldman-sachs.html?pagewanted=all)

------
fsckin
I would rarely suggest this, but if something is so incredibly broken that
you're loosing money at a rate of 800 million dollars per hour, screw the
customers.

Turn it off at any cost. If you are forthcoming and transparent, customers
will understand.

~~~
pheon
Point is you dont know what the loss is.

1) They bought too much stock (incorrectly)

2) realized WTF, stopped everything

2a) more likely their clients said WTF is wrong first

3) had to sell the stock for the rest of the day.

Its only after they sold everything did the $440MM price tag surface.
Hopefully they sold most of their positions to goldman (instead free market)
so one of their investors made a boatload of cash.. giving them favorable
terms for a line of credit.

~~~
sp332
This is wrong. The algorithm was buying and selling constantly, sometimes
losing small amounts of money (usually about $15) each time, sometimes as
often as 20-40 times per second for each of about 150 symbols.

------
retube
There seems to be a lot of confusion around market making, brokering,
execution algorithms and HFT in this thread.

------
teyc
I read the nanex article. Regardless whether it is true or not, the general
trend is towards development of more sophisticated load testing programs.

The most benign ones were developed for use in IT systems. E.g. Apache bench.
While these can cause disruptions if aimed at production services, this does
not necessarily threaten the health of an entire enterprise.

However, the trend is that all software sectors are starting to adopt this
particular technique of testing software with not sufficient regard to what
happens if it is released into live systems.

For example, we have chaos monkey, from Netflix, which randomly shutdown
services in a cloud based system.

What would happen if software which simulated meltdown at a nuclear facility
was accidentally bundled into the build system by a tired operator? Or some
one does the same with flight software?

The main software running trading platforms would presumably be supervised by
another program to ensure that bad algorithms do not lose e company too much
money. However there was no such tool for the component that generated the
test data.

To me, it sounds like the supervision should be done at a higher level, e.g. A
wrapper around existing APIs. All software running against live systems must
call into the wrapper.

Secondly, test software should conduct some kind of verification. E.g. Check
for evidence that it is testing against a Test system. This might be the
presence of a nonexistent company, et c.

I am more than happy to compile any other ideas you may have so that the IT
industry is able to build more fail safes into software.

We are starting to see some of these fail safes in practice. E.g. When you try
to send out an email to everyone in the organization, email software may warn
you if you are sure you want to do that. The problem is we haven't thought
enough about these scenarios that we don't adequately address them.

Incidentally, over in Australia, the Commonwealth Bank suffered a major
downtime when it's outsourcer HP accidentally pushed out system wide updates
instead of doing this to select machines as originally intended.

------
sanxiyn
Pure speculation. Maybe there was an off switch, which used to work, but not
regularly tested, and silently broken? Wouldn't surprise me.

~~~
CaveTech
Highly doubt it.

------
sahilz79
This was apparently an infrastructure problem of some sort:
[http://www.bloomberg.com/video/tom-joyce-knight-is-open-
for-...](http://www.bloomberg.com/video/tom-joyce-knight-is-open-for-
business-1ZpjUmh0TlevIFSAZ2UwxA.html)

Infrastructure changes can be notoriously difficult to back out by simply
using an "off" switch, particularly if this was some type of a firmware
upgrade that impacted all of their production servers. Backing it out at a
minimum would require some type of a reboot, which would cause problems with
an active trades. It could very well be that they were running an Active-
Active environment, they had to go Active-Passive, back out the changes from
the passive environment, reboot, and surgically cut over to the passive
environment. This could easily take 30 minutes.

------
elmarks
Doesn't this mean that others made a killing, taking advantage of all the
mispriced orders?

~~~
jonah
Most likely. c.f. the recent JPMorgan losses[1].

[1] [http://www.pbs.org/newshour/businessdesk/2012/06/who-
benefit...](http://www.pbs.org/newshour/businessdesk/2012/06/who-benefited-
from-jp-morgans.html)

------
jonah
"Knight is also working with Goldman Sachs to help unwind the trades behind
its extensive loss, according to people briefed on the matter.

"Goldman has agreed to buy, at a discount, the shares that the trading firm
had accumulated. Such a move would help Knight by taking the portfolio off its
hands and freeing up capital."

What does this mean? Why would GS do this? Why would Knight do this? Couldn't
they just sell them on the open market at a better price instead?

~~~
jkimmel
Yes and no. Given the kind of volume Knight purchased during the faulty
trades, it could be difficult to offload that many shares on the open market
in a timely manner. Maybe those stocks are hot Monday, maybe they're not.
Knight needs capital yesterday to keep floating, so they're likely looking to
sell everything in one basket.

As for GS's motivation, they're buying at a discount. Due to the time
sensitive nature of Knight's predicament, they're probably trading the
portfolio to Goldman at a reduced rate. Unlike Knight, Goldman has the cash to
sit on it for a while and sell the shares directly out into the open market,
even if it takes a few days. Given the discount they bought the shares at,
they're likely selling with a decent margin.

~~~
jonah
So GS is primarily the go-to since they're willing to leisurely unload stock
they bought at a discount and they've got the cash in hand to do so. Makes
sense.

------
teekarja
Strange article. Lots of text but missing the main thing I was looking for.
What kind "erroneus trades"? where did the money go? If you buy stock at the
market you did not intend to buy, why not just sell them the next day?

~~~
jonah
Seems like maybe they couldn't hold on to the stock for long enough to unload
the enormous volume they were dealing with. It sounded like at one point they
were doing AS MUCH VOLUME AS EVERYONE ELSE on the exchange combined.

<http://news.ycombinator.com/item?id=4337750>

~~~
Devilboy
Since there's 2 parties to every trade doesn't that make 50% the limit?

~~~
sp332
Nope, sometimes they bought stock they sold themselves!

~~~
Devilboy
If you sell stock to yourself... why even bother to go through the exchange?
Why would the exchange even allow such trades?

~~~
sp332
Knight has different programs running. It was handling >10% of all NYSE, so it
must have been running a lot of servers. When the berserk algorithm wanted to
buy or sell a stock where Knight was the only "market-maker", another Knight
server would usually intercept the order after it had been posted on the
exchange. Here's one way it might have happened:
<http://www.nanex.net/aqck2/3525.html>

------
0x0
Why isn't automated high-frequency trading banned already?

Does it not go directly against the spirit and purpose of having a stock
market with proper investors?

~~~
retube
this is issue nothing to do with speculative trading. Knight is a broker. They
provide an interface to the market for retail brokers, spread betting outfits
and so on. Their algos execute orders placed by their clients. Their new algo
had a bug. That's it.

~~~
0x0
I don't know the details of retail brokers vs HFT, but this writeup
<http://www.nanex.net/aqck2/3522.html> has a lot of charts showing trades with
25 millisecond intervals.

Just looking at things in a big perspective, the fact that the system is
designed for allowing trades at such frequencies makes it seem like markets
these days no longer exist for the benefit of the listed companies.

Then again, maybe I don't know wtf I'm talking about :-S I guess I don't
understand how real value can be created from such a system.

~~~
asmithmd1
Check out this TED talk about how these algos. are actively changing the
surface of the planet:

[http://www.ted.com/talks/kevin_slavin_how_algorithms_shape_o...](http://www.ted.com/talks/kevin_slavin_how_algorithms_shape_our_world.html)

The speaker makes an arguement that no one can understand how these algos.
interact and are being studied like natural phenomena.

------
reddiric
Can someone smarter than me please explain why a system where trades got
matched / executed at a granularity of once per second or once per several
seconds wouldn't work? What would be the problem with exchanges accumulating
and keeping secret buy and sell orders and executing them at a reasonable
interval?

------
sgt101
I wonder if the lack of a kill switch was linked to the power black outs in
India - what if they set it going, got cut (power) and then only came back
online 45 mins later? Could be that just one link - say a local exchange or
power for an FTTP line failed.

------
tlogan
I don't believe the problem was caused by "bug" - this reminds me of "rouge
trader" stories.

It is kinda weird that all problems in Wall Street are caused by "bugs in
software" or "rouge traders": while executives are never hold accountable.

------
SeanDav
This is an extreme example of what what can happen when what should be a
software company thinks it is some other sort of company. I am sure they
thought they were a trading company and software development was the necessary
evil required to get things done.

Well 400 million odd dollars in the red later I doubt they still feel that.

I do feel sorry for them and they probably didn't deserve this huge loss.
Hopefully valuable lessons can be learned.

------
brokenparser
Can anyone provide some context on this matter? What happened wednesday and
where?

~~~
davvid
[http://blogs.wsj.com/marketbeat/2012/08/02/knight-capital-
tr...](http://blogs.wsj.com/marketbeat/2012/08/02/knight-capital-trading-
error-cost-firm-440-million/)

Basically, they deployed a new HFT algo and it started buying high and selling
low. oops!

