
Knight Capital Says Trading Glitch Cost It $440 Million - rkaplan
http://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/?hp
======
pmb
These sorts of problems will occur in any distributed system which runs at a
high rate without humans in the loop. There is no chaos monkey for the stock
market, and transaction rollback is available only in the most extreme
circumstances.

I have yet to see a convincing rationale for how high frequency trading adds
value to the system - it certainly doesn't seem to add pricing stability.
Because this is hacker news, I suspect a few people reading this either work
in the industry and/or have strong opinions about it, and I would love to hear
why I am wrong. Because I do like it when clever computer scientists make
money, and that seems to be pretty much the only social benefit of HFT, at the
cost of flash crashes and things like the example above.

~~~
patio11
_These sorts of problems will occur in any distributed system which runs at a
high rate without humans in the loop._

This implies that humans are actually better than machines at making data-
driven decisions at high rates. They're not. They're _astoundingly_ not once
you compute the cost of humans versus the cost of machines on a per-decision
basis.

Liquidity in the market used to be provided by large groups of sweaty,
overpaid alpha males yelling at each other. We replaced them with dueling
robots. The robots can provide liquidity for a fraction of the price. The
alpha males really, really hate competing with robots, because the alpha males
invariably lose, so they complain that robots are stealing the money that the
alpha males used to extract from their customers by right of being the one
with a license to be shouting and sweaty at a particular physical location.

Occasionally a robot blows up. Not a problem -- robots are easy to replace.
Besides, humans blow up all the time. We only ignore their ridiculous
strictly-inferior-in-every-way-at-this-task nature because they look more like
us than the robots do, and because these _particular_ humans being displaced
used to be rich, whereas e.g. telephone operators tasked with manually doing
call routing (also clearly inferior to _highly reliable distributed
algorithms_ ) were poor.

~~~
danso
Another way to look at this: it's a well-known phenomenon that expert
radiologists will misdiagnose an X-ray image, even contradicting a past
judgment they made on the same X-ray. It's feasible that a properly machine-
learned computer could make better judgments on a long-running average basis.

But when a computer screws up an edge case, versus the many, many times a
doctor will screw up a diagnosis, which instance will get more attention.
People are scared of black box machines and will take particular note of the
times an algorithm has screwed them over, regardless of the times a human has
screwed them up.

Also, when the possibility of examination exists, computerized decision making
is far, far easier to audit (i.e. determine blame)

~~~
jlgreco
Exactly. It is the same as the problem with robotic cars.

Cars with human drivers kill thousands, but nobody blinks. The day a robotic
car kills a single person, all the news networks will have a field day.

~~~
mc32
I think you're right about this. Maybe it has to do with appearance of
control/self determination. It was a mistake (an accident), but it was my
mistake/accident (as opposed to something one didn't have control over).

------
veyron
This is incredibly strange.

You get back an acknowledgement with each fill. There are risk checks in place
that should check whether the order was a take or add liquidity fill (as per
rule 15c3-5 amongst others)

This is a risk check that should have been handled by a component which
appears not to exist. They could lose FINRA and SEC authorization if it is as
bad as it sounds ...

------
sp332
Dark Knight Rises spoiler alert:

Isn't it really odd that a company called "Knight" makes all these nonsensical
trades just a couple weeks after the movie where a similar thing bankrupted a
company?

~~~
iag
Hah this is the best comment I've heard in a while, what a great way to start
the day. Thank you sp332.

------
incision
I love it.

Looks like someone's HFT system backfired. I'm looking forward to the Nanex
analysis of this one.

Nanex on the 2010/5/6 'Flash Crash':

[http://www.nanex.net/FlashCrashFinal/FlashCrashAnalysis_Theo...](http://www.nanex.net/FlashCrashFinal/FlashCrashAnalysis_Theory.html)

~~~
fr0sty
> I'm looking forward to the Nanex analysis of this one

Already posted:

<http://www.nanex.net/aqck2/3522.html>

~~~
joezydeco
Wow, you can almost see the bug happening as you read this. Amazing stuff.

------
itsprofitbaron
It seems that Knight Capital's ended up buying at the offer and selling at the
bid. In order words, Knight Capital essentially collected the opposite of a
rebate by paying someone for no reason aside from attempting to create
liquidity in the market which resulted in these losses.

As a result, they were pushing stocks higher and higher which causes a huge
incentive for a system like Knight Capital have. Again, this explains not only
some volume surge which was seen on the market but also massive moves in
certain stocks like China Cord Blood Corporation (CO) [1] who rose several
hundred percent, until someone finally stepped in & the important thing here
to note that there isn't a reason under the current SEC order cancellation
methodology to bail out Knight Capital and its algorithm.

Similarly, they were trading Exelon Corporation (EXC) [2] and were losing
$0.15 on every single trade and when its being done 2.4k times per minute, its
very easy to lose money.

[1]
[http://finance.yahoo.com/q?s=co&ql=1](http://finance.yahoo.com/q?s=co&ql=1)

[2]
[http://finance.yahoo.com/q?s=EXC&ql=0](http://finance.yahoo.com/q?s=EXC&ql=0)

~~~
samstave
WTH is "China Cord Blood Corp" - they collect cord blood for stem cells I
suppose - id be very wary of any such product like that from china, both for
quality and humanitarian reasons.

Take a look at the Vice documentary on Tiger part harvesting. China states
that tigers are endangered, yet there are these secret farms that breed
thousands of them and the make tiger dick wine, all sorts of things with their
other parts as well.

I wouldn't think any biological products being produced in china on the level.

------
dhyasama
My first job out of college was at a small HFT firm. The night before rolling
out my first changes is forever imprinted in my memory. Lots of tossing and
turning and images of exactly this kind of disaster.

~~~
danso
How did its testing/security practices measure up to other places you've
worked? I understand that the tech division of a financial firms might be top
notch, but the culture of the moneymaking dept. may mismanage them.

~~~
veyron
SEC/FINRA have required practices. The risk controls have to be firewalled
from the trading desks precisely so that this type of blowup is contained.

~~~
dhyasama
Is that new? Back then it was two devs and two traders in a room. See a
problem? Press the big red button. Deploy change. Press the big green button.
Cowboy style. I STRONGLY recommend against doing it this way. I was young and
stupid and I'm pretty sure it took years off my life.

~~~
veyron
The issue regards naked access. At that time the broker had requirements. Now
that two-dev shop has to worry about risk etc.

------
manishsharan
I am highly skeptical of this being a software glitch and here is why: I have
been working for banks' brokerage and capital markets groups for years and the
amount of checks and reviews processes that they have put in place is just
incredible. One does not simply deploy a new software into production without
there being several small pilot runs, staging tests etc. etc. There are
committees with onerous questionnaires and test results compiled by QA that
need to be documented and approved before a new deployment is approved.
Therefore I don't see this being a software issue. If the checks were not in
place then it seems that the management was lax in safeguarding its capital.

------
salman89
I don't understand how it went on for 45 minutes without human intervention.
Do they really not have a live person at the very least monitoring trades?

~~~
showerst
It probably _was_ noticed, but once it started you can't just pull all of the
plugs since that you make you lose even more...

------
amykhar
Thank heaven I didn't write that code.

~~~
rabidsnail
Or deploy it. Imagine being the last one in the push queue that day.

------
rcavezza
"Trading Glitch" = a very good excuse for very bad trades. I'm not implying
this happened, but if I was an analyst at Knight Capital and we just lost $440
million, I'd blame it on a "trading glitch".

~~~
incision
Yep.

They're calling it a "glitch" to suggest something unavoidable, a problem that
could have happened to anyone.

It's same vein of bullshit we heard in 2008 "We couldn't possibly have known!"
and more recently with the string of "rogue traders".

In every case, its just a bunch of folks who don't want to take responsibility
for their gambles when they lose.

~~~
jwoah12
You are completely wrong. Try looking into the circumstances before you make
some unfounded accusation about what caused this. Comparing a 45 minute
isolated incident at a firm with an otherwise-excellent reputation to the
housing crisis or rogue traders is moronic.

~~~
incision
You're missing the point entirely.

They gambled on an algorithm and everything related to engineering and
operating it.

On some level or another they failed spectacularly and they're calling it a
"glitch" - a minor malfunction, a transient error, a spurious little blip.

Say what?

$440 million dollars pissed away over the course of 45 minutes is not a glitch
by any stretch of the imagination.

It's a failure of epic proportions at multiple levels. Failure to test, to
review, to react and finally failure to own up.

~~~
sseveran
They did not gamble. Market making doesn't involve gambling. Someone checked
in an egregious error that testing failed to find. The fact that it is in a
trading system makes the cost of the bug easy to measure.

~~~
incision
What on earth are you talking about?

Market makers gamble on their ability to set spreads that will produce a
profit.

They're using code to generate spreads, execute head fakes and stuff quotes to
their advantage. They're "players" as much as anyone.

------
romey
I've always wondered, what happens to the actual programmer[s] responsible for
a bug like this? Assuming they find the exact issue, and then go back through
the version control logs to the exact commit, how responsible, if at all, is
the individual programmer? Or the QA team, for that matter.

Incidents like this make me think that I'd have a panic attack with every
commit, if I worked in the financial industry.

------
bhurt
I think this talk is relevant: <https://ocaml.janestreet.com/?q=node/61>

~~~
create_account
Why?

You might be far less likely to blow up the way Knight did, but you're hardly
immune.

~~~
sseveran
It is reasonable to use technology to make the universe of potential errors
significantly smaller.

------
jagermo
I'm not sure, if it was a technical failure, but here is a great talk from Def
Con 19 about security and high trading systems
<https://www.youtube.com/watch?v=ncpm6vRi9F4>

------
NelsonMinar
Is there anything good written about failsafe programming? I try to code
defensively so terrible bugs like this can't snowball. A heuristic like "only
make 1000 trades a minute" could have saved Knight $400 million!

I try to build global failsafes into my code, particularly network code. For
instance, a recent Twitter project I wrote has various loops calling their
API. That code has a global counter that says "this program can only ever make
25 calls" that's enforced in the bottom level HTTP calling function. If the
rest of my code is correct I'll never trip that limit, I have application-
specific logic that means I should only ever need to make 10 calls. But I
appreciate having the failsafe in case that more complex logic fails.

~~~
fr0sty
> A heuristic like "only make 1000 trades a minute" could have saved Knight
> $400 million!

A heuristic like that would not allow Knight to function. Knight is an
electronic market-maker (one of the biggest) and is probably making markets in
thousands of symbols and likely updates those quotes several times a second.
Their normal quoting rate is on the order of millions of messages/minute and
in volatile times may spike 10x or more and still be within 'normal' limits.

~~~
justincormack
A heuristic like "do not lose more than $100m a day" would have saved the
company.

~~~
veyron
PNL limits are required risk checks. Clearly Knight didn't implement them ...

~~~
beagle3
Stop-losses are not guarantees of execution. If you're in an illiquid market,
or one that moves faster than you're prepared for, then your PNL check may
alert you, but it won't save you from the loss.

------
brianbreslin
My initial skepticism makes me want to think this was maliciously
orchestrated. Where one firm stands to lose, someone else would undoubtedly
gain.

~~~
fr0sty
Interesting theory, but there is no implicit parity between # of winners and #
of losers. Whereas 1 firm lost $440M there were thousands (maybe tens of
thousands) of firms and individuals that made hundreds or thousands of
dollars.

~~~
jonknee
Unless someone comes in and scoops up Knight for cheap--the stock is trading
down 65% or so over night.

~~~
justincormack
They are out of cash apparently. Likely to be sold or file for bankruptcy
shortly it seems.

------
swalsh
Is it possible to purchase a ridiculously expensive insurance policy
protecting yourself against the risk of dangerous bugs like this?

~~~
justincormack
No. Buying the insurance would probably increase the risk as checks would get
more lax. And you can't hedge it.

------
eps
I'd love to see what the patch looked like.

------
codegeek
I wonder whats going on with the IT executives/CIOs of Knight at this point .
They must be hanging by a thread?

~~~
debacle
They're probably very close to hanging from a thick rope.

------
jberryman
easy come, easy go.

