
How Power Peg Brought Down a Knight - minimax
http://www.futuresmag.com/2014/11/01/how-power-peg-brought-down-a-knight
======
PhantomGremlin
This is a little long, but it's a _must read_ for anyone interested in real-
life consequences of computer bugs.

Basically, new software was pushed to seven of eight servers at Knight, one of
the biggest players on Wall Street. A flag activating the new software was
repurposed from very old inactive software that was still on the servers.

The new software ran fine on seven servers, but the flag activated the old
software on the eighth server, which promptly started doing something
completely insane as far as financial markets are concerned. A few minutes
later people removed the new software. But didn't change the flag. So then:

    
    
       (the old software) jumped in to fill that void
       and was now running on all eight servers
       simultaneously, buying on the offer,
       selling on the bid.
    

Knight lost over $400 million in under an hour, and was forced to eventually
sell out to one of their biggest competitors.

~~~
dclusin
45 minutes to be precise:
[http://www.sec.gov/litigation/admin/2013/34-70694.pdf](http://www.sec.gov/litigation/admin/2013/34-70694.pdf)

------
rmason
A little surprising to me that when you're trading hundreds of millions of
dollars an hour that the software wasn't built with a NASA style of redundancy
built in that would have caught the mistake.

~~~
kasey_junk
NASA style redundancy comes with a significant cost, that is delivery times.
The opportunity costs of time to markets in electronic trading are huge. Some
opportunities disappear in weeks (though these systems were not targeting
those sorts of opportunities).

Also, redundancies/qa are hard to quantify from a cost benefit perspective and
trading firms are very short term P&L driven (pay is frequently heavily bonus
based).

~~~
PhantomGremlin
> The opportunity costs of time to markets in electronic trading are huge.

That's probably 90% of it right there. But perhaps the remaining 10% is that
the "big swinging dicks" that run the brokerages have nothing but contempt for
anyone who isn't a broker or a trader. IT is a cost center, nothing more.

Even after the debacle, one of the executives said:

    
    
       “The IT guys were arrogant.”
    

Well, maybe they were. And, maybe, because of the arrogant attitude the
executives had, they couldn't attract any non-arrogant IT employees?

