

Analysis of the NYSE "Flash Crash" - jancona
http://www.nanex.net/20100506/FlashCrashAnalysis_CompleteText.html

======
yummyfajitas
This guys explanation of "quote stuffing" seems incorrect to me:

 _"What benefit could there be to whomever is generating these extremely high
quote rates? After thoughtful analysis, we can only think of one. Competition
between HFT systems today has reached the point where microseconds matter. Any
edge one has to process information faster than a competitor makes all the
difference in this game. If you could generate a large number of quotes that
your competitors have to process, but you can ignore since you generated them,
you gain valuable processing time."_

I don't think this is correct, since it's not a trivial matter to ignore your
own quotes. Say there are two orders about to come in, and you place a third.
The market data feed then tells you [order 1, order 2, order 3]. It _does not_
tell you which order is yours. So you need to separately run code which
remembers when you placed orders, and then guess that "look, order 1 and order
2 match the order I placed, I guess order 1 is mine, lets ignore it when doing
the rest of our calculations". That's not fast code.

Besides, if your broker thinks you are DOSing the markets (as well as _the
broker's_ system), they will shut you down FAST.

My "stupidity, not malice" explanation: novices didn't throttle the
oscillations in their code and feedback loops developed. Some were self-loops
- basically, you _just don't have_ the self-order detection code I described
above, and try to trade based on your own actions. Some were loops between
traders - trader 1 places an order, trader 2 cancels in response, trader 1
cancels in response to 2, trader 2 places an order in response to the cancel,
etc. The band-saw patterns look like bidding wars on wildly undervalued stocks
(e.g., one system places an order for accenture at 0.05, another at 0.06,
another at 0.07, etc), which is actually a very good thing since it pushes
prices back up to their proper levels.

~~~
ewjordan
Technically speaking, it may not be a trivial matter to ignore your own
quotes, but it's not very difficult, either - if it takes M operations to
handle your typical fully processed quote, and N operations to hash the quote
and compare it to a table of quotes that you've delivered, then as long as M >
N you've got some room for manipulation as long as you're able to pump enough
bullshit into the system that everyone else has to deal with; I assume that in
these situations M is usually substantially greater than N, probably on the
order of at least 10:1 unless the quote is discarded as obviously irrelevant.
It's just awful hard to imagine that the amount of processing done per quote
in a HFT system can be less than a few hash table lookups...

I'd imagine the real difficulty would be making sure that the quotes you're
putting in are a) relevant enough not to be immediately and trivially screened
out by your competitors, b) not in danger of actually getting acted upon, and
c) seemingly legitimate even in the face of after-the-fact scrutiny so that
you avoid getting butt raped by the SEC for manipulation. a) and b) would
likely be pretty easy to satisfy if you didn't have to worry about c), so...I
don't know, this doesn't seem terribly implausible to me.

Then again, I don't know much about HFT, there may be factors in play that
would make it a lot trickier to manipulate things in this way. And I'm
certainly not convinced that this is what was behind the crash - "stupidity,
not malice" is a good rule of thumb, and I'd have to see more evidence to
assume that someone deliberately manipulated the markets here.

~~~
yummyfajitas
If you are doing high frequency trading, processing a quote isn't that hard
either. It can't be, or you'd never respond fast enough. I suppose the best
way to do it would be to ignore quotes X distance from the book, and then do
your DOS X+0.01 from the book, though this might not be compatible with your
regular strategy. Whatever DOS you want to do has to be compatible with your
already existing strategy. I'm not saying it's 100% impossible, only that it
isn't easy.

Additionally, your broker starts getting annoyed at you when your fill rate
drops below 0.5% or so. If you DOSed the market, you wouldn't even be in that
ballpark.

~~~
ewjordan
FWIW, looking at those graphs, it does appear that a lot of the extraneous
action was happening right on the edge of the inside bid/ask.

 _Additionally, your broker starts getting annoyed at you when your fill rate
drops below 0.5% or so. If you DOSed the market, you wouldn't even be in that
ballpark._

That's definitely true, and I'm sure it would be a serious problem with doing
anything of this nature.

Is there always a middleman that cares about what quotes you're pushing
through, though, or do some people have closer connections that wouldn't be
watched very closely?

~~~
yummyfajitas
Some larger funds become a broker/dealer, so they can cut their brokers out of
the loop. I really doubt that anyone big enough to do that would be willing to
risk their business on a DOS attack, however.

I fully agree that it's possible to do. I just find other explanations, like
badly written algorithms, much more plausible.

------
gruseom
Nanex is a data feed company that was started by Eric Scott Hunsader who is
one bad ass programmer and, based on my limited interactions with him when I
used to program against a previous API he designed, a decent person. He was
able to deliver results back then that other people couldn't dream of, and
Nanex (called GenerationV at the time I think) was his "build it all again
from scratch based on what you've learned to date" data feed engine. It's
interesting to see the name come up here. How big of a player are these guys?

~~~
brownegg
They're not, really. Not on the HFT landscape, anyway. It's tempting to say
they're being slept on, but the cream rises to the top VERY fast in HFT--there
have been multiple instances of products brought to market and the company
being purchased outright by the first would-be customer (although this is
much, much more frequent on the networking and hardware sides).

I think the analysis and borderline-accusation of what is termed "insincere
quoting" by the exchanges is... laughable. It doesn't speak well to the
authors' knowledge of their industry. I've had an exchange disconnect my
session for sending 1000+ orders per second by myself, and I would have been
quite happy to execute any of them (or else they wouldn't have been sent). To
be honest, I'm surprised 5000 is the biggest number they could find. I would
have guess an order of magnitude higher.

What happened on 5/6 was that the NYSE finally learned that they aren't the
boss in the equities markets anymore. Trying to slow down trading without
regard for the other routing destinations was nothing but quaint.

You know those videos you see of suburban shoppers at 6:00a on Black Friday,
rushing through the doors to get one of the three available Xbox 360s or
whatever? The NYSE was the hapless security guard whose watch was slow telling
everyone the store wasn't open yet.

~~~
gruseom
Very interesting. Thanks!

I wrote too hastily when I said Hunsader was producing results at the time
that others couldn't dream of; that was a sloppy exaggeration. But his work
was very good.

------
dandelany
Short conclusion: "In summary, quotes from NYSE began to queue, but because
they were time stamped after exiting the queue, the delay was undetectable to
systems processing those quotes. On 05/06/2010 the delay was enough to cause
the NYSE bid to be just slightly higher than the lowest offer price from
competing exchanges, but small enough that is was difficult to detect. This
caused sell order flow to route to NYSE -- thus removing any buying power that
existed on other exchanges. When these sell orders arrived at NYSE, the actual
bid price was lower because new lower quotes were still waiting to exit a
queue for dissemination."

"This situation led to orders executing against whatever buy orders existed in
the NYSE designated market maker (DMM) order book."

This is actually the most cogent explanation I've yet heard.

~~~
gxti
The scary part was that you could watch it unfold:
[http://partiallystapled.com/~gxti/charts/2010/05/06-spread.p...](http://partiallystapled.com/~gxti/charts/2010/05/06-spread.png)

When you can see the spread like that then something has gone terribly wrong.
As I read it, that spread is between NYSE on the bottom and the rest of the
market on the top.

------
lrm242
You might want to start from the main page as there is an intro and other
overview info: <http://www.nanex.net/20100506/FlashCrashAnalysis_Intro.html>

------
pdx
So the NYSE has been blaming the other exchanges for their behavior, when this
was really a NYSE problem.

However, it was only a NYSE problem because they were effectively being DOS'd
by some of the other exchanges.

    
    
         On the subject of HFT systems, we were shocked to find 
         cases where one exchange was sending an extremely high 
         number of quotes for one stock in a single second -- 
         as high as 5,000 quotes in 1 second!
    

So the NYSE was right to blame the other exchanges, but they were right for
the wrong reasons.

~~~
gxti
I don't think they mean that one exchange was DoSing another.

Quotes within a single exchange are disseminated immediately to all interested
parties -- if one party was spamming thousands of quotes at a stock, that
stream would be repeated for everyone with a feed from that exchange. There's
no way to tell which party is sending it, so the only thing you can measure is
the quantity being sent by the exchange. Thus it is technically accurate to
say that "one exchange was sending an extremely high number of quotes". It's
in the exchange's best interest to disseminate quotes as quickly as possible
in order to attract order flow and thus collect fees.

On the other hand, inter-exchange feeds are federally mandated and used
primarily to comply with Regulation NMS rules that prevent trades from
executing at a price inferior to the NBBO (national best bid/offer). In other
words, if I send a buy order to NYSE but NASDAQ has a lower price, NYSE is
required to either reject the order or route it to NASDAQ for a fee (it's up
to the customer which method is used). It's not necessary to send 5000 quotes
per second to another exchange because the primary motivation is compliance.
Generally, institutions will send their orders directly to the exchange with
the best price in order to avoid the extra fee for having it routed by the
exchange.

I may be way off on the inter-exchange feeds as I am not a professional, but
there are undoubtedly many here who are so please correct me.

------
hop
Can anyone say why this momentary "flash crash" was a big deal? Who cares if
people or computers trade securities under market value for a short amount of
time? It only affects the idiot that is selling them at that undervalued
price. The value of the underlying company is unaffected and market forces
will always quickly correct it. The people or firms that sold the securities
at those prices are probably either fired or lost their own money and they
chance it happens again is unlikely. Use buy and sell limit orders, you should
be already.

Should we worry about all online retailers because Zappos had a computer error
and lost some money? - [http://www.walletpop.com/blog/2010/05/24/pricing-
error-costs...](http://www.walletpop.com/blog/2010/05/24/pricing-error-costs-
zappos-1-6-million/)

~~~
nl
_Who cares if people or computers trade securities under market value for a
short amount of time? It only affects the idiot that is selling them at that
undervalued price. The value of the underlying company is unaffected and
market forces will always quickly correct it._

Well, the stock price _is_ the market value - if it's not then what is?
Detecting errors as opposed to genuine changes in value is a non-trivial
problem.

 _Use buy and sell limit orders, you should be already._

The flash crash happened so quickly that a lot of limit orders weren't
executed, because the price dropped past the limits too fast.

If you had an order to sell something at $20, and the price dropped from $25
to $1 in seconds then there's a good chance there were no buyers for you at
$20, but instead your trade might have been executed at $1. That makes a
pretty big difference.

~~~
yummyfajitas
A limit order sell at $20.00 will never be executed at $1. It will never even
be executed at $19.99. It will simply go unfilled until someone is willing to
sell at $20.00 or higher.

------
anamax
Note that a crash of similar magnitude and duration happened in the 60s, well
before computerized trading.

~~~
yummyfajitas
Could you provide a source?

~~~
anamax
I'm looking... (It isn't the first two places where I thought that I saved
it.)

~~~
petewarden
Here's a link discussing the May 1962 equivalent to the flash crash:

[http://online.wsj.com/article/SB1000142405274870395760457527...](http://online.wsj.com/article/SB10001424052748703957604575272791511469272.html)

------
Judson
These kinds of writings are why I read HN. The article is pretty technical,
well written, and removes any political BS that you would most definitely hear
on TV.

Notice that "circuit breakers" wasn't offered up as a solution to the problem.

------
zmmz
The recommendations miss the most obvious fix:
[http://en.wikipedia.org/wiki/May_6,_2010_Flash_Crash#Circuit...](http://en.wikipedia.org/wiki/May_6,_2010_Flash_Crash#Circuit_breakers)

These things have been working fine and dandy across Europe for several years,
where such a situation would not happen.

For the Eurex implementation see here: <http://is.gd/d2f1Y> [link to google
quick view of the pdf]

~~~
jazzdev
Fixing the quotes to be timestamped properly seems to be a much simpler fix
than relying on additional circuit breaker logic.

~~~
zmmz
While time stamping solves the problem of inconsistent quotes, it does not do
anything against bogus trades or somebody accidentally adding an extra 0 when
typing in their data.

A circuit breaker would prevent the absolute free-fall that happened during
the flash crash. At that stage, nobody knew what the "real" price of a stock
was and everybody was guessing, the algos could not handle the situation. If a
circuit breaker was triggered, it would lead to a trading halt on the stock
and an auction to set the price.

Time stamping quotes solves system errors, but not mistrades or market panic.

------
tmsh
i don't know what all the big deal is about this 'flash crash'. they happen
all the time in the futures markets. they are the risk premium for strategies
that make a market automatically.

this is not a threat to the economy if that's what the concern is. it's only a
'threat' in conjunction with a perceived fear. in this particular case, it had
to do with greece. in the 87 crash's case, it had to do with a collapse in the
bond markets.

all high-frequency trading or program trades do is quicken the whole process.
one can implement safeguards to make the whole thing slower -- or you can let
the market participants take care of themselves and/or fix the underlying
problems and ignore mass hysteria which will reset as soon as the absence of a
rational reason for a correction is made clear.

i feel like the 1987+ 'crashes' are very different from the 1929 crash, etc.
they are not in themselves a problem. so far they've always popped back.

which is not to say that our economy has been in good shape since 1987. but
that there are deeper issues -- having very little to do with hft.

------
d_c
also: <http://news.ycombinator.com/item?id=1456746>

~~~
klochner
and also, ever so slightly before that one :)

<http://news.ycombinator.com/item?id=1456414>

~~~
d_c
a low delay post :)

------
wheaties
Sweet. If the 50ms rule goes into effect HFT traders could conceivably be set
up anywhere in the US, no?

~~~
yummyfajitas
They already can be. Tradebot lives in Kansas City, for example.

~~~
marcinw
At the moment, you're at a serious disadvantage if you're not across the
street from the Exchange due to latency (obviously).

~~~
yummyfajitas
You're at a serious disadvantage if your _server_ is not across the street,
it's true.

That actually wouldn't change with a 50ms rule. You still want your order in
the queue before the other guy, even if he can't pull his order for 50ms.

~~~
marcinw
Right, sorry for not making that distinction. That is what I meant, but the
latency between brain and fingers is under some delay :)

------
yread
_the HFT systems detected the sudden price drop and automatically went short,
betting on capturing the developing downward momentum_

This is the scariest part

