
HFT Quote Stuffing is a software bug - jpdus
http://www.chrisstucchio.com/blog/2014/quote_stuffing_is_a_software_bug.html
======
kasey_junk
So again with all HFT conversations we have to be clear on what we mean with
the language we are using.

I've seen quote stuffing used to mean what Chris mentions here. The
implication being that HFTs were using large volumes of low quantity orders
for nefarious purposes such as DoS the exchange & SIP or confusing your
competitors by making the order flow to heavy to process. These kinds of games
did occur, but it was very briefly a long time ago because the exchanges
themselves had a vested interest in stopping it and an easy mechanism which to
do so (fill ratios). I prefer the term quote spamming for this, and I agree
with Chris that most of the time when this happens it's just a bad order
management bug.

Quote stuffing can also refer to the practice of putting a few large orders
into the order book with no intention of trading them in order to make the
volume on the book look larger than it is, and then cancelling them quickly to
take advantage of the "real" volume. This is a predatory activity designed
explicitly to take advantage of market making strategies and is already
illegal. Your mileage may vary when it comes to enforcement...

~~~
throwaway13qf85
The second kind of quote-stuffing you mention, where you place larger orders
outside the best bid/offer with the intention of distorting the order book
(aka spoofing, see comment below) is certainly nefarious. But it's a special
kind of nefarious behaviour that really only affects other market participants
who can react quickly enough to the information - i.e. it's mostly HFTs taking
money from other HFTs.

~~~
hft_throwaway
I think it is more short-term day traders taking money from naive HFTs. I'm
sure some people automate this type of manipulation, but I seriously doubt
that any of the "big name" market making firms do it. They are under a lot of
regulatory scrutiny, make more than enough through legitimate trading, and
have the technology & data to enable their internal compliance auditors to
detect it.

I also don't think it can make that much. It may work for a little while, but
no algo is going to sit there bleeding money to you forever without hitting a
risk limit or being reviewed/adjusted. If your business model depends on
somebody else doing something dumb, it's not going to work for the long-haul.

~~~
lrm242
The cases I know of where the SEC has cracked down were all manual traders
leaning the book and essentially taking advantage of naive MM algos. Proceeds
< $5mm each time I believe.

------
rsync
I always assumed that step 2 was "DoS your competitors".

This is crudely done with volume of traffic, but more elegantly done with
frequency and amplitude (and so on) given some set of (possibly wrong)
assumptions about how competing algorithms will respond to this rapid fire
order placement.

I know almost nothing (beyond ZH and chapters 1-5, so far, of _Dark Pools_)
about HFT, but I can trivially think of ways you could be evil with quote
stuffing as it is described in the OP.

~~~
hft_throwaway
Manipulation definitely happens where traders will show non-bona fide interest
on one side of the market to provoke other traders or algos to react. It's
known as layering or spoofing and like any behavior where you're placing
orders to influence others rather than out of an interest to trade, it's
illegal.

Most cases of this or similar behavior like stop running have involved human
traders at loosely regulated click trading shops or smaller electronic firms
that don't play by the rules:

[https://www.finra.org/Newsroom/NewsReleases/2012/P178687](https://www.finra.org/Newsroom/NewsReleases/2012/P178687)

[http://www.cftc.gov/PressRoom/PressReleases/pr6649-13](http://www.cftc.gov/PressRoom/PressReleases/pr6649-13)

Aside from the regulatory risk, it doesn't fit the business model for most
"HFT" firms, who generally are making many low-risk, low-reward trades with a
small statistical edge based on proprietary models. Spoofing usually requires
showing large false interest which entails considerable risk if you actually
get elected. If you are fast enough to make markets or arb crude oil, you can
make way more doing that than manipulating the DOM hoping to push algos into
doing something stupid until they figure your tricks out.

Predicting markets is hard. Predicting how thousands of heterogenous agents
will react to your order is even harder, if not impossible.

~~~
rsync
"Predicting markets is hard. Predicting how thousands of heterogenous agents
will react to your order is even harder, if not impossible."

My very small amount of knowledge on this topic leads me to believe that the
idea that one algo would specifically target another algo is not at all
farfetched. Who cares how the market responds to your weird trading if you're
trying to create distortions for one, or a few, actors ?

~~~
hft_throwaway
Because the market has many actors. If I try to push the market down to shake
out stop losses, another trader may say, "Hey, this price move doesn't make
sense considering recent activity, how the broad market is moving, etc., I'm
going to buy more" and prevent me from doing so. Likewise if I'm shifting the
depth with a big false order. I might induce some actors to do bad trades, but
another actor might see my order as a great opportunity to trade in size and
trade against it, so I'll take a loss.

Liquid markets with a diverse set of actors are more resilient to manipulation
attempts. It's hard to do unless you have more capital or are willing to take
risk that other actors in aggregate are not.

~~~
dllthomas
_' "Hey, this price move doesn't make sense considering recent activity, how
the broad market is moving, etc., I'm going to buy more"'_

Betting you know more than the guy moving large size rarely a good move,
though...

------
hft_throwaway
[http://zacharydavid.com/2014/04/on-hft-part-ii-bugs-
features...](http://zacharydavid.com/2014/04/on-hft-part-ii-bugs-features-and-
aggressive-incompetence/#A-Bug)

This blog post details similar issues that can lead to rapidly canceled
orders. Generally, firms that are smart avoid moving their prices around "too
much" since:

a.) You lose your spot in the queue. If I am near the front of the price/time
queue buying at 100.01 and move to 100.00 and back up again, now I'm behind
everyone else and will either lose opportunity or face more negative selection
(I'll only trade when someone wants to sell all the contracts bid at 100.01,
whereas before I would get some trades where someone only wants to sell a
few).

b.) Sending orders is more computationally expensive than doing nothing. If
you send enough orders you will slow down your own system.

c.) Depending on market rules and behavior the messages you send may cause
your gateway to queue up or block you from sending more messages which hurts
you.

I don't think quote flicker can provide any "evil" benefit to the person doing
it. It doesn't make sense to do it to slow down competitors, since most feeds
are multicasted to everyone and you still have to process all the orders you
send, both from the gateway and over the data feed. If you are faster at
handling this situation than your competition then you are going to beat them
anyway so what's the benefit in slowing them down?

I do think it can be disruptive in the sense that it places extra load on the
exchange and can annoy participants who have trouble taking prices on their
screen. Some markets have quote/trade rules to encourage more efficient
messaging which is their right and not a bad idea if the limits are set
reasonably. Rules like this make more sense than a minimum resting time, which
just penalizes makers at the expense of fast takers. Imagine I am putting
quotes out in SPY, an S&P 500 ETF, but they need to rest for half a second.
Someone sees the S&P 500 futures or the individual stocks move and then my
quote becomes a free option to trade at an unfair price. In a market like
this, market makers would be forced to put up wider prices to compensate for
these losses.

A high quote/trade ratio in itself is not disruptive. A lot of the Nanex
charts use thinly traded derivative products like ETFs as their whipping boy.
The underlying value of these products will change whenever another stock or
commodity changes, so market makers need to update their quotes quite rapidly
to avoid trading at unfair prices, even if no trades have occurred in the
product itself.

~~~
lmm
> If you are faster at handling this situation than your competition then you
> are going to beat them anyway so what's the benefit in slowing them down?

Suppose your fund has invested a lot in some hefty compute hardware but your
trading strategies are on the weak side, whereas you know your competitors
have better strategies but their computers are slower. Isn't it then in your
interest to stuff the order feed and make everyone more dependent on computer
power?

~~~
hft_throwaway
Maybe? Most guys who are really fast are also really smart. It's hard to make
money purely being fast these days.

I can't speak for everyone but most "HFT" models are built around providing
some kind of service to the market, generally either making markets to try to
make the bid-offer spread or arbitraging away inefficiencies between related
products.

Even if you are evil, building a model that solely seeks to take advantage of
another trader doesn't make sense in the long run. Eventually they will alter
their behavior or exit the market. It's not sustainable.

------
jpdus
Nanex linked an interesting CS research piece[1] which also discusses other
possible reasons for (malicious) quote stuffing including:

\- walking others into the book,

\- creating false midpoints and

\- trying to cause stale pricing and slowing others down.

[1][http://de.scribd.com/doc/116761218/CS-HFT-
DETECTION](http://de.scribd.com/doc/116761218/CS-HFT-DETECTION)

~~~
hft_throwaway
But even they admit that a market flashing between 100.01 offer and 100.00
offer is sometimes beneficial for them. It says their execution algo will
attempt to "pick off" the 100.00 offer with IOCs if they are representing a
buyer.

Flashing orders with the intent to manipulate is wrong and is illegal already.
It doesn't matter if you do this by hand or with a computer. Putting up a
tighter best price, even if it doesn't stay put very long, can only be
beneficial. At least some of the time, other traders will get filled at a more
advantageous price than they would otherwise.

------
mikeash
That sure is a lot of hand-wringing for what is a fairly simple and obvious
problem. You're reacting to the extremely short-term derivative of a noisy
signal. Of course things are going to go crazy from time to time.

It's odd that he touches on the obvious solution of rate limiting, but doesn't
actually discuss it as a way to stop this problem. If quote stuffing is a bug
that harms the people carrying it out, then why don't HFT programs have a rate
limiter on them that make them stop? He even goes so far as to say that you
can get shut down for doing this, so why isn't there a tripwire to make sure
you can never, ever do it?

~~~
lucisferre
The low level of opacity when it comes to electronic trading makes it
difficult to make any sense out of exactly what is (or perhaps isn't)
happening, which I think is fundamentally the problem.

~~~
sambe
Do you mean the high level of opacity? There is considerably more opacity than
in non-electronic trading, surely (everything on record, all data public, no
effect from personal relationships etc)?

~~~
tptacek
I can't tell whether you're suggesting that "HFT" (heavily automated
electronic trading) is more opaque than human trading (ie, the specialist
system), but for what it's worth there's a right answer to this question:
human market makers are _WAY_ less transparent than algorithms are.

Here's a benchmark: human market makers rigged the public markets so that
tradable instruments were quoted in fractions of a dollar, rather than in
pennies, so they could surreptitiously increase spreads. It's hard to get more
brazen than that.

~~~
minimax
_human market makers rigged the public markets so that tradable instruments
were quoted in fractions of a dollar, rather than in pennies, so they could
surreptitiously increase spreads._

I don't think this is right. Erh, obviously U.S. equities traded in quarters
and eights (and later sixteenths before decimalization), but to say that they
were using a weird fractional system _because_ they wanted to rip people off
isn't documented anywhere in the literature. They could have moved to decimals
and still kept the minimum price variation above a penny†, or they could have
just added more fractions (32nds, 64ths, 128ths) that would have moved the
minimum price variation into roughly the range you get by quoting in decimals.
If you think quoting in 128ths sounds silly I won't argue with you, but it's
something they do actually do for the US treasury futures contracts on the
CME††.

I suspect that they just used what worked. In a market of human traders, it's
much easier to deal with large round numbers. I think that's why odd lots were
treated specially too. E.g. It's a lot easier for humans to keep track of 5000
shares at 50 1/2 than in 100 small orders from 200 shares to 5 shares all
priced between 50.40 and 50.60. As markets adopted technology, they naturally
moved to a smaller minimum price variation (and eliminated the special
treatment of odd lots) because computers don't have any problems dealing with
an aggregate of small orders vs one big order.

† They are actually talking about doing this for some small cap stocks later
this year.
[http://online.wsj.com/news/articles/SB1000142405270230427530...](http://online.wsj.com/news/articles/SB10001424052702304275304579396861123381346)

†† ctrl-f "Quotation Practices" \--
[http://www.cmegroup.com/education/files/understanding-
treasu...](http://www.cmegroup.com/education/files/understanding-treasury-
futures.pdf)

~~~
tptacek
[https://www.sec.gov/litigation/investreport/nd21a-report.txt](https://www.sec.gov/litigation/investreport/nd21a-report.txt)

Search "odd-eighth".

Decimalization drastically reduced spreads. Scott Patterson suggested in _Dark
Pools_ that it literally put many human market makers out of business; in
other words: those human market makers were kept afloat by the gross
inefficiency of quoting prices in 8ths and 16ths, an inefficiency that was
taken out of the hides of investors.

~~~
minimax
I was really just trying to make the points that 1) the minimum price
variation of 1/8 of a dollar probably made sense at some point in the past
(NYSE being started in the early 19th century) 2) sometimes it's more than
just market makers who want larger tick sizes (consider this post from a
managing partner at Andreesen Horowitz†) and 3) you can reduce the minimum
price variation (the "tick size") without doing decimalization (i.e. smaller
fractions) and vice versa -- you can set the minimum price variation at sizes
larger than a penny even with decimalized quotations.

† [http://blog.pmarca.com/2013/03/26/unshackle-the-middle-
class...](http://blog.pmarca.com/2013/03/26/unshackle-the-middle-class/)

~~~
tptacek
I'm not sure what this comment means. You said, earlier: "but to say that they
were using a weird fractional system because they wanted to rip people off
isn't documented anywhere in the literature." I think I refuted that,
effectively.

Can you artificially inflate spreads without making inflated spreads the
explicit law of the market by quoting in 1/8ths? Sure. Is that what happens?
No.

~~~
minimax
I thought you were making a distinction between quoting in fractions of a
dollar vs quoting in pennies. It isn't really material. If you quote in
fractions but allow the minimum price variation to be 1/128th of a dollar you
have a market that allows for tighter spreads than one in which the minimum
price variation is a penny. There is nothing wrong with prices in fractions of
a dollar. What matters is the minimum price variation a.k.a. the tick size.

~~~
tptacek
It _is_ material, because the fractions of dollars we're talking about were
bigger than 1/100th of a dollar. Clearly, they could have moved to sub-decimal
fractions, but the point isn't "fractions" versus "decimals"; it's "human
market makers fixed the fractions at 1/8ths and 1/16ths of a dollar to
artificially increase the spread", versus "algorithmic trading competed the
spread down below a penny".

You implied that maybe humans had been using fractions because they were
easier to deal with. No. They were using the fractions they were using in
order to skim extra money off trades, and they fought tooth and nail to
prevent competition from reducing spreads below 1/8ths --- in fact, if you
read the link I showed you, it wasn't even 1/8ths; it was 1/4s!

~~~
minimax
What I meant to imply was that the emergence of fractional dollar tick sizes
was probably due to the fact that they were easier to deal with before the
advent of computerized execution, clearing, and settlement. By the mid
nineties most of that had been automated and there was therefore no good
reason for such large tick sizes. When humans were still doing all trade
processing manually, it made sense to have a system that incentivized fewer
large trades (vs more smaller trades). Large minimum price variations and
round lot requirements would have been a means to that end.

------
anigbrowl
_Except, once in a blue moon this happens when the mean signal approaches the
threshold and stays there for a while:_

I'm having some trouble with the contention that this is just such a rare
thing. I do a lot of audio signal processing so I'm used to dealing with noisy
signals at Khz rates. Threshold fluctiations are a fact of life if you're
trying to do things like pitch detection, envelope following, or sidechain
compression (all of which have obvious analogs in the trading context).

Obviously you don't want to impose too brutal of a low-pass filter on your
input because you would miss important signals, but what prevents you from
running the input signal to an LPF in parallel and keying a noise gate when
the amplitude of the low-frequency (trend) signal drops? This is such a common
problem in dealing with audio that we use specific devices called de-essers
for recording vocals, because sibilant sounds otherwise tend to throw off
other components of the signal chain: [http://en.wikipedia.org/wiki/De-
esser](http://en.wikipedia.org/wiki/De-esser)

Ironically, you mention using XKCD-type graphs because 'nobody reads' the
explanations and caveats, but just looks at pictures. I wonder if you aren't
missing a trick here by over-engineering your HFT systems for speedy response
at the inevitable price of positional confusion. Is it possible that you're
using Infinite Impulse Response filtering and running into denormalization
problems as a result? If so, can you not rectify the problem by adding a small
amount of noise?

I get your point about the negative aspects (for the HFT trader) of quote
stuffing, how it can easily lead to being on the wrong side of a trade or
punishment by the broker for flooding the output channel with unintended
orders. But, you want to make money from automated signal analysis, that's
your problem in much the same way that a sound engineer is responsible for
preventing transient pops and feedback howls in an audio network. Sure, these
problematic signal conditions are statistically rare, but when you're dealing
with a high volume of samples you'r going to run into statistically rare
conditions on a regular basis - and indeed you mention that this is the sort
thing you might expect to happen once a day.

~~~
hft_throwaway
I think you are right on the money and most traders do use techniques
analogous to the ones you describe when designing models:

[http://www.decal.org/file/2945](http://www.decal.org/file/2945)

"Alpha is often nothing more than taking commonly available data and
mathematically encoding it in a signal correctly. Correct often means
something as simple as using an rate of change instead of a difference,
normalizing a value, smoothing a chaotic signal with an EMA, using a
heteroscedastic weighted linear regression instead of a simple regression, or
handling all numerical errors or edge cases."

It's not always easy to get this completely correct in every situation and
traders face a reward function that doesn't necessarily reward correctness so
much as avoiding Type II errors.

~~~
anigbrowl
I feel like I'm in the wrong job - clearly I should be routing ticker prices
into my mixing desk and playing the market like a cheap violin instead of
being an underpaid sound engineer.

------
lmg643
this is an interesting way to look at the problem but in my reasonably
informed opinion, flawed.

it seems like this is working backwards from the assumption that quote
stuffing must be benign activity, and then trying to rationalize it with
signal/noise theories about finding the fair price level.

it is totally inaccurate to say that there is more load on the sender than the
general public, and also misses the mark on what the DOS theory alleges.

the exchange bears considerably more load than the sender of orders. the
exchange must (1) ack the order, or reject if any flags are populated out-of-
spec, (2) add the order to the order book, (3) check for potential matches
based on price of order, if so, cross and update feeds, (4) if non-marketable,
publish out to (a) depth-of-book feed, (b) SIP feed, etc.

the DOS theory is that excessive activity in a single partition can slow down
the publishing of data in that symbol range, which would mess up the quoted
price for the exchange in a symbol, which would affect orders priced on the
exchange, or anyone using the price in a derived manner (at a dark pool etc).

~~~
yummyfajitas
I'm not making the _assumption_ that it's benign. It's based on my own
experience - I've quote stuffed hundreds of times in simulation, and I'm sure
it happened in production as well. It's a hard bug to completely avoid.

Interesting - the DOS attack is on the exchange rather than other market
participants. But could you be more specific on how this helps you make money?

~~~
kasey_junk
There used to be games where you would attempt to get one order connection on
one exchange gateway and then another connection on a different exchange
gateway. You'd spam with one connection to slow down all the other
participants on that gateway, and use your other connection for your real
trades.

Other games involved spamming open auctions, settlement trades etc.

This gaming was a very brief phenomenon (because it is stupid, doesn't make as
much money as traditional trading, and trivial to foil). The exchanges got rid
of it almost immediately but it still gets passed around as folklore for one
of the "predatory" activities that HFT does.

------
austengary
This is fun

[https://twitter.com/nanexllc/status/458663948800520192](https://twitter.com/nanexllc/status/458663948800520192)

------
gd1
Obviously, well written HFTs avoid this by introducing some hysteresis to the
signal. i.e. Insert an order when the signal crosses 0.5, but then only delete
it if the signal falls below 0.4.

Chris: You've also missed the most common cause of these issues in my
experience - feedback from identifying your own orders in the public stream.
HFTs need to filter their own orders out of the orderbook they are trading on,
or a feedback loop can begin. This can be difficult, since some exchanges
don't provide a clear way to identify your own order, or it may be the case
that you receive the public UDP packet before the private TCP packet that
contains the ID required to match the two.

For example (a basic one), say I have an algo that calculates the midpoint
based on the best bid and ask volumes. And can only insert within x distance
of the midpoint. I add a bid to the top of the book for a 1 lot, get the UDP
public depth update, don't know whether this is my order or someone elses. So
I re-calculate the midpoint, which pushes the midpoint upwards (greater bid
volume), which means I then pull my bid, which means I recalculate the mid
again, then re-insert, etc. etc....

~~~
yummyfajitas
I actually mentioned this briefly - see the discussion of the "reconciliation
step" in the "Quote Stuffing is not a DOS attack" section.

I guess I assumed everyone's reconciliation system was as good as the one I
worked on. But you make a very good point, as well as a meta point - this
stuff isn't easy.

~~~
gd1
Sorry, missed that bit

------
personZ
The 2. in the series, if I understand it right, is that they overwhelming the
derelict SIP feed between market locations, increasing the benefits that their
own high speed connections brings. If true, that does make money because they
can act "in advance" at other markets before the artificially slowed SIP data
gets there.

