
The Idiot's Guide to High Frequency Trading - jerryhuang100
http://blogmaverick.com/2014/04/03/the-idiots-guide-to-high-frequency-trading/
======
JumpCrisscross
The NYSE has always required market makers continuously buy and sell the
stocks they specialise in. They must do this in all market conditions, even if
it means running a loss. In exchange, they get privileged access to order flow
information. This is why those seats are valuable. NASDAQ challenged that
model by removing the physical trading floor. Instead of humans standing in a
pit there were humans sitting behind screens. But in both cases there is an
institution guarding access to the market's nerve centres. Barriers to entry
were raised and the incumbents protected.

HFT takes the abstraction one step further. Democratised is that access to
privileged order flow - anyone who can pay can get it. Lost is the role of a
designated market maker - HFTs can pull out when markets get rough. This is a
valid debate.

But don't confuse yourself. The order flow privileges being criticised have
always accrued to institutional market makers. It is no surprise that those
incumbents are the ones driving the present lobby.

~~~
AndrewBissell
> The NYSE requires designated market makers to buy and sell the securities
> they specialise in. They must do this even in violent market conditions

This "mandate" tended to break down when tested by actual market panics. Bids
dried up in 1987, and histories of the 1929 crash often mention clerks and
floor runners who entered stink bids at $0.01 for stocks that had recently
traded at $30 or $40, and got filled because the specialists had abandoned
their posts.

Whether human or computer, market makers know better than to stand in front of
a freight train.

~~~
JumpCrisscross
Market makers can declare "self help" in the event of technical difficulty.
Exchange rules prohibit declaring self help to avoid adverse market
conditions. In practice, self help declarations have a peculiar habit of
cropping up in rough markets. Nevertheless, "guarantors of market liquidity"
remains a rallying cry for traditional market makers.

------
tptacek
If HFT GUARANTEES profits, why are the profits to HFTs declining so sharply,
and why do HFT firms make such a small fraction of what the buy-side firms
make?

Much more discussion here:

[https://news.ycombinator.com/item?id=7531429](https://news.ycombinator.com/item?id=7531429)

~~~
mayukh
Declining profitabilities caused by increased competition. What market risk
are the scalp-style of hft strategies (not all hft's are scalping) taking ? If
they can cancel orders at abandon and make pennies if they win the race
(against other hft's), but can simply x out of their order if the price doesnt
go their way is as close to a riskless profit as it gets.

virtu's prospectus as a case in point.
[https://www.sec.gov/Archives/edgar/data/1592386/000104746914...](https://www.sec.gov/Archives/edgar/data/1592386/000104746914002070/a2218589zs-1.htm)

~~~
lrm242
Manoj Narang of Tradeworx has stated their average holding time is up to 10
minutes. Is he taking risk?

[http://washpost.bloomberg.com/Story?docId=1376-N2CB0F6TTDTQ0...](http://washpost.bloomberg.com/Story?docId=1376-N2CB0F6TTDTQ01-3SAG95I8F8SD907LL6IA4CLK0P)

~~~
mayukh
'Average' holding times don't help understand the issue. You could have one
position that was a long-term bet

edit: I did not imply that none of the hft strategies were taking market risk.
The ones that scalp certainly seem to.

~~~
tptacek
This is the second time you've used the word "scalp", as if all liquidity on
the public markets for the last century weren't funded by "scalping".

In the absence of "scalping", trading in stocks works like trading in houses.
There are lots of buyers. There are lots of sellers. In the majority of cases,
they disagree materially on the correct price. Therefore, it (a) takes forever
to enter or exit a position, and (b) often forces people to accept terribly
unfavorable pricing.

The "scalp" market makers take is the market price for always having a
counterparty willing to trade with you at a price near the true market value
of the trading instrument.

If you want the markets to work more like the real estate market, you can do
that: place limit orders. The fact that market orders carry a premium price
isn't a subtle detail of the market; it's trading 101.

All things being equal, you want the "scalp" to be as thin as possible. The
wider the spread, the closer the scalping blade comes to the skull. Liquidity
has a price, and investors want that price to be as low as possible.

So now, an exercise for you: at the height of HFT profit-taking, was the price
of liquidity (a) lower or (b) higher than it was during the 1990s?

~~~
dllthomas
Though houses also have the additional complication that they're not (at all)
fungible.

------
theorique
Complete idiot's guide is right.

Exchanges offer price-time priority. A 'better' price (higher bid, lower
offer) gives you priority over a 'worse' price. Given two orders at the same
price, an earlier order gives you priority over a later order.

If you spend millions of dollars on computers, data feeds, and salaries to
skilled personnel, to predict the motion of markets _correctly_ and work
within the system to make money, then you've earned that money fair and
square. Companies that don't operate within the rules get shut down fast and
hard by the SEC and/or FINRA. This is a _highly_ regulated and policed
industry.

HFT is hardly different from Warren Buffett identifying a company with a
10-100 year time horizon. Should we penalize Buffett because he's smarter and
better at that game than everyone else in the world? Is he scamming the person
from whom he bought shares in 1984?

As Knight Capital showed us two years ago, you can spend those millions, and
_still_ screw up and lose $10M per minute for the better part of an hour until
your firm is bankrupt. No one is guaranteed to make money.

Posting an order and canceling it is fair game. When the order is in the
market, it's a live intent to trade. When it's canceled, it's no longer a live
intent to trade. Why is this "moral" when it's a person doing this on a time
scale of seconds or minutes, but "fraud on the market" when it's a computer
doing it on a time scale of micro or milliseconds? If someone hits that HFT
computer's bid, there's an _obligation_ to fill that order - nothing is
different about such an order just because it's placed and canceled quickly.
The only objection is fear of the unknown. And the only difference here is
that it's being done based on an automated strategy, faster than people can
react. Your broker has access to HFT tools. Your 401(k) manager has access to
HFT tools. If you're a day trader, your order is probably being routed through
HFT tools.

~~~
a8da6b0c91d
It's more like taking a peek at the other players' cards, when only you have
the ability to do that.

We have a bunch of laws that establish the pretense that everybody in the
market is equal. I suspect we should get rid of most of the laws and drop the
pretense. Folks should participate with the understanding that there are
unfairly advantaged operators at all levels. But until that happens it's hard
to morally square aspects of HFT.

~~~
lrm242
No one can see the other players cards in trading unless the player shows
them. Brokers sending IOIs to dark pools releases info to the market. Hitting
venue A and then venue B in a serial fashion will release info the market,
letting some traders cancel before you get to B.

There is no mechanism on the lit markets for anyone to see an order BEFORE it
interacts by either posting to the book or being crossed with another order
and generating a trade. Anyone who says otherwise is unfortunately
misinformed.

~~~
foobarqux
Flash orders

~~~
lrm242
No longer available on lit US equities. A failed experiment by exchanges to
compete with internalizers by providing the opportunity for price improvement
to those orders who elect to be shown to participants prior to posting or
taking.

Can you think of one active in today's market?

~~~
foobarqux
On which instruments are flash orders still operational?

~~~
lrm242
In US equities? None on lit venues. I'm sure some dark pools still handle
IOIs.

~~~
foobarqux
Outside of US equities.

~~~
lrm242
No idea. My game is US equities.

------
nbouscal
If you want to know what the actual effects of HFT are on individual investors
rather than hypotheticals, a study was done in Canada last year on that exact
question. Spoiler alert: HFT is good for the little guys.

[http://qed.econ.queensu.ca/pub/faculty/milne/322/IIROC_FeeCh...](http://qed.econ.queensu.ca/pub/faculty/milne/322/IIROC_FeeChange_submission_KM_AP3.pdf)

~~~
dangerlibrary
1\. That paper hasn't been published or peer reviewed.

2\. One of the authors used to work for HSBC as a derivatives trader.

3\. That study analyzes the impact of a fee-per-order regulatory regime. It is
not a study, as you claim, of a world with high frequency traders vs a counter
factual world without them.

------
njharman
"With these changes the fastest players were now able to make money simply
because they were the fastest traders."

Directly contradicts "speed is not a problem".

Speed directly translates into being able to "front of trades from slower
market participants". If they did not have the speed, they could not
algorithmically predict the actions of (or react to activity on one exchange
before it reaches other exchanges) and jump in front of slower participants.

There's other issues on top of this, such as paying for early or privileged
access. But, if you are faster, you can successfully arbitrage across time or
distance(really same thing as time)

------
arjn
Correct me if I'm wrong but couldn't this problem be solved by some simple
rules or act of legislation ?

For example, make it a rule that when you buy a stock, you have to hold it for
at least some time - say a minimum of 5 minutes.

Or you could just impose a fee/penalty on traders that buy sell the same stock
within one day of trading.

(note - I'm merely speculating here and have no specialized knowledge)

------
stygiansonic
My (non-expert) take: HFT is a tool and it can be used for good or bad.
_Potential_ good uses may include: reduction of bid/ask spreads, improved
liquidity and faster arbitrage of things like ETFs.

However, it can also be used for bad:
[http://www.sec.gov/News/PressRelease/Detail/PressRelease/136...](http://www.sec.gov/News/PressRelease/Detail/PressRelease/1365171484972#.Uz75MfldXm4)

In that link, the SEC charges that a certain firm employed a trading strategy
using non-bona fide orders to attract interest on another (bona fide) order.
This involved submitting orders on the other side[1] to simulate interest.

HFT makes such strategies viable, or at least more viable.

1\.
[http://www.nanex.net/aqck2/3598.html](http://www.nanex.net/aqck2/3598.html)

EDIT: Corrected to remove inaccurate information.

~~~
lrm242
The action you link to was not perpetuated by HFT. The vast majority of
Lightspeed's business is good ole fashioned screen traders, and that is likely
what this was. Previous actions for spoofing and layering were also the result
of manual traders using regular day trading tools.

~~~
stygiansonic
My mistake. I assumed HFT was involved because the timescale of the "layering"
orders and subsequent cancellation (following the bona fide order execution)
was on the order of several hundred milliseconds.

------
herbig
John Stewart had a really interesting interview with Michael Lewis this week
about HFT:

[http://thedailyshow.cc.com/guests/michael-
lewis](http://thedailyshow.cc.com/guests/michael-lewis)

------
mianos
Just jump to the second comment for the actual details. Welcome to the
internet, where if you are popular you can write any shit you like and people
will read it, even if you don't actually know what you are talking about. (I
wrote the software that runs some of the major stock exchanges. I did that for
13 years).

------
tlrobinson
Is "registering algorithms" really a good solution? It seems incredibly hard
to analyze or simulate such a complex system.

------
arasmussen
I really wish the "Idiot's guide to high frequency trading" would start out by
explaining what high frequency trading is to me like I'm an idiot...

------
Havoc
Step 1) Have deep pockets, big balls and the right skills

------
cgmoore120
read a great article on this the other night

[http://www.nytimes.com/2014/04/06/magazine/flash-boys-
michae...](http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael-
lewis.html?_r=1)

------
jseliger
Note that this:
[https://news.ycombinator.com/item?id=7524315](https://news.ycombinator.com/item?id=7524315)
is a better place to start than the linked post.

------
thikonom
The title of the article sounds oxymoronic.

------
transfire
The solution is very simple: a Tobin Tax.

------
asbestoshft
I have a lot of experience with HFT. Lets clear all of this up.

How exchanges work for dummies.

All of the modern US stock exchanges work as a limit order book. Orders are
filled on a price-time basis. In other words, you put in an order, if it isn't
filled immediately it gets put in the limit order book with your order going
to the end of the line ( lowest priority ) at that price. That's it, that is
how it works. Lets look at an example.

I want to buy 250 shares of ABC. Right now there are orders in the book. 5
orders to buy at 9.95, each one for 100 shares. There are also 5 orders to
sell at 9.96 also each order for 100 shares. I have several options.

(1) I can put in a market order to buy 250, when this order gets to the order
book sell orders will be filled starting with the oldest ones, the ones that
have been in the book longest, first. My order will be matched, this is why
they call it a matching engine, with two of the sell orders and then a partial
on the third one. As orders are filled in the order book messages are sent out
to all subscribers of the data that those orders were filled so everyone can
keep track of the state of the book. It is important to note that when my
order gets to the matching engine/order book there is no chance for the orders
in the order book to get cancelled. Those orders aren't told "you are about to
get filled do you want to cancel". (2) I can put in a limit order to buy at
9.96 in which case the order will be filled as described in 1 unless the sell
orders at 9.96 are cancelled before my order gets to the matching engine. (3)
I can put in a limit order to buy at 9.95 in which case my order goes into the
order book at 9.95 and I have to wait for the orders ahead of me to either get
filled or cancelled and then I have to wait for someone to send in a sell
order, either market or limit, that will get matched with my buy order.

That's it! Repeat, all day, every day. OK, that is a lie, there are special
order types and that is one way exchanges try to differentiate themselves but
that is a simplified way of how this works.

So where is the problem that Brad Katsuyama saw? I don't know him all I know
is what I've seen and read. One thing to note is that he talks about buying
large blocks of stock. Very large. From
[http://www.nytimes.com/2014/04/06/magazine/flash-boys-
michae...](http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael-
lewis.html?_r=0) "It used to be that when his trading screens showed 10,000
shares of Intel offered at $22 a share, it meant that he could buy 10,000
shares of Intel for $22 a share." 10k shares at $22k/share, that is $220k in
value. So he isn't talking about some small retail order, this is, as he
points out, institutional flow.

So lets think about what happens in this kind of case. There are 13 exchanges,
all running limit order books. They are geographically disperse and internally
they are implemented differently and have different latencies and different
message protocols etc. At some instance in time all 13 exchanges have some
orders to sell at $22.00. Think about that. In some sense that is awesome.
Look how efficient the market is. If you went to 13 gas stations what are the
chances that a gallon of regular gas will cost exactly the same at every
station? Anyway, back to our problem. You want to buy 10,000 shares. You fire
off your order to exchange A to buy 10,000 shares. This hits the matching
engine and you get filled on 10,000 shares ( THERE IS NO WAY FOR THOSE SELL
ORDERS THAT ARE IN THE BOOK TO BE CANCELLED!!!! The orders in the book aren't
sent a message "hey, a 10,000 share order is coming in and about to hit your
order do you want to cancel your order first?". THAT DOES NOT HAPPEN!!! ). The
thing is that those 10,000 shares that got matched, lets say each order in the
book was 200 shares. Your 10,000 share order just took out 50 orders. The fact
that these orders were executed is sent out to every data subscriber who is
listening for that data. All messages are timestamped, sometimes down to the
nanosecond. From that the recipients can tell "wow, all of these 50 orders for
10,000 shares were executed at the same time so that was one order."

Now, if you had orders in at other exchanges to sell at $22 what would your
response be? For a second lets think about something more mundane, like TVs.
Suppose you work at Target and you have a friend who works at Walmart. You're
working one night and a guy shows up driving a semi-trailer. He walks in and
says "How many 50 inch flat screens do you have? I'll buy them all, as
advertised at $400." You sell all 20 that you have to him. As he's walking out
the door he says "Now I'm going to Walmart." Just then your buddy at Walmart
texts you and says "what's up? slow night right." and you go "Hell no, just
sold 20 50 inch TVs to a guy, he bought everything I had and says he is headed
for your Walmart to buy more." Now what does your friend do? In a world where
prices are set by supply and demand Walmart could look at that and go "wow,
there is a big buyer of 50 inch TVs out there, lets increase our price from
$400 to $410." And so that is what Walmart does and sure enough the guy comes
in and buys all the TVs from Walmart for $410. Does the TV buyer say the
market is "rigged"?

In any case that is very similar to what happens in this case. All of the HFT
and market makers see this big order rip through a lot of sell orders at
exchange A and they go "wow, there is a big buyer, I better change my sell
order from selling at $22.00 to selling at $22.01 or $22.10 or $25 or
whatever". Brad Katsuyama looks at that and goes "I've been front run, this
market is rigged." And then he has a piece of software written so that all
orders arrive at the different markets at the same time, as opposed to leaving
his trading workstation at the same time, and suddenly there isn't any front
running. There isn't any front running at all, what he is seeing is the
reaction to supply and demand.

HFT and market makers live in fear of something called "adverse selection".
Market makers are required to post a two sided market, they are supposed to
put up a price and quantity that they are willing to both buy and sell at
right now. They publish this and they can't back away from it, they have to
live up to it if another order shows up to buy or sell at a price that matches
they have to execute the trade. The risk to them is that they are trading with
someone who has information that they don't have. Just like playing poker or
any game, you're going to lose if you play with people who are better than
you. In this game though sometimes just having a big order is enough to move
the market even if the person behind the order doesn't actually have any new
information. Market makers and HFT either want to buy on the bid and sell on
the offer and make the spread or they want to do arbitrage. That is how they
make money. They lose money if they trade with people who are smarter or
faster than they are. As soon as they see big orders come in they get really
skittish and so they immediately move their orders to reduce their risk.

I've written a lot and bored almost everyone. Ask away and I'll do my best to
answer your questions.

A couple other comments:

1\. I'm not saying HFT is good or bad, I don't know. 2\. Lots of people
mention costs. Just to ballpark the whole thing. For the technology you need
to Arista switches, that is $500/port, get a 48 port switch, that is $24k. For
servers you need something decent, not crazy, dual CPU, 64 GB, running linux,
maybe solarflare for kernel bypass networking, custom software. The hardware
will cost you $6k/server. Then you need a connection to an exchange, varies
but lets say $10k/month, then you need market data, $5k/month, then you need
an order entry port, that is maybe $1k/month. That's it. Well, that is at one
exchange, then you need low latency 10 Gbps cross connects between exchanges
and you need to do that at each exchange. Then you need to find a clearing
firm and you need to become a broker dealer ( exchanges and clearing firms
only talk to BDs ). This is a highly regulated environment. You must have a
compliance person, you will get audited yearly by SEC, your exchange, FINRA,
all kinds of stuff. You will need a serious accountant. You have to have
written procedures. 3\. And how in the world do you avoid the Knight type of
meltdown? Well, when you send in an order the order goes to the exchange. But
the exchange also sends the order out via a "drop copy". This might go to your
clearing firm so they know you aren't losing tons of money. But you can also
get the drop copies and then you can constantly do your own risk checks that
your orders you think are open are actually open but the ones you cancelled
are actually cancelled. Most exchanges also have "cancel on disconnect" so
when you really don't know what is going on you shut off your connection to
the exchange and any open orders in the order book are cancelled for you.

------
a8da6b0c91d
Some aspects of HFT seem highly analogous to insider trading to me, and
insider trading is currently illegal. Abusing the ability to cancel orders
fast in order to ping for non-public data about other orders sure seems like
an insider advantage. As I understand it this "pinging" is central to HFT
strategies. Am I missing something?

~~~
lrm242
There is nothing about HFT that is analgous to insider trading. Unfortunately
the people describing it in public like Cuban simply don't understand what's
going on. There is no non-public information at work here.

~~~
guelo
The information that the HFT traders pay millions of dollars to acquire before
the rest of the market does is not public at the time that they acquire it by
any definition of public that does not include having to pay millions of
dollars to acquire the information.

~~~
lrm242
That's fundamentally wrong. Direct feeds are available to anyone who wishes to
purchase them. Do you have a source for a piece of data purchasable by a firm
that isn't on the public side of the information pipe?

~~~
erichocean
> Direct feeds are available to anyone who wishes to purchase them.

Out of curiosity, how much does a direct feed cost, and is there a practical
limit to the number of simultaneous buyers (for example, physical server space
in a data center)?

~~~
lrm242
Price lists are all public. Look at nasdaqtrader.com or batstrading.com.

------
notastartup
If HFT adds no value, rigs the game, adds cost to everyday investors and
traders looking to make an honest fair buck from the markets, will it be
allowed in the future?

I almost feel like SEC will come down on them hard.

Basically, I'm bummed at the idea that as individuals, we can't innovate in
this field without ponying up several million dollars in startup cost.

~~~
lrm242
It doesn't cost several million dollars. It does require some capital, likely
less than your average valley angel deal. It is not easy and you'll probably
fail, but you can certainly try. Phone up Nasdaq, they'll happily give you all
you need to know along with pricing information on their website.

