
Barbarians at the Gateways: High-frequency Trading and Exchange Technology - rupelloy
http://queue.acm.org/detail.cfm?ref=rss&id=2536492
======
cs702
So much talent... focused on the _buying and selling of securities_ , instead
of _creating new things_ that will make the world better in a directly
measurable manner.

Virtually all trading volume today consists of buying and selling _old_
securities -- essentially, legal claims on _existing_ assets. The sale of
_new_ securities issued to finance the creation of new products and services
-- for example, a company selling new shares via an IPO or issuing new bonds
for investment in physical infrastructure -- represents only a minuscule
portion of total trading volume.

I'm not sure having so many of our best and brightest minds going to Wall
Street (and into high-frequency trading in particular) is a good thing, from a
societal perspective.

\--

PS. Whenever I read anything about high-frequency trading, I'm reminded of the
following passage, written by John Maynard Keynes in 1936 -- 77 years ago (!):
"Of the maxims of orthodox finance none, surely, is more anti-social than the
fetish of liquidity, the doctrine that it is a positive virtue on the part of
investment institutions to concentrate their resources upon the holding of
'liquid' securities. It forgets that there is no such thing as liquidity of
investment for the community as a whole. The social object of skilled
investment should be to defeat the dark forces of time and ignorance which
envelop our future. The actual, private object of the most skilled investment
to-day is 'to beat the gun,' as the Americans so well express it, to outwit
the crowd, and to pass the bad, or depreciating, half-crown to the other
fellow."

Source:
[http://ebooks.adelaide.edu.au/k/keynes/john_maynard/k44g/cha...](http://ebooks.adelaide.edu.au/k/keynes/john_maynard/k44g/chapter12.html#chapter12.3)

The more things change, the more they stay the same!

~~~
dopamean
Someone said something similar about a lot of people in the startup community.
That we were sending our best and brightest out to create products that
ultimately only served the purpose of pushing ads (Instagram, Facebook,
Twitter, etc). I don't know how true this is. There are a lot of very smart
people in the world. The luxury of having that many smart people is that we
can have a lot dedicated to designing amazing infrastructure that sends
satellites into orbit and we can also have Twitter and HFT.

~~~
31reasons
Then why we haven't cured cancer yet or malaria or global warming or
psychological disorders ? There are a lot of smart people in the world but we
really can't afford to work on silly things.

~~~
SkyMarshal
Some of those problems are "Nine women can't make a baby in one month" type
problems. Sometimes the solutions are stumbled upon by accident, rather than
by concerted effort, especially in medicine.

In fact it wouldn't surprise me if some sophisticated neural net algorithm
developed for HFT was repurposed and used successfully for other things -
medical research, climate modelling, brain research - basically anything
involving non-linear dynamic systems.

Only problem of course is that all those HFT algorithms aren't FOSS, but the
incentive to develop them in the first place would be greatly diminished since
they'd lose whatever edge they had almost immediately after publishing.

The markets are a pretty decent lab for experimentation and learning about
complex systems, which we're still not so great with, as long as systemic risk
can be prevented and every player forced to take their losses instead of
passing them off to governments. I personally don't think HFT is remotely as
much a problem as what caused the financial crisis - a few flash crashes here
and there plus an "HFT tax" on asset prices, vs the arguably-fraudulent
origination and dissemination of trillions of dollars of bad debt resulting in
the near total annihilation of the global economy. I think we'll gain a lot
more (or at least lose a lot less) as a society by prioritizing the latter
problem.

------
chollida1
> Some facilities such as the Mahwah, New Jersey, NYSE (New York Stock
> Exchange) data center have rolls of fiber so that every cage has exactly the
> same length of fiber running to the exchange cages.3

I remember the first time someone told me that. I thought they were kidding.
Then I actually got to see the data center.

The exchanges, and HFT firms, take this very seriously. The speed at which
they execute is just unbelievable.

The amount of technology and networking stack knowledge is just mind blowing.

I remember tuning my linux distribution's networking stack to gain an
advantage only to find out that everyone had moved onto using asic's and
infiniband hardware.

I continue to be amazed at the sheer talent and brain power that goes into
this field:)

~~~
rfnslyr
How do I get into this field? I'm a 20 year old making a shit ton of money by
doing mobile work but I find it so boring. Nothing about it is innovative. I
feel like a high level computer construction worker. It's brainless for the
most part, it just takes _time_.

I want to get into some really cool stuff. I've been wanting to get into
investment banking from a technical standpoint, HFT, algorithmic trading etc,
but honestly don't know where to start.

What are the de facto books/websites/reads for this field?

~~~
sseveran
The worlds of HFT and the rest of algorithmic trading are very far apart. For
HFT pickup the exchange documentation for each exchange. Memorize how order
types. Get a copy of RegNMS. Memorize that as well. To be effective an HFT
needs to exploit the market structure. Learn how processors work. How do you
avoid a pipeline stall or a cache miss? Become an expert in c++. Maybe pickup
some verilog for those pesky FPGAs.

The rest of algorithmic trading offers a lot more flexibility. In either case
read 'Trading and Exchanges'. It will give you a good overview of the
mechanics of all different venue types. Then its thinking about how to build
better algorithms then the next guy, or get better data.

~~~
shogunmike
I second the recommendation of 'Trading and Exchanges'. It's a great book to
get insight into how the underlying exchanges work and how to exploit certain
architecture to produce profitable strategies.

------
pasbesoin
I'm going to say this again: Exchanges should be put on a stepped clock. I'd
suggest a one minute interval. Orders pile up during that interval, and then
they're processed in random order. If things start to go really, _State-
concerning_ hinky, then authorities "pull the plug" before the next interval
is up.

Anything less, to my mind, serves point-shaving and profiteers, rather than
real investment.

Then, all the "gearheads" can go back to doing useful science. And maybe
getting paid decent wages and benefits for doing so.

/grump

~~~
gd1
By the way, this is an awful, awful idea that you hear quite a lot. Think
about what would happen. What incentive would a participant have for placing
an order early in the one minute interval? If you can't cancel it, you're a
sitting duck. And if you can't be executed now, why insert the order now? By
doing so, you show your hand. You'd end up with a rolling one minute e-bay
auction where everyone tries to insert at the last possible microsecond.

~~~
MereInterest
Why would it matter when people place the orders if there is no extra
information to be gained during that time? If the orders placed are only
revealed after they have taken effect, why bother waiting for the last moment?

~~~
phyalow
YOu have just reinvented a 1 minute cross on a unlit platform...

------
kylebrown
The sub-penny rule (decimalization) was also mentioned by Chris Stucchio (aka
yummyfajitas).[1] And it is extensively discussed in the book "Dark Pools".[2]
Because order queues only exist at whole-penny prices, the bots are forced to
compete on speed. If it weren't for that rule, there would also be competition
on price because you could place offers and get price-priority in the sub-
penny interval.

The playing field is basically a DDoS arms race, where the main technique is
full-throttle quote-stuffing to push anyone without equivalent hardware and
connection away from the front of the queue, and out of the game. (Dark Pools
also mentions a conspiracy of special undocumented order types on certain
exchanges). The ACM article even admits that the algos aren't even
particularly intelligent, just very speedy: "In my experience, most high-
frequency algorithms are fairly straightforward in concept—but their success
is based largely on how quickly they can interact."

A distributed order book like the one in the Ripple protocol[3] is a much
fairer system. The network charges a transaction fee in order to prevent quote
spam/DoS attacks, price fractions can be as small as 6 decimal places, and
offers are processed in an effectively random order (within each discrete
ledger).[4]

1\.
[http://www.chrisstucchio.com/blog/2012/hft_whats_broken.html](http://www.chrisstucchio.com/blog/2012/hft_whats_broken.html)

2\. Dark Pools: The Rise of the Machine Traders and the Rigging of the U.S.
Stock Market by Scott Patterson

3\. [https://ripple.com/distributed-fx/](https://ripple.com/distributed-fx/)

4\.
[https://ripple.com/wiki/Arbitrage#Rapidly_changing_offers](https://ripple.com/wiki/Arbitrage#Rapidly_changing_offers)

~~~
gd1
Mind you, if they lower tick sizes, that means a lot more order cancellations.
A market-making algo is simply a control system, taking in a set of inputs and
spitting out a fair price that it is willing buy/sell something for. The
number spat out is a rational one, that is then rounded to the nearest
discrete tick. Smaller ticks = more jumps in the discrete output price = more
cancellations/modifications. Which is fine, except the anti-HFT brigade seem
to think a trading system that cancels 10000 orders and trades 100 times is
the devil. It's not, it's about as remarkable as any other control system. You
may as well express outrage that the cruise control system in your car adjusts
the throttle ten thousand times a second. The only downside is that the
exchange needs to cope with that level of traffic.

------
dopamean
I'm wondering if someone here can explain to me the value in HFT for anyone
other than the people doing the trading. I went to Wharton and some old
friends are in HFT and the one thing they all say, and I hear repeated
everywhere, is that they provide liquidity to the market. Something about it
just doesn't ring true to me. If it is in fact true I'd love to hear an
explanation.

~~~
kasey_junk
Liquidity has certainly gone up.

Also, it is hard to prove a causal relationship, but with the rise of
electronic trading and HFT, price spreads and fees have come down.

I can't prove it, but I believe that insider trading is also much less rampant
in modern electronic systems than it was in the older pit based markets.

~~~
beagle3
Re: spreads & fees: Price spreads and fees have come down as a result of
electronic trading being open to everyone, and is probably independent of HFT
(at least, modern HFT): E.g., The decreased spreads and fees were happening in
europe in the early 2000s, when fastest updates were at 1/4 second, and slower
updates were at 2 seconds. The nanosecond scale was not the reason for this.

Re liquidity: it depends how you define liquidity. If you define it by "the
average size at the bid or offer and next few price levels" or "volume
traded", then yes - HFT has helped liquidity tremendously. If you define it as
"the probability that a large order can complete", then liquidity has NOT gone
up. With HFT, it's the same 100 shares/futures changing hands thousands of
times per day, and they disappear often in times of uncertainty. If you need
to execute a large order, you observe that you do not have the amazing
liquidity everyone in HFT is talking about.

Re insider trading: I wouldn't bet on it. With electronic trading, it's now
easier to do insider trading slowly without attracting much attention.
Furthermore, HFT has created new forms of fraud, so far unregulated: see e.g.
[http://www.nanex.net/aqck2/4329.html](http://www.nanex.net/aqck2/4329.html)

~~~
AndrewBissell
> If you define it as "the probability that a large order can complete", then
> liquidity has NOT gone up.

People seem to have this idea that back in the days of floor-based trading you
could just call up the NYSE and say, "sell 1 million shares of Citigroup!" and
the market makers would just kindly oblige you, without widening their spreads
or trying to eke out a bigger gain from a transaction which inherently carries
significant risk for them.

It's possible that the situation for executing giant orders in one fell swoop
hasn't gotten any _better_ under HFT, but it hardly seems to have made it
worse. And as the large flow traders become more sophisticated and start to
implement algos of their own, they'll get better at moving volume at a fairer
average price.

> With HFT, it's the same 100 shares/futures changing hands thousands of times
> per day, and they disappear often in times of uncertainty.

I don't know about "often," unless you're talking about trading individual
issues where news events introduce a high degree of uncertainty. There's
really only been one case of a market-wide drying up of liquidity, which
lasted for about 15 minutes during the flash crash.

There's also nothing new about market-maker liquidity drying up in times of
uncertainty or severe volatility. Plenty of stocks went "no bid" during the
1929 crash. Floor-based market makers are no more interested in standing in
front of freight trains than algos are. If you're looking for
someone/something to blame when markets crash and there are no bids to be
found, focus on the Fed and its attempts to manipulate the credit cycle, which
periodically fail in spectacular fashion.

~~~
tanzam75
> _It 's possible that the situation for executing giant orders in one fell
> swoop hasn't gotten any better under HFT, but it hardly seems to have made
> it worse. Plenty of stocks went "no bid" during the 1929 crash._

1929 is an obsolete example. Heck, the SEC didn't even exist in 1929. A more
relevant example is 1987 -- how many stocks went "no bid" on Black Monday?

Here's how HFTs operate today. Bid furiously while the going is good. If you
find yourself on the wrong side of a trade, you can always hit up the actual
market-makers for a penny loss. And if the markets collapse, you pull out.
What could be better? Supply liquidity when it's not needed, and stay out
precisely when liquidity is most needed.

Meanwhile, the registered market-makers are pulling back, or pulling out. And
why not? They used to count on having the good times to balance out the bad.
But now, the HFTs are siphoning off their profits during good times. Thus, the
cross-subsidy has gone away. In periods of market stress, the HFTs go hide
under a rock and don't participate in absorbing losses.

The effect is: There are now fewer market-makers in periods of market stress.
That's how HFTs have made things worse.

------
tucson
""It's legalized front-running. I think it is basically evil and I don't think
it should have ever been allowed to reach the size that it did," he said. "Why
should all of us pay a little group of people to engage in legalized front-
running of our orders?""

Charlie Munger,
[http://www.cnbc.com/id/100705820](http://www.cnbc.com/id/100705820)

~~~
smoorman1024
What's the difference between HFT and traditional Human Market Makers?

May I suggest that Electronic trading and HFT have made it possible for a few
people to do the work of thousands. This has brought down the cost of trading.

------
3327
This is a well informed and put together article. Up to date technically and
the Author knows what he is talking about. As a former HF trader I endorse it,
great piece.

~~~
rurounijones
Out of curiosity. Why "Former"?

~~~
3327
I walked away and moved onto more exciting things like entrepreneurship. As
the author states, HFT is like rebuilding gravity models every week, so your
product is not sustainable and doesn't grow, cash generation from HFT also is
usually not scalable on a per strategy basis.

~~~
rurounijones
Thank you, interesting insight.

------
elecengin
A great overview of the space. The numbers and techniques are realistic and
the article refrains from the moral debate (which you can find elsewhere...)

------
ForHackernews
God, what a waste of talent and technology.

~~~
thejosh
Not really, what comes out of it is nice.

------
001sky
_By 2005 we were approaching the one-millisecond barrier for latency
arbitrage_

Everything you need to know, in one soundbite.

~~~
gd1
Why? What's the response time on an airbag release in an accident? On the ABS
in your car? Your graphics card manages to rotate, scale, texture and shade a
hundred million polygons per frame in around 1000/60 = 16ms. We're talking
here about processing one UDP frame, usually under 256 bytes.

Why is this the only topic discussed by HN where technology suddenly becomes
this frightening bogeyman?

~~~
001sky
Its not the engineering, its the objective. Latency arbitrage is code-word for
front running. Woops.

~~~
gd1
You'd best look up the definition of "front-running". Reacting to an event the
fastest makes you the quickest behind-runner, but you're still behind. Not
front-running at all. Where did you even learn this term? Are you just
regurgitating something you've heard elsewhere?

~~~
001sky
Try looking up the definition of 'arbitrage'

~~~
gd1
Wtf? An example of latency arbitrage is say, two identical fungible securities
listed on different exchanges. An order arrives causing the price on one
exchange (A) to spike up (the event), you react _after_ that event and buy the
security on the other exchange (B). You reacted fastest to public knowledge.

Front running would be if you were say sitting on a broker desk and one of
your clients rang up and said to buy lots of A. Then you put in your _own_ buy
order _before_ the clients one hits the market. That is front-running. You got
in before that knowledge become public. Do you see the difference?

~~~
001sky
_An example of latency arbitrage is..._

Arbitrage requires that you cross the trade. You've merely described a
following Algorithm. One that leaves open long positions.

~~~
gd1
And you've merely tried to change the subject. Something can't be front-
running if it happens after a publicly disseminated market event. Understood?
Flash orders = front running. Trading ahead of client flow = front running.
Latency arbitrage != front running. HFT market making != front running.

~~~
001sky
Arbitrage is precisely the subject. Sorry you don't understand that.

~~~
easytiger
It isn't front running, stop being a dick

~~~
001sky
Actually, it is "synthetic" front running.

------
icedchai
Good to see an article about real engineering, not the usual web app / JS
bull...

------
manishsharan
Can someone please tell us what is the return or for an hft fund vs. the
market in the long run say 5 years. Also , are there hft funds available to
general public for investing.

~~~
HockeyPlayer
> Are there hft funds available to general public for investing?

No, one of the characteristics of HFT is that the returns are high and the
capacity is low.

~~~
chollida1
> one of the characteristics of HFT is that the returns are high and the
> capacity is low.

The second part is correct, the first, not so much anymore.

[http://www.businessweek.com/articles/2013-06-06/how-the-
robo...](http://www.businessweek.com/articles/2013-06-06/how-the-robots-lost-
high-frequency-tradings-rise-and-fall)

------
unabridged
Would there be any loss of liquidity if trades were matched in 1 or 2 second
intervals? This would eliminate all front running and colocation bias.

~~~
kasey_junk
This doesn't solve the problem in anyway. The issue is what if you have a
different amount of buy to sell qty. Who gets their order filled? This is
called matching and each market matches in different ways.

The typical way (and one of the leading drivers of latency races) is first
in/first match. If you continue to FIFO match on your intervals there is still
a tremendous advantage to being first into the interval.

If you aren't going to FIFO match that is fine, you just need to determine
(and publish) your matching routine.

