
Donald MacKenzie on high-frequency trading - joosters
http://www.lrb.co.uk/v36/n17/donald-mackenzie/be-grateful-for-drizzle
======
ord-lhr-nyc-nrt
I worked on these systems and I can categorically say this is the best summary
of HFT I've ever come across.

Many of the systems, fiber networks, data centers, and software that I touched
are mentioned here. I've never heard the term "bit fucking" before, but the
explanation of having to understand what happens under the metal is very true.
You have to have a good understanding of software, compilers, assembler, CPU
micro architecture, network protocols, and operating systems to be able to
operate in this space.

I don't really know what else to add, except that I and many others I worked
with in this business think that batch auctions are the solution. Although the
work is technically challenging and interesting, I certainly don't like the
arms race aspect of it, and I wish the resources (natural and human) used by
this sector could be better allocated. One of my big concerns was the added
CO2 emissions of constantly spinning programs doing no useful work - all in
the name of minimizing latency. I did some research into reducing the carbon
footprint of our code by reducing idle spin cycles. Alas, I never really had
the time to get it working well before I decided to pack up and move on.

Happy to answer any questions anyone may have.

~~~
Faint
Why would the batch auction need to be FIFO? What about something like this:

1) When opening a "new tick", release data about trades in last tick, and
start accepting new orders. Information about orders submitted to this tick is
kept secret. Allow setting order price to practically arbitrary accuracy. 2)
After tick closes, match orders in _random order_. That is, it should not
matter at all on at which point in time inside this tick order arrived 3)
repeat

Allowing arbitrary accuracy would help turn the competition from speed or
stuffing the order book with huge number of orders to "spoof" the random
matching, towards competition of just providing lower spread. I understand
that it would still advantageous to hold submitting trades until very end of
the tick, to take into account as much outside information (other markets,
etc.) as possible. To counter this, maybe the exact length of the tick could
be randomized also. What do you think?

~~~
kasey_junk
So the biggest argument against random matching (whether in batch auction or
open market book orders) is that the easiest way to game it is to figure out
your expected fill rate. Then you just raise your order volume such that your
expected fill rate matches your risk profile.

For the advantage of removing the speed advantage, you get the downside of
providing advantage to people that have the loosest risk profiles and you
further complicate the audit burden such that it is impossible to follow.

Think of it from the perspective of a value investor that set a market order 6
months ago. Under the current system once their price becomes top of the book
they have a virtual certainty of getting filled. Under a (fair) random match
they have the exact same chance of getting filled as someone who literaly just
put in their order.

------
tfinniga
I don't understand why HFT is still around, if it's considered a problem.

Naively, couldn't all the craziness about HFT be resolved by enforcing a
1-second tick on the markets?

Like, right now it appears that trades occur instantly. Instead, queue
requests and have the trades occur only once a second. Or 5 seconds.

Would this work? I'm assuming that either there's no motivation to solve the
problem, or there are difficulties that I'm unaware of.

\---

I've been wondering this about HFT for a while and this seemed like a good
place to ask. Then I went back and finished reading the article, and this
solution is proposed at the end. Curious about whether it would work on not.

~~~
crdoconnor
>I don't understand why HFT is still around, if it's considered a problem.

Primarily because regulatory capture has, until now, overruled all attempts to
reign it in. The banks and hedge funds engaged in this activity wield vast
amounts of political power.

>Naively, couldn't all the craziness about HFT be resolved by enforcing a
1-second tick on the markets?

Not really. That just changes the game from "who can trade a picosecond before
you?" to "who can trade a picosecond before the tick?".

An almost microscopic tax on transactions would prevent it, though while
having a minimal impact on all other trading activity.

~~~
yummyfajitas
This is silly. HFT is a $10-50B/year industry, depending on who's numbers you
go with. (You can ballpark the numbers by multiplying trade volume by the
spread.)

For comparison, Morgan Stanley alone made $8B in profit in the last _quarter_
[1]. HFT is tiny, a drop in the bucket compared to the rest of the financial
industry. Further, many big players in the industry are against HFT since HFT
reduces their information advantage over smaller players [2].

The danger in a 1-second tick is that a) a massive change like this is hard to
do [3] and b) it would likely increase volatility. A stylized fact of
dynamical systems is that continuous feedback reduces volatility, whereas
discrete step sizes increase it - witness how many numerical ODE solvers can
blow up even while the underlying ODE is stable.

[1]
[https://www.google.com/finance?q=NYSE%3AMS&fstype=ii&ei=bVYI...](https://www.google.com/finance?q=NYSE%3AMS&fstype=ii&ei=bVYIVNiUNcqfrAHHp4GoCA)

[2]
[http://www.chrisstucchio.com/blog/2014/fervent_defense_of_fr...](http://www.chrisstucchio.com/blog/2014/fervent_defense_of_frontrunning_hfts.html)

[3] Bloomberg, Excel and other calculators contain mathematical errors (e.g.,
"weekends are sometimes treated as a business day for the purpose of interest
accrual") which _cannot be fixed_ because the change would be too disruptive.

~~~
crdoconnor
>This is silly. HFT is a $10-50B/year industry, depending on who's numbers you
go with. For comparison, Morgan Stanley alone made $8B in profit in the last
quarter [1].

I'm not sure what your point is here? There's more to finance than HFT? There
are other, bigger rip offs out there?

~~~
kasey_junk
His point is that small industries (especially ones with very low margins like
HFT) cannot wield the political muscle required to invoke "regulatory
capture". For instance BlackRock, a large hedge fund that has been very vocal
in it's anti HFT stance made more profits in 1 quarter last year than the
entire HFT industry. If regulatory capture was going to happen, don't you
think they would be in a better position to accomplish this?

Bill Ackman is another giant player who is highly vocal against HFT and is
known to spend donation money to attempt to make trades go his way.

HFT is the little guy in this story, the little guy doesn't get to create
regulatory environments.

~~~
crdoconnor
The few larger players who came out against it have done so only relatively
recently, and in Goldman's case, only subsequent to being under investigation
by the AG for its own HFT practices.

~~~
kasey_junk
This is simply untrue.

[http://www.reuters.com/article/2009/12/02/us-
highfrequency-i...](http://www.reuters.com/article/2009/12/02/us-
highfrequency-idUSN173583920091202)

A Senator quoted against high frequency trading at least 5 years ago.
Principal Financial is also quoted.

~~~
crdoconnor
One Senator, huh?

~~~
kasey_junk
Found in a 2 minute google search. Your position is false, that no one was
against HFT until recently. Large participants have been actively against it
for a very long time, further pointing out how incorrect your position is on
regulatory capture.

You have an ax to grind and an incomplete understanding of how the market and
HFT work. It makes you look foolish.

~~~
crdoconnor
>Found in a 2 minute google search. Your position is false, that no one was
against HFT until recently.

My position was that no large banks or hedge funds or other concentrations of
political power were against it until relatively recently. Which is completely
true.

An individual senator is not capable of making vast changes to the law by
themselves. Your belief that this IS possible is completely absurd.

>You have an ax to grind and an incomplete understanding of how the market and
HFT work. It makes you look foolish.

Frankly I don't know how you're still defending barclays for selling an
service promising freedom from HFT predators and then introducing HFT traders
to it so they can collect additional fees.

Your belief that these traders were dumb and did not know what they were
buying and therefore deserved to be lied to by Barclays is beyond retarded.
They knew exactly what they were doing. You do not.

~~~
kasey_junk
"My position was that no large banks or hedge funds or other concentrations of
political power were against it until relatively recently. Which is completely
true."

I mentioned several giant hedge funds (orders of magnitude bigger than the
entire HFT industry) that have been arguing against HFT for a very long time.
Further, the first article I found on a 2 minute google search, mentions a
large fund that was speaking out against HFT in 2009. This is as simple of
proof as I can provide that your position that HFT is not opposed by large
finance groups as untrue.

"Frankly I don't know how you're still defending barclays for selling an
service promising freedom from HFT "

Not once in this whole thread have I defended Barclay's and trying to paint me
with that brush is either a sign of extreme obtuseness or terrible rhetoric.
What I argue is that the reason Barclay's needed to defraud their customers is
that their claim (and all other dark pool's) claim that they could provide a
better trading environment by excluding HFTs has been historically proven as
false. Traders are not dumb, they move to the venues that provide them with
the best trading environment and that has continued to be venues that allow
market makers EVEN IN THE FACE OF FRAUD.

No matter how many times you paint HFT as "predators" or "sharks" or as
"spearing whales" you've yet to describe a single behavior attributed to HFTs
that is in any way unethical or illegal.

[edit: having to work around HN reply rules, I apologize] To the reply below
about BlackRock. Do you really think it is a realistic comparison to compare
BlackRocks recent profit (2013) to one of HFTs best year (2009)? At the very
least you should have mentioned that it was different years.

"Were this true, dark pools would not exist in the first place"

Except that there are all manner of product offerings that don't make sense
for their clients. The easiest analogy in this case is extended warranties.
"If extended warranties were so bad why do so many people buy them?" Of course
this is a ridiculous argument. People buy extended warranties to hedge risk
that is over rated due to warranty salespersons. The exact same way dark pools
are.

" There is a link I posted below in response to you that describes such a
strategy"

It does no such thing. It hand waves a lot of semi-technical non specific
jargon around an idea that is easy for people to get upset about. I could just
as easily write an article about how multicast networks were dangerous. It
wouldn't make it so.

~~~
crdoconnor
>I mentioned several giant hedge funds (orders of magnitude bigger than the
entire HFT industry)

You mentioned Blackrock, whose net income last year was $3 billion.

[http://www.wallstreetandtech.com/trading-technology/the-
real...](http://www.wallstreetandtech.com/trading-technology/the-real-story-
of-trading-software-espionage/a/d-id/1262125)?

"TABB Group estimates that annual aggregate profits of low-latency arbitrage
strategies exceed $21 billion, spread out among the few hundred firms that
deploy them."

>What I argue is that the reason Barclay's needed to defraud their customers
is that their claim (and all other dark pool's) claim that they could provide
a better trading environment by excluding HFTs has been historically proven as
false.

Were this true, dark pools would not exist in the first place. Traders would
have stuck to the public exchanges and paid their (lower) fees.

>No matter how many times you paint HFT as "predators" or "sharks" or as
"spearing whales" you've yet to describe a single behavior attributed to HFTs
that is in any way unethical or illegal.

False. There is a link I posted below in response to you that describes such a
strategy (spearing the whale) in precise detail.

~~~
kasey_junk
"You mentioned Blackrock, whose net income last year was $3 billion.
[http://www.wallstreetandtech.com/trading-technology/the-
real...](http://www.wallstreetandtech.com/trading-technology/the-real...)?
"TABB Group estimates that annual aggregate profits of low-latency arbitrage
strategies exceed $21 billion, spread out among the few hundred firms that
deploy them.""

Thanks for using 1 number from now and another number from 5 years ago. Are
you willing to concede the rest of your rhetorical high ground?

Even if we discount the market impact numbers that you have to juke, I found a
senator in 2009 speaking out against HFT. Can you find one defending it?

"Were this true, dark pools would not exist in the first place. Traders would
have stuck to the public exchanges and paid their (lower) fees."

Maybe if the buy side firms didn't engage in fraud to keep their clients out
of public exchanges that would be true.

------
joosters
IMO this is one of the best articles I have read on HFT. It gives a good,
balanced overview of the technology and the fundamental principles behind how
automated trading works, without any particular viewpoint or opinion directing
the text.

The section near the end on FPGAs and programming styles was an eye-opener to
me. I wonder how many other firms have taken risk checks out of their code to
save some nanoseconds...

~~~
conistonwater
> without any particular viewpoint or opinion directing the text

You say that like that's a good thing.

I know that a lot of rubbish has been written about HFT and algorithmic
trading, but it's sad that no one seems to be able to write about its merits
without starting a flame war.

~~~
Retric
It's an outgrowth of poorly designed matchmaking software at exchanges. The
could completly distroy HFT without costing society anything or adding
transaction fees.

~~~
conistonwater
Do you have any specific issues in mind when you say that?

A lot of the rules that markets operate by come from standards and
regulations, not software design. The software design reflects the kind of
markets exchanges want to set up, so it's not a software problem. The people
who wrote the standards had specific markets and specific outcomes in mind
when they wrote them.

~~~
harmegido
I work in this industry. I've come to the conclusion that I will be leaving
very soon. There are a number of reasons, but the primary one is because it's
become clear to me that I spend the vast majority of my time on trying to
figure out how to take advantage of flaws in exchange hardware/software.
Sorry, I can't go into specifics.

That being said, I don't know if it is possible to create an ungame-able
matching engine.

------
mer10z
I can't help but think of that article yesterday about how we're avoiding the
Big Problems
([https://news.ycombinator.com/item?id=8261098](https://news.ycombinator.com/item?id=8261098)).
It's crazy how much money and brain power people have invested in HFT which is
essentially just turning time into money with no benefit to society. What
could the human race have achieved if the same amount of effort was focused in
other areas?

~~~
fchollet
It could be that our current economic system implements a pretty terrible
resource allocation algorithm, where many workers' jobs consist in undoing the
efforts of employees from concurrent companies.

In trading, a quant's job is to outsmart or outspeed the other bank's team.
They invest considerable brain power, effort and money into producing
incremental improvements that are quickly rendered meaningless by the work of
the team in the building across the street.

Another prime example is advertising, a zero-sum game where competing brands
fight to undo other brands' brainwashing efforts and replace it with their own
brainwashing. A waste of the time and energy of million of humans.

Some more ideas on the subject: [http://www.sphere-
engineering.com/blog/15-hour-work-week.htm...](http://www.sphere-
engineering.com/blog/15-hour-work-week.html)

~~~
Retric
Advertizing and HFT are only zero sum for competing products in an established
market. For new products it can be a net benifit ex: "talk to your doctor
about ED."

------
guiomie
Can someone help me understand the following statement (specifically the don't
branch part) ?

"Don’t touch the kernel. Don’t touch main memory … Don’t branch.’ (That third
commandment means don’t fill your program with ‘if’ statements – ones with the
generic form ‘if A then do B else do C’ – because they get in the way of a
modern microprocessor’s capacity to do several things in parallel.)"

~~~
bluecalm
Obligatory link: [http://stackoverflow.com/questions/11227809/why-is-
processin...](http://stackoverflow.com/questions/11227809/why-is-processing-a-
sorted-array-faster-than-an-unsorted-array) see most upvoted answer for short
lecture about cost of branching in modern CPU's.

------
NickStefan
HFT is a two sided coin. For every person who complains that it's screwed them
out, they're forgetting all the times they've saved money on narrower spreads
due to extra liquidity. Possible?

------
tobltobs
Goldman Sachs’ Blankfein on Banking: ‘Doing God’s Work’. This kind of self-
deception seems to quite prevalent with those banksters.

~~~
GFK_of_xmaspast
They're totally doing god's work, it's just that the god in question is
Mammon.

