
A High Frequency Trader's Apologia, Part 1 - yummyfajitas
http://www.chrisstucchio.com/blog/2012/hft_apology.html
======
losvedir
Huh, I'm glad this was posted!

1) Didn't realize when I read this it was by yummyfajitas, whose comments I
always respect.

2) Do I understand correctly that the order book is public (albeit
anonymized)? That surprised me -- it seems like this would lead to meta-games
and jockeying for position and such, as opposed to different parties just
submitting into a black box what some security is worth to them.

3) Are there any good papers/articles modeling different market set-ups? For
instance my black box one above. Or one where orders are matched randomly
rather than chronologically. Or one where trades happen in one batch once per
day. I can't say I'm opposed to HFT, it just seems to sap a very large amount
of engineering brainpower for not that much societal marginal benefit anymore.
If I could snap my fingers and give up a bit of the liquidity and get all
those engineers back I probably would.

4) I once worked in a sell-side equity research shop which traded stocks the
old fashioned way (based on fundamentals) and had a non high-frequency trading
desk, etc. That type of company is getting hurt by the lower spreads offered
by HFT. But I never did quite comprehend why it made sense to pay for our
product (research) with trading commissions. Seemed to cross two unrelated
services, although that kind of business model seemed deeply entrenched in the
market.

~~~
yummyfajitas
_Do I understand correctly that the order book is public (albeit anonymized)?_

Yes.

 _...submitting into a black box what some security is worth to them._

These are Dark Pools. <https://en.wikipedia.org/wiki/Dark_liquidity>

_Are there any good papers/articles modeling different market set-ups?_

These are often called Crossing Networks.
<https://en.wikipedia.org/wiki/Crossing_network>

~~~
losvedir
Wow, that's exactly what I was asking about. Thanks for the interesting links.

------
adamsmith
The way the system is set up makes it a tax on global efficiency. All of these
resources (crazy smart people, money, equipment) are needlessly wasted on
getting lower latency when really it adds no systematic value.

Instead, just as the system disallows fractional penny bids, it could
discretize order times to the nearest e.g. 5 seconds. Boom. Suddenly you have
more MIT/etc grads solving better problems for the world.

~~~
traversal
Whenever HFT comes up on HN, someone asks why we don't just remove the
incentive for low latencies by matching orders less frequently. It's a fair
question, but I think there are good reasons not to do this.

Let's say we discretize the exchange to, say, 1 Hz, so all the orders are
queued up and then executed simultaneously at the next clock tick. Now, on a
given tick, there will almost always be a mismatch between the number of buy
orders and the number of sell orders, so some orders will go unfilled. How do
we choose which ones? (For this discussion I am ignoring price. Obviously we
will fill more aggressively-priced orders first; the question is how to
prioritize orders of equal price.)

We could give priority to the ones that arrived first, but of course then
we're back to traders racing each other. The only other scheme I can think of
[1] is to fill every order in proportion to its size. So if Alice wants to
sell 200 shares, Bob wants to buy 100 shares, and Charlie wants to buy 300
shares, we give 50 shares to Bob and 150 shares to Charlie.

Well, this scheme has its own problems -- arguably worse ones than the current
system! If Bob really wants to buy 100 shares, and expects to be competing
with Charlie, he has an incentive to place a much bigger order: in this case,
if he knew Charlie's bid size, he would also ask for 300 shares, expecting to
get 100. Of course, Charlie will be playing the same game and inflating his
own bid size. The equilibrium is that everyone asks for way more size than
they actually want.

There are two things wrong here:

1\. Traders are spending mental energy trying second-guess each other's order
size; those who don't are crowded out of the most competitive (and, typically,
profitable) trades.

2\. Since traders can't guess correctly every time, they will sometimes end up
buying or selling much more than they want, which means they have to turn
around and do the opposite trade, paying the spread in the process. If this
happens during during a large price movement, these traders can lose even more
money, and in their haste to reverse their trades, they will drive the market
even further in the same direction.

And by the way, this is not just speculation. While I'm not familiar with any
exchanges that discretize their clocks this way, there is a product with a
similar system: the CME Eurodollar contract [2]. In this contract, unlike the
other futures products traded on the CME, resting orders do not execute on a
first-come-first-served basis; instead they are allocated "pro rata", or in
proportion to their size, much as I described above. Based on my conversations
with several people who trade them (including HFTers), the results are also as
I described: traders routinely over-order and suffer the consequences.

Hopefully this sheds some light on why this seemingly obvious solution is not
widely implemented.

[1] Of course I can think of many other schemes, but they either have obvious
problems or basically reduce to this one.

[2] There are other products matched this way, but I'm not as familiar with
them.

~~~
pak
Why not use a random order for order fulfillment at each tick? No skewed
incentives with that rule.

If that creates too many concerns over security of the RNG, then use
deterministic rotating ranks. Traders may be able to predict who they will be
able to beat out on the next tick, but the high ranks rotate throughout the
herd, so everybody gets a fair shot at beating out others at the same price.

~~~
jsnell
A random ordering will create an incentive to split orders into smaller
pieces. Which at the limit is going to devolve to the the proportional
fulfillment case once everyone submits only orders of the minimum size.

~~~
Drbble
Weight the orders by size.

~~~
sans-serif
That's exactly what pro rata does.

~~~
Robin_Message
I was gonna post a comment saying what they meant was obvious, but then I just
went "ohhhh." * That's actually quite an annoying little problem there.

One work-around might be to add rules to the exchange forbidding the placing
of multiple bids by the same entity at the same price, and then allocating
randomly.

This is also reminding me of the Talmud's descriptions of how to pay off
debtors when there isn't enough money to go around:
[http://mindyourdecisions.com/blog/2008/06/10/how-game-
theory...](http://mindyourdecisions.com/blog/2008/06/10/how-game-theory-
solved-a-religious-mystery/) Such a system could work for exchanges too,
although it still encourages over-bidding.

Surely also over-bidding has its own risk/reward curve so it would be
reasonable to allocate pro-rata and let people work out how much to over-
allocate.

* And realised what the implications were.

~~~
traversal
What you see as "an annoying little problem", I see as a sign that the system
is consistent and robust. It's a good thing that the system doesn't
distinguish between a hundred one-share orders and a single hundred-share
order; such a distinction would be totally arbitrary. (By contrast, ranking
orders by time is _not_ arbitrary. All else being equal, it's better for
things to get resolved faster; the only question is whether we're giving too
much of a reward for too little of an improvement.)

Elsewhere in this thread (<http://news.ycombinator.com/item?id=3856015>) I
argue against treating orders differently based on the "entity" that placed
them.

In general, we want traders to spend their time thinking about asset prices
and risk, not market structure and game theory. The price-time priority system
is a very simple one that rewards traders for deciding what they want and then
announcing it right away. All the modifications that have been proposed in
this thread encourage traders to play games, second-guess one another, or
otherwise work around the system.

There are definitely tradeoffs, but personally I think it's better to use the
cleaner system and accept the latency arms race than to add a layer of
artificial incentives -- and for what it's worth, it appears that nearly every
major electronic market has come to the same conclusion.

Finally, thanks for the link to the Talmud article: it was a very cool
application of game theory to history. (Of course, I don't see any indication
in the article that the Talmudic system would be an improvement over pro
rata.)

~~~
Robin_Message
Just referring back to this from the comments on the second part, and just
wanted to mention I meant "annoying little problem" in the sense that it seems
a shame there can't be other solutions, like how Arrow's Impossibility Theorem
prevents achieving a perfect voting system
(<http://en.wikipedia.org/wiki/Arrow%27s_impossibility_theorem>). I definitely
see what you mean about this being the most optimal solution from the point of
view of actual pricing and risk. The downside though is there is profit to be
made in slicing time every quicker which can't be prevented without breaking
something else, but the money spent on this time-slicing is broadly wasted
because it is just an arms race.

So I don't think the current solution is necessarily wrong; just annoying we
can't get all the benefits and _also_ avoid the arms race.

No problem, I very much like the Talmud article and how simple the explanation
works out to being.

------
lrm242
A few notes:

(a) Orders for stocks priced below $1.00 are permitted to display in sub-penny
increments. Not that this is really relevant to HFT, but it is often a point
of confusion.

(b) Orders may rest on the book and display at a price but execute at a
separate price. For example, if a market is 20.01 / 20.02 and I believe true
value is 20.017 I can bid 20.01 but tell the matching engine that I'm willing
to accept 20.015. That you are willing to trade at the midpoint also places
you at a higher priority in the queue, above those only willing to buy at
20.01.

~~~
javert
So basically you don't know the actual midpoint where the market is trading.

Doesn't that mean people can get an advantage by guessing where that point is
and bidding just below it? (Not too far below, because then you're wasting
money.) And if so, do they use strategies to try to determine where the
midpoint is? (e.g. offer a very small number of shares at many different
points, see which ones execute)

~~~
cheddarmint
What the linked article doesn't discuss is that it is possible to get a real-
time notification when a trade occurs. The price of the last trade is the mid-
point that you are referring to. Algo trading/HFT shops are listening for
trades as well as quotes - this is how they know the price that trades are
actually happening at.

~~~
javert
So what you're saying is that the subpenny price that the trades are actually
executing at is publicly available?

~~~
cheddarmint
Yep. Here's an example:

<http://www.google.com/finance?q=ABIO>

When I checked this page, the last displayed trade was $0.737.

~~~
retrogradeorbit
But isn't this just the aggregate? How is every trade visible? Why, if its all
free and public, do you have to pay so much for live data feeds?

~~~
bradleyjg
All trades are made publicly avaiable _eventually_. The lower latency you want
the more you have to pay. And some trades aren't avaiable at low latency for
any price.

------
radarsat1
What I don't understand about HFT is why its efficiency does not seem to be
limited by transaction fees. I've tried a bit in the past to play with some
algorithms that buy and sell (on paper) and it seems like the faster you go,
the more you need the market to move in order to scrape a few pennies of
profit, because the movement has to be sufficient to cover the spread, the
transaction fees, _and_ provide some profit. Of course with a simple example
like "buy at $20, sell at $20.10", it seems simple enough to extract some
profit, but in fact it would need to be enough profit that it covers the
transaction fees, which seem to be a kind "friction" that limits the smallest
change in price that is useful. From what I can tell this kind of movement
just doesn't generally occur in the sub-millisecond time frame, so how does it
work? Or is it just that they are trading so much $$ in one shot that a very
tiny change in price covers the transaction fee without issue? (Meaning they
are taking very large risks..)

~~~
drubio
The OP provides a good technical explanation. Unfortunately he fails to
mention another closely related term to high frequency which is what makes
this type of trading highly lucrative: 'Front running'. See
<http://en.wikipedia.org/wiki/Front_running> .

Front running is illegal, but if you look for successful cases of high
frequency trading they are generally tied-to/accused-of front running. And as
you might imagine, in order to do front running you need to be high-up on the
food chain (i.e. be a market maker)

The missing key about front running in the article is the 'anonymous' bid-ask:
"The matching engine takes his order and displays it (anonymized) to all other
traders with a data feed." and "She places her orders, and it is again
displayed to the world (anonymously) and stored.".

If you have forehand knowledge of the bid-ask (i.e. non-anonymous) the market
maker can front-run and with high-frequency make a considerable profit.

~~~
tptacek
Frontrunners have knowledge of specific trades that are about to occur. You
frontrun a specific block trade. The definition you imply here suggests that
all market makers are "frontrunners".

------
deathflute
Thanks, this is a very clear article on the most important ingredients of HFT.
I am curious to hear the apology but I am not sure there is anything to
apologize for. The market marker is taking on substantial risk of adverse
selection and is being compensated for it. As long as legal means are used to
achieve that end, this is not really different from any other business
strategy. Causal implied benefits to society from any business are subjective
and the most important consequences are usually latent.

Anyways, I am curious to know if elaborate prediction models are used here or
it is mostly a game of adjusting prices based on correlated instruments? Also,
how does one build a simulator for such strategies?

~~~
wglb
Apology in the sense explained in comment in another thread:
<http://news.ycombinator.com/item?id=3852595>

------
anonDataUser
This offers no insight into HFT and is not an apology. Why is it here?

For people interested in HFT, start your exploring with a google search for
"latency arbitrage".

~~~
firebones
It's an apology in the second sense of the word apology:

"2. a defense, excuse, or justification in speech or writing, as for a cause
or doctrine."

Basically, apology as "explanation and defense of an idea".

~~~
cynicalkane
The word "apology" almost always means saying sorry, even to people who know
the other meaning.

The author might consider the word "apologia", which does not obviously mean
saying sorry and has the additional benefit of sounding Greek, stirring up
memories of Socrates' famous defense at his trial. But again, there isn't even
a defense in the author's post, just the promise of one forthcoming.

~~~
yummyfajitas
I was alluding to Hardy's book, "A Mathematician's Apology", not the ancient
greek one. I've updated the title on HN as per your suggestion.

------
achy
From my understanding of this, HFT basically makes it impossible for a non-HFT
trader to buy at anything but his maximum buying price and to sell at anything
but his lowest selling price. This seems like it would significantly undermine
the profitability of the market for non-HFT traders.

~~~
tptacek
HFT makes things very bad for day traders. It makes things marginally better
for people who want to take or sell a long-term position. It doesn't change
limit orders.

~~~
UK-AL
I guess a lot of anti hft literature comes from traders? Yes?

~~~
wglb
And the former middlemen who lived on the $0.10 spread that existed before
hft.

------
sid6376
What I would love to hear in the follow up to this post is what are the
typical technical challenges which a typical HFT encounters? If someone wants
to be an HFT what are the core skills one must have? (Strong in algorithms,
good c++ skills?). How much math should one know and what kind of math?

~~~
radikalus
I think to be successful your team needs to be strong in four key areas:

\- Trading (Strategy) \- Software \- IT \- Research

There isn't necessarily one of the four that trumps the rest, and different
players have different relative strengths across those four disciplines.
(Which are as you would imagine, not terribly specific to HFT)

I imagine the question is developer-focused, so on the software side, I think
I value most highly:

\- Don't fuck up. This is obvious, and the trading adage of "Don't lose money"
is overdone...but essential. Assume you will fuck up and have nets in place to
catch you. Doing safeties well isn't sexy and doesn't add any edge, but I've
witnessed "unlikely" safeties save trading systems.

\- Understand the hardware. Undestand the strategy. You cannot write fast
"generic" code. Have a sense for how fast you could optimally be on the
hardware and network stack you're using. This requires interfacing with IT
and, sometimes, trading, as some "strategies" traders come up with aren't easy
to do FAST; you might be able to suggest something similar that is potentially
much quicker on the critical path.

\- If you really want to compete in low-latency, C++. If you just want to be
pretty fast and your strategies aren't competing for specific orders with the
other market makers, I don't think it matters. You can probably do a lot of
strategies in 100us in any language. GUI can be done in anything, but it
should be able to scale. Most of the GUI pieces are pre-existing. Sortable
tables for fills. Ladders for displaying markets, etc. I've used Java/C#/C++
GUIs -- C++ has always worked best for me, but I wouldn't be opposed to doing
the front-end in JS/HTML honestly.

\- Understand the critical path. This and, to some degree, only this, has to
be FAST. Obsessing about how to optimally build a theoretical value cache more
quickly etc etc is often not that big of a deal. (It's not on the path)

\- Linux kernel and TCP/UDP stack DETAILED understanding. (Or verilog I
guess...)

\- Locks. When does a mutex really need to be locked? Obviously depends on
system architecture. And can often dictate architecture.

I think almost anything else I'd say would be either vague or generic. I think
there's a lot of niches one can carve out in HFT and, while I personally value
generalists who can holistically understand the goal (make $), there's a ton
of people doing really well by just being the most badass at X/Y/Z.

~~~
jrockway
C++? Everyone is now building their own hardware instead.

~~~
radikalus
A pretty small subset of HFTs are building their own hardware from my
perspective; there's a lot of people _claiming_ to be doing awesome stuff on
FPGAs.

------
dbecker
I've seen so many news articles full of hyperbole about HFT, and this is the
first legitimate explanation I've seen. Thank you.

------
johnohara
_The astute reader will probably ask this straightforward question:_ \--
"Jubal, does that seem right to you?"

~~~
tptacek
The concept of a market maker? Market makers are not an HFT "innovation";
they're kind of central to the whole concept of a trading exchange.

He hasn't yet explained anything more than what a market maker is or where
HFTs fit into them. This post is the first in a series. It's background
material. It would be a little weird if anything in this particular post got
your hackles up.

~~~
johnohara
Mal, Inara, Jayne, Kaylee, River, Simon and Wash are all characters from the
television series "FireFly." Jubal Early (Richard Brooks) is the bounty hunter
in the last episode who ultimately ends up as an object in space.

Throughout the episode, his hypothetical situations always end with the
rhetorical question, "Does that seem right to you?"

My comment was a tip of the hat back to Chris Stucchio. I enjoyed the article
and also look forward to more.

~~~
tptacek
D'oh!

------
someben
Kinda an obvious introduction to order books. Later I hope he gets into the
rebate mechanism:

[http://www.interactivebrokers.com/en/accounts/fees/NYSEstkfe...](http://www.interactivebrokers.com/en/accounts/fees/NYSEstkfee.php?ib_entity=llc)

Note that you get PAID for placing certain types of orders.

------
jboggan
Will the following installations address the role of high frequency trading in
flash crashes? Although high frequency trading seems like a very good strategy
when arrayed against human opponents I have to wonder what happens when all
the inputs and outputs are controlled by HFT algorithms trading against each
other. How do you debug a HFT algorithm?

~~~
wglb
The rapidity of recovery from the flash crash can be credited to HFT.

~~~
toomuchtodo
Lets all rejoice in how fast those who created the disaster fixed it. _rolls
eyes_

~~~
runeks
How is the price of a stock market plummeting a "disaster"? For those who own
the stock it's of course not advantageous, to those who wish to buy it, it is.

I'd like the HFT traders to get into milk and apples. Maybe they can crash the
price of these too so I can get some cheap groceries.

~~~
GFischer
Groceries do have huge variations (at least here in Uruguay), and there is a
futures market in them.

There's definitely talk of fruit futures :)

[http://www.goodfruit.com/Good-Fruit-
Grower/October-2011/Futu...](http://www.goodfruit.com/Good-Fruit-
Grower/October-2011/Futures-market-for-concentrate-coming-soon/)

"The status of the long-awaited futures trading contract in apple-juice
concentrate hasn’t changed. It’s still coming “soon.” "

Milk doesn't only because it's regulated (here at least).

------
photon137
Can you say how feasible it is to build an HFT startup - the operating costs
of execution/connectivity/colocation(!) aside - do you think regulatory
authorities make the barrier-to-entry too high?

SEC/FINRA have Series 7/56 requirements for employees, UK has FSA-related
principal investment regulations - both require a company to sponsor a person
to trade. Also, the MiFiD regulations in EU require an "adequate capital" of
700k euros - is it really possible to get venture/seed capital covering these
bare costs so that you just remain "legal", let alone operate?

~~~
veyron
Thanks to sponsored access and the large trader rule, you really just need the
blessing of a clearing firm and pay some colocation fees. As a one time cost
anyone can get in with 200K deposit (on a 6:1 levered setup) and about 50K in
startup expenses, with a 25k/mo recurring charge.

P.S.: that is what I am doing now :)

~~~
photon137
That's useful to know - thanks!

------
caladri
Men buy, women sell, and also if women are sad you can feed them strawberries
and get off on it. Got it. Now I understand HFT perfectly.

~~~
francoisdevlin
It's a firefly reference.

~~~
caladri
That much was obvious. When you build your examples around a sexist experience
of sexist media, your examples will themselves end up sexist. I mean, why
isn't Zoe buying, too? What is it that Inara's selling? I think we know, and
from the heternormative misogyny rises the situation in which a cheeky
description of market making has exclusively men buying and exclusively women
selling.

~~~
yummyfajitas
_What is it that Inara's selling?_

Shares of Blue Sun. <http://firefly.wikia.com/wiki/Blue_Sun_Corporation>

I'll be sure to have Zoe and Book make an appearance in part 2. Sorry that my
fanfic missed your favorite character.

------
ChristianMarks
I'll believe that HFT adds liquidity to the market when the typical start of
employment to retirement period is 15 milliseconds.

~~~
lrm242
An expected and uneducated statement. It's simple really: any participant who
rests an order on the book is adding liquidity. Period. Many HFT strategies
are largely passive (market making, for example) and therefore entirely rely
on resting orders (as Chris' article points out). Therefore, they add
liquidity.

Now, if you want to get into a discussion as to the quality of that liquidity
then you have to go educate yourself a bit more. Once you do you'll find that
the liquidity being added is not toxic nor fleeting. However, what you and
most others don't understand is that it is also significantly more informed
passive liquidity than it used to be.

This may lead folks to believe that the liquidity is fleeting because they can
cancel away before you can execute. However, that simply means they are better
and faster than you, not that they are doing anything wrong.

Why, then, don't we simply require them to rest their orders for a longer
period of time? Simple: you want tight spreads. As spreads shrink market
making becomes less profitable. When market making is less profitable market
makers have to be more risk averse to avoid getting adversely selected and
losing money. More intelligent models (they become more informed) and less
latency (they can leverage that informedness to avoid being selected) is how
market makers stay profitable in a market where the average spread is 1 penny.

~~~
_delirium
How about rather than adding regulations, just discretize the market's clock?
Would a market that performed a trade-resolution tick once every 100ms lose
anything important? I believe arbitrage improves efficiency, but I'm less
clear on the benefits of extremely high temporal resolution. Currently that's
what the incentives encourage optimizing for, among other things. If you
predict a market movement 10ms ahead of time, you can profit from that brief
10ms temporal-arbitrage window. And like any arbitrage, that does indeed
improve the pricing signal in an absolute sense, in this case by taking a
price change that was 10ms later than it "should've been", and moving it up in
time via your trading. But it's not clear that 10ms-level pricing
inefficiencies are actually something particularly important to smooth out via
arbitrage, when you can just define them out of existence by going to a
discrete-time market (which are fairly well-studied in the mathematical
literature).

~~~
lrm242
I think if you consult the literature you'll find that the prevailing trend is
for markets to become more continuous rather than more discrete. Continuous
markets help with true price discovery. Our markets are continuous in time
right now and we've pretty much squeezed the spreads to their maximum, so
while prices aren't continuous they are about as close as we can get w/o
eliminating the profitability of the primary market participant (market
makers) that make it function.

You can scheme up any number of possible microstructures that sound
interesting on the surface. There's a reason why none exist. Remember, there
is nothing to stop you from implementing a time-discretized market place.
Current regulations (Reg-ATS) allow for you to do so. You'll find it hard to
compete. Some markets do perform large scale discretization for block orders.
Read up on POSIT and related ATSs and dark pools.

~~~
MarkPNeyer
what exactly is 'true price discovery' ?

can a price be false?

~~~
roel_v
With insufficient trades, you cannot know if a certain price is really what
the 'market' wants to pay, just what that one person at one point in time
wanted to pay. Overgeneralized, the more trades, the more accurate the price
gets.

------
RobertKohr
So if you remove the restriction on subpenny increments, you remove the need
for HFT?

~~~
RickHull
HFT would still serve a role with subpenny increments. Nonetheless, the
subpenny rule is a very important mechanic in the construction of HFT.

~~~
AndrewHampton
I agree. I think there is currently a balance to be struck between accuracy of
the price determined by the algorithm and the speed at which it obtains that
price. I think removing the subpenny rule would just reduce the importance of
speed and increase the importance of accuracy. I'd imagine trades would still
happen extremely fast though.

------
xaa
Do HFTers pay any sort of commission or transaction fee (to the exchange)?

~~~
sailfrog
The common way transaction fees are handled in US equity markets is called the
"maker/taker" model. When a participant adds liquidity to the exchange order
book, AND that order is executed, they are given a very small rebate per share
(they are the "maker"). The other side of the order, the firm who "removed"
liquidity from the order book, is charged a very small fee. The difference
between the rebate and the fee is the spread the exchange earns for executing
the trade.

Note that not all orders that add liquidity qualify for rebates, the rebate
may depend on particular order properties as well as the way in which the
trade executes.

------
joejohnson
Last week, Bloomberg published this op-ed defending HFT:
[http://www.bloomberg.com/news/2012-04-10/high-speed-
trading-...](http://www.bloomberg.com/news/2012-04-10/high-speed-trading-is-
progress-not-piracy.html)

The piece is largely propaganda. A nice rebuttal is found here:
[http://www.ritholtz.com/blog/2012/04/hft-pirates-and-
their-a...](http://www.ritholtz.com/blog/2012/04/hft-pirates-and-their-
academic-friends/)

Can anyone explain why multiple exchanges exist (besides for historical or
political reasons)? It seems anti-competitive that exchanges can be private at
all, let alone that exchanges can be allowed to operate as monopolies in a
market or country.

~~~
javert
_It seems anti-competitive that exchanges can be private at all_

"Anti-competitive" is a false (i.e. nonsensical) concept. I mean, if I am a
really good golf player, and I just keep getting more and more competitive, do
I suddenly cross a bright red line that makes me "anti-competitive" because
I'm just too competitive?

It's just a made-up term used to attack business that lots of people have
unknowingly bought into.

If it were to actually mean something, it would mean using the government to
grant a monopoly. That's the opposite of any kind of private business where
two or more parties are _voluntarily_ agreeing to trade. Like, for example, a
private exchange.

~~~
Cushman
Anti-competitive behavior would be literally cheating at golf.

In both cases, it has been observed over time that allowing individuals to do
whatever they want has a negative effect on the game. You are not required to
believe that the rules are necessary, in either business or golf, for them to
exist.

~~~
javert
_Anti-competitive behavior would be literally cheating at golf._

Right, and "cheating" in the real world is defined by fraud and physical
violence and their myriad manifestations, _not_ mututal trade to mutual
benefit (as on a proprietary exchange, for example). Or offering the best
product at the lowest price.

But, that's not at all the meaning of "anti-competitive" as people use it.

~~~
Cushman
It is the position of the United States government, among others, that
cheating includes the use of monopoly or oligopoly to unfairly restrict
competition in any of several ways which are neither violent nor fraudulent.

You're welcome to argue that that position is misguided, but it is certainly
not nonsensical.

~~~
javert
_You're welcome to argue that that position is misguided, but it is certainly
not nonsensical._

I think it's a very commonly held position, and it's understandable that
people hold that position.

I think monopolies are impossible or next to impossible except when granted by
the government. BTW, I think (not sure) that that's actually the origin of the
word "monopoly."

So, yes, monopolists are cheating, and the government is cheating. Of course,
if the government permits cheating, it's "dog eat dog," so you have to cheat
and/or might as well in some cases. (That, in fact, is what makes the Occupy
Wall Street protests so inane, IMHO.)

I believe that in a truly capitalist system, oligarchies would be unable to
weild undue influences for the same reasons I think monpolies would be unable
to.

~~~
rayiner
> I believe that in a truly capitalist system, oligarchies would be unable to
> weild undue influences for the same reasons I think monpolies would be
> unable to.

"Truly capitalist system" is not an economic term, it's a political term. As
such it's not very useful in this discussion. It is important to distinguish
"free markets" in the theoretical sense from "unregulated markets" in the
political sense. Conflating the two just results in confusion.

Monopolies and oligarchies and all sorts of anti-competitive structures arise
naturally in unregulated economies. For example, monopolies arise in any
market where the marginal cost of providing a good or service continues to
decrease without hitting a floor. Natural monopolies also arise in situations
with significant network effects. Telecoms are the pedagogical natural
monopoly.

~~~
javert
_Conflating the two just results in confusion._

Completely disagree with you here. I use "capitalism" to refer to an economic
and political system, and I think people who use the terms the way _you_ are
are the ones introducing confusion. There are economists on both sides of
this, as well as philosophers, so having clarified, we should stop this kind
of discussion. What I mean is, if you are already so convinced of the
rightness of this many things that I disagree with, I don't think HN is going
to be a productive place for further discussion.

 _Monopolies and oligarchies and all sorts of anti-competitive structures
arise naturally in unregulated economies._

I completely disgree, but I think we disagree on which usage of _most_ of the
words in that sentence is proper and which is invalid. (All the ones that
aren't just glue words.)

I know you've read this stuff in economics textbooks. I'm already aware that I
disagree with those particular textbooks.

------
lifeformed
Not making any value judgements here, I just find it fascinating how much we
have abstracted away the original concepts of the stock market.

~~~
grandalf
Why do you say that? How would a basic marketplace work without the same
rules?

~~~
lifeformed
I'm not saying different rules are necessary, just that it's interesting how
abstract the whole concept is. I think about stock like how it was explained
to me a long time ago, in very simple terms: Bob runs a restaurant worth
$9000, and he needs $1000 more to buy a new oven. 10 of his friends lend him
$100 each, in exchange for 1% ownership of the restaurant. Now with the new
oven, Bob can bring in more money and he shares 10% of that with his ten
friends.

Now we have this gigantic hurricane of integers that we call the stock market,
and people make millions by running complex programs that find numbers that
are slightly bigger than other numbers, and then run some code. An electrical
signal is sent across the country, and in a database somewhere, transistors
that represent somebody's bank account are changed into a different
configuration. Everyone is sitting behind a desk, or standing at a terminal,
pushing buttons. Based on what pattern of colors the screen flashes, people's
lives are ruined or made.

~~~
grandalf
Well articulated comment. Indeed it has become much more abstract. I guess one
way to think of it is:

If someone is trying to raise money for an oven it may be fairly difficult to
find people willing to invest. If investors are found, it may be difficult to
agree on a fair arrangement.

The stock market has grown precisely because it offers price discovery and
liquidity. Putting shares on an open market lets the fair price be discovered,
and when investments are priced fairly they become more liquid, which means
there is less risk to the investor b/c he/she can change his/her mind about
the investment decision and not be stuck with an investment that no longer
meets his/her objectives.

This lowered risk leads to more investment, which in turn leads to even fairer
prices, and so on.

I think the key point about HFT is that to the extent that HFT provides
liquidity (which it does by adding orders to the book at competitive prices)
it helps with price discovery.

Since the "fair" price is a function of the risk various parties are willing
to take by their willingness to risk trading at the wrong price, HFT not only
makes prices fairer but its participants risk their own capital to vouch for
the prices. Clearly if things turn out unexpectedly, the HFT firm is footing
the bill b/c of its devalued inventory.

------
toddh
Accepting the earliest bid seems quite arbitrary. How would the system change
if a random participant was selected? Or perhaps you could buy preferential
treatment or use an algorithm weighted by some good actor metric.

~~~
mayoff
You can buy preferential treatment: If you set a higher price for your buy
order, then you are preferred over every lower-priced buy order.

If you select a participant at random, you are encouraging each trader to
place many orders (at the same price level), to increase the probability that
one of his orders is chosen randomly.

If you use a pro-rata algorithm based on order size, you are encouraging each
trader to inflate his order quantity, and cancel very rapidly after all (or
nearly all) of his desired quantity is filled.

------
polynomial
> The rise of algorithmic trading is merely a special case of machines
> replacing humans.

One of the article's more salient points.

------
redwood
The author describes changing bid/sell prices based on new information (like a
new press release putting pressure on value, say). He doesn't mention whether
people are feeding the algos up/down pressure based on these press releases or
if they're automatic.

I assume they're at least partially automated. In other words for every
article that comes out about a given company, some algo (or human) gives it a
plus/minus ranking and magnitude based on what it means for the company.

Assuming this is partially done by algos that crawl press releases looking for
signals, or similar (in order to be first to market with latest news) one can
imagine a new type of hacking.

Hack the press releases and watch the algo trades go haywire....

------
MikeCapone
Question: Are all HFTs only market makers like this, or do some of them use
other strategies (front-running?) that actually make transactions more
expensive for others, or move markets a lot (especially in small caps, I would
guess)?

~~~
Nrsolis
HFT is a badly abused term that gets applied to any firm that uses computers
to trade.

In reality, there are many other strategies that use automated techniques to
model the values of securities using internal metrics and then trade when the
price is above/below the internally calculated value. Such conditions usually
indicate that the market has "mispriced" a security and will quickly realize
it's mistake and return to fair-value pricing.

Many "algorithmic traders" use statistical techniques to determine optimum
times to trade so they capture some profit margin with a high degree of
certainty. Those statistical techniques can involve anything from automated
balance sheet analysis and natural language processing of news items, to
pattern-matching techniques designed to notice when a large buyer/seller has
entered the market and needs to move a large block of shares.

------
javert
I would be really interested to know the author's motivation for leaving HFT.
For example, did you feel that you could make more money elsewhere?

 _Eventually I’ll even put forward a policy prescription which I believe could
cause HFT to focus primarily on socially valuable activities_

I'm really against the government making rules ot try to force people to serve
society in some fashion. That's totally contrary to the principles of
individual rights and pursuit of happiness upon which the country was founded.

~~~
yummyfajitas
_I would be really interested to know the author's motivation for leaving
HFT._

I left to do a startup. I am hoping to make considerably more money than I
would have in HFT.

 _I'm really against the government making rules ot try to force people to
serve society in some fashion._

I'd suggest you should stay tuned to the next post, since you will likely not
object to my proposal. It's not a punitive measure designed to express
disapproval, it's a minor tweak to market mechanics.

~~~
deathflute
It is interesting that you believe that E(compensation in a start up) >
E(compensation running a HFT strategy). The popular media is awash with
reports of billions of dollars being made by HFT traders. Are they completely
off?

What about HFTs that are running market taking strategies?

~~~
bearmf
Well, that is what ends up in the news: guys making billions. What articles
are you referring to?

There are also plenty of people making not so much money in HFT and plenty of
people losing money. This is a competitive market and it has become more so.

------
LinXitoW
While the article itself is very interesting to someone like me that knows
squat about diddledy, i must commend the author for including FireFly
references.

------
deathflute
I am also confused about how latency arbs work. If you are co-located at one
exchange, wouldn't you be further away from other exchanges?

~~~
tolitius
market makers work (have deals) with exchanges. e.g. a market maker M
guarantees to provide liquidity (e.g. orders) for a certain set of securities
to an exchange E, and gets certain benefits in return.

This market maker M makes sure that he is as close as possible to the exchange
metal, so he has an upper hand.

The importance of colocation is directly proportional to the liquidity (amount
of orders) you provide. e.g. if you send orders manually every minute or so,
it does not really matter where you are in comparison to a large bank that
sends orders every several microseconds..

------
djhworld
Not sure where the apology is...or if there is even one needed in the first
place.

I really enjoyed your article, it explained (in laymans terms) something
that's often been bewildering to me as an outsider to the financial sector.

Maybe this is why people outside of the sector are hostile to the idea of HFT
- because they haven't got a clue how it works?

------
ptolosav
Very interesting. As an undergraduate looking to learn more and hopefully get
some knowledge so i can get that "edge" in the field when i graduate, what
books do you guys recomend on trading? I know HFT is very specific and
probably there isn't lots of literature on it, but something more general will
do.

Thanks a lot.

~~~
surement
As I understand it, it's better to learn assembly, algorithms and parallel
computing. Some firms even look for PhDs in machine learning. The finance side
you can learn when you find a job.

------
tjmania2003
Really enjoyed reading this, great stuff. Even better Chris spent some time at
Snevets :)

------
Fando
very interesting

