
Online Algorithms in High-Frequency Trading - stevewepay
http://queue.acm.org/detail.cfm?id=2534976
======
tptacek
The sad thing about any thread including the term "High-Frequency Trading" is
that we won't be able to talk about the technology, and instead will need to
relitigate automated trading.

I think we'd have been better off if the title of this thread had been "Online
Algorithms in Automated Trading Systems". As far as I understand, that title
has exactly the same meaning, but doesn't set off a crowd of HN commenters who
have never placed a limit order jumping in to crow about _Flash Boys_.

I'm a little frustrated, since this is a technical topic that happens to be
very relevant to me right now (I'm not a trader, FWIW).

~~~
kylebrown
The High-Frequency Trading Arms Race: Frequent Batch Auctions as a Market
Design Response[1][2] is a must-read. The tldr; is that a continuous mechanism
(sequential trade processing) only looks efficient in time-space, but is
actually inefficient in volume-space (most of the volume trades at stale
prices). The solution is discrete batch auctions (batch trade processing).

1\. slides: [http://faculty.chicagobooth.edu/eric.budish/research/HFT-
Fre...](http://faculty.chicagobooth.edu/eric.budish/research/HFT-
FrequentBatchAuctions-Slides.pdf) 2\. paper:
[http://faculty.chicagobooth.edu/eric.budish/research/HFT-
Fre...](http://faculty.chicagobooth.edu/eric.budish/research/HFT-
FrequentBatchAuctions.pdf)

~~~
tptacek
This is really interesting, but doesn't it presume that there's only one
trading venue for each instrument, and that instruments aren't correlated?

You can make time discrete for a single auction, but the time between auctions
still has to be continuous unless everything's traded in a single exchange,
which can't in reality happen even if we want it to (and we probably don't).

As I understand it, most of the most dramatic HFT stuff is already targeting
multi-venue trades or correlations.

~~~
kylebrown
I think it is modeled in terms of endogenous vs exogenous entry. But I might
be wrong, please correct me if I misunderstood.

~~~
tptacek
It's obvious that they see the issue, since the examples in their slide deck
are about correlation between different securities and between the same
security trading on different exchanges. What I don't see is how batch
auctions get around the distributed systems problem; we're still talking about
an eventually-consistent system, right? There still must be opportunities for
sniping, right?

(I'm not smart enough to know the answer, which is why I'm asking).

------
abeppu
The first section of this which relates HFT to one-pass algorithms is kind of
useless. Online estimation algorithms are much more broadly applicable than
that, and I'd expect that their relevance to HFT would be immediately obvious.
I've never worked in HFT, but I've probably written the online variance
algorithm at 3 of 4 companies I've worked. If you care about metrics such as
"how quickly are each of my customers sending me requests?", then this stuff
is relevant to you.

But more to the point, is there anyone in these fields who _doesn 't_ know
this stuff? At a basic level, the things that you can do a really good job
estimating in an online way are closely related to sufficient statistics,
which every undergrad stats course should cover. And hopefully anyone actually
involved in writing HFT systems has way more stats background than that --
right?

~~~
tptacek
I'm a systems programmer with a pretty primitive intuitive sense for maths. I
"agree" with you, in the sense that the algorithms this article mentioned were
intuitively obvious to me and thus seemed like they couldn't be sophisticated.
What's the state of the art for online estimators?

I've been playing with simulations of market maker algorithms from the
literature, and simple Bayesian stuff and MLEs are all I've really seen, which
makes me think that the interesting stuff isn't in the literature.

~~~
cdavid
There is a lot of literature, but it may not be easy to find out, and
different domains use different names for it.

One keyword to look for is 'stochastic approximation', with a long (> 50
years) history in engineering and stats. Another one is 'stochastic gradient',
which got a resurgence following the whole deep learning craze. There is also
'particle filtering' in signal processing.

One generic article I may recommend is
[http://leon.bottou.org/publications/pdf/online-1998.pdf](http://leon.bottou.org/publications/pdf/online-1998.pdf),
it does not require much math knowledge, but that may be less practical than
what you want.

The 'onepass' keyword mentioned in the original article is another keyword to
look for (see e.g. interesting work to do Monte-Carlo based linear algebra
operations for very large matrices, this used to be a reference, not sure
what's the state of the art is: [http://cs-
www.cs.yale.edu/homes/mmahoney/pubs/matrix1_SICOMP...](http://cs-
www.cs.yale.edu/homes/mmahoney/pubs/matrix1_SICOMP.pdf))

~~~
hcrisp
> ... different domains use different names ...

I agree. In engineering, many of these algorithms can be coded using digital
filters. Here's the exponential weighted variance:

    
    
      import scipy.signal as dsp
      import numpy as np
      
      b = np.array([alpha])
      a = np.array([1., -(1 - alpha)])
      
      # Compute exponential weighted average        
      signal_ = dsp.lfilter(b, a, signal)
      # Compute exponential weighted variance
      ewv = dsp.lfilter(b, a, (signal - signal_)**2)

------
expertonkappa
Is there a mirror? Seems like too many connections at the moment.

~~~
galaktor
google cache should do the trick if it's down

[http://webcache.googleusercontent.com/search?q=cache:WqysSpS...](http://webcache.googleusercontent.com/search?q=cache:WqysSpS-
SBoJ:queue.acm.org/detail.cfm%3Fid%3D2534976+&cd=1&hl=en&ct=clnk&gl=ie)

------
crazypyro
Needs a date stamp (2013).

------
amelius
Could somebody explain what the use is of HFT to society? Because I don't get
it.

Links are also welcome.

~~~
reverend_gonzo
Before HFT, there was only a few market makers, which meant the bid-ask was
high (if you wanted to buy or sell right now, you'd pay more), liquidity was
low (it'd be harder for you to get in or out of a large position and in hard
times it would take longer for the market to recover - See 80s crash vs flash
crash).

With HFT, there's a lot more competition over the bid ask spread. Trading
firms are essentially in an increasingly expensive fight over pennies, which
is good for everybody else, since it's cheaper prices, more liquidity, and a
more efficient market.

The people it's bad for is the old-school style brokers (which are effectively
gone now) and large investmant banks who have a fundamental view on the market
(they need to be smarter about executing since the market is more efficient
and reacts much more quickly to their buying and selling).

~~~
djrobstep
Which brings us to the next question: What is the social benefit of a slightly
smaller bid-ask spread and a few milliseconds of faster liquidity? And are
these benefits worth the massive amount of wealth extraction from the rest of
us that HFTers are engaged in?

~~~
henrik_w
The book "Flash Boys" by Michael Lewis chronicles the rise of HFT, and
basically argues that it doesn't really provide that much of a benefit.

[http://www.amazon.com/Flash-Boys-Michael-
Lewis/dp/0393244660...](http://www.amazon.com/Flash-Boys-Michael-
Lewis/dp/0393244660/)

~~~
kasey_junk
Standard PSA about "Flash Boys". If you are interested in HFT you are better
off quite literally not reading that book than reading it. It is very biased
and largely inaccurate.

If you want to read narrative non-fiction about HFT, "Dark Pools" by
Patterson, which also has its mistakes, is much better.

~~~
Judson
I would also recommend reading "Flash Boys: Not so Fast"
([http://www.amazon.com/dp/B00P0QI2M2](http://www.amazon.com/dp/B00P0QI2M2)).

~~~
CyberDildonics
Written by a high frequency trader. Are there any third parties that have
deeply explored HFT and come out in defense of it?

~~~
kasey_junk
Sure. Vanguard has the widely publicized view that HFT helps to bring down the
cost to retail investors for instance

[http://www.ft.com/cms/s/0/ff8c6486-cb37-11e3-ba95-00144feabd...](http://www.ft.com/cms/s/0/ff8c6486-cb37-11e3-ba95-00144feabdc0.html#axzz3ReBeAyhg)

------
CyberDildonics
Anyone who thinks HFT is something needed should go read Flash Boys. HFT is
basically front running with some variations. There is a small amount that is
necessary arbitration that will have to be done at some speed, but by and
large it is not performing any sort of a service, it is taking advantage of
the people trying to buy and trade stock by being a manipulative middle man.

~~~
tptacek
There are three sentences in this comment. The first two are wrong, and the
third is hard to understand, but if I understand it correctly, it too is
false.

1\. _Flash Boys_ has been panned by technical experts and makes very little
sense to people with a basic understanding of how the markets work. I can link
to technical reviews if you'd like to see them. I'd be interested in _any_
technical reviews of the book that defend it; I haven't seen any. As a fan of
Michael Lewis: this book was surprisingly bad.

2\. Front-running is an agency problem. It's what happens when your broker
knows you're about to move a lot of stock and, acting on that insider
knowledge, jumps in front of your trade. Market-making isn't front running. To
make a colorable argument that HFTs are frunt-runners, describe a hypothetical
sequence of trades in which that happens ("BOB BUY 10 @ $10, CAROL SELL 10 @
$10.01, &c"). If HN history is any guide, that exercise will quickly deflate
the notion that there's front-running (or even any unfairness) happening.

3\. HFTs were preceded in the markets by the human specialist system, in which
people deliberately increased the spreads to scalp money from trades. HFTs
fight over pennies. Human specialists rigged the stock markets so that every
share included a premium denominated in dimes and quarters paid directly to
the specialists.

~~~
CyberDildonics
Nothing in this comment contradicts or disproves that front running can and is
happening. Flash Boys described a large order going to an exchange and HFT
firms seeing that, buying stocks on other exchanges before the real purchaser,
and selling them back to the real purchaser after the first exchange they get
to can't fill their large order. HFT firms know there will be buy orders at
the other exchanges and they run out in front of those buy orders to become a
middle man. That is the very definition of front running and it was obviously
happening. Do you have information that directly contradicts this?

~~~
tptacek
First, that's not the definition of front running, and second, the expectation
that a large order can trade atomically has never been valid, and, if you
consider the computer science behind it, _can 't_ be valid.

~~~
CyberDildonics
If that isn't front running, then what is? And if you wouldn't call that front
running, what would you call it? You haven't offered anything to back up any
of what you've said.

All I'm saying is that sub-millisecond HFT is only useful to the people doing
it and no one else. People defending HFT as being necessary and helpful is
ridiculous. Even an exchange where orders go in over the course of a second,
are matched up within the exchange, then traded, would not offer latency to a
person, but would offer everyone a chance to sync with the exchange cycle and
submit their orders with enough time to be matched. If all the exchanges
someone was trying to use worked this way it would offer no disadvantage to a
human trader, but would make the HFT arbitrage much less lucrative.

~~~
tptacek
Like I said upthread: front-running happens when a broker takes an order for a
client and, before submitting it to the market, trades against that order.
Front-running is an agency problem.

