I think we'd have been better off if the title of this thread had been "Online Algorithms in Automated Trading Systems". As far as I understand, that title has exactly the same meaning, but doesn't set off a crowd of HN commenters who have never placed a limit order jumping in to crow about _Flash Boys_.
I'm a little frustrated, since this is a technical topic that happens to be very relevant to me right now (I'm not a trader, FWIW).
1. slides: http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...
2. paper: http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...
You can make time discrete for a single auction, but the time between auctions still has to be continuous unless everything's traded in a single exchange, which can't in reality happen even if we want it to (and we probably don't).
As I understand it, most of the most dramatic HFT stuff is already targeting multi-venue trades or correlations.
(I'm not smart enough to know the answer, which is why I'm asking).
And to everyone who feel the need to voice their disapproval of HFT: if this was an article about improving the penetration of M1 Abrams, are you gonna be voicing your disapproval about ... something? Why/ why not?
But more to the point, is there anyone in these fields who doesn't know this stuff? At a basic level, the things that you can do a really good job estimating in an online way are closely related to sufficient statistics, which every undergrad stats course should cover. And hopefully anyone actually involved in writing HFT systems has way more stats background than that -- right?
I've been playing with simulations of market maker algorithms from the literature, and simple Bayesian stuff and MLEs are all I've really seen, which makes me think that the interesting stuff isn't in the literature.
One keyword to look for is 'stochastic approximation', with a long (> 50 years) history in engineering and stats. Another one is 'stochastic gradient', which got a resurgence following the whole deep learning craze. There is also 'particle filtering' in signal processing.
One generic article I may recommend is http://leon.bottou.org/publications/pdf/online-1998.pdf, it does not require much math knowledge, but that may be less practical than what you want.
The 'onepass' keyword mentioned in the original article is another keyword to look for (see e.g. interesting work to do Monte-Carlo based linear algebra operations for very large matrices, this used to be a reference, not sure what's the state of the art is: http://cs-www.cs.yale.edu/homes/mmahoney/pubs/matrix1_SICOMP...)
I agree. In engineering, many of these algorithms can be coded using digital filters. Here's the exponential weighted variance:
import scipy.signal as dsp
import numpy as np
b = np.array([alpha])
a = np.array([1., -(1 - alpha)])
# Compute exponential weighted average
signal_ = dsp.lfilter(b, a, signal)
# Compute exponential weighted variance
ewv = dsp.lfilter(b, a, (signal - signal_)**2)
It goes a bit further into the mechanics of market maker profitability. Does your simulator reflect what your queue position would be in a real electronic market? Does your market making algorithm have logic try to scratch out when it looks like the market is going to move against its position?
b) Given the reactionary view that many HFT participants have to information sharing, widely known/documented algorithms are often thought of as secret sauce and are frequently not talked about.
c) I agree with your sentiment specifically to one-pass algorithms being not terribly novel but am very happy that people with HFT backgrounds are publishing anything at all as it will hopefully help to remove the backwards thinking around secrecy so prevalent in the industry.
regarding b), the details of how exactly to use the algorithms are the secret sauce and those details matter.
I have no particular position on the elite status of those working in HFT. Certainly there are some smart people there, but I don't know how to generalize. I don't have anything like a representative sample of either HFT or programming more broadly to compare, despite having worked in HFT and out of it.
Links are also welcome.
With HFT, there's a lot more competition over the bid ask spread. Trading firms are essentially in an increasingly expensive fight over pennies, which is good for everybody else, since it's cheaper prices, more liquidity, and a more efficient market.
The people it's bad for is the old-school style brokers (which are effectively gone now) and large investmant banks who have a fundamental view on the market (they need to be smarter about executing since the market is more efficient and reacts much more quickly to their buying and selling).
What about index funds rebalancing their portfolios? (Honest question)
Index funds don't have a view on specific securities. They don't perform research to try to get an edge. Instead, they believe that the market is generally efficient, the and they try to lower risk by diversifying across a large portfolio of somewhat related securities, and get a return that closely matches the overall market. As such, HFT helps them by givings them better bid-ask spreads and also providing prices that are closer to the actual value of an asset.
Note that those performing fundamental analysis and buying millions of dollars worth of assets aren't SOL. They just have to be smarter about how they execute.
Unless you were a specialist market maker or pit trader on an exchange there hasn't been any "wealth extraction". Quite the opposite, HFT has taken a system built on being tall, having good connections, and understanding an arcane set of hand signals (not to mention very high fees) and replaced it with a ruthlessly efficient automated system based on being good at writing software and planning network setups (and brought the fees down with it).
For the vast majority of market participants HFT has been a massive upgrade over the old system and dramatically lowered the cost of trading.
>And are these benefits worth the massive amount of wealth extraction
What "massive amount of wealth extraction"? Are you just making shit up?
I think he was probably just confused about how HFT works, since unless you're familiar with it it is very easy to assume it is screwing someone.
The thing that most people seem to be upset about is that now market makers generally have few to no obligations to provide liquidity and they are drastically smarter than they used to be.
Ultimately, unless you are slinging enormous blocks, if you send a marketable order through a retail broker these days you are almost always going to get a better fill than the displayed market. Likewise, if you are a large trader you have to be more sophisticated about how you fill your orders, you can't just lift 100k shares even when 300k are displayed unless you do so with a semi-intelligent execution algo (note: these are everywhere these days, and not using one brings into question the trader/brokers abilities and whether they are even qualified to be in the market).
I just got an email that the expense ratio of my Vanguard target date fund is going down, so costs were reduced somewhere.
I mean, if a supermarket sells strawberries for cheaper than your local greengrocer, is that "a non-negligible amount of money for a service with questionable value"? It seems to me that the onus there isn't on the supermarket to show that strawberries are worth the price.
Because automated market making is intensely competitive, the vast majority of that "wealth" was transferred... wait for it... to retail and value investors.
Did you know, in the past it was NOT POSSIBLE for you to post a limit order and have it displayed in the order book unless you were an anointed market maker on an exchange?
[edit: this was meant as a reply to gdl not tptacek]
People choose to cross the spread and take liquidity. Ask yourself why they might do this (hint: google "execution risk").
Of course if you are time insensitive, you can certainly play games. This basically means running your own HFT strategy, but biased in the direction of the trade you want. You'll also opportunistically take liquidity if the fill you want is in the book.
If you want to read narrative non-fiction about HFT, "Dark Pools" by Patterson, which also has its mistakes, is much better.
For technical books:
- Trading & Exchanges by Harris (a little out of date)
- Algorithmic Trading & DMA by Johnson.
On the narrative non-fiction side Dark Pools is basically it.
The second link specifically talks about social utility.
The (quite common) liquidity argument isn't a very good one IMHO, given that HFT have no interest in holding positions for any long time, so they won't buy what they're not reasonably sure they can sell quickly (ie. where there already is liquidity).
The other argument here is that before HFT, traders wanted even larger fees. HFT can't be the cheapest possible option out there, if they can afford to lay their own fiber between continents to operate.
So what is it that makes HFT desirable enough that we (as a society) shouldn't find means to make trading happen without them?
The great thing about this is, automated traders don't capture the value they're creating by bidding the spread down: the buy-side does! Automated traders are getting paid to liberate money that used to go to fees for sell-side middle-men.
The sell-side is, as you can read subtextually from _Flash Boys_, pretty furious about this.
Is it their own (really public) instance of Core War (https://en.wikipedia.org/wiki/Core_War)? Is it the milliseconds advantage gained through custom transatlantic fiber?
If either of these aren't core to what enables them to dump prices, we could get them down even more by getting rid of these toys.
Lower costs by having computer programs rather than human traders. Two or three programmers can do the trading that would previously have taken a hundred traders, and there's a corresponding saving in salary/bonuses/etc.
> If either of these aren't core to what enables them to dump prices, we could get them down even more by getting rid of these toys.
How would you do that? Having algorithms is core to it. Having custom fiber is only core to beating the other HFTs (they can't compete on price so they compete on latency), but what are you going to do, pass a law against installing optical fibres?
Prices are already as low as they're legally allowed to be (sub-penny rule). Which is what causes a lot of the weird millisecond-chasing.
The needs of HFT are such that technology barriers are pushed to the limits. Some of the technologies developed in the process make it back to the general public.
Where is the commitee that decides on the allowed actions?
Luckily that is not allowed (nor technically possible) on any credible exchange.
> An earlier HN post that I can't find had a discussion on how this type of behavior can be countered by sending fake buy offers without intent to actually buy, but for whatever reason, that is illegal.
You probably meant: https://news.ycombinator.com/item?id=8977198
Spoofing, which is what you are describing, is not useful for countering HFT, it is used to change the market price of an instrument by exaggerating the demand for it. This has obvious detrimental impact on any one, HFT or otherwise, who is trying to price the instrument.
This is again how scientificism goes. We think machines improve everything. "If only we trade faster, that will help the markets!" It's not hard to see this is BS.
There is no possibility to "back away when a matching order comes in" because the first time you find out about the matching order is when you are informed about the trade that you just did!
What can happen is that a market maker will post liquidity on multiple exchanges A, B and C, and will remove their orders from exchanges B and C if they are filled on exchange A (assuming they are fast enough to do it!)
This is not shady or nefarious. In fact, it is the only way that a distributed market could possibly work.
The market maker has some finite capacity to provide liquidity (e.g. they have position limits, they need to post margin, they have certain risk tolerances). They want to maximise the chances of their orders being traded on (i.e. providing liquidity), so they post liquidity on all available markets.
When they match on one of those markets, their capacity to provide liquidity has diminished, until they manage to offset their position somewhere (either on the same market, or on a different one). Moreover, they have just received a new piece of information about the supply/demand imbalance for this particular stock. For both of these reasons, it is natural to assume that they would want to adjust the prices they are offering on other exchanges. The market cannot function any other way.
1. _Flash Boys_ has been panned by technical experts and makes very little sense to people with a basic understanding of how the markets work. I can link to technical reviews if you'd like to see them. I'd be interested in any technical reviews of the book that defend it; I haven't seen any. As a fan of Michael Lewis: this book was surprisingly bad.
2. Front-running is an agency problem. It's what happens when your broker knows you're about to move a lot of stock and, acting on that insider knowledge, jumps in front of your trade. Market-making isn't front running. To make a colorable argument that HFTs are frunt-runners, describe a hypothetical sequence of trades in which that happens ("BOB BUY 10 @ $10, CAROL SELL 10 @ $10.01, &c"). If HN history is any guide, that exercise will quickly deflate the notion that there's front-running (or even any unfairness) happening.
3. HFTs were preceded in the markets by the human specialist system, in which people deliberately increased the spreads to scalp money from trades. HFTs fight over pennies. Human specialists rigged the stock markets so that every share included a premium denominated in dimes and quarters paid directly to the specialists.
Again: a single positive technical review would be interesting. I'm not being hyperbolic: I haven't read a single one. The sense I get is that everyone who's ever formatted a FIX message thinks _Flash Boys_ is terrible.
All I'm saying is that sub-millisecond HFT is only useful to the people doing it and no one else. People defending HFT as being necessary and helpful is ridiculous. Even an exchange where orders go in over the course of a second, are matched up within the exchange, then traded, would not offer latency to a person, but would offer everyone a chance to sync with the exchange cycle and submit their orders with enough time to be matched. If all the exchanges someone was trying to use worked this way it would offer no disadvantage to a human trader, but would make the HFT arbitrage much less lucrative.
HFT ate the lunch of the people who benefited most from the old system: giant broker-dealers (investment banks) who made grotesque fees for the simple service of shopping large blocks on behalf of pension and mutual funds. Those funds now get better service without special fee-collecting agents.
_Flash Boys_ was not just panned by HFT people. Where are you sourcing that objection from?
Also, why do you keep putting underscores around Flash Boys?