After writing that I joined Headlands, where I've been working for the past 8 years as we grew from a startup to a relatively large firm (trading is actually a very small industry). If you want a more up to date description of low-latency trading in particular, please see this more recent post:
Feel free to email me, my email address in the pdf is still valid.
Indeed, "work shall set you free" is not a great reference to include.
It doesn't for people who haven't grown up with Looney Tunes. You're on a fairly international site here. Similarly, you can't expect everyone's education about the Holocaust to have covered this specific thing in a way they'd reliably remember and avoid an embarrassing mistake like that.
Now, if the quote had been "Arbeit macht frei" I would understand the outrage, but
to me the English translation sounds more like a reflection of the Calvinist/Protestant work ethic from the time when the saying was minted, at least 100 years before the Nazis rose to power. To some people the quote in itself holds appeal, and why shouldn't it? If read without context it is actually quite inspiring.
Is there any authoritative literature on the execution aspect of algotrading? I'm talking specifically about the things that are hard, if not possible, to analyze via backtesting. As I've moved to shorter and shorter timeframes, I feel like there's a huge execution gap that I'm suffering from. I've got good strategies that work but can definitely be improved with better execution. For example:
How to optimize limit pricing to get better fill rates or better entries or exits
Market microstructure analysis and modeling. How to analyze book depth and order flow to predict short term (shorter than 5-10s) behavior
Applications for and optimizations of various order types like IOC, MIT, LIT, FOK, Iceberg, Basket, etc.
How to handle the million ways that algotrading can do unexpected things: market gaps, trading halts, network outages or latency spikes, news events and volume explosions, etc.
Metastrategies: creating strategies that "trade" strategies to turn them off or swap them out based on market conditions.
Statistical process control, or other ways to detect when an algorithm is no longer working as expected.
Use of metainformation to dynamically inform strategies, such as implied volatility or any of the option greeks.
BTW- not sure why they wanted to patent this, since that just makes it all public information.
My biggest takeaway was just confirmation that really "many people are doing the same things" or looks at the market in very similar ways.
One good way to get an edge is to think how would someone in a similar position approach the problem. Now iterate again and think of it differently.
Its the game theory of trading.
I am also closely following the MQL5 market studying who the retail thinks. I see how the waves of people gather one strategy/signal/EA that seems to be having a good streak, and then jump to the next one once the previous one 'just' failed. There is great ignorance in the space, it feel like 80% of the people I talk to in retail haven't even read a book and/or never heard of Kathy Lien or monitor the news on BB or Reuters.
I will definitely absorb those 57 pages and will stary monitoring HT blog.
I’d venture to say that trading might just be the most competitive field in existence, with all the smart and cunning that goes with it.
However, honest advise: if you actually really want to trade in this way, start with a solid course in mathematical finance. It’ll teach you how to model the space, think about arbitrage, portfolios, risk, factors and the sheer important of theory and hypothesis in real strategy.
Building a backtester is exceptionally hard to do if you include scenario analysis, slippage, event risk, lag, multi hypothesis testing, data snooping, etc. Testing is a huge source of competitive advantage.
In summary, if you’re starting out, go learn the basics like the back of your hand, and he basics are mathematical finance.
One of the most successful hedge funds, Renaissance Technologies, also takes this approach. Regarding its employees: "a third have PhDs, not in finance, but in fields like physics, mathematics and statistics. Renaissance has been called “the best physics and mathematics department in the world” and, according to Weatherall, “avoids hiring anyone with even the slightest whiff of Wall Street bona fides." (https://en.wikipedia.org/wiki/Renaissance_Technologies). Similarly some of the best HFT hires I've been have been people with a maths or science background.
A quote sourced to Renaissance Technologies from https://www.quora.com/What-is-the-secret-behind-Renaissance-...: "We have some Tier One mathematicians and a lot of Tier Two mathematicians. Other quants funds have mostly Tier Three mathematicians, and worse, they don’t even know that these tiers exist! Someone has to create the correlations in the markets." Aim to be one of those Tier one mathematicians!
Why not go even further and just not bother with algotrading, dumping money into index funds like VTI instead? It certainly gives a lot more value for your time.
While we're here, though, I'm genuinely curious: how accessible is HFT to the average retail investor, let alone those who aren't investing at all? Lastly, can HFT consistently beat a total stock market index like SPTMI year-over-year?
As for beating indexes, it goes back to the age-old question of being able to identify the great automated trading systems.
As for the funds you mentioned, are there any that have a minimum investment of less than $1,000?
Investing directly in a hedge fund requires a much higher initial investment as well as a high net worth. I would not recommend it. It is very difficult to select a portfolio of hedge funds that can outperform a simple low-cost index fund tracking the S&P 500.
A nice post on the topic by someone who most certainly is a Tier One mathematician: https://terrytao.wordpress.com/career-advice/does-one-have-t....
Then the applied part of it is simple but different: different analysis, different theory, different data, extraordinary results.
Edit: just read your article ... kind of poo poos my theory if what he says is true ; at least at the level of the mega model.
But HFT is inherently low capacity. You can't put a billion dollars on an order book and expect to make the same returns as you would with a thousand. That's the reason HFT firms are almost always proprietary trading firms...they don't need or want more capital.
A hedge fund, like Rentech, is typically on much higher timeframes because their size necessitates higher capacity strategies. This could mean holding periods of minutes for the million dollar funds to days for the billion dollar funds. As you get to higher and higher timeframes, your mathematics are going to need to be dramatically more sophisticated in order to beat the market. The math you are looking at in that quora link is about pairs trading which is where pairs of instruments mean revert over time. I would expect Rentech to be doing this type of trading over long timeframes and holding a trade for days to weeks.
There are some quality uses for NN, but price or volatility prediction is not one of them.
From there, “learn as much relevant applied maths as possible” would be more time efficient.
And from there, “learn financial maths first” proceeds as an obvious corollary.
So I kinda feel we agree :)
In all seriousness, though, it's crazy that we're at a point where these hyperactive algorithms can perform price discovery fairly decently.
As an initial focus, I suggest understanding discrete time models well. If you've got some LA and calculus then I'd suggest just hoping into Introduction to Mathematical Finance by Pliska. It's concise, zero filler, but really clear.
My all time favorite article related to this topic, I always try to promote it Guys like the author started this whole rodeo through some really ingenious hardware/software/networking jerryrigging.
HFT, at worst, adds exactly zero value to society. I don't buy into the memes of "we provide liquidity!" or "we make markets efficient!" being some noble mission. But HFT certainly does no harm
Regular tech ultimately just pushes ads or products on people, which I consider poisonous to society
Or you don't worry about contributing to the world.
Or you enjoy intellectual entertainment above what you might add to society.
For some reason this type of comment comes up quite often on algo trading threads. Keep in mind most people are not deciding on their careers based on what they contribute to the world.
At least ideally, successful trading makes markets more efficient, and making markets more efficient ripples out making it easier for many people to make many products that create value for their users.
Even if all of that is true though, it's an unnatural framing for people to work with and is much harder to find existential meaning in abstract changes to the system that make people's lives better in complex and hard to trace ways than it is to find meaning in a direct value proposition to your users.
For an extreme version, look up Effective Altrusim. Some argue that the best way you can help others in this world, is to earn as much as possible (for example at a trading desk) and then give all your surplus earnings to the most charitable causes.
Personally, I would like to earn money so that I can work on projects that I think are worth doing and have a chance of using my particular skills and drives to good effect.
For example, I feel ready to take open hardware to a new level by building a small-scale chip (IC) factory capable of manufacturing open source designs (or any designs) at a realistic cost for customisation. And that's just one step in a series of related things, each of which will contribute something to the world if they are successful. That's just an example: Other people have their own grand projects they think are worthwhile.
But that venture is both challenging and expensive, and I'm still paying off debts from the last failed bootstrap. So my next big project is paused, until I have earned enough to be able to bootstrap all over again.
Something like HFT would be rather useful at providing the money, and its on my list of things to explore in the short term, because it's also a good match for my experience in several areas. But it's not something I'd want to do for a long time for its own sake. It would be for the money, so that I can do other useful things down the line.
The role of capital markets is (among others) to integrate all available information from the real world into a price. A trader does this (for example) by studying company financials, their corporate strategy, expectations for the performance of their products and then buying or selling stock if the current price does not reflect this information. In a capitalist economy capital market serve as the major signal on the performance (another word for efficiency) of a company.
I think it's hard to show, but coding is of paramount importance. General skills like keeping things in version control, writing simple code, and making things modular. But also quite advanced specific topics like CRTP or cache optimization. Could be a whole book in itself without the financial parts.
If you want to be the guy who actually knows how to program (the guy translating the Python into C++), then I think a backtester would be a nice project. Look at the python libraries zipline and pyfolio and then try to create something yourself (ideally not in python).
Sorry for the dumb questions- I have genuinely spent significant time Googling and could never find the answers, thought I'd use the opportunity to ask here. I am obviously not a market professional, just one guy with most of his assets in index funds. I can rephrase questions if necessary
Investing in equity markets is very different from trading. Corporations pay you dividends and capital gains in return for the capital you provide them. That's why long term investments in passive index funds are typically the best bet for someone who is not a finance professional. Rather than trying to beat the professionals at pricing assets, you get to sit back and enjoy the compensation from a mix of companies working hard to increase the value of their stock and/or generate dividends. You just need to ensure you don't take on excessive risk. Stay away from margin at all costs.
Having said that, in addition to the other recommendations in this thread, lookup SEC settlements. The Athena trading one from a few years back regarding their "Gravy" on close strategy is particularly memorable.
Edit: https://www.sec.gov/news/press-release/2014-229 SEC order at the bottom of the press release
A successful trader won't give out the thing that if replicated by 10.000 people will attract attention that may deem this inoperable any more.
Part of my 'hobby' is to find EAs and put them to test in other pairs, timeframes, and try to optimise and get better configurations for the triad EA-pair-timeframe. When I get one, it goes straight to MY 'parking lot', NOT a blog :)
I'll also add that the commercial market for them is about as scummy as it comes. You want to know how to game backtest metrics? Have a look at what they're doing with EAs.
This XKCD is always a good reminder not to get to enthusiastic:
Numbers is just one aspect, then we got 'the News' (and the specualator, the other EAs that are programmed to kill 'your' EAs), and then we got politicians twitting (I won't name names), and semi-democratic governments that have other agendas unrelated to financial prosperity, and so on, and so forth.
Basically unless one is running algos with a target of more than 20% per annum, they should be losing sleep.
I was taken aback by the 'work shall set you free' quote at the top of Section 2. That seemed in very poor taste.
I could see how that specific saying is tainted in German but I do not think it should be tainted for all of history and in all translations because of some unfortunate coincidence. I believe it has wisdom to share
It’s not like it was completely disassociated with Nazi history and then they decided to put it on their camps because it sounded cool. They literally wanted to work people to death, i.e. freedom.
>When you laugh, the world laughs with you. When you cry, you cry alone.
It was in the film Oldboy but it's actually from an English-language poem. Ella Wheeler's Solitude:
Laugh, and the world laughs with you;
Weep, and you weep alone.
For the sad old earth must borrow its mirth
But has trouble enough of its own
What are the salaries now-days for a BackOffice Technology (C++ and everything else related)?
> The phrase “algorithmic trading” has a different meaning depending on if you are in Chicago or New York. In Chicago it will mean using a computer to place trades to try to make a profit. In New York it means to use a computer to work client orders to try to minimize impact.
Max was a fresh out of college graduate at the time, so his usage was a bit off. Knowing this distinction is a sign of actual experience on the street.
However, loads of people (and, according to Max’ document, Chicago) refer to automated trading as “algo trading”. Cue the descriptivist vs prescriptivist linguist debate.
"..is a type of trading done with the use of mathematical formulas run by powerful computers.."
"..makes use of much more complex formulas, combined with mathematical models and human oversight, to make decisions to buy or sell financial securities on an exchange.."
Ps: In simple english, I go to work, that thing makes profit (if done right). I go to sleep, that thing makes profit. Someone important twits, that thing either makes profit, OR goes bust.
It’s not uncommon to take an HFT trade that is competitive in the US or German markets and go around applying it to markets like Tokyo, London, Singapore, Etc. But the converse is not usually as successful.
So if you start in Tokyo, it might be harder to move up and out just due to more restrictive regulatory frameworks and less efficient markets there.
For the large organizations, they got far more complicate systems/servers/software, but these practically do the same thing, just on a massive scale.
The point of all this is that algorithmic trading in the industry sense of the term is a product that banks and brokers can sell to buy side funds. "You make the decisions, and we'll handle the execution." If a large org is using algorithmic trading, there's a good chance they're paying someone to do it for them.
Let’s take HFT for example. HFT firms are the reason that Mom and Pop investors can get efficient execution on the public markets. Compared to the inefficient markets that existed before, they have generated many many billions of dollars to pension funds, mutual funds, and individual investors.
I don’t think it’s possible to overestimate the positive effect efficient markets have had on everyone’s lives, from the richest to the poorest.
The world isn’t a zero-sum game, and markets aren’t either, everyone profits from a good free market.
To be candid, I'm fully on the same page as you: what's going on here is no different than what a really good pit trader did from 1800's -> 2000. It's just the digital, semi-invisible version, and because volumes are so large, the money is exponentially larger as well.
But, that's not all the HFTs get up to; a lot of it doesn't hit public markets, or isn't particularly helping w/ efficiency.
In regards to money, I think that decimalization actually really hurt market makers. There has been massive consolidation in the HFT space with most players either going out of business or being acquired. Profits are way down and latency arbitrage is almost impossible nowadays without billions of dollars of trading infrastructure. Profits have dropped by something like 90% in the last 10 years and the super-normal profits now (after consolidation) are just "normal."
The good thing about HFT is that with the right infrastructure, it's (almost) risk-free. The problem with HFT is that the returns don't compound, there's only a fixed amount of pennies that can be vacuumed up. This is in contrast with other non-HFT quant strategies, which can run tens or even sometimes hundreds of billions of dollars in capital and can make over 10% annually.
The story with HFT is the story with any new market. Players get in, make a ton of money. More players get in, now they are making a little less money. Even more get in, and now they aren't making enough money. Then the people not making enough get bought up or go out of business. And now the equilibrium has been reached: everyone is making "normal" money and everybody forgets about them.
And all of this is good, it's the natural market cycle. People like Elizabeth Warren were talking about how unfair HFT was blah blah, but now, no one cares. Fortunately, the entire cycle resolved before the government could get their sticky little fingers on it.
Just to be clear, I mostly think that's fine and whatever, who cares, but there has been a lot of deception in the space.
I could respond by explaining that HFT facilitates liquidity and price discovery, making trading cheaper overall. I could also counter that adtech isn't any better than fintech, or that it's a fallacy to suppose every intelligent person needs to maximize their ethical production in a capitalist society.
But I'm not going to engage in any of those rebuttals. Instead, why don't you take a look at the massive charitable work bankrolled and empowered by James Simons and David Shaw? Take a moment to read through the organizational mandates and scope of research for the Simons Foundation and DE Shaw Research (DESRES). These two men have poured billions of dollars from their personal wealth into making the world a better place in a variety of ways. Both of them explicitly cite cancer research as a core focus - what do you consider more ethical than that?
While we're at it, let's circle back to your specific example. You're talking about sending rockets into space - that's great! I bet you're a fan of Elon Musk and SpaceX then? Are you aware that Elon Musk made his first fortune through PayPal? Do you consider payment processing to be a virtuous industry? Was Musk maximizing his ethical potential by working on PayPal?
People (both on HN and at large) seem to believe that HFT is somehow unfair, probably because of Flash Boys or some uninformed NYT article.
But back to the point, unless you believe in communism, then the best way to help the world and add value is to make money. Simply put, if you are making money, you are doing good (there are exceptions, of course and they usually involve the government). It is folly to measure social contribution in any other way (charity isn't doing good, it means whoever you are giving it to is doing good).
People, especially with HFT, think the world and the markets are a zero-sum game. I think the problem people have with HFT is that it doesn't seem "fair." Mom and Pop don't have a co-located server with ASICs churning out latency arb trades so they shouldn't be able to do that either! But why would you expect Mom and Pop to be able to compete in the first place? They probably wouldn't be able to compete with dentists, race car drivers, or scientists, so why should they be able to compete with professional investors?