On: Using Reinforcement Learning in the Algorithmic Trading Problem

[traderthrow454]

If you want to read useful academic papers about trading there is one author in particular who is actually not bad - Zura Kakushadze. Most of his stuff is applicable to mid-frequency trading, not HFT. He worked at WorldQuant (reputable trading firm) and the founder of WQ, Igor Tulchinsky, is a coauthor on one of his papers.

Example of a pretty interesting and accessible one - is "101 Formulaic Alphas" [0].

[0] - https://arxiv.org/pdf/1601.00991.pdf

[young_rutabaga → traderthrow454]

This paper is a hilarious dump of WQ's randomly generated formulas that (hopefully) happen to pass in-sample test.

  Alpha#33: rank((-1 * ((1 - (open / close))^1)))

This formula trivially reduces to

  rank(open/close - 1)

which is an example of a mean-reversion strategy. But: 1) nobody bothered to simplify this formula, 2) as any mean reversion, it is extremely difficult to trade.

[astrofinch → young_rutabaga]

Why is mean reversion difficult to trade?

[young_rutabaga → astrofinch]

High turnover, high costs. You flip your position too often

[stainforth → young_rutabaga]

If the market flips against me, I just double my bet on the next play.

[dkersten → stainforth]

the market can stay irrational longer than you can stay solvent

Martingale is a sure-fire way to lose all your money rather quickly.

[vecter → stainforth]

That's a great way to go broke very quickly.

[anjc → vecter]

Only if you don't have infinite money on the side, in which case it's guaranteed to be profitable

[WJW → anjc]

You also need the other side of your trades to have infinite money. But in any case, why are you trading at all if you already have infinite money?

[rsecora → anjc]

Martingala with transaction costs are not profitable.

[huac → traderthrow454]

picking a random one out of the pile:

> Alpha#90: ((rank((close - ts_max(close, 4.66719)))^Ts_Rank(correlation(IndNeutralize(adv40, IndClass.subindustry), low, 5.38375), 3.21856)) * -1)

I wonder how these magic numbers get picked (4.66719, 5.38375, etc) -- I guess there is some optimization solver which attempts to find the most profitable variables for a given alpha formulation, but isn't this approach also very vulnerable to overfit?

[traK6Dcm → huac]

Yup, it's probably just the output of an optimizer and then tested on held-out future data. Not overfitting is the key here and what's really hard. You need to be careful about the number of parameters and the amount of validation data you have.

These alphas will likely be only profitable for a short time period as long as the market data distribution (i.e. strategies of other market participants) doesn't change. So you would need to continually optimize and update them.

The way I think about it is that you are essentially finding the right parameters to "exploit" the combination of algorithms of all other participants, where algorithm could also be a human looking at charts and following certain rules, with a lot of random noise from retail traders thrown in.

[laxatives → huac]

Seems kind of rudimentary. Namely

> (sign(delta(volume, 1)) * (-1 * delta(close, 1)))

That's crazy. Would be interesting to see WTF a "mega-alpha" actually does using these strategies.

[smdz → huac]

I believe they may have used something on the lines of genetic-programming to create this equation - not sure about the high precision constants. The search space is compute intensive. Many years back, I used that technique to generate a profitable strategy. These things work and are different depending on the timeframe/sampling, stock, trend and money management.