
Most trading strategies are not tested rigorously enough - ilamont
http://www.economist.com/news/finance-and-economics/21644202-most-trading-strategies-are-not-tested-rigorously-enough-false-hope?fsrc=scn/tw/te/pe/ed/falsehope
======
sz4kerto
"Most trading strategies are not tested rigorously enough"

After having spent many-many years in the financial sector, I don't even know
whether I should laugh or cry. :) The industry is not based on science, well,
99% of it isn't. Traders can be considered being the master of the universe
just because pure luck. Well-researched, tested strategies are thrown out
because they're not profitable enough to the senior management. If a model
seems to bring in high profits then even the very makers of the model do not
want to let it into production, the management will just use it because nobody
cares of tail risk. And so on.

Great industry. :)

~~~
IndianAstronaut
My experience across industries is that people don't know basic statistics or
the value of statistics. I even came across a manager of a data science team
at a major company who did not know anything about statistical testing.

~~~
superuser2
... what

Data science is a buzzword substitution for statistics.

How you could possibly have someone in a "data science" role without a
statistics education is baffling.

~~~
stdbrouw
It's really not. Statisticians usually receive only minimal training in
statistical programming, data cleaning, data gathering and warehousing and so
on – yet those are all essential to data science. On top of that, when dealing
with bigger data sets, many common statistical tests become irrelevant, as
their main purpose is to make it possible to reason about small sample sizes.

Of course, I don't mean to imply that a data scientist shouldn't have at least
a basic statistical grounding: they do. But there's roles in data science for
people with varying skills in programming, ops, statistics, ML, visualization
and so on.

------
wpietri
And I'd say that this situation is an advance. 20 years ago when I was a naive
techie working in trading, most of the trading strategies I saw were tested by
some young, high-testosterone guy wading into a pit, shouting a lot at people,
and feeling vindicated if he made some money. (Or, in a pinch, feeling
vindicated by whatever positive signal he received before getting creamed.)

For those interested in this, I strongly recommend Nassim Nicholas Taleb's
"Fooled by Randomness", which gives the reader a feel for how much supposedly
mathematical and rational finance runs on intuition, survivorship bias, and
plain bullshit.

------
icu
A lot of the comments here seem to be from people with institutional
experience or wannabe retail traders. As a retail trader (who used to phone my
broker to make trades) that has progressed to an algorithmic retail trader (I
do all my own development with some mentoring help from professionals) I can
say that it is possible to make a good return on risk.

This did not happen overnight. I've spent thousands on my education and by
that I mean I've been scammed, gone to useless seminars, read nearly 100 books
on the subject and made terrible trading errors and trading losses.

I only started getting serious traction after a confluence of events that led
to being tutored by an ex-JP Morgan quant and a software developer friend who
has been developing trading software for the big Bank trading desks in London.

Moral of the story? Persistence.

How does this relate to the article? Persistance eventually overcomes a lack
of testing to eventually lead to a robust testing methodology.

So yeah I do agree that most (retail) trading strategies are not tested
rigorously but that's due to the difficulties around acquisition of inter-
disciplinary knowledge and the balls to get real experience my putting money
on the line.

~~~
infinite8s
I guess you wouldn't be interested in going into more detail?

------
consz
I don't know anybody that uses anything as weak as 2-sigma results in HFT, at
least. Most of our valid signals have 20-50+ sigma.

~~~
jeffreyrogers
Since you seem pretty well acquainted with HFT, I'm curious about how much of
a constraint capacity is? From what I understand the amount earned per trade
is very small (this paper[1] suggests $1.45 per $10,000 traded). And since HFT
is already a large fraction of the daily volume it seems that the natural way
to increase profit (i.e. just trade more) isn't an option in most cases.

[1]: [http://faculty.haas.berkeley.edu/hender/hft-
pd.pdf](http://faculty.haas.berkeley.edu/hender/hft-pd.pdf)

~~~
consz
$1.45 per 10k traded seems really high. A good S&P futures strategy (one of
the biggest products in the world) is typically going to make 60-80 cents per
contract traded. Since each contract is approximately 100k, thats closer to
.06-.08 cents per $10k traded? Even for equities (I don't touch US equities,
so I'm not entirely sure how good the best strategies perform), making a full
price tick per contract is still < $1earned/$10k traded... and good HFT
strategies are more on the order of 5-10% of a tick per contract traded.

In any case, capacity is probably the first constraint you hit. Most firms
have reasonably accurate simulations, so most HFT strategies are scaled up to
as large as they can possibly trade (ie. quote the largest amount passively or
aggress with the largest amount you possibly can) within a few days of being
released -- once you can confirm that your live trading is at least mostly
matching simulation, you usually try to simulate the maximum possible size it
can trade and just start live trading that. Since you're typically scalping a
tick at a time, your maximum size is typically some fraction of the zero level
bid/offer -- relatively small. Typically the way you scale up is either have
better execution (know when to size up/size down appropriately) or better
prediction quality -- since you're adversely selected, your bad trades get
filled at a much higher percentage than your good trades, so as your have
better prediction quality, a smaller percentage of your volume is bad and you
can start to fire larger and larger.

~~~
jeffreyrogers
Thanks, that's really interesting. I hadn't even thought of the adverse
selection problem. And you're right the $1.45 was high. I looked at the paper
again and that number was before adding in costs. With trading costs the
number was much lower, but still profitable (can't remember the exact number
off the top of my head), and that paper was just US equities.

------
rsiqueira
From Wikipedia: "Some people claim that by recognizing chart patterns they are
able to predict future stock prices and profit by this prediction; other
people respond by quoting 'past performance is no guarantee of future results'
and argue that chart patterns are merely illusions created by people's
subconscious."

~~~
balazsdavid987
In my experience, the patterns are not illusions, they are very much real, but
they don't have predicative power.

~~~
fma
[http://www.bloomberg.com/news/articles/2015-02-20/high-
frequ...](http://www.bloomberg.com/news/articles/2015-02-20/high-frequency-
trader-virtu-extends-nearly-unblemished-streak)

Predictive enough that when combined with money management strategies you can
make money every single day of the year.

~~~
throwaway183839
What Virtu does to make money (market making and latency arbitrage on a milli-
and microsecond timescale) is very, _very_ far away from recognizing chart
patterns and trading on them!

------
alfiedotwtf
All types of trading, no matter what system, is theoretical until it hits the
market. The market is the only test.

------
cgold
To what extent does the industry use Bayesian methods? It seems to me that,
for a given trading strategy, trading companies are really interested in the
mean profit impact of the strategy and its distribution (e.g. "there is a 10%
change we will lose more than $20M"). Bayesian methods will naturally produce
such an answer.

Bayesian methods won't magically solve all problems (e.g. fitting to
historical data) but could make the assumptions more clear.

~~~
throwaway183839
I can't speak for other teams, obviously. But almost everything we do would
fall under the heading of "Bayesian methods". It's such a broad term that it
would be hard to write a trading algorithm that couldn't be described as
Bayesian.

------
known
Past Performance is Not Necessarily Indicative of Future Results; ‎

------
biomimic
When trading real money in a real market, predictions based on historical data
go out the window.

Historical data will never be able to truly simulate manipulation or
sympathetic, symbiotic or parasitic relationships. Ever back-test a trading
system that simulates a Market Maker letting low block go under the bid or
dialing down the sensitivity of the bid vs. the ask? Speaking from experience.

That's why I'm developing an algorithmic trading system based on sympathetic,
symbiotic and parasitic hidden connections.

Also Ref: Contagious Speculation and a Cure for Cancer: A Non-Event that Made
Stock Prices Soar -
[http://www0.gsb.columbia.edu/whoswho/getpub.cfm?pub=1555](http://www0.gsb.columbia.edu/whoswho/getpub.cfm?pub=1555)

~~~
pbowyer
> That's why I'm developing an algorithmic trading system based on
> sympathetic, symbiotic and parasitic hidden connections.

That sounds fascinating. Do keep us informed!

~~~
biomimic
[http://www.cymetica.com/recommend/app/hidden_connections?que...](http://www.cymetica.com/recommend/app/hidden_connections?query=linux&db=all)

