1. Systematic trading doesn't necessarily require an algorithm. For instance this rule might work (over a 5 year horizon, don't try it monthly): "buy very large cap stocks if their P/E goes below 4 and sell if it goes over 10." But you don't need an algo.
2. The market has long bull runs. So you might have an algo that has some long bias. It will seem to perform above chance. You should compare it to just holding the market.
2b. If the market is going through a bull run and your algo has some leverage built in, it will outperform just holding the market.
3. If the market had a massive crash in the data set and your algo has a short bias, then you should check it against just shorting the market.
4. The issue of models, markets and biases mirror the same debate in science theories, data and statistics.
maybe I'm being naive but "buy very large cap stocks if their P/E goes below 4 and sell if it goes over 10." sounds kind of like an algorithm to me. Maybe it's just a ruleset?
If the entire stock market P/E goes below 4, you could use it. However when a single stock goes that low it implies that somebody knows something. Don't invest in a company with a P/E below 4 if they are going out of business. (the obvious example that probably never existed: a y2k company in 1999 - they are probably making a ton of money this year, but next year they will go out of business).
There are also "cyclical companies". They have a long history of having boom and bust years, it is well known that you buy when the P/E is high - the bust years when prices are low - and sell when the P/E gets low - those are the boom years when prices are high - but the boom will not last.
There are many other company specific things that can get in the way of any formula.
1. Systematic trading doesn't necessarily require an algorithm. For instance this rule might work (over a 5 year horizon, don't try it monthly): "buy very large cap stocks if their P/E goes below 4 and sell if it goes over 10." But you don't need an algo.
2. The market has long bull runs. So you might have an algo that has some long bias. It will seem to perform above chance. You should compare it to just holding the market.
2b. If the market is going through a bull run and your algo has some leverage built in, it will outperform just holding the market.
3. If the market had a massive crash in the data set and your algo has a short bias, then you should check it against just shorting the market.
4. The issue of models, markets and biases mirror the same debate in science theories, data and statistics.