This isn’t ML. It’s cargo-cult performance of words and ideas that ML people use.
See Hyndman's fpp2 — https://otexts.org/fpp2/accuracy.html
Also, his description of rolling window validation: https://robjhyndman.com/hyndsight/rolling-forecasts/
What I've seen from companies marketing to Higher Education is - we have a lot of data, you set arbitrary flags to the data that you believe indicate 'x' (or even better, they have pre-built data expectations) and you will get 'y' outcome.
And none of it is actually based on anything real. It's all anecdotal applied to extreme amounts of actual data.
And when I read about ML on here, it seems to confirm my experiences.
You have to be very selective about what you consider "ML" to come to that conclusion. There has been a constant parade of incredible, mind-blowing results out of ML over the past decade, advancing the state of the art by leaps and bounds both in research and in real applications.
Do you not remember how terrible speech recognition and speech synthesis were just a few short years ago? Did you not see DeepMind finally crack Go? Check out BigGAN . Try search in Google Photos. See translation getting better every year.
Yes, there are quacks and charlatans and people who are just plain wrong. But ML is real, and it solves real problems that people failed to solve any other way despite decades of concerted effort.
That's because they have the platforms and applications that people are using at scale. So ML is a force multiplier if you already have a consistent and strong user base for a good product.
If you're trying to get a product or company started, unless you're a pure ML research company like Clarifai (and arguably even then), ML is probably going to cost you more than you gain.
You don't have to invent or implement the algorithm yourself. You can use AI/ML intelligent/cognitive services provided by the big-3 cloud companies to reduce your starting cost significantly.
To have positive ROI, your problem complexity should have crossed the threshold where common-sense traditional solutions don't work any more.
If you do embark on ML research yourself, then be sure to walk the path from simpler models to complex ones while carefully establishing performance metrics.
1) work a LOT better for specific problems than statistics or statistical learning ever has (and at this point, I think we can safely say: ever will)
2) a lot of methods either can't be explained, or outright shouldn't work, according to statistical theory.
The use of statistics in machine learning is limited to evaluating performance and individual element performance (and even that is tenuous at best in many cases). If you ask, say, why would an autoencoder, with an LSTM on it's compressed representation and Q-learning evaluation have somewhat decent performance on half the computer games humans ever designed ? Statistics will not be useful in formulating an answer.
If you ask extremely valid questions, like "why would an LSTM predict anything ?". Statistics draws a blank. There is no good reason to assume an LSTM will ever converge (and on a truly random dataset, it won't, whereas statistical methods will still allow you to say something).
I think there's 2 reasons for this
1) the "upper limit" of complexity a human can understand in a statistical model is lower than the upper limit a neural network can "understand". In statistics the human understanding is critical to getting to a valid model, in machine learning ... it is not. Meaning machine learning can learn relationships a human mind cannot.
2) There must be some fundamental property of the world we live in that matches neural network architecture. In order for backprop to work on real-world problems, it has to be the case that almost all real world phenomena are continuous, both "raw" and in the frequency domain. If this wasn't the case, machine learning would never be able to learn anything.
When people invented a steam powered engine, some other people had probably said: "Modern physics can't explain how it works. It shouldn't work. It's too complicated". Then a few decades later physicists discover laws of thermodynamics.
50$ increase from 50$ is 100% increase
50$ decrease from 100$ is 50% decrease
e.g., if the model finds 50$ increase/decreases, that actually corresponds to very different wealth changes
- edit: miscounted number of LSTMs
There's a good reason the most successful 'quantitative' trading fund, Rennaisance Technologies is so secretive and subjects its employees to a lifetime NDA/non-compete.
If there is edge in the market (and I think there are anomalies that can be traded on), I would assume it to come from correlating proprietary datasets with price. Or datasets that have a high barrier to access/analyze. One example that I'd bet still works in some industries would be counting trucks from a supplier to estimate product demand before the companies announces earnings ;)
I play in the stock market a little and my best wins have been on small cap stuff where I've had some edge with unique knowledge of the industry and have taken the time to read SEC filings / keep tabs on earnings reports / closely watch competitors, acquisitions, etc.
To be clear, I still think this article is great from a learning about Keras perspective. That said, to anyone who thinks building some ML models and outperforming the market is easy, remember that edge is only as good as the number of participants in the market who don't have it.
- Pick Em strategies on DraftKings - https://medium.com/draftfast/evaluating-possible-strategies-...
- Thinking in multiples - https://medium.com/draftfast/thinking-in-multiples-7e7c76ee2...
Funds like Two Sigma that haven't had as good (or scalable) returns are actually the ones that focus on novel data sources.
In reality, everyone in this industry is drowning in data, and the real edge comes from learning how to more efficiently parse and analyze data rather than acquiring more of it. The top performing firms work at a level where the data is merely an abstraction; above-market returns can be achieved from nothing but public data sources, if you've automated the process of extracting signals.
For what it's worth, Renaissance (remarkably, in my opinion) advertised a role in the most recent Who's Hiring thread.
How do you even enforce that? If I get a job somewhere else, there's no reason for me to disclose where I'm going.
By suing your new employer for $20 million .
(On the other side of the coin, they offer like a month of paid vacation a year, plus longer sabbaticals every few years, plus tons of benefits...if Long Island and math are your thing, it's a very nice place to work.)
But in this world, it actually makes sense for those companies to stop doing their normal business and just go into the business of selling their shares.
This reality sounds absurd, but you could argue that BTC market is there. Enough of the market thinks that "always buy" is a good investment, regardless of the real-life value of the asset. Or maybe, less controversially, gold is that market. Any asset that is always increasing in value and not related to the real value of that asset is just a store of value.
BTC provides a number of services. From money changing and international money transfers to actual investment brokerage. Granted, the number of securities available in BTC is less than spectacular, but it's not zero. I pay for both those services. Now you may argue that those are unregulated services and therefore have trust issues, but one might argue that all markets have trust issues, and the only difference is the level. BTC, so far, seems to be more trustworthy than, for instance, the ECB (e.g. the Greek payment limits and the Cypriot bail in, one of which affected me, and both of them used MY money to achieve political aims, without my approval).
Gold provides a store of value, with a good story behind it. I pay my bank, I believe, around $40 per year for that same service. With frankly, not as good a story behind it (as I trust my bank less than I'd trust a bar of gold under my pillow when it comes to still having value tomorrow. Not that I have the kind of spare change to make that a pressing issue, but ...)
So given that both BTC and Gold provide services that clearly people are willing to pay for, who's to say they shouldn't have a valuation based on that income like every other financial service provider in the world ?
And if you have a model that people believe which predicts the movement of stocks but doesn't have any good reason for those predictions then it may very well move the price of those stocks in the ways that it predicts. But in doing so the money of the people who believe its predictions will be transferred to the people who don't until it stops having an effect.
Which is say, belief in a direction can make the direction happen but belief in a particularly shaped curve won't make the market resemble that curve.
Not quite lifetime, unless they expected him to drop dead within a year.
On the flip side, employees are compensated very well, and the turnover is very low.
Most of them are also qualified for general software engineering roles; some of the people with more theoretical backgrounds might not be.
However the Virtu Financials, the Citadels, etc, will always exist, and will be doing exceptionally well whether people realize it or not.
The issue is that these business are capital limited, so you can't reinvest your massive returns to compound them.
Medallion fund is profit maker but their strategy is obviously very size limited. Their strategy does not scale.
Betting syndicates can make similarly massive ROI, but they are very growth limited.
Except Jakob explains the folly of the method.
Whereas the author of this piece says the following:
From the plot we can see that the real stock price went up while our model also predicted that the price of the stock will go up. This clearly shows how powerful LSTMs are for analyzing time series and sequential data.
I am sure plenty of people have done lstm on the stock market but the form of the code looked similar between the two.
Yes I've noticed on HN recently ML and data science have become popular topics but I'm surprised a post like this has so many votes.
I recommend doing something similar to the original post. Neural networks tend to produce valid looking output for a stock price easily, a random walk does too. You have to find correlations, causations, and then review the results carefully.
For reference, I wrote my own financial advisor (not directly utilizing deep learning, that functions relatively well):
It works relatively well when checking the "causality" which also has it's limits: https://blog.projectpiglet.com/2018/01/causality-in-cryptoma...
EDIT: Added some evidence:
* 2016 stocks: https://imgur.com/a/j8YWR
* Early 2018 crypto: https://blog.projectpiglet.com/2018/01/30-weekly-returns-usi...
* Early 2018 stocks: https://twitter.com/AustinGWalters/status/976347632439209985
* All 2018 on Robinhood only: https://imgur.com/a/2CxEFqI
* Also (I am lettergram on HN), created: https://hnprofile.com and https://redditprofile.com
I do not see enough data to backup the claims of how well it works. And if you can't provide that - what good is it as a tool?
Somehow my network always learned to output a delay of the input no matter how hard I tried to shape it. I've searched through literature briefly and some examples. Some blog posts even claimed they had good time series prediction when I clearly saw they were having the same problem.
What I mean to say is, be critical of what you find online, and be critical of your results. This is definitely not an easy problem, and some argue that it's unsolvable all together :)
Setting aside that first bit of hyperbole, I don't see the danger in the bridge example. I mean, if engineers are building bridges based on information they've gleaned from internet articles, there are larger issues at play.
As far as markets, retail traders basically exist and have existed within a massive bubble of misinformation since forever. There's a reason order flow is so valuable.
I refer to engineering code, because it is clear to me we don't want our bridges build by those not skilled enough to make safe bridges. Yet, it is not so clear with ML/AI, while the potential for damage may be even greater.
One of the biggest problems facing someone trying to assemble a numerical model of some thing or process is whether or not their model is adequate to describe the phenomena they are observing.
This is why psychology remains a statistically modeled science, largely relegated to anecdotal research.
(Note that my intent is not to speak ill of psychology and its cohort, only illustrate that just because you can slap numbers on a thing doesn't mean you've described it well.
Look at the transition of understanding from Newtonian mechanics to relativity and quantum mechanics. Newtonian mechanics was enough until someone saw Mercury was doing weird things it shouldn't.
And before Newtonian mechanics we had a broken description of the solar system with more exceptions than rules. We were throwing numbers at a system and failing, in some part, to describe it with consistency.)
(For anyone who didn't get the reference, you have an excellent movie & coding soundtrack to catch up on: https://www.youtube.com/watch?v=ShdmErv5jvs)
It just lags the current price. If there is no alpha in the price signal, then the rational thing to do is to just stay at the current price, which is what the algo seems to have learned to do.
Many data science thought pieces don't quantify the results, and it's suspicious that they're not included in detail (typically, the R^2 of a simple stock market prediction model is super low, making it impractical putting actual money on the line as it's barely better than guessing randomly: https://twitter.com/minimaxir/status/1021885939361042432 )
In the stock market something that is consistently better than guessing randomly is very valuable! You don't need much of an edge to make a lot of money.
(I'm very skeptical about approaches like the OP though)
That’s not playing catch up, if the prediction was made at time 0.
> previous price of a stock is crucial in predicting its future price
This is a poor and incorrect model. The stock price has plenty other variables and many of which are unknown. What's used here is just a single variable.
Working out a function that passes a split test is inevitable and easy tbh. That function then making money is highly unlikely. This is especially true if your data sources are public.
Only comparing the prediction accuracy against simpler models like ARIMA or VARMAX (as the author suggest) can tell you if the model has any use.
What DL is good at, is to automate tasks that are easy for humans (e.g. telling cat from dog, understand a sentence from sound wave, translation, etc) but hard for machines without DL.
Now, if a task is even hard/borderline impossible for humans to achieve (e.g. predicting stock market with higher accuracy and consistency), why would we believe DL could do a better job than humans?
It can be used to recognize patterns and train to solve problems better than humans. Though in this case since the market is people making predictions about a prediction I’m not sure how much it’d help.
There’s probably something about humans using a rough estimate of what the stock has been historically to inform where it could go though so it’s value is probably non-zero.
AlphaZero is a generalized successor, and it does just that:
DL could potentially do a better job by recognizing patterns in the dataset that lead towards winning (making more money) that humans might miss. Like how AlphaZero can recognize moves in Go or Chess that humans don't understand are the best moves to make.
It's not obvious to me how this would be done, but I think it's plausible that some clever implementation could help.
That said, the pitfalls of using ML in trading systems are numerous.
Look at Long Term Capital Management. It had big names behind it and the model/strategy made a lot of money, until I believe the Russian bond crisis caused the markets to act irrationally. Then everything they had invested in quickly went down the toilet, and it was big enough to almost cause a financial crisis.
Other examples include analyzing the words/sentiments in earning calls, to see if certain words indicate a bearish or bullish signal. For example if a CEO mentions the word "headwinds", does that historically lead to a much better quarter next time?
Some other companies do analysis with public/private data. For example, you can use a weather dataset to calculate the average temperature for all cities where there are restaurants owned by Cheesecake Factory in a quarter, and see if it will negatively or positively impact sales.
Read Tsay's Financial Time Series Analysis for a better idea how to forecast financial data.
The biggest problem you will face is stationarity: the statistical properties of the data is not constant over time. For example, the mean and std dev is not constant over time. Using returns instead of raw prices helps to make better financial forecasts.
Two methods to explore:
1. You are better off predicting stock prices by predicting future returns and then forecasting is the current price plus predicted future return.
2. You could use your neural model to predict absolute size of returns using realized volatility.
I would be less skeptical if it was more humble, and specific in its constraints. For example, could it predict the % probability a stock would up X% if it had N consecutive negative days? Or, how many minutes/hours after earnings are announced is the after-hours price usually indicative of what the price will be the next day?
You can't predict everything, but you might be able to have a probabilistic prediction of specific scenarios. though even then I'll still be skeptical.. daily stock movements are totally random.
Skipping to the "results" of this experiment, I am seeing nothing. It looks like a slow, short period moving average (SMA)... which does nothing in terms of predicting anything. It just averages out the chaos over time.
Is laughter or tears the right reaction here?
In addition, major factors are external and trying to predict further price only based on past price is bound to disappoint.
Seriously tho', articles like this need to come with a big banner at the top saying warning: do not try this at home.
On a positive note, nice showcase of how simple implementation with Keras is.
Before clicking I thought "I bet they do stacked LSTM to predict the _next_ time step" :|
Without that you are just regurgitating some model that does something.
Imagine having the whole Twitter stream, maybe categorized by some network in an automatic way.
Also prequential evaluation.
-- Yogi Berra
It is like predicting Trump by Nickson or, even better - predicting a next Trump.