Having used GPT4 for some time now, I found it quite useful for many things. However, I think the context window size is the big bottleneck currently. Right now GPT4 is an exceptional tool. But for it to be revolutionary AI, it will need higher and higher context sizes.
e.g For it to be impactful in finance, being trained is not enough. It will need to take tons of new data in.
I get the impression OpenAI is quarantining their AIs from real-time data, disallowing it from crawling the web, as part of an alignment/safety precaution.
Let's put it this way: someone is, surely, already doing this. If, a couple years from now, there is not some Warren Buffett-sized new wunderkind of the stock market, then you will know that ChatGPT is not able to do this.
Renaissance Technologies: "Renaissance's flagship Medallion fund, which is run mostly for fund employees,[10] is famed for the best track record on Wall Street, returning more than 66 percent annualized before fees and 39 percent after fees over a 30-year span from 1988 to 2018"[0]
[0] https://en.wikipedia.org/wiki/Renaissance_Technologies
coming full circle, Peter Brown and Robert Mercer of Renaissance started off doing statistical language modeling at IBM and their 1992 paper in Computational Linguistics is cited in the BloombergGPT paper.
They haven't taken outside investments for many years because of capacity constraints with their strategy. It goes beyond that actually: they force people to take dividends of their profits, because the strategy can only handle so much capital before marginal returns plunge-- otherwise they would quickly have too much capital if they kept reinvesting their cumulative profits.
I thought the strategies varied over the years: early adopter by jumping on different bleeding edge strategies. Strategies restricted by limited fund size.
To add: “So, how does the Medallion Fund make money? It finds individual patterns in data and exploits each pattern just enough to turn a small profit. And when you add up all of those small profits, you end up making a lot of money.”
Wait til they feed it your social feed combined with other collected PII from the last 20 years and use it to psychologically tailor ads, Edward Bernays[1] style, just for you, which it will also generate in real time.
Wait till I fine tune it on 20 years of my own social feed, texts, emails, and forum posts, and create a me-bot that can answer my say-to-day emails and probably do half my job too…
This is the real issue with privatized AI imo. The limitations and guardrails exposed to the end user do not apply to the developers, who can alter results in ways that would be difficult to detect at best and invisibly manipulative at worst.
Overall a good article. I learned that the financial news company Bloomberg has built a ChatGPT called BloombergGPT trained on a corpus of 700 billion tokens.
this is bloomberg's marketing department trying to cash in on the ai hype. It isn't that deep. Also and LLM doesn't have goals, can't reason, etc. It just spits out tokens likely to come after the tokens you type in. So it can't "predict" the market.
It's not really outperforming the market. It's mirroring the market, and the market has been down since it started, so it's up. But it looks like if the market keeps going generally up, it will go generally down.
A random number generator would outperform Cramer (and every other such investment personality). His job isn't to be talented but to get clicks and views. The benchmark for comparison should be the top hedge fund managers.
Over the past decades we have had both bull and bear markets. So unless the LLM is trained very fast on the latest news and programmed to give more recent news higher value, I suspect that it would produce approximately equal numbers of buy and sell recommendations.
If LLM's have hidden models like people say they do, a hedge fund might be able to replicate the success that RenTech has had with their hidden Markov models. I mean, they recruited NLP engineers to achieve the greatest level of success Wallstreet has seen.
Applying something like ChatGPT to stock prices together with quarterly reports, industry research, and daily (minutely) news articles seems like a no-brainer -- I'm surprised this is the first time I've seen something like this mentioned.
And so I can't help but wonder if this would outperform the market significantly due to seeing more patterns and holding more information than any individual trader or even whole desk ever could... OR if its total lack of any actual reasoning/deductive capabilities makes this just a total non-starter here. Or maybe it might be useful for day trading but not any kind of longer-term investment?
> Applying something like ChatGPT to stock prices together with quarterly reports, industry research, and daily (minutely) news articles seems like a no-brainer – I’m surprised this is the first time I’ve seen something like this mentioned.
Customized neural networks for stock prediction have been around for decades, and for obvious reasons its been a domain that’s attracted a lot of attention and energy. LLMs obviously have utility in automatically processing news and the narrative portion of reports, but I don’t know that they are particularly likely to be useful as the main prediction engine rather than a tool to process text into inputs for specialized trading models (I’d be surprised if there hasn’t already been AI news sentiment analysis, etc., being used to supply input for these models, even before ChatGPT was available.)
Any edges, though, in that field get eaten up quickly in the arms race.
> Applying something like ChatGPT (...) seems like a no-brainer
Why? It's famously bad at math after all. It can spit out quarterly reports and industry research back (so if analyst says "buy", ChagGPT will parrot that), but other than that it would - I think - work as a glorified sentiment analyser. Of course I may be wrong.
And we also need to remember that AI is used heavily in finances (and the best cutting edge methods are probably not published by the companies using them, since this is a zero sum game).
> Applying something like ChatGPT to stock prices together with quarterly reports, industry research, and daily (minutely) news articles seems like a no-brainer
It has been done many times. It turns you predicting the flip of a coin can be done with extremely high levels of accuracy using traditional methods.
You seem to assume that one can do better than random in that field... That's already debatable (or maybe not: wasn't it proven that hedge funds don't do better than rolling dice?)
Well, there's Warren Buffett's famous bet that the S&P 500 (or an index fund tracking it) would outperform hedge funds after all their fees over the course of a decade, which he won in a landslide: https://www.cnbc.com/2022/10/03/billionaire-warren-buffett-s... . There's also something in there about almost all active funds underperforming (84% after 5 years, 90% after 10 years) and being worse than luck.
Basically. Because if you think you can figure out a better investment strategy than Warren Buffett, well, good luck, you're gonna need it. Monkeys, cats, and small children often pick stocks better than the pros: https://prosperitythinkers.com/personal-finance/three-monkey... . In fact, here's the Guinness World Record for a chimpanzee who was the 22nd most successful money manager in America by throwing darts at a list of companies: https://www.guinnessworldrecords.com/world-records/most-succ... .
I think quants have squeezed any alpha that can be gleaned from historical data, which seem useless in predicting the future, remember you need to be accurate in the price and timing, not just proce
There's a chance that the "quants" are using the same technology that "language models" were using, before they were revolutionized just a couple of years back. I have no idea if this is true, but if so, then what they've gleaned so far might be nothing compared to what an LLM-style neural architecture could do.
It would finally expose the whales and show clearly the correlation between news and stock moves. The whales own the news outlets, news which placed intentionally at the right time will incite fear for them to buy low or generate excitement to sell high. Looking at you Elon Dodgecoin
It's already being trained on centuries of mining data, decades of geophysical data, terrabytes of geochemical data, and a century of mineral lease acquisition data.
Rio Tinto et al have been warehousing data from the get go (circa Roman Empire with the oginal mine in Spain).
Just for kicks I think we should start from the other direction. Ask GPT-N to create a perfect random number generator by using Stock-market data. All the failed attempts in doing so would basically be exploitable 'hedged' investment opportunities.
Dumb question: What does it mean to "train" an LLM on something? I've used Llama Index to create an index based on a large dataset which I can then query. Is that what it means? Or is it something else?
In this case they trained a model from scratch, which means they built their own LLM, which is what's called a transformer, on their own financial data. How transformers work in detail is a little bit more involved.
That's not the same as indexing which is a way to tune an existing model for a particular context.
The current crop of services and pdf chatbots that you see going around using "train" on your own data aren't doing anything of the sort. They are essentially finding relevant sections of your docs and stuffing that into the prompt context window. Think of llama as the trained base model (not particularly useful for any one thing by itself) and alpaca is a fine tuned variant of that to give it chatgpt like "powers".
The irony is that once this becomes common-place, it no longer becomes effective. The only reliable way to make money on the stock market is to bet that the whole stock market will go up over a long enough time.
Wouldn’t it have to be continuously trained on new info moment by moment as economic news unfolds? Isn’t training the most computationally expensive part of the whole process?
Also, my understanding of LLMs (tiny) means they will only give you the most probable next sequence of words to your prompt based on the words in the training set. Has an LLM ever produced output that wasn’t buried in the training set somewhere? I.e. a novel answer to a novel question?
> Wouldn’t it have to be continuously trained on new info moment by moment as economic news unfolds?
It would benefit from access to current news, but not continuous training (though more training is better).
> Also, my understanding of LLMs (tiny) means they will only give you the most probable next sequence of words to your prompt based on the words in the training set. Has an LLM ever produced output that wasn’t buried in the training set somewhere?
This is a misunderstanding. Its a pattern recognizing predictor, not a Markov chain, its not reproducing its training set, its predicting what a plausible document would look like from the universe from which the training set was sampled would look like if it started with the prompt, and the conpletion cna be vert different from anything in the training set if the prompt is, and (depending on tuning like “temperature”) potentially even if the prompt is exactly represented.
"Bullshit has its uses" is kind of a way of saying "it may be very valuable and useful, but I don't particularly appreciate it".
I feel like every time people equate this sort of thing with human consciousness, it is a way of saying "yes, bullshit is the totality of human existence".
If it was additionally trained on decades of old Playboy magazines, it would be an AI only capable of comprehending money and sex, thus rendering humanity 99% redundant.
a fun fact is Bing AI already gives pretty good answers on the tasks Bloomberg cites as motivation, like who is the CEO of various companies, "write a bloomberg excel blp formula to return industry subgroup of ibm apple microsoft google"
another fun fact from the paper, it trained for 1.3m GPU hours and didn't get through the whole corpus, did 80% of 1 epoch (p. 16).
another fun fact, according to Twitter rumors GPT-5 is already training on 25,000 A100s w/8 GPUs each, so 4.8m GPU/hours per day.
it's a good paper with a lot more detail than OpenAI is releasing. also confirms OpenAI is quite a bit ahead. They compare with benchmarks they ran with Bloom and open source GPT-NeoX which they were able to run on same data, and reported benchmarks for GPT-3 since you can't run it yourself (well I guess you could sample the API), and BloombergGPT was competitive with GPT-3 which is a couple of years old now.
my takeaway was, this stuff is hard and OpenAI is crushing it.
I wonder if they will just use it for terminal features, or make the API available, and at what cost, and if they will license the model and weights for people who want to run it internally and maybe train incrementally on their own data. (not sure how feasible that is)
> another fun fact, according to Twitter rumors GPT-5 is already training on 25,000 A100s w/8 GPUs each, so 4.8m GPU/hours per day.
If true that is incredible. From what I understand there isn’t a function that relates the increase of training time/effort to performance. For example, twice the training doesn’t mean twice the performance. It could mean no increase or 1000x increase, no one really knows. Does anyone have any idea what to expect in terms of capabilities when this training is finished?
the bloomberg paper does talk about how they sized the model based on their compute budget and research on time to train ... megatron has 1t parameters
i'm intrigued by some of the knowledge graph performance benchmarks and wonder about training it alongside an explicit knowledge graph, instead of building its own implicit knowledge graph, despite the 'bitter lesson' about just letting simple models and massive computation do their thing.
How does it go? The trend is your friend — until it bends in the end. I’m sure trading firms are already fervently working to have the new AI to identify these trends faster and sooner, and when exactly the trend will bend in the end. Probably not too crazy of an idea to imagine the distribution of profitable stock following the power law and the entire market getting dominated by superhuman stock-picking AI. As long as the chat prompt optimizes for some human quality like “lessen human suffering” or “improve the human condition” then let’s give this experiment a go.
Depends what that word "data" in the title means. If it is live satellite images showing shipping movements, flight data - including private jets, public minutes of millions of meetings, weather patterns and so on, it might see what we can't. We may not have that going back decades, but they could start building such a set now.
It will be like you are at the horse race, a punter reads the horses form in the local paper. Versus the computer runs a tuned up linear regression on thousands of variables about past horse races, weather, tracks, horse injuries and so on going back 10 years.
It doesn't really matter. You can feed a generative AI model all the live data in the world, but it will still only be able to interpret it in context of how such a situation played out in the past.
That statement is tautological. It is equivalent to "getting information from the future is impossible".
What I replied to was:
> It'll do great. As everyone in the industry knows, past performance is always indicative of future results.
Assuming sarcasm, you are saying:
"It may not do great, because past performance is not always indicative of future results."
This is an adage to warn the everyday person that, say, while MSFT has produced great returns in the last 10 years, it does not guarantee it will carry on producing the same returns.
Which is fine.
But a sophisticated operation running machine learning on large live datasets to predict stock market movements doesn't need this adage. And in addition, it is possible for them to beat the market if they can make predications and connections that no one else is making.
Understand that is not "they can't lose any bet". It is more "they will make money over all bets on average that gives a higher return than the market".
Open question as to whether the generative model can do this by turning on a switch, or whether it is tuned and modified by experts to understand something about current affairs.
I tend to agree, but maybe AI can find patterns/correlations/etc that Humans wouldn't think of? Not sure we're there just yet, but this might be a step in that direction
TLDR - Bloomberg trained an LLM 700 billion words of their own clickbate, and now it produced clickbate even better that chatGPT. Also, it's an ad for bloomberg terminals.
For people new to this or not reading the text, the key words are "list of internet companies" during the dot com bubble. The monkey throwing darts had very little to do with it.
I highly recommend the book "A Random Walk Down Wall Street" for further insight on this topic[0]. It provides compelling evidence demonstrating that the majority of professional fund managers tend to underperform or match the results of random stock picking strategies.
Thinking Fast And Slow by Daniel Kahneman among many other things explores the answers to "Given the same information presented to hedge fund managers, why does one buy and another sell" and concludes relying on extensive research the answer is: their performance can't be distinguished from rolling a dice. The book is very strongly recommended.
I'm wondering, as I read this over and over again about random picks performing as well or even better than professionals in the field, how come we never see rigorous scientifc studies about this for various timescales?
If we could really attribute this task to randomness wouldn't this save a lof of money to financial institutions considering how much the same people earn for these tasks?
This is what "whole market funds" or "S&P 500 funds" are all about. No stock picking, other than identifying some pre-existing known set of stocks, and just investing in them without discrimination or effort. They have become quite popular over the last 15 years as the "random walk down wall st" idea has spread within (US) culture.
I have read, and heard from multiple financial professionals, that these work best in low interest rate bull markets, and are more likely to underperform active management strategies in more turbulent and difficult financial environments, such as the one we are entering.
The phase you're looking for is tax loss harvesting. A good money manager on a good fund will be able to take advantage of that in a down market, leading you to greater riches when there's a bull market. You can do it yourself, but you can also pay a professional to do that work for you, hence the fees on a given fund.
> I'm wondering, as I read this over and over again about random picks performing as well or even better than professionals in the field, how come we never see rigorous scientifc studies about this for various timescales?
Erm, Kahnemann got a Nobel price in economy for his work with Amos Tversky over a decade which this was a part of. I am not sure what you need here, do you need a list of papers by Kahnemann and Tversky? With Terry Odean thrown in for good measure? I am not sure who the "we" are here who do not see these papers but they do exist.
consider a stock picker who starts with 256 people, tells 128 of them that next week Tesla will go up and Twitter will go down. Whichever happens, take that 128 strong cohort, tell half of them that the next week, Ford goes up and Chrysler goes down.
Etc.
repeat for 8 weeks.
At the end, the stock picker has a true believer: someone who thinks they saw the stock picker successfully "know" for 8 weeks in a row which stocks would go up.
Change the numbers around a bit, don't run until the true believer pool size is one, and you've got ... the real world.
From this, most people conclude that financial managers cannot outperform the market. Which is obviously false since there are a bunch of managers that have consistently beat the market for many decades.
The actual conclusion is that most financial managers are fakes, and laypersons cannot tell the legit from the fakes.
We have this problem in Computer Science as well, where randomly picking objects performs just as well, if not better, than a some logical huristic or algorithm. Most notably in cache eviction. When the cache is full, how do you pick what to evict, given incomplete knowledge of the future? There are various algorithms, evicting the least recently used - LRU - object being one popular one. It's often beat in benchmarks by an algorithm that randomly selects an object to evict though.
Still, BloombergGPT has knowledge far in excess of a monkey, and if it's anything like ChatGPT, is able to provide reasoning as to how it arrived that that conclusion. Time will tell if, granted a brokerage account, how it will perform relative the market. It may or may not outperform a monkey throwing darts.
If I showed up somewhere and everybody there thinks I'm an all-knowing, all-capable oracle, I'd know how to make money easily...
A group of men in suits would ask me, oh Oracle, what should I invest in? And I'd tell them "My intuition tells me.. buy ACME stocks". Another group would show up and ask the same thing, and I'd answer the same, and the first group of men would make a profit, and huzzah, the oracle is right again!
But of course I'd tell a good friend to buy ACME stock before all of them did, or maybe this friend would be the one telling me which stock I should recommend to the people...
The Inverse Cramer Tracker ETF (the “Fund”) seeks to provide investments results that are approximately the opposite of, before fees and expenses, the
results of the investments recommended by television personality Jim Cramer.
The Fund is an actively managed exchange traded fund that seeks to achieve its investment objective by engaging in designed to perform the
opposite of the return of the investments recommended by television personality Jim Cramer (“Cramer”). Under normal circumstances, at least 80% of the
Fund’s investments is invested in the inverse of securities mentioned by Cramer.
This worries me a bit - is everybody in on the joke? People putting real money in?
I mean, "do the opposite of a stupid thing to succeed" is traditionally funny because it obviously doesn't work - because there are generally infinite ways to fail...
e.g For it to be impactful in finance, being trained is not enough. It will need to take tons of new data in.