Honestly the whole thing with these chat AIs not continually learning is the most disappointing thing about them and really removes a lot of utility they provide. I don't really understand why they are essentially fixed in time to whenever the model was originally developed, why doesn't the model get continuously improved, not just from users but from external data sources?
Do we need for the model to be be continuously updated from data sources or is it good enough that they can now figure out either by themselves or with some prompting when they need to search the web and find current information?
The links aren't loading for me, but there is a difference between the output when the AI is trained vs having it google something for you, no? Having the ability to google something for you vs just making up an answer or being unable to answer is definitely a step in the right direction, but isn't the same as the AI incorporating the new data into it's model on an ongoing basis as a way of continuous improvement.
Idea I had: given user question, run model twice – once with web search assistance, once without. Then, continuously fine-tune the model in the background on every question (or a random sample of them) to prefer the web-search-augmented response to the non-augmented response. Maybe update the model every day (or even once every few hours) based on that fine-tuning.
For questions about current events it could be very helpful. I mean, suppose {Biden|Trump} wins the 2024 US Presidential election. You want the model to know who the winner is the next day, as part of its background knowledge, not wait 6 months for a new version to be released whose background knowledge includes that fact.
It is right that humans don’t try to remember everything, but if one only did this for a random sample of conversations, you wouldn’t remember everything - you’d probably remember key developments because they’d come up a lot (in many questions hence high probability of being sampled). You’d likely also remember some obscure trivia just because it got sampled (just like humans do too)
Another option would be to have the model evaluate each conversation and have it decide whether to include it in the fine-tune (e.g. did the web search reveal important new developments the model didn’t know about?)
One could also do some kind of clustering to find common gaps in the model’s background knowledge and use that to guide sampling for fine-tuning
What I am saying that today with ChatGPT 4x, if it detects that its model doesn’t have the current information, it will search on the web and summarize the results from different websites.
One example I have was “what’s going on in Atlanta over the next two weeks”.
I had another example I just posted where I asked it “who are Amex’s and Chase’s current transfer partners and which ones do they have in common”.
It searched the web and summarized the information like you would expect.
Suppose I ask it questions like “who is Joe Biden?” and “who is Donald Trump?”. It is going to say “current President elected in 2020” and “former President elected in 2016”. And I imagine it doesn’t need any web search to answer those questions, it already knows.
But, is it smart enough to think “if today’s date is 2024-11-06, I need to do a web search to find out results of yesterday’s Presidential election?”
For the average adult human (even very many non-American adult humans), “who is current US President” is something they know off the top of their head, not something they have to always look up. I think inevitably it is going to perform worse at some point if it has to always look that up if it has changed recently. At some point the model is going to not look it up - or not pick it up from the results of its search - and hence get it wrong.
Suppose, hypothetically, Biden unexpectedly dies tomorrow (of natural causes) and Kamala Harris becomes President. Because the model won’t be expecting that, it wouldn’t necessarily think to do the web search necessary to find out that had happened.
I think the problem I am talking about is more likely to come up if you ask a question which isn’t directly about who is the current President, but rather some other issue in which the President just happens to be mentioned.
Suppose hypothetically Trump wins in November, and on 2025-01-20 he is sworn in. On 2025-01-21 I ask it some question about a policy issue. It finds some articles from mid-2024 responding to the question. Those articles talk about “former President Trump” and its background knowledge says he is a former President, so in its answer it calls him the “former President”. Except (in this hypothetical) as of yesterday that’s no longer correct, he’s now the current President.
Whereas, a reasonably well-informed human would be less likely to make that mistake, because if Trump is inaugurated on 2025-01-20, you expect by the next day such a human would have added that fact to their background knowledge / long-term memory, and if they read an article calling him “former President”, they’d know that would be an older article from before his return to the White House
I don’t know why the links aren’t working correctly.
The idea is that if an LLM can now search the web and analyze data, it will be more up to date than training it.
Another unrelated improvement with the newer versions of ChatGPT is that while LLMs are notoriously bad at math, they are pretty good at writing Python. ChatGPT can write Python code to solve a problem, run it and give you an answer based on the code.
You can also verify the code yourself. I used ChatGPT to model financial scenarios and the answers were correct in 4x. I verified the generated code it was using.
But here is what it answered when I asked ChatGPT “What’s going on in Atlanta this weekend”?
> This weekend in Atlanta, there are several exciting events happening:
1. Lakewood Antiques Market (May 31 - June 2): A large market in Cumming featuring antiques and collectibles .
2. Bark Meow-Ket (June 1): A craft and pet vendor market at The Lawn at Uptown Atlanta, with food and pet adoptions .
(and a few other events)
But a better example and where I forced it to look on the web is
“Compare Amex’s current transfer partners to Chase’s partners”
For context, when you earn credit card points with Amex and Chase, you can transfer them to airline and hotel loyalty programs.
It searches the web (and provides the links it used to find the information) and then answers with:
——-
### American Express (Amex) Transfer Partners
Amex has a robust list of transfer partners, spanning both airlines and hotels. As of 2024, Amex Membership Rewards points can be transferred to the following partners:
Chase Ultimate Rewards points can be transferred to a variety of airline and hotel partners. Here are the current partners for 2024:
#### Airlines:
1. Aer Lingus (AerClub)
2. Air Canada (Aeroplan)
(a list of airlines)
#### Hotels:
1. Hyatt (World of Hyatt)
2. IHG (IHG One Rewards)
3. Marriott Bonvoy
### Comparison
Both Amex and Chase offer a strong selection of transfer partners, but there are some differences:
- *Unique to Amex*: ANA, Avianca, Cathay Pacific, Delta, Etihad, Hawaiian, and Qatar.
- *Unique to Chase*: Southwest Airlines, United Airlines, and Hyatt.
- *Common Partners*: Aer Lingus, Air Canada, British Airways, Emirates, Flying Blue (Air France/KLM), Iberia Plus, JetBlue, Singapore Airlines, Virgin Atlantic, and Marriott Bonvoy.
Amex tends to have a larger selection of airline partners, especially those in the SkyTeam and oneworld alliances. Chase, on the other hand, includes Southwest and United, which are popular with domestic travelers in the U.S., and Hyatt, which is highly regarded among hotel loyalty programs
Probably because it's expensive, slow and adds more complications to their guardrails that humans need to check. It takes time and a lot of resources (human and compute) to make these big models. If the model is constantly being updated, how can you be sure it won't degrade in some unknown ways?
Afaui, training takes a lot of time and money for base models and it is not incremental. Dataset elements don’t simply pile up at the end, they must be baked into a model together with every other element. You can make additions through loras, but that’s limited. There’s also a risk of model crash by including its own (or derivatives) generated texts.
It strikes me what would be possible in pics/video AI, if training was quick or incremental.
