> After trying some queries, I found ChatGPT is good at finding public data, sometimes it may not be accurate, but overall, pretty good.
GPT-4o is no more reliable for this than GPT-4 or GPT-3.5 before it.
One of the many unintuitive things about LLMs is that they are terrible tools for looking up specific facts about the world - which is weird, because that's traditionally one of the things computers have been best at.
If you want to build a working version of a system like this the trick would be to provide it access to tools for looking up data in citable, reliable data sources - things like the CIA World Factbook.
Then display the charts along with a link to that underlying reliable data.
4-turbo should have a knowledge up to December 2023. But it doesn't know anything about the cli tools you created before then (except shot-scraper).
Compare 4-turbo to 4o
llm -m 4t "Tell me about these cli tools and link to them:
- llm:
- ttok
- symbex
- shot-scraper
"
Among the tools you've listed, "shot-scraper" is the only one that I could provide detailed information about based on widespread use and documentation available as of the last update. As for "llm," "ttok," and "symbex," there isn't readily available, widely recognized information on tools precisely matching those names, so it's possible they might be lesser-known tools, newly released, or possibly regional or specialized tools not widely documented in widely-used platforms.
(Truncated)
llm -m 4t "Tell me about these cli tools and link to them:
- llm:
- ttok
- symbex
- shot-scraper
"
As of my last knowledge update in September 2023, here is the information on the CLI tools you are asking about:
1. *llm*:
- I couldn't find a specific command line tool named "llm" that is widely recognized or used. It could be a lesser-known tool, a new tool, or potentially a typographical error. Please verify the tool's name and provide additional details if possible.
2. *ttok*:
- As with "llm," there isn't any widely known CLI tool named "ttok."
(Truncated)
llm -m 4o "Tell me about these cli tools and link to them:
- llm:
- ttok
- symbex
- shot-scraper
"
Sure! Here are brief descriptions of the CLI tools you mentioned along with links to their official repositories or webpages:
### 1. llm
`llm` is a command-line interface (CLI) tool to interact with large language models like OpenAI's GPT-3. It allows you to query these models directly from the terminal, making it easy to integrate AI capabilities into your workflow or automate tasks.
### 2. ttok
`ttok` is a CLI tool for working with tokenizers. It's a tool under development aimed at helping users understand how tokenization processes text, which is essential for preparing data for machine learning models, especially in NLP applications.
### 3. symbex
`Symbex` is a CLI tool for performing symbolic execution on Python bytecode. It can analyze Python code to find potential bugs, vulnerabilities, or logical errors by exploring all possible execution paths.
llm -m 4o "Tell me about these cli tools and link to them:
- llm:
- ttok
- symbex
- shot-scraper
"
Sure, here is a brief overview of each CLI tool along with links to their respective repositories or documentation:
1. *llm (Language Model CLI)*
- *Description*: The `llm` command-line tool leverages large language models, such as OpenAI's GPT-3, to make it easier to incorporate AI functionalities into your command-line tasks. You can use it to generate text, answer questions, and assist with coding or other language-based tasks directly from your terminal.
- **Link**: [llm on GitHub](https://github.com/simonw/llm)
2. *ttok*
- *Description*: `ttok` is a simple command-line utility that provides tokenization for text, particularly useful for breaking down input for language models. It helps to preprocess text, count tokens, or manage tokens more efficiently from the command line.
- **Link**: [ttok on GitHub](https://github.com/simonw/ttok)
3. *symbex (Symbolic Execution)*
- *Description*: `symbex` is a CLI tool designed for searching codebases. It employs symbolic execution to analyze code paths and find functions, variables, or other code elements. This can be particularly useful for developers looking for higher-precision searches in large codebases.
- **Link**: [symbex on GitHub](https://github.com/paulgb/symbex)
(Truncated)
Clearly, openai did a lot more web scraping for this model.
Cute, but can I see sources on the data? I tried some query on some belgian statistics and it gave me just... very weird numbers that i do not trust, but zero way to check them.
Unless it's very "public common", otherwise, the numbers can be used for trustworthy work. In general, I think this is just for "fun play" most of the time.
This is a great point. It's easy to spin up an LLM data analysis demo, but it's really hard to trust the outputs.
I think we've (https://www.definite.app/) done a decent job here. Once the LLM generates results, you can inspect the query all the way back to the source (SQL).
Yeah I just tried it with numbers I knew a bit and it seems totally made up. The generated chart showed a linear downwards trend while in reality there isn't one and the numbers seem way off.
'daily unique visitors to openai.com by month since 2022 to 2024'
It gave a graph with
1) time axis decreasing left to right,
2) visitor numbers which can't be real (near prefect linear trend)
3) points in the future, going out to Dec 31, 2024.
Inspired by that I tried to ask it for “Price of bitcoin from year 1900 to year 2300”
I had to press the “refine” button several times which turned the query into “Bitcoin price 1900-2300”
The result was a bar chart with five data points.
All of the five data points were labeled “2K”, and had differing heights.
The dates along the X axis were in a short span of time – just a few days – and were seemingly randomly ordered. Or like, sorted with even dates in descending order followed by odd dates in ascending order. Very.. peculiar.
2023-10-04
2023-10-02
2023-10-01
2023-10-03
2023-10-05
Also the link copy button is not working for me in Safari on iOS, and the url doesn’t change from the op link either, so I can’t share a link to the graph.
The graph in that page is completely different (e.g. the sharp drop in 2015). This seems to be just linearly interpolating between start and end populations.
So it's a pretty simple wrapper of LLM model in use (currently gpt-4o), it does not add much technical stuff in it.
It does not use database for any "random search", but yes, columns.ai is a data analytics tool that allows you to connect supported live data sources like Google Spreadsheet, Airtable, Notion Database to create visual stories.
The analytics engine is home built (https://github.com/varchar-io/nebula) but it is not a database. And I don't use LLM agents, just build logic how to purify data returned by LLM, and fit them into an optimized visualization.
Nice one, though I can't scroll the list horizontally thus can't read columns beyond screen bounds (iOS 17 Safari, tried "best ingredients that go with bacon" query and the result was a table view)
I can't believe there are no sources for the data. A few charts I tried seem off, but even in the cases where it looks like it might be right, trusting it without a source is a big no.
Trusting is a problem, to be honest, mostly for fun. If you want to trustable source, feel free to copy text paragraph and let it parse for you, there is another mode for that.
How is this in any way different than an RNG? The numbers are completely made up, does slapping an AI label on a RNG somehow make this unique or interesting?
I get that free will means you can do evil. But, given the choice, why would you not only do it this baldly, but then go out of your way to show it off?
I think what the world really needed right now was an LLM making up fake statistics, presented in a convincing high quality way.
There is just no way this could ever be a problem. Surely everybody knows that if presented with some data they need to do a deep dive into the actual sources instead of blindly trusting a graph.
I seriously want to know what was going on in the mind of the person who made this.
I still have a faint hope that proliferation of AI generated content will make it all so absurd that people will turn back to traditional editorially reviewed content. But I know that's probably too naive - I recently had someone bring up "here's what Chatgpt told me" as an argument in a discussion, in manner like Wikipedia was quoted before
It also made sense that being able to quickly look up evidence online should have ended nonsense conspiracy theories… it did not quite work out like that.
This might be a fun project, but it's going to cause extreme misunderstanding for many people. People trust graphs, and users of this site are going to unwittingly spread falsehoods to for example r/dataisbeautiful or even less reliable social media pages.
I'd urge you to take this site down as it will be net negative to the world.
The correct data, collected from the central statistical bureau, is almost the opposite, with the winter as the most wind time (thankfully, as that's when Sweden needs the power) https://imgur.com/a/OX1608V
The political bias in the results is astonishing and shows why these models should not be used for educational purposes. Just ask it a "contentious" question and notice it give biased and sometimes nonsensical results - 'crime rate by political orientation' shows three 'republican' states coming out on top with three 'democratic' states filling out the bottom. When necessary it just seems to make up data to get the desired results, e.g. 'murder rate by political orientation' talks about 'Country A´ and 'Country B' with, of course, 'right' being far more murderous than 'left'. It claims that 'democratic New York' has the lowest crime rate. 'IQ by political party' is another interesting example. Compare this to 'trust in LLM output by political party' and maybe those 'dumb republicans´ (who do not trust these models, together with the supposedly super-smart independents by the way) suddenly seem to be a lot smarter than all those 'bright "democrats"'.
This site is a harbinger of the agitprop factories which are about to flood the 'net due to the general availability of LLMs.
To the downvoters: instead of just pressing that downvote button on things which you do not agree with it would be beneficial for the discourse if you laid out your disagreement.
> After trying some queries, I found ChatGPT is good at finding public data, sometimes it may not be accurate, but overall, pretty good.
GPT-4o is no more reliable for this than GPT-4 or GPT-3.5 before it.
One of the many unintuitive things about LLMs is that they are terrible tools for looking up specific facts about the world - which is weird, because that's traditionally one of the things computers have been best at.
If you want to build a working version of a system like this the trick would be to provide it access to tools for looking up data in citable, reliable data sources - things like the CIA World Factbook.
Then display the charts along with a link to that underlying reliable data.