The authors claim that LLM are reducing public knowledge sharing and that the effect is not merely displacing duplicate, low-quality, or beginner-level content.
However their claim is weak and the effect is not quite as sensational as they make it sound.
First, they only present Figure 3 and not regression results for their suggested tests of LLMs being substitutes of bad quality posts. In contrast, they report tests for their random qualification by user experience (where someone is experienced if they posted 10 times). Now, why would they omit tests by post quality but show results by a random bucketing of user “experience”?
Second, their own Figure 3 “shows” a change in trends for good and neutral questions. Good questions were downtrending and now they are flat, and neutral questions (arguably the noise) went from an uptrend to flat. Bad question continue to go down, no visible change in the trend. This suggests the opposite, ie that LLMs are in fact substituting bad quality content.
I feel the conclusion needed a stronger statement and research doesn’t reward meticulous but unsurprising results. Hence the sensational title and the somewhat redacted results.
What if LLMs are effective enough at assisting coders that they're spending less time on SO and instead pushing more open source code, which is more valuable for everyone?
While this article doesn’t really seem to be hitting what I am about to say, I think someone on HN a while back described a related phenomenon (which leads to the same issue) really well. The Internet is Balkanizing. This is hardly a new concept but they were drilling down specifically into online communities.
People are electing to not freely share information on public forums like they used to. They are retreating into discord and other services where they can put down motes and raise the draw bridges. And who can blame them? So many forums and social media sites and forums are engaging in increasingly hostile design and monetization processes, AI/LLM’s are crawling everywhere vacuuming up everything then putting them behind paywalls and ruining the original sources’ abilities to be found in search, algorithms designed to create engagement foster vitriol and controversy, the list goes on. HN is a rare exception these days.
So what happens? A bunch of people with niche interests or knowledge sets congregate into private communities and only talk to each other. Which makes it harder for new people to join. It’s a sad state of affairs if you ask me.
Yes, it's sad. On the other hand, I think it's a good thing that people share knowledge less, publicly and free of charge on the web, because there is so much exploitation going on. Big corporations obviously capitalize on the good will of people with their LLMs, but there are also others who take advantage of the ones who want to help. A lot of users seemingly expect others to solve their problems for free and don't even put any effort into asking their questions. It's a massive drain for energy and enthusiasm, some even suffer from burnout (I assume more in open-source projects than on SO but still). I rather want it to be harder to connect with people sharing the same passion "in private" than having outsider who don't contribute anything profit off of activities happening in the open. This frustratingly appears to become the main reason for corporate open source these days.
If the inference is drawn from the number of claims the first question that comes to mind is, how easy is it to file a claim with Waymo?
Also, do people behave differently when involved in an accident with a driverless car?
For example, at McDonalds, the automatic checkouts don’t simply substitute expensive workforce but actually boost sales of addons because people are less inhibited by a screen than by a human.
So, do people just drive off if it was a minor bump?
Finally, does Waymo initiate claims? If it were me, I wouldn’t. Id simply partner with a mechanic and fix issues internally as part of a fleet maintenance system.
If that’s the case, I’d roughly half the stats before doing any additional inference.
I agree there is another side of the medal, if that’s what you mean. In fact, if we accept that innovation is also the recombination of existing knowledge, then AI will help up discover entire fractals of new stuff.
Good candidates for what I am looking are OpenAI Playground (once it supports versioning and variables for chat), PromptLayer (once it supports chat prompts), EveryPrompt (chat prompts missing). More dev oriented, LangChain https://js.langchain.com/docs/getting-started/guide-chat which can then be easily versioned.
Thanks for pointing towards the right direction. I'll edit the original question.
To rephrase, I am looking for a tool to do model lifecycle management https://github.com/kelvins/awesome-mlops#model-lifecycle and wonder if there is any one in particular that you'd think is better suited for prompts, i.e. an array of objects with templated text
However their claim is weak and the effect is not quite as sensational as they make it sound.
First, they only present Figure 3 and not regression results for their suggested tests of LLMs being substitutes of bad quality posts. In contrast, they report tests for their random qualification by user experience (where someone is experienced if they posted 10 times). Now, why would they omit tests by post quality but show results by a random bucketing of user “experience”?
Second, their own Figure 3 “shows” a change in trends for good and neutral questions. Good questions were downtrending and now they are flat, and neutral questions (arguably the noise) went from an uptrend to flat. Bad question continue to go down, no visible change in the trend. This suggests the opposite, ie that LLMs are in fact substituting bad quality content.
I feel the conclusion needed a stronger statement and research doesn’t reward meticulous but unsurprising results. Hence the sensational title and the somewhat redacted results.
reply