Can anyone comment on the issue of hallucinations? The author only mentions them briefly and I cannot gather how big of a problem this is. Apart from the literal quote the LLM hallucinated, wouldn’t all the other information have to be double-checked as well?
IMO, hallucinations make it basically unusable for things it should be very good at. For example, I have asked two different AIs what the option is for changing the block size with HashBackup (I'm the author). This is clearly documented in many places on the HashBackup site.
The first time, the AI said to use the -b option to the backup program, with examples etc. But there is no -b option, and never has been.
The 2nd time, the AI said to set the blocksize ("or something similar"; WTF good is it to say that?) parameter in hashbackup.conf. But there has never has been a hashbackup.conf file.
From examples I've seen, AI tends to do a passable job spewing a long-winded response where asking several different humans would give similar long-winded responses that contained a lot of judgement and opinions, some of which could be valid or not.
I'll echo this and say that I've run into very similar issues when evaluating local LLMs as the author of a popular-ish .NET package for Shopify's API. They almost always spit out things that look correct but don't actually work, either because they're using incorrect parameters or they've just made up classes/API calls out of whole cloth.
But if I set aside my own hubris and assume that my documentation just sucks (it does) or that the LLM just gets confused with the documentation for Shopify's official JS package, my favorite method for testing LLMs is to ask them something about F#. They fall flat on their faces with this language and will fabricate the most grandiose code you've ever seen if you give them half a chance.
Even ChatGPT using GPT4 gets things wrong here, such as when I asked it about covariant types in F# a couple days ago. It made up an entire spiel about covariance, complete with example code and a "+" operator that could supposedly enable covariance. It was a flat out hallucination as far as I can tell.
Yes, this. If the form of a plausible answer is known it is likely to be invented. API method names, fields in structures, options to command lines, plausible inventions that have a known form.
Similarly references of any kind really that have a known form, like case law, literature, science, even URLs.
I’ve had very similar problems asking technical things. I wish it would do like humans do, and say “not sure have you tried an option that might be called Foo?”. a good human tech repi doesn’t always have the precise answer, and knows it. unfortunately LLMs have mostly been trained on text which isn’t as likely to have these kinds of clues that the info might not be as accurate as you’d like.
I’ve found for technical things, I’m happier with the results if I’m using it as clues to getting the right answer, and not looking for an exact string to copy and paste.
There are many recently published or preprint research papers around that, not necessarily that hard to read I think.
As a consultant this totally prevents me from making any professional use of LLMs at the moment (edit: aside from actual creative work, but then you may hit the copyright issues).
But even without hallucination, using non scholar sources for training is also a problem, Wikipedia is great for common knowledge but become harmful at a certain point where you need nuanced and precise expertise.
The other problem of Wikipedia is it being a target of hostile politically motivated attacks that attempt to rewrite the history. It it will normally self-correct but time to time there are pieces of information that are maliciously incorrect.
Apparently this was an architectural fad in the middle ages: rich families built "skyscrapers" to show their wealth and look out for enemies. At one point, Bologna had more than 100 of these towers [1]. Today, the small town of San Gimignano [2] is well known for these structures which do not seem to have a name in English (in German, they are called "Geschlechtertürme" and have their own Wikipedia [3] entry).
Also, I encourage everyone to visit Bologna, it's an amazing city!
[1] this is a figure you will often read, but it seems to be a bit inflated - apparently it's double counting towers from different time periods
I highly recommend Bethany McLean’s "The Smartest Guys in the Room" to read up on Enron. Also, Lucy Prebble - who is now a consultant on HBO‘s succession - has written a nice play on the matter.
Sgitr is a great book and gives a real appreciation of how Ernon could have happened and will happen again. The perfect storm of hands off management and belief that ideas are more important than execution, plus the usual greed and slippery slope of covering small frauds with increasingly bigger ones. Other than Skilling and Fastow, they weren't all bad guys, I hope I'm never caught in a situation like that.
Also interesting is that sgitr is almost identical to "Bad Blood" by John Carreyrou (sp?) about Theranos. Most of the same elements were present in that fraud.
