We should pivot the culture to one that is pro-liberal arts again. These people know how to read and write better than STEMs in general.
CS as the only path to programming was always too narrow, and often people with a broader education are better at creative solutions. With AI-assisted programming I'd argue they have an even clearer advantage now.
I’ve always noticed that truly excellent programmers with strong lexical instincts also tend to be formidable with their native language. Not necessarily verbose, but capable of clear and structured writing.
I’d even go so far as to argue that if someone has poor writing skills in their native language, they’re probably not a very good programmer.
In theory, it isn't that difficult, in practice writing _accessible_ text takes a lot of practice and feedback. Letting go of your own biases towards base levels (curse of knowledge) is something that already trips most people and what people find really difficult to overcome.
Which is why the statement you are responding to is often more true than you might realize. Because these people have had a lot more practice in that specific area. Although not all of them, that would be a generalization in itself.
Some much focus on fine-tuning when it can actively make performance on reasoning and planning benchmarks worse (over a baseline of already worse-than-coin-toss).
Why not give us nice things for integrating with knowledge graphs and rules engines pretty please?
I know the article title says "integration tests" but when a lot of functionality is done inside PostgreSQL then you can cover a lot of the test pyramid with unit tests directly in the DB as well.
The test database orchestration from the article pairs really well with pgTAP for isolation.
Kind of a tangent but I'm really interested in why statements like:
> if you have carbon-based life forms, you will have water and CO2.
..can lead to statements like:
> it is just way more likely than any other form
I totally agree on the observation, but what is fascinating to me is why a deductive statement can be considered to indicate likelihood in probability. It seems there is a bit of abductive reasoning going on behind the scenes which neither the deductive logic or inductive probability can really capture on their own.
I don't see that particular statement "leading" to the second statement. You quoted very selectively and didn't quote the part that is the reason why I believe carbon-based to be much more likely, i.e. the fact that silicon is such a bad candidate and that no other molecule allows complex chemistry. Maybe that helps with your fascination. Obviously I skipped all the actual arguments, but they are easy to find by the interested reader in standard literature.
There's a third kind, which is when unscrupulous business managers or politicians use it to make decisions that they would not be capable of auditing for a rationale when otherwise required to know why such a decision was made.
It's more of an ethics and compliance issue with the cost of BS and plausible deniability going to zero. As usual, it's what humans do with technology that has good or bad consequences. The tech itself is fairly close to neutral as long as training data wasn't chosen specifically to contain illegal substance or by way of copyright infringement (which isn't even the tech, it's the product).
$10k is nothing for rich people. Just putting them in debt for it doesn't mean nothing of value was provided.
The reverse would be true where poor people could be ruined, unless the value provided is worth significantly more than the debt created, which seems doubtful.
There is a lot of synthetic content about academic subjects on YT now, and it's very low quality. I used to search for lectures to listen to while walking or driving but now need to wade through tons of enshittified spam. Even if it's reading wikipedia or other long form articles, the voices and graphics are bad.
Actually I paid for Blinkist recently and really enjoyed it at first. They have a lot of "blinks" that state at the end that the voice was synthetic and I was legitimately surprised at the quality, having not even noticed until they told me.
This seems like a good move for YT to maintain a basic level of quality (which I'm amazed can actually get worse), but I suspect it's a pretext to avoid paying out to "illegitimate creators" for commercial reasons in a way that makes them look like they care about people.
After using RAG with pgvector for the last few months with temperature 0, it's been pretty great with very little hallucination.
The small context window is the limiting factor.
In principle, I don't see the difference between a bunch of fine-tuned prompts along the lines of "here is another context section: <~4k-n tokens of the corpus>", which is the same as what it looks like in a RAG prompt anyway.
Maybe the distinction of whether it is for "tone" or "context" is based on the role of the given prompts and not restricted by the fine-tuning process itself?
In theory, fine-tuning it on ~100k tokens like that would allow for better inference, even with the RAG prompt that includes a few sections from the same corpus. It would prevent issues where the vector search results are too thin despite their high similarity. E.g. picking out one or two sections of a book which is actually really long.
For example, I've seen some folks use arbitrary chunking of tokens in batches of 1k or so as an easy config for implementation, but that totally breaks the semantic meaning of longer paragraphs, and those paragraphs might not come back grouped together from the vector search. My approach there has been manual curation of sections allowing variations from 50 to 3k tokens to get the chunks to be more natural. It has worked well but I could still see having the whole corpus fine-tuned as extra insurance against losing context.
It's not impossible that fine-tuning would also help RAG. but it's certainly not guaranteed and hard to control. Fine-tuning essentially changes the weights of the model, and might result in other, potentially negative outcome, like loss of other knowledge of capabilities of the resulting fine-tuned LLM.
Other considerations:
(A) would you fine-tune daily? weekly? as data changes?
(B) Cost and availability of GPUs (there's a current shortage)
My experience is that RAG is the way to go, at least right now.
But you have to make sure your retrieval engine work optimally: getting the very most relevant pieces of text from your data: (1) using a good chunking strategy that's better than arbitrary 1K or 2K chars (2) using a good embedding model (3) Using hybrid search, and a few other things like that.
Certainly the availability of longer sequence models is a big help
It's never the technology that's the problem, it's the owners and operators who decide how to use it.