> since anyone using them in such a serious manner knows they are quite unpredictable
"anyone" is doing a lot of work there, also what is "a serious manner"? There are plenty of people who use it quite a lot and end up in a mental health crisis, sometimes from trying to use it to solve a mental health crisis; or find themselves in something equivalent to people who fall down conspiracy rabbit holes and all the fallout that causes in their lives.
You may understand why it is not a good idea to use them that way, I may too, but I think you underestimate the general populations (outside HN) misunderstanding of LLMs, especially in light of the marketing stunts these companies themselves pull.
"anyone" is doing a lot of work there, also what is "a serious manner"? There are plenty of people who use it quite a lot and end up in a mental health crisis, sometimes from trying to use it to solve a mental health crisis; or find themselves in something equivalent to people who fall down conspiracy rabbit holes and all the fallout that causes in their lives.
You may understand why it is not a good idea to use them that way, I may too, but I think you underestimate the general populations (outside HN) misunderstanding of LLMs, especially in light of the marketing stunts these companies themselves pull.