They’re comparing LLM output to professional creative writers? That seems like a high bar to start with. I’d think that would be the highest bar they could achieve. How does it compare though to your average college student?
Also I feel like I haven’t heard people say that much about their ability to be creative. It’s more about their ability to generate human like content — and most human content isn’t that creative either.
Yes, it's a testament to how powerful these models have become in a such a short time that we've gone from being impressed that they can write a coherent paragraph to being critical that they can't win a professional writing competition, all in the span of 5 years.
> They’re comparing LLM output to professional creative writers?
I mean that seems a fair experiment to judge based on the rhetoric employed by AI enthusiasts, who talk of things like "democratizing art" (as though art as it exists is somehow anti-democratic?) and letting any layman slam some text into a prompt and get a work of shakespeare, a painting by Da Vinci, or a Mozart symphony back.
Yeah GPT-4, the current Sota isn't generally professional/expert level in most domains so this isn't exactly surprising revelation even if true.
This paper also suffers from the "single pass problem". LLMs aren't that new anymore. Most researchers should know that whatever you get on a single pass can be/is often much lower quality than if taking the effort to be more specific, provide examples or breaking the task down and going through multiple passes.
From my experience getting several LLMs to write, this doesn't hold any less true in the domain of creative writing so there's a very good chance this paper severely underestimates the potential of even current LLMs.
Also I feel like I haven’t heard people say that much about their ability to be creative. It’s more about their ability to generate human like content — and most human content isn’t that creative either.