I think it is important that the designer gets a feel for the character and “vibe” of the book by actually reading the manuscript and gathering as much context information as they can, ideally in direct exchange with the author(s) to align with their artistic vision. Book cover design can be an art in itself and as a reader, I appreciate it if it is done well and communicates some of the subtleties and nuances of the content in an interesting way. Of course, as you mentioned, it should not be about the ego/self-marketing of the designer, but that does not exclude artistic expression.
I believe it is a disservice to the author and the reader to focus solely on the marketing side of things. After all, the design of the book is part of the product, which, as a whole, sets the frame for the reading experience. Physical books today have to compete with digital content and ebooks, so as a reader I would expect that more effort is being made to make a product that also looks and feels good and not only carries the content to my eyes. That also applies to the layout and typesetting of the inner pages, which is unfortunately often careless and not aligned with the cover design (let alone the choice of paper and binding, which is of course seldom in control of the designer).
Physical books compete with digital versions - that would be a reason for LESS budget for the paper cover. Some has to be devoted to the digital marketing.
Some authors have the luxury of going all in on the artistic. That does exist. For most though, the book needs to sell or there won't be a next book. Constraints like that are what a marketing or design brief is supposed to explain. It's stuff the publisher understands long before they call in the cover artist or layout designer.
I'm with you on the surprising choices we often encounter. But then these problems were common when paper was king. The distractions of the move to digital won't help. For example, far too little margin next to a stiff binding. For example, ebooks with ludicrous epub rendering (on book readers that don't sit on the publisher's desk). Puzzling.
I believe it is a disservice to the author and the reader to focus solely on the marketing side of things. After all, the design of the book is part of the product, which, as a whole, sets the frame for the reading experience. Physical books today have to compete with digital content and ebooks, so as a reader I would expect that more effort is being made to make a product that also looks and feels good and not only carries the content to my eyes. That also applies to the layout and typesetting of the inner pages, which is unfortunately often careless and not aligned with the cover design (let alone the choice of paper and binding, which is of course seldom in control of the designer).