And yet J.K. Rowling became one of the richest women in England by writing a book.
I think it's not so much that the market for fiction is shrinking, I think it's that it's becoming more hit-driven. It used to be that reading was a solitary pursuit, and you checked a book out because it looked interesting on a library shelf (I still do this, but I'm one of the few folks I know of that do).
Now, most people read books because their friends do, and they want to read what their friends are reading. So popular books become more popular, while unpopular books get no sales or readership. This also increases the drive towards universally-applicable themes (a book that flies over the heads of 99% of the population isn't going to get word-of-mouth), which is why you see them as being less challenging.
Same goes for games. When I first started playing MM0RPGs in 1993, it was impossible to find an average person on the street that played them. Now, half my friends are addicted to WoW. Video games are a social enterprise, so they need to be a lowest-common-denominator. If you want text-only RPGs or Civilization 4, they're still out there. But if you want to play with your friends, you have to put up with your friends' varying skill levels.
And yet J.K. Rowling became one of the richest women in England by writing a book.
More specifically, a children's book. Written engagingly, but with little by way of moral ambiguity or challenge to the reader. And yet this book became suddenly popular, and took over the adult charts as well. If there were ever a literary example of Guitar Hero over learning to play an instrument, this is it.
If you ask me, your example is evidence in favour of DaniFong's argument rather than against it.
And yet. And yet Harry Potter says that Voldemort isn't the only evil, that the Ministry that Harry's expecting to take care of him is just as awful in that it hides facts to raise morale. And yet, part of the moral of the book is that if you fight for what's right, there's a chance you're killed for your beliefs. And yet, the father-figure is revealed in the end to be a racist and a bigot and partly a coward. Some of the villains go unpunished. Some innocent people are tortured and killed. The moral of Harry Potter might be that good wins in the end, yeah, but it absolutely offers ambiguity. Never with Harry, but with other people. And that's fine, literarily speaking: it makes Harry and Voldemort into foils, into pure good and pure evil, and from there you can debate the exact nature of good and evil. The story's deeper than it lets on because it's so well written (for the most part) that you can ignore a lot of what's going on because it puts itself very subtly into the background.
Furthermore, it ended a lot of the publication limits on what constitutes kid's literature. Suddenly the stories don't have to all end happily ever after, lengthy series with big books are acceptable, and heavily political and religious themes are allowed. That's big: it let kid's lit writers have a lot more respect for their work.
And yet. And yet Harry Potter says that Voldemort isn't the only evil, that the Ministry...
I almost put this point into my original comment, because I anticipated this response:
Go back and read the first two books. They contain no hints of this sort of thing. It was only after the series became a runaway success that Rowling started to beef up the storyline with such "adult" topics. (And even these themes are fairly bolted-on to a world still fundamentally dominated by Absolute Good and Absolute Evil - or, at least, they started out that way).
My argument is that the introduction of these themes, and the retconning of a fairly simplistic world into something more ambiguous, resemble the rereleasing of the original books with "adult covers". They were an after-the-fact response to the way that a children's story became massively popular in the adult world.
(Which isn't to say that the books aren't engagingly written, or fun to read. And video games are fun to play - I'm with Randall Munroe on people who look down on those having fun just because they are insufficiently highbrow. But I simply disagree with your argument that the popularity of the original Harry Potter books represents the rise of challenging mental stimulation.)
The Harry Potter books are deliberately designed to have a more "adult" outlook as they go on because Rowling's core audience was, literally, growing into adolescence and young adulthood as the books came out. The books were published over ten years. Anyone who read the first one at age nine would be nineteen when the last one came out.
Is this strategy blatantly commercial? Maybe. The first rule of being a professional writer (as opposed to a self-published wannabe whose Great Novel is found, posthumously, in a desk drawer) is to write stuff that people want to buy. If that's a sin, most pros would go to hell with happiness, as opposed to remaining stuck in their day jobs.
Commercial or not, Rowling's change of tone has a perfectly good literary rationale, because her main characters explicitly grow up as the books go on. The books are told from those characters' point of view. At the beginning, they're kids, and they see the world in a kidlike way. At the end, they're adults, and they see an adult world. The consistent yet evolving sophistication of the point of view is one of the things that makes the writing strong.
The series was plotted out in full before the first book was published. Some things probably changed in the rewrite, but you can't suddenly take it in a different direction without ruining the coherency of the story. (Just look what happened with the Wheel of Time, or A Song of Ice and Fire, or Heroes Season 2).
There're also little details in the prologue of book 1 that only make sense in book 5/6/7, so it's clear that she had the later books in mind when she wrote the first one.
"This also increases the drive towards universally-applicable themes [...], which is why you see them as being less challenging."
I doubt this. The Sorrows of Young Werther is about unrequited love, after all. I think it is the emotional, moral, intellectual and psychological content of literature that challenges. Not the theme. Pride and prejudice is considered the epitome of the romance novel by many, after all. 55% of the novels in America are sold in the genre. And here's what the Romance Writers of America say about the genre:
"the main plot of a romance novel must revolve around the two people as they develop romantic love for each other and work to build a relationship together. Both the conflict and the climax of the novel should be directly related to that core theme of developing a romantic relationship, although the novel can also contain subplots that do not specifically relate to the main characters' romantic love. Furthermore, a romance novel must have an "emotionally satisfying and optimistic ending." Others, including Leslie Gelbman, a president of the Berkley Group, use a more shortened definition, that a romance must make the "romantic relationship between the hero and the heroine ... the core of the book." In general, romance novels reward characters who are good people and penalize those who are evil, and a couple who fights for and believes in their relationship will likely be rewarded with unconditional love." [1]
In this way, reality is sanitized for mass consumption.
I think it's not so much that the market for fiction is shrinking, I think it's that it's becoming more hit-driven. It used to be that reading was a solitary pursuit, and you checked a book out because it looked interesting on a library shelf (I still do this, but I'm one of the few folks I know of that do).
Now, most people read books because their friends do, and they want to read what their friends are reading. So popular books become more popular, while unpopular books get no sales or readership. This also increases the drive towards universally-applicable themes (a book that flies over the heads of 99% of the population isn't going to get word-of-mouth), which is why you see them as being less challenging.
Same goes for games. When I first started playing MM0RPGs in 1993, it was impossible to find an average person on the street that played them. Now, half my friends are addicted to WoW. Video games are a social enterprise, so they need to be a lowest-common-denominator. If you want text-only RPGs or Civilization 4, they're still out there. But if you want to play with your friends, you have to put up with your friends' varying skill levels.