That fact alone--that, out of all of the characters, Tom alone has power of the Ring--is enough to captivate the reader's attention. Add to that the whole business of being "Oldest" and "Fatherless", yet being neither a Vala, nor a Maia, nor even Eru Iluvatar, is enough to make him worthy of deep scrutiny, research, conversation, and speculation.
Actually, if I remember right, it was discussed (briefly) giving the ring to Tom. But Gandalf said the ring would eventually consume Tom in the end. I always wondered just how Gandalf knew this. Given that he confessed not really knowing what Tom was.
I always thought Tom was sort of "outside reality". Maybe Tom was the physical representation of Eru itself. Or maybe some aspect of Eru.
> Actually, if I remember right, it was discussed (briefly) giving the ring to Tom. But Gandalf said the ring would eventually consume Tom in the end.
You don't remember right. :)
I think what you are remembering is when the Council discusses leaving the Ring with Bombadil, at the end of which they conclude that it wouldn't work; even if you could make him accept, he would lose it, or forget the necessity and importance of it, and even if he didn't, then all of Middle-Earth would fall to Sauron (who can torture even the hills), last of all Tom: Last as he was First. There is no suggestion that the Ring would ever afflict Tom.
> "Could [Sauron's] power be defied by Bombadil alone? I think not. I think that in the end, if all else is conquered, Bombadil will fall, Last as he was First; and then Night will come."
I doubt his popularity was the issue. Time was. Tom is singular. He doesn't appear anywhere else in the story. And he's only mentioned in conversation. Easy to edit him out. And save precious time for the film.
Good film adaptations of novels, especially books as rich in backstory as LOTR, almost always have to be simplified even if it does drive a lot of fans crazy. And, of course, even well-regarded and popular novels might well have benefited from tighter editing. Not saying this is the case with Tom Bombadil in LOTR, but it's probably true of back half of Return of the King. Even with the scouring of the Shire cut, the film version of Return of the King really went on too long after the climax.
For me it wasn't so much him as all of the songs IIRC. Once I started just skimming the songs/poetry he became much less annoying (I'm horrible at deciphering meter and it just trips me up). The first words out of my mouth upon seeing the first film was "they left out Bombadil".
Had been looking forward to hearing one of those songs.
I didn't really like him either. He clashed with the overall feeling of Fellowship while at the same time breaking its logic. If he just clashed, he'd be comic relief. If he just broke the logic, he'd be a mystery wrapped in an enigma. But by being both at the same time, he becomes the big-lipped alligator moment of the whole series.
I don't see any reason to think the author dislikes Bombadil! If anything, they'd have to like him quite a bit to research his background and come up with this sort of wild theory about him.