Those numbers are obviously bothering you but they are definetely not needed. Stop focusing on that. They are only one way, largely inherited from vi's ancestry, but there are so many other ways. I don't use counts much for the same reason as you (and I find that counting breaks my train of thought) but I use search and "advanced" motions a lot and quite naturally. The ability to jump to arbitrary positions in my code and directly work (without selection) on things like inner parenthesis and other text-objects have been extremely useful in my day to day work.
Once you are used to those text-objects and motions, using any other editor is a monumental pain in the ass, however advanced and modern they are.
But, as I alluded at the end of my comment, I don't mean to convince you or anyone of the superiority of Vim because I don't care: I have no interest in the success or failure of Vim. I use Vim and a couple of IDEs in my work and I'm totaly fine with you using and prefering other tools.