It is not a logical fallacy nor ad hominem to take into account the number of times someone has been incorrect about a specific subject matter in order to estimate how much one should weigh the probability of said individual being correct on future related matters. That is, unless one has complete and correct information about the future.
Nope, that's a logical fallacy. Lots of people do it, yes. I'm guilty of it myself. But that doesn't mean that it's correct.
http://en.wikipedia.org/wiki/Ad_hominem