But I have a friend who is in his late 40s who is coming up to speed in Number Theory, and doing well. But he is serious, and is putting in the time.
With age comes maturity, that can help, but also a more set way of thinking, which doesn't. Also, it depends on what kind of mathematics you want to learn ...
... and why.
My advice is to find a place/community where you can ask "stupid" questions without being judged too much. For this kind of questions, every once in a while, I like to ask on usenet, on the sci.math newsgroup.
* What do you want to study?
* How much time can you dedicate to it?
* How will you study - will you have mentors?