Personally I can’t see how Brady is not a top athlete. It’s like judging a jazz musician on the skills needed in pop music or vice versa. You have to look at success within the genre or sport.
It seems like this is more about the semantics of what we mean by athleticism then?
It sounds like for you, being a top athlete simply means being very good at a sport.
I've always generally understood athleticism to be about raw physical traits, like speed, strength and agility (and is therefore only part of the range of attributes that makes up the overall profile of a sportsperson).
Out of interest would you consider people performing at an elite level in high-skill, relatively low-physicality sports like golf to be top athletes?