This pisses me off too. It's why I don't give money to my Sports Franchise undergrad.
I remember my mother speaking to some college admissions person at a semi-prestigious school in Pennsylvania, many years back when I was in the throws of applying. When speaking to some admissions confusion with respect to my high school, the guy said, "Well frankly, it's not called the Ivy League because of its academics." That's always stuck with me. Athletic scholarships? You betcha. Academic scholarships? What are those?
As for sports having an impact... They may not let in a 900 SAT football player, but 1200 may be enough for an Olympic rower.
What are you talking about? Ivy League schools do not offer athletic money and all (now, not when I was applying) are need-blind, giving tons of money to those who need it.