As phones got faster, the situation improved, until a larger screen was added, which brought things back down again as the CPU burns to keep up. The milliseconds latency here and the stuttering FPS there decreases the usability significantly in my opinion.
I think that there is a group of users who don't notice these latency and animation issues. But I do, and I think many users still unconsciously attribute their dislike for Android to these issues. (example: my brother in law, used several Android Froyo phones extensively before settling on a BlackBerry because it felt "faster".)
My EVO 4G gets about 24 hours battery life, with moderate usage during the day. (But again, this is because I don't use HTC's build. With their build, the battery life is barely long enough to boot the phone.)
Responsiveness and framerate-wise, even a simple cursory examination will tell you: WP7 and iOS run laps around poor old Android. I got to play with a Nexus S recently, and the responsiveness is greatly improved over Froyo on N1... but is still noticeably less smooth than either Apple or MS's platforms.
It's an odd day when MS's UX is more polished than somebody else's...
One area where Android's lacklustre performance really hurts usability is diving into menus. iPhone came up with the novel idea of sliding menus - a simple animation that communicates a lot of context to the user (A belongs under B triggers modal window C, all based solely on the direction of the swipe animation). On Froyo+N1 this animation is jerky enough, frequently enough, that this relationship can easily be missed, resulting in more user confusion and worse UI comprehension overall.
Google needs to realize that we are in the year 2011, not 1995. "UI design" is no longer about placing the right buttons in the right places, and animation is often now the defining line between an easily comprehensible user experience and an obtuse one. IMHO graphics/animation performance is the #1 obstacle in Android's way, and the main advantage everyone else seems to have on it (and it's a huge advantage).
When I use a 2.2 phone, my impression is one of annoyance. I don't know if my touches will be detected, and even if they are, I don't know how long I'll have to wait for the phone to respond. Heck, I enjoy using WP7 phones more than Android because of the performance, despite the fact that WP7 is significantly less featureful than Android! Navigating WP7 is a pleasure, and it never feels like the phone stands in your way. I have yet to get this feeling from Android.
Also, is it really fair to compare the N1, a phone that's over a year old, to brand-new WP7 phones?
I don't have any problems with responsiveness or battery life.
As for battery life, it depends entirely on where you live. I experimented on it for months before finally figuring out that well over 50% of the power draw on the system is the data network (mobile network option when you hold down power). If you have it on when you have really weak/no reception, the phone will burn through the full battery in about 4 hours and it's pretty simple to lose 20% battery in 15 minutes. If you always have a decent connection, your battery life will be fine, if you happen to live in a place where you have holes in your coverage (e.g. NYC urban canyons), battery life is terrible.
After realizing that, I set up a set of Locale rules to shut down the network when I'm on the wireless network at home/work and to shut it down when I lose signal. Combined with the undervolted kernel, I normally get 48 hours on a charge for my normal use (~350MB/mo).
Google has done a lot of cool stuff for developers in the last few iterations of Android. The NDK has evolved into something quite decent, definitely good enough for writing games now. But responsiveness is the major user experience issue that they've completely failed to address.