Agreed. I currently hace a 4.3" phone which feels like it is pushing the limits for me. That said, I will be picking up a Nexus 4- because the phone buttons are now on-screen instead of hardware the 4.7" figure is misleading. But I think a 4" phone is the ideal for me, sadly no Android manufacturer is interested in it making a phone that is both powerful and a reasonable size.
The Razr I (or Razr M, depending where you are) is worth a look if form factor is an issue. The Intel processor in the Razr I isn't high-end, but I've not had any compatibility issues and games (Minecraft, GTA III, World of Goo) run fine - not to mention great battery life. Although I would personally go for the Nexus 4 if I hadn't already bought one of these - simply because it has great specs, is "pure Google" and is so darn cheap.