It's true the SRBs were powerful, but I wonder how much of that was compensated by structural elements needed to live with their vibrations. I remember the original Ares I concept would vibrate the astronauts to death before the solid 1st stage burned out.
https://en.wikipedia.org/wiki/File:Srbthrust2.svg