To me, it was the UI and magnitudes of toolbars that would come with every addon (even the useful ones) that would then need manual disabling, and the need to restart browser over and over again (at times due to new addons being installed, disabled/enabled or browser updates) that pushed me to Chrome.
This isn't a problem anymore, however, and hasn't been for a long time - at least, not unless you really try.
Both chrome and firefox have excellent session restore, so even if you need to update (which you usually don't), you won't really notice. Even session cookies are properly restored, as are partially filled forms, though some pages' scripts cause the form restoration to fail.
Toolbars have always been rare, and they certainly are now; it's probably possible to still install a toolbar extension, but I can't remember the last time I did. That may be more a change in typical extension style than in the actual browser, however.
