In smaller company you are closer to person with the decision power. It is much easier to build closer relationship and have high visibility to avoid being laid off.
While that is true, if any given person has some X% chance of being an asshole, the more layers of management there are above you, the higher the chances you are at the mercy of at least one (and possibly multiple) asshole(s).
You could use the exact same argument in the other direction: the more layers of management there are above you, the higher the chances a high-level managers will protect your team from armageddon. Sometimes, it's only a matter of a single email or a chat with a CxO.
I stand by my original point: no obvious correlation between company size and quality of life inside it.