As someone responsible for Cloud security for the last several years at a couple of companies, I can tell you exactly how this happens.
You, the sec-ops engineer, propose your pie-in the sky, compartmentalized fortress. There is no one security domain, but many, with barriers in between them, so that having one part of your org compromised leaves the rest intact.
Then, you talk to your peers, and have many discussions of the form, "yes, that's secure, but it's really annoying to work with, and my engineers won't be able to monkey-patch the production environment to test stuff". you say, that's the point, the CTO says, you can't slow down dev.
So, you come up with something that's as secure as you can make it within the limited ability you have to impose inconvenience. Then it comes to implementing the thing.
You don't have time to implement everything yourself, so you delegate. Some people now have credentials to the production systems, and to ease their own debugging, or deployment, spin up little helper bastion instances, so they don't have to use 2FA each time to use SSH or don't have to deal with limited-time SSH cert authorities, or whatever. They roll out your fairly secure design, and forget about the little bastion they've left hanging around, open to 0.0.0.0 with the default SSH private key every dev checks into git. So, any former employee can get into the bastion.
Now, that's what happens when someone designs something secure from the outset.
When you start out with everything being in the default VPC, initially brought up by hand, with a subnet setting which default-assigns public IP addresses, you're basically boned, but that's where most companies which roll service in the cloud start.
You, the sec-ops engineer, propose your pie-in the sky, compartmentalized fortress. There is no one security domain, but many, with barriers in between them, so that having one part of your org compromised leaves the rest intact.
Then, you talk to your peers, and have many discussions of the form, "yes, that's secure, but it's really annoying to work with, and my engineers won't be able to monkey-patch the production environment to test stuff". you say, that's the point, the CTO says, you can't slow down dev.
So, you come up with something that's as secure as you can make it within the limited ability you have to impose inconvenience. Then it comes to implementing the thing.
You don't have time to implement everything yourself, so you delegate. Some people now have credentials to the production systems, and to ease their own debugging, or deployment, spin up little helper bastion instances, so they don't have to use 2FA each time to use SSH or don't have to deal with limited-time SSH cert authorities, or whatever. They roll out your fairly secure design, and forget about the little bastion they've left hanging around, open to 0.0.0.0 with the default SSH private key every dev checks into git. So, any former employee can get into the bastion.
Now, that's what happens when someone designs something secure from the outset.
When you start out with everything being in the default VPC, initially brought up by hand, with a subnet setting which default-assigns public IP addresses, you're basically boned, but that's where most companies which roll service in the cloud start.