That's not a fact that the recipients are users that signed up for it. As someone with a first letter + last name @ gmail account, I can't tell you how many transactional emails I get for things I never signed up for. Try getting a bank to take your email address off of someone else's account—damn near impossible.
It is if you use an email verification step as part of your sign-up flow, which we do. At most, someone could sign up with someone else's address and generate a confirmation email, but it'd only be once per address and that's the price of an open Web.
Anyway, I'm aware that not everyone does this, but at a certain point there's not much an ESP can do, beyond which there's essentially trust and monitoring.