Not if you are securing your email with a security key and enforcing that you can only access it from devices that meet security guidelines (as is easy to do with Office 365 and almost easy to do in GSuite). Unlike SMS, where you could lock down your phone and take every precaution and an attacker could still compromise your SIM via your carrier.
