The CAN-SPAM law makes a clear distinction between “commercial electronic mail message” and “transactional or relationship
In most cases, if an email isn't commercial in nature, it's excluded from the CAN-SPAM requirements. Now, whether or not it annoys your users is another discussion...
One relevant excerpt:
"These requirements do not prohibit transmission of “transactional or relationship” content. Even if a recipient opts out of receiving messages with a commercial primary purpose from a particular sender, that sender may continue to transmit other types of messages. Therefore, recipients who invoke their rights under the opt-out mechanism required by CAN-SPAM will continue to receive valuable “transactional or relationship” messages. This is important because transactional or relationship messages are communications that Congress has determined to be per se valuable to recipients."