As others have mentioned, it's not really the technical aspects of SMTP that make hosting your own email difficult. It's the entire deliverability ecosystem that's developed (ostensibly to protect from spam), such as most residential providers blocking outgoing port 25, or most mailservers filtering incoming mail from dynamic IPs.
I used iRedMail to setup department mail server and it just worked. I had no problems with configuration: IMAP, SSL, SMTP - everything works out of the box. Spam protection is also adequate, at least for a relatively low-volume usage.
You're not addressing what majelix mentions. How about DKIM? SPF? Do you know if/when you end up on any block-lists? What about deferrals from the major providers?
It is by no means impossible that everything "just works" for you, but depending on your e-mail patterns, and what ISP you uses, and whether or not there's any spammers or spammer-look-alikes on the same netblocks as you, the level of hassle in ensuring your outbound e-mails actually gets delivered to your recipients inboxes rather than spam-filtered can be immense.