yes yes yes!
I 100% agree.
It would be interesting to hear from someone with some legal knowledge. I suspect in many countries it's the case that if you host your email with a third pary (iCloud, gmail, FB, etc) you give up some rights to privacy ... whereas if you host your email on your own server that you physically control, for example in your personal home, your rights to privacy are greater.
Even on Ubuntu Server it's very very easy to setup a mail server. Linode has guides, and there are others on the interwebs as well.
Just make sure to implement a regular backup regime.
In my experience IMAP to my own mail server is WAY faster than dealing with Gmail (even web interface).