Assuming you're a C-programmer, read the libsodium docs first. https://download.libsodium.org/doc/public-key_cryptography/s...
If you're using higher level language, use a library that provides bindings for it https://download.libsodium.org/doc/bindings_for_other_langua...
By using libsodium, you're not rolling your own crypto. Rolling your own crypto would mean
-trying to find new one way functions for public key crypto
-trying to implement RSA from textbook
-trying to implement RSA-OAEP from papers, RFCs, books etc.
Using a library is not anywhere near those. There are other ways to fail cryptography too, from not doing public key authentication, to storing private keys in insecure places.
So it's highly recommended you take time to read a book on the topic. The best modern book currently availalbe is https://www.amazon.com/Serious-Cryptography-Practical-Introd...
That's debatable. Libsodium does not have a proper authenticated key exchange. The key exchange does the job somewhat, but it has worse security properties than a properly crafted interactive protocol.
Problem is, designing your own key exchange protocol is more delicate than most construction. I know, I designed my own, and made several serious mistakes in the process (one of which voided an important security property).
Granted, rolling your own constructions is generally less error prone than rolling your own primitive. But it still shouldn't be done without at least having followed and fully understood an introductory course in cryptography (I recommend https://www.crypto101.io/). I mean, I did quite a bit more than that, and I still don't fully trust myself.