There are still some dangerous things you can do (e.g. mismanage keys, reuse nonces).
I wrote a library called Halite for PHP developers that wraps libsodium and makes it even harder to misuse. My philosophy was, "You shouldn't even need to know what a nonce is to use it securely."
From elsewhere in the thread, PyNaCl takes a similar approach. So how dangerous "just using libsodium" is, with respect to the "don't roll your own crypto" guidance, depends a little bit on which binding you're using.
Also, a lot of tasks might require a specific protocol (PAKEs, VPN protocols, searchable encryption, etc.) that libsodium isn't suitable for.