I would strongly encourage you to just wrap `libsodium`. The authors have thought about this problem a lot, and have done the hard work for you — there is a surprising amount being done behind the scenes to give the kinds of guarantees cryptographic keys warrant.
If CGo isn't acceptable, at least use their implementation to guide the design your go-native version.
If CGo isn't acceptable, at least use their implementation to guide the design your go-native version.