It's true that you can only handle them in defer funcs, but the defer func could be anywhere up the stack, not just the current function you're inside.
(I'm not sure if that was your understanding or not)
"The panic and recover functions behave similarly to exceptions and try/catch in some other languages in that a panic causes the program stack to begin unwinding and recover can stop it. Deferred functions are still executed as the stack unwinds. If recover is called inside such a deferred function, the stack stops unwinding and recover returns the value (as an interface{}) that was passed to panic."
(I'm not sure if that was your understanding or not)
"The panic and recover functions behave similarly to exceptions and try/catch in some other languages in that a panic causes the program stack to begin unwinding and recover can stop it. Deferred functions are still executed as the stack unwinds. If recover is called inside such a deferred function, the stack stops unwinding and recover returns the value (as an interface{}) that was passed to panic."
https://code.google.com/p/go-wiki/wiki/PanicAndRecover