The latency increased, but it doesn't have to be drastic. Ilya Grigorik has a great piece on how to get the handshake down:
https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-...
We turn on as much of that stuff as we can for our servers, and I'm pretty happy with it.