Too bad there isn't a stable version of Plex with GPU transcoding. (It's beta right now). I would love to do this on my 1U short depth PMS server enclosure.
Plex just uses ffmpeg, the version bundled does include GPU support although I believe it depends on the codec, from memory it does x264 but not x265 or vp9 but I could be wrong.