I've done computational physics at the grad level. There, PDEs are converted to finite difference which basically leads to giant sparse linear matrices. These are solved using SOR or even more advanced numerical techniques. These techniques tend to be quite GPU friendly.
Well, if you're just doing a standard finite difference method, and you have to keep shuffling your matrices between CPU and GPU because other operations don't work well on GPUs, you actually won't have any speedup.
Where GPUs shine for PDEs is if you have a lot of extra work for each node, for instance if you have complex chemical reactions or thermodynamics, or if you have a high-order method that requires lots of intermediate computations.
If you don't believe me, you can download the PETSc code and test the ViennaCL solvers versus the regular ones.
