I can't comment on XUbuntu, but I went from LXDE to Debian+Xfce and finally settled on Debian+OpenBox.
If you're really only interested in Xfce vs. LXDE, I'd go with Xfce -- but only because it should be less of a memory hog & I doubt you'll miss much from its heavier cousin. Honestly though, either would probably work fine.
Or hey, give OpenBox a crack. :)
(Be warned I'm a keyboard junkie, so my love of OpenBox might be tainted by that :) Between the customizable shortcuts + menu & some of my own stuff [http://tomlee.co/code/ginvoke] I rarely have a reason to reach for the mouse outside of the browser.)