Performance isn't the problem. If any of your drives ever corrupt a single bit then all of your data is trashed (with most setups) and that's a big problem.
This is no the case with filevault, when it is set up with a login account. I have lost encrypted disk images under OS X due to drive problems... but have had a filevault make it thru a failing disk without trouble.
I use Filevault. Never had ANY issues with it so far. My password is also pretty strong (10 alphanumeric characters with punctuation sprinkled) so I doubt anyone would get much out of trying to crack it.
Everyone who travels with their laptop should use some kind of an encrypted FS if their data is worth anything.
It requires far more HDD space than not using it because it doesn't seem to clean up after itself for a while. And if you get low on hard disk space your system grinds to a halt.
I dumped all of the non-important stuff (videos, music etc) into /Users/Shared and use Filevault only for dev and the size of it is fairly small. I haven't noticed any of the things you mentioned...
