For public key exchange, it doesn't matter who gets the keys along their path from phone to phone. But, for the paranoid, you would use Bluetooth or NFC, and show a key fingerprint on both phones (ideally in the form of an algorithmically generated graphic plus the original hexadecimal) so the users can compare the images side by side and make sure the keys are valid.
As for endpoint security being the weak link, I tend to agree with you, but it also depends on who is trying to snoop on your conversations, and what level of resources they have.
The Android application "TextSecure" for encrypting SMS using Elliptic Curve public key cryptography, by Moxie Marlinspike, allows you to validate the authenticity of keys by displaying a QR code on your phone, which the verifier scans using their phones camera.
As for endpoint security being the weak link, I tend to agree with you, but it also depends on who is trying to snoop on your conversations, and what level of resources they have.