I trust the app itself, since you can indeed verify that code, but the idea is that you don't have to trust the (Keybase-managed) server.
So what you'd verify is the encryption key. If we do a Diffie-Hellman key exchange and our shared key is abcxyz, then both phones should show that key. If an attack is going on, the key would have to be one known to the attacker rather than your conversational partner.
Simplified, DH is quite easy: you pick a base number and a modulus (public knowledge), both parties generate a random number (let's call it Rand), they do base^Rand or Rand^base (I forgot which way around), apply the modulus, send the number to the other person, and apply the same exponentiation with the number you received (and apply the modulus). The resulting number is only known to both parties, even though anyone could have observed the public parameters and the numbers that were sent to the other side. If a person wants to intercept this, they need to pick a Rand themselves and do the operations with that, replacing the number that gets sent over with their own. Because they can't know the Rand of either legitimate party, they will necessarily have a different result, and so the resulting encryption key is different. Both parties would (upon verifying their key out of band, for example by holding their phones next to each other in real life) see different encryption keys.
It's not about verifying the source code, but verifying that the server which I'm talking to is not malicious (for example if someone compromised it). That's the one property which makes end to end encryption "end to end" :)
Similar schemes can be made with different types of encryption (Diffie-Hellman is one of a few good methods), but with end to end encryption, the end devices are always the ones that verify each other.