Checking for encrypted packets wouldn't tell you anything about the binaries' trustworthiness. A back door might just encrypt the data with a second key, or more specifically, encrypt the key that encrypts the data with another key. A reliable way of testing binaries doesn't seem very feasible to me. It's like antivirus vendors trying to find new viruses: the malware authors can always obfuscate their code just a little more, do it just a little bit differently, and now it does the same thing while escaping detection.
Authors of open source software who want to distribute trustable binaries should include instructions for how to reproduce the binary exactly from the source. A third party verifier could reproduce the binary, then publish a digital signature affirming that they reproduced it, allowing anyone who doesn't want to compile it to check with a trusted third party.
But all of that is a moot point if the source code isn't being very carefully checked.
Authors of open source software who want to distribute trustable binaries should include instructions for how to reproduce the binary exactly from the source. A third party verifier could reproduce the binary, then publish a digital signature affirming that they reproduced it, allowing anyone who doesn't want to compile it to check with a trusted third party.
But all of that is a moot point if the source code isn't being very carefully checked.