Whether or not it's open source is irrelevant to the trust issue. You would still have to look at the binary to check whether its behavior matches that of the putative source code. Either way, you end up examining the behavior of the compiled code.
Build it yourself and host/use your own app builds and server instances. But then I'll have trust the company I bought my VPS from to host my Matrix instance, isn't it?
It is not irrelevant. If you can't build the binary from code yourself you have no way to know if it is trustworthy. Step one in finding out if it is is to look at the source and then compile it. After that you can look if it does strange things because of something you missed in the code but without step one you might as well not start at all. It will always be at most guesswork.
Virtually no one who uses software compiles it themselves, so this is not a very interesting rebuttal.
Meanwhile, it's not 1994 anymore, and people who know how to look for bugs can (I know this is hard for some people to wrap their heads around) look inside of binaries and draw conclusions about how programs work. There's a name for it; it's a kind of engineering.
