This can be summarized quite quickly (and for all federated/distributed protocols). Should you need to deploy some kind of feature or security upgrade to the protocol, a centralized system lets you do this for everyone (and more or less force them into it). A distributed system has the potential to end up like TLS, where "ciphersuite agility" can be a negative in that you end up supporting something relatively weak or otherwise problematic because one member of the network refuses to turn it off "just in case" one of their customers need it.
This is the distributed vs centralized part. From a security perspective, Moxie would also likely take issue with the server-side metadata in Matrix. If your "homeserver" is hacked (or otherwise accessed), my understanding is that it would likely contain a fair amount of information on who you communicate with and what rooms you are in, or you would be able to piece it together. Here's the schema for data stored server-side in the reference server: https://github.com/matrix-org/synapse/blob/master/synapse/st...
Signal by contrast is designed precisely to minimize the amount of data users expose even to Signal's own servers. Profile information, group membership etc, it's all encrypted.
Finally, Matrix doesn't E2E by default - not all clients support its double ratchet implementation. This has been getting better over time, but I'm not sure if you can still negotiate cleartext connections all the time or force clients to downgrade. Experience has shown leaving users to consciously turn on encryption is a bad idea. This is also a problem in PGP, XMPP/OMEMO/OTR, Telegram, SMIME etc. If you use Signal, WhatsApp or Wire, you cannot fail to turn on the crypto, which means non-technical users benefit from it without even realizing it.
Edit: We're also working on minimising metadata by storing it all clientside and running peer-to-peer; see https://fosdem.org/2020/schedule/event/dip_p2p_matrix/
I personally would love to see the blog post on this.