See https://news.ycombinator.com/item?id=19418111 for a detailed response to these accusations.
The lack of a simple, common library is a real sticking point. Each client & bot shouldn't have to implement the baseline network protocol and then build e2e atop it, rather the clients should be focused on what to do with the messages after receiving them, how to display them, etc.
I say this as a daily non-Matrix.org user of Riot, the functionality of the Riot ecosystem on its own is good, but I'd be hard pressed to port my own Signal bots to Matrix, let alone build a featureful client in a performant toolkit (eg: Not Electron) for Matrix today.
There isn't a viable ecosystem outside Riot/Synapse currently, which makes it feel like Matrix is just a self-hosted Slack/Hipchat solution with just Riot as usable clients. IMO there needs to be diversity in the Matrix ecosystem just like the Fediverse has with Mastodon, Pleroma, PeerTube, PixelFed, NextCloud, Rustodon, etc all inter-operating in a sane, functional manner.
We are starting work this week on closing the E2E gap for all the rest.