On Matrix practically everyone is matrix.org. The other public servers I tried are relatively slow and suffer hickups when joining rooms.
On XMPP, many people run servers in their companies, projects, or homes. Which are the "top 3-4 XMPP servers"? I probably do not even know them.
But you are right: XMPP is used mainly for IM, i.e. one-to-one dialogues or smaller, non-public chatrooms. I'm not aware of any huge, public chatrooms in XMPP, and for that I'm on IRC — via a gateway from XMPP called biboumi.