I'm pretty glad they're not using DTLS. TLS is the sane and (usually) correct choice for normal client-server protocols, but DTLS is not necessarily the conservative choice for group messaging systems.
This isn't a particularly sophisticated cryptosystem --- AES-GCM with keys broadcasted between the trusted participants --- but it's simple and, while GCM is nobody's favorite AEAD, it is at least a formal AEAD mode, leaving not a whole lot of room to screw the basics up.
This isn't a particularly sophisticated cryptosystem --- AES-GCM with keys broadcasted between the trusted participants --- but it's simple and, while GCM is nobody's favorite AEAD, it is at least a formal AEAD mode, leaving not a whole lot of room to screw the basics up.