I've run both XMPP and Matrix servers myself. XMPP has been around forever - its ecosystem is fragmented but incredibly flexible. You can pick a client that works for you and it just works.
Matrix has better E2E encryption out of the box which is a real plus. The federation works but feels more controlled than XMPP. With XMPP servers can talk to each other with just a few XML config files.
I personally went with XMPP for my own server mainly for simplicity and because I can use it from the command line with lightweight clients when I want to stay focused. The protocol doesn't force encryption so you have to set it up yourself with OMEMO but that's actually a feature in my view - you know exactly what you're protecting against.