Most people use either Matrix or XMPP. Both work.
There is a nice overview of chat protocols here: https://www.messenger-matrix.de/
I mostly use matrix as of today. I think it’s alright. It’s a bit difficult to explain encryption and device verification to other people… I think that could be designed better. But apart from that it works very well. So does XMPP which I’ve used before that. Have a look at the messenger matrix and all the options before deciding on an ecosystem. I’d take one of the friends and do some evaluation before dragging the whole group in. You can do that with some pre-existing servers before learning how to host the server part.
I think what you’re trying to do is called a VPN. Set up a VPN that tunnels all the torrent traffic to and from the virtual server.