You can do this during fights. The question is who will control the party when just walking around and talking to NPCs or exploring the world. Who will make the story changing decisions?
Also things like leveling up your character, or changing equipment may be more painful depending on everyone's patience. Being in a multiplayer campaign is already quite the commitment but having to wait for your turn at the controller to change equipment or spells might make it unenjoyable.