So you are asking about something that seems simple, but is actually many different components working together. Apple and google have really made this integrated for a long time.
What you want is:
- caldav/cardav server (radicale is good)
- integration into your email client (Thunderbird can do this)
- share-able webDAV service
- some auth in front of this
I've left out all the plumbing needed to either support your access to this, or provide secure integration with a 3rd party email service.
This is a hard problem to solve for self-hosting. I have a self-hosted radicale instance and I get around the inter-connectivity by simply exporting ICS files and sending them to folks. Updating meeting times, setting calendar sharing is all very difficult because of above.