Container platforms (docker, lxc, podman)

I recently migrated my services from rootful docker to rootless podman quadlets. It went smoothly, since nothing I use actually needs to be rootful. Well, except for caddy. It needs to be able to attach to privileged ports 80 and 443.

My current way to bypass it is using HAProxy running as root and forwarding connections using proxy protocol. (Tried to use firewalld but that makes the client IP opaque to caddy.) But that adds an extra layer, which means extra latency. It's perfectly usable, but I'd like to get rid of it, if possible.

I'm willing to run caddy in rootful podman if needed. But from what I understand, that means I can't have it in the same rootless network as my other containers. I really don't wanna open most of my containers' ports, so that's not an option.

So, I'm asking whether any of these three things are possible.

1. Use firewalld to forward ports to caddy without obscuring the client's IP.
2. Make rootful caddy share a network with other rootless containers.
3. Assign privileged ports to caddy somehow, in rootless mode. (I know there's a way to make all these ports unprivileged, but is it possible to only assign these 2 ports as unprivileged?)

Or maybe there's a fourth way that I'm missing. I feel like this is a common enough setup, that there must be a way to do it. Any pointers are appreciated, thanks.