Thanks for taking the time to reply!
The host setup has eth0 as the physical interface to the rest of the network, with br0 replacing it completely. br0 has the same MAC as the eth0 interface and eth0 just forwards to br0 which then does the bridging internally. br0 being a bridge means that incus is able to split it off without MACVLAN but rather its nic device in bridge mode which "Uses an existing bridge on the host (br0) and creates a virtual device pair to connect the host bridge to the instance." That results in a network interface that has its own MAC and is assigned a local IP by the DHCP server on the network while also being able to talk to the host.
Incus accomplishes the same goal as Proxmox (Proxmox has similar bridge network devices for its containers/VMs) just without Incus needing to be your OS/distro like Proxmox does, it's just a package.
As for the Docker, the parent interface is br0 which has supplanted eth0. MACVLAN is working as it is intended to in Docker, as far as I can tell. The container has a networking device with its own MAC address, and after supplying the MACVLAN network device with my network's subnet and gateway and static IP address in the Docker compose file it works as expected. If I don't supply a static IP in the Docker compose file, Docker just assigns it the first IP in the given subnet - no DHCP interaction. This docker-net-dhcp plugin (I linked to the issue about it not working on the latest version of Docker anymore) was made to give Docker network devices the ability to use DHCP to get an IP address, but it's clearly not something to rely on.
If I'm missing something about MACVLAN that makes DHCP work for Docker, let me know! Hardcoding an IP into a docker-compose file adds an extra step to remember compared to everything else being configured on the centralized DHCP server - hence the shoddy implementation claim for Docker.
Thanks for the link to using another MACVLAN and routing around the host<-/->container connection issue inherent to MACVLAN. I'll keep it in mind as an alternate to Incus container around another container! I do wish there could be something like Incus' hassle-free solution for Docker or Podman.
This was very insightful and I’d like to say I groked 90% of it meaningfully!
For an Incus container with its unique MAC interface, yes if I run a Docker container in that Incus container and leave the Docker container in its default bridge mode then I get the desired feature set (with the power of onions).
And thanks for explaining CNI, I’ve seen it referenced but didn’t fully get how it’s involved. I see that podman uses it to make a MACVLAN interface that can do DHCP (until 5.0, but the replacement seems to be feature-compatible for MACVLAN), so podman will sidestep the pain point of having to assign a no-go-zone on the DHCP server for a Docker swath of IPv4s, as you mentioned. Close enough for containers that the host doesn’t need to talk to.
So in summary:
I’ve got Docker doing the extent it can manage with MACVLAN and there’s no extra magicks to be done on it.
Podman will still use MACVLAN (no host to container comms still) but it’s able to use DHCP to get an address for the MACVLAN container.
If the host must talk to the container with MACVLAN, I can either use the MACVLAN bypass as you linked to above or put the Docker/Podman container inside an Incus container with its bridge mode.
Kubernutes continues to sound very powerful and flexible but is definitely beyond my reach yet. (Womp womp)
Thanks again for taking the time to type and explain all of that!