DNS isn't instantaneous - and it's hierarchical. If your systems are configured with pihole #1 (let's say x.x.x.21) as primary and pihole #2 (x.x.x.22) for secondary and 21 fails, there will be a time when DNS requests time out. IIRC, for Linux it's something like 4 seconds.
I know this because I saw a really weird thing at a work lab where requests were taking 8 seconds to complete for all the internal sites, but internet sites worked immediately. Turns out two of three PDNS systems in our cluster weren't configured properly for local lookup and systems would timeout on primary (+4 sec), timeout on secondary (+4 sec), then resolve with the tertiary server.
I'm not saying that's what's happening here, but if this is a recent occurrence you could start with your primary/pihole and check system logs, updates etc. Or you could take a system that has this behavior and either swap the pihole order, and/or remove the pihole from the DNS all together to see if it's even your gear causing the issue.