Lately my home internet service has been experiencing brief periods of failure and I'm trying to understand what the root problem is and if there's anything I can do to fix it.
I have two piholes running on my home network providing DNS service and blocking. Lately I've been encountering brief internet outages, approximately once a day for the last week. Work laptop disconnects from VPN saying no internet and other devices show no internet either. Cable modem doesn't show it's lost connection though, and logging into my router it shows it has internet connectivity. Trying to ping a website shows the dns resolution isn't working. I can ping by IP address but not name. The outage usually resolves itself in approx 5 minutes, sometimes a little longer.
My piholes are setup to use Google, Cloudflare and Open DNS for dns service so I'm puzzled how my dns resolution suddenly starts failing. In theory if one's down, it'll use the others right? So how am I experiencing a dns outage? Is this a problem with something outside my network or something internal, or is there anything else I should do to diagnose and fix it?
DNS isn't instantaneous - and it's hierarchical. If your systems are configured with pihole #1 (let's say
x.x.x.21) as primary and pihole #2 (x.x.x.22) for secondary and21fails, there will be a time when DNS requests time out. IIRC, for Linux it's something like 4 seconds.I know this because I saw a really weird thing at a work lab where requests were taking 8 seconds to complete for all the internal sites, but internet sites worked immediately. Turns out two of three PDNS systems in our cluster weren't configured properly for local lookup and systems would timeout on primary (+4 sec), timeout on secondary (+4 sec), then resolve with the tertiary server.
I'm not saying that's what's happening here, but if this is a recent occurrence you could start with your primary/pihole and check system logs, updates etc. Or you could take a system that has this behavior and either swap the pihole order, and/or remove the pihole from the DNS all together to see if it's even your gear causing the issue.
Looking at the dashboard for each pihole I can see that both show failures happening at the same time... the queries skyrocket during the outage as devices retry their dns lookups. If it was only one pihole experiencing a problem, they wouldn't both show the outage I don't believe.
Regarding pihole order, both are DHCP providers to the network and I have each make itself the first dns provider and the other second, so order varies across my network. Whichever pihole satisfies the DHCP request is the primary for the device it satisfies and the other secondary. The outage is for every device and both piholes though, as both show their query requests go through the roof at the same time.
Wait, both are DHCP providers? Admittedly, I haven't looked too deep at pihole clusters - but do they synchronize their leases? If not, there's a chance you might have duplicate IPs which can cause some interesting network behaviors.
If your logs say both are having issues at the same time, I'd go with the second troubleshooting option - accept DHCP, but statically set DNS to the same servers your Piholes use.
If there's an issue, it's with the upstream DNS (reachability or their rate limiting)
No duplicate IP concerns. They both work with the same static list and when they need to assign dynamically - because requester isn't in the static list - they assign from a non-overlapping range. Works well.
I think it is the upstream dns but strangely only the v6 pihole is reporting it, not my v5 one.