How can we stagger DNS resolution queries

[nirvanagit]

We use Istio, which uses Envoy, which uses the c-ares library for DNS resolution.

It is observed that Envoy does DNS resolution in a matter of 1-2 seconds, which overwhelms our DNS servers when there are thousands of entries to resolve.

Can the DNS resolution be spread out across a time window so that the systems are not overwhelmed?

[bradh352]

Currently there is no way to do that within c-ares itself as c-ares doesn't employ any sort of queuing mechanism, it sends requests immediately. It is known that many DNS servers do, however, employ rate limiting. There have been thoughts about adding a queuing system to c-ares but nothing has come to pass. I'd recommend contacting the upstream users of c-ares to see if they can rate limit the DNS queries at all.

[crrodriguez]

Modern DNS server software is able to handle thousands of queries per second on very modest hardware.. soemthing does not seem right with the way you are trying to resolve your problem.

[bradh352]

What's funny is a lot of DNS servers have very low defaults for rate limiting, even on recursive name servers, or caching proxies like dnsmasq, which may make it appear that a server is overloaded when really its a configuration issue. However if you don't control the upstreams, that can be an issue.

[crrodriguez]

dnsmasq assumes you are using a potato to run it, all limits are extremely low.
nsd: defaults to 200 queries per second if built with --enable-ratelimit
unbound : ratelimit is off be default
bind : rate limit off by default