ProxySG DNS misconfiguration leads to performance issue in forwarding

If you are using ProxySGs in chains like an internal proxy and an internet facing proxy you must configure a rule on the internal proxy when he should forward traffic to the upstram proxy towards the internet.

In this case the internal ProxySG tries to resolve the target if not possible he forward the request to the internet proxy.

This customer had a misconfigured DNS server, the DNS servers are not presenting a root hint zone, why is that a problem?

Example to explain the issue: Your internal proxy server has 2 dns servers configured and 4 dns suffixes in the dns suffix search list.

If the internal proxy server begins to resolve for example google.com the following will happen when no root hint is present:

1. proxy sends request google.com to dns1 answer response code: 2 »> SERVFAIL
2. proxy sends request google.com to dns2 answer response code: 2 »> SERVFAIL
3. proxy extends with first suffix google.com.suffixA.internal to dns1 answer: host not found »> NXDOMAIN »> no second try because authoritative answer come back until cache times out
4. proxy extends with second suffix google.com.suffixB.internal to dns1 answer: host not found »> NXDOMAIN »> no second try because authoritative answer come back until cache times out
5. proxy extends with third suffix google.com.suffixC.internal to dns1 answer: host not found »> NXDOMAIN »> no second try because authoritative answer come back until cache times out
6. proxy extends with forth suffix google.com.suffixD.internal to dns1 answer: host not found »> NXDOMAIN »> no second try because authoritative answer come back until cache times out
7. decision: forward to internet proxy

In an healthy environment it looks like this:

1. proxy sends request google.com to dns1 answer response code: host not found »> NXDOMAIN
2. decision: forward to internet proxy
3. Next request to same domain will be anwered by dns cache

Because of the fact that the internal DNS servers response with SERVFAIL instead of NXDOMAIN the ProxySG will not cache the result and on the next request the show begins again. This waiting time to find an DNS answer blocks the proxy server and the performance in an high load environment decreases dramatically.

In this environment 54 million dns requests are fired against the internal DNS servers, after fixing the root hint the requests drop down to 971k per day from the proxys.

The SERVFAIL should be around ~ 1-2% that is normal. You can also see it in the statistics on the proxysg, goto https://your-proxy:8082/TCP/DNS-Query-Stats

TCP/IP DNS Query Statistics
...
UDP queries                          1.300.000.000
Server failure responses          451.000.000   >>> **33%**
...

References

#

How does the DNS resolution work on the ProxySG? (broadcom.com)

The top four DNS response codes and what they mean – BlueCat Networks

What you can learn from an NXDOMAIN response – BlueCat Networks

Fix

#

Fix your DNS environment and present a root hint which can authoritative answer to not existing domains with NXDOMAIN, this can fix also other DNS resolution delays you may have.