Do you use let's encrypt?





7738 votes ~ 25 comments

 

Speedup DNS requests with a local cache

Posted by Steve on Tue 25 Apr 2006 at 08:20

One common server bottleneck is DNS lookups. Many common server tasks such as from looking up hostnames to write Apache logfiles and processing incoming mail require the use of DNS queries. If you're running a high-traffic system it might be useful to cache previous lookups.

There are several different packages you can use for caching DNS requests - including bind, djbdns, dnsmasq and pdnsd.

The pdnsd package is a very simple and lightweight tool for DNS caching. It will, like many of the other systems, act as a small DNS server forwarding requests to a "real" DNS server and caching the responses.

When pdnsd is stopped it will save all the lookups which have been made against it so they may be reloaded when it starts again.

Installation is very straightforward:

apt-get install pdnsd

Once installed the software is configured via the file /etc/pdnsd.conf.

To configure the software you must do two things:

  • Configure pdnsd so that it will forward requests it doesn't know about to a real DNS server, letting it cache those results.
  • Update your system so that DNS lookups against the newly installed cache, or proxy.

Once you've completed these two steps all DNS lookups upon your system will be cached, and your DNS lookups should be much faster.

Upon your Debian GNU/Linux system you configure the DNS server(s) which are being used by means of the file, /etc/resolv.conf, this file will contain a list of name servers to query, perhaps along with a search domain to be used for unqualified hosts.

To tell your server to make DNS queries against the freshly installed server you would update that file to read:

nameserver 127.0.0.1

The next thing to do is to edit the pdnsd configuration file /etc/pdnsd.conf to specify which DNS servers the cache should use for its own lookups - these will most likely be your ISPs nameservers.

Locate the section of the configuration file which starts with server and add the IP address:

#
#  Specify the IP address of the real DNS server to query against here:
#
server {
        ip=11.22.33.44;   
        timeout=30;
        interval=30;
        uptest=ping;
        ping_timeout=50;
        purge_cache=off;
}

With this setting updated you can restart the caching service:

root@itchy:/etc# /etc/init.d/pdnsd restart
Restarting proxy DNS server: pdnsd.
root@itchy:/etc# 

If you wish to add more DNS servers to query against you can add them seperated by commas, or you can add multiple ip= lines such as these two examples:

       # Several IPs seperated by commas.
       ip=11.22.33.44,111.222.333.444;

       # Easier to read - one per line:
       ip=11.22.33.44;
       ip=111.222.333.444;

For more details of the supported options please consult the documentation by running "man pdnsd.conf".

You can test the cache is working by issuing a manual request to it:

root@itchy:/etc# dig  @localhost example.com mx


;; QUESTION SECTION:
;example.com.                   IN      MX

;; AUTHORITY SECTION:
example.com.            86400   IN      SOA     dns1.icann.org. hostmaster.icann.org.

;; Query time: 2224 msec
;; SERVER: 192.168.1.50#53(192.168.1.50)
;; WHEN: Sun Apr 23 21:47:41 2006
;; MSG SIZE  rcvd: 90

Here we used the dig command (part of the dnsutils package) to lookup the MX record of the domain name example.com. Notice at the bottom it shows "Query time: 2224msec"? Lets run that same query again - if our cache is working correctly it should be significantly faster:

root@itchy:/etc# dig  @itchy example.com mx |grep time
;; Query time: 1 msec

Much faster :)

(Yes DNS queries are ordinarily cached to a certain extent; so you'd expect the speedup even without our explicit DNS caching server...)

 

 


Re: Speedup DNS requests with a local cache
Posted by Anonymous (137.222.xx.xx) on Tue 25 Apr 2006 at 11:25
pdnsd couples nicely with the resolvconf package so if you are a mobile computing user you can do (wpasupplicant)+ifplugd+guessnet+resolvconf+pdnsd for lots of fun.

I've been using that setup quite happily for a while now -- I occasionally have some troubles with pdnsd not resuming properly when I suspend-to-disk, but they are quite easily fixed when they happen.

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by spiney (85.124.xx.xx) on Tue 25 Apr 2006 at 18:02
"...looking up hostnames to write Apache logfiles..."

Is there really any reason to do this in real time, i.e. enabling HostnameLookups in the Apache configuration?

I mean, the DNS names are quite handy when creating webserver stats to guess where visitors are coming from etc., but most if not all statistics software for webserver logs can do DNS lookups. So where's the point in doing it in real time?
--
Debian GNU/Linux on an IBM Thinkpad T43p

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Steve (82.41.xx.xx) on Tue 25 Apr 2006 at 18:35
[ View Steve's Scratchpad | View Weblogs ]

Honestly? I always disable real-time lookups.

I chose it as a reasonably common thing that other people frequently do ..

Steve

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by spiney (85.124.xx.xx) on Tue 25 Apr 2006 at 19:25
Oh, ok, I was just wondering whether I wasn't aware of some scenario where it might prove useful. :)

Otherwise, I have to agree on your other comments, some kind of DNS cache when using some slow link is really useful, even UMTS/3G connections (with its sometimes lousy latency, at least here in Austria) are improved a lot.
--
Debian GNU/Linux on an IBM Thinkpad T43p

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Anonymous (213.164.xx.xx) on Thu 27 Apr 2006 at 13:51
Apache hostname lookups are useful if you need the hostname of the remote host at the time of the request, rather than the hostname that it is a week (or whenever) later when you do the stats run.

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Arthur (84.19.xx.xx) on Tue 25 Apr 2006 at 18:20
[ View Weblogs ]
If HTTP server performance is critical, it's time to set HostnameLookups Off, don'tcha think?

I've given pdnsd a whirl a few times, but I was dissatisfied by its pipe backend, which I needed to use in order to establish horizons. It failed in a very sneaky way (offering up incorrect answers!), and I finally clocked enough hours tinkering with it to satisfy myself that it was time to go back to DJB's trusty dusty dnscache.

Installing dnscache (from djbdns) at least used to conflict with courier-mta due to both ucspi-tcp and courier-mta providing /usr/bin/addcr -- the easy way out is to compile ucspi-tcp from source so everything ends up in /usr/local/bin. The two versions of addcr seem to be compatible, so there's no harm done that I've detected in quite a long time of running both Courier and djbdns together.

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Steve (82.41.xx.xx) on Tue 25 Apr 2006 at 18:37
[ View Steve's Scratchpad | View Weblogs ]

I always disable realtime lookups of Apache requests too, I just chose that as a common example.

I've noticed real appreciable speedups using the cache on dialup, and even over a slow ISP link - without any problems. I guess your mileage may vary, but it is a simple thing to setup and if it doesn't work out easy to reverse.

dnscache I've not tried, but I'm not a fan of the DJB-philosophy so that really is personal bias...

Steve

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by lollipop (198.63.xx.xx) on Tue 25 Apr 2006 at 19:14

if you are running debian testing or unstable, another option is to use nscd with persistent databases something like this:

grabbed from Laptop LDAP

server-user nscd
debug-level 0
reload-count unlimited
paranoia no
enable-cache passwd yes
positive-time-to-live passwd 2592000
negative-time-to-live passwd 20
suggested-size passwd 211
check-files passwd yes
persistent passwd yes
shared passwd yes
enable-cache group yes
positive-time-to-live group 2592000
negative-time-to-live group 20
suggested-size group 211
check-files group yes
persistent group yes
shared group yes
enable-cache hosts yes
positive-time-to-live hosts 2592000
negative-time-to-live hosts 20
suggested-size hosts 211
check-files hosts yes
persistent hosts yes
shared hosts yes

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Arthur (24.122.xx.xx) on Tue 25 Apr 2006 at 19:18
[ View Weblogs ]
I'm also "not a fan of the DJB-philosophy", would rather go without mail than run qmail (the reigning king of backscatter), and would have loved to use PowerDNS if it weren't for the flakiness in the pipe backend. Or, perhaps, the flakiness of the guy who wrote the backend I used -- me.

I've gone down that PowerDNS trail three times now, and from scratch each time hoping to find that the pipe backend problems were my own mental misfiring. I'll probably try again later this year, and keep trying once or twice a year until the darn thing will work for me. Because I'm not a big DJB fan.

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Steve (82.41.xx.xx) on Tue 25 Apr 2006 at 19:30
[ View Steve's Scratchpad | View Weblogs ]

Just a quick comment to point out that pdnsd is not the same as PowerDNS - you might know that already, but it isn't 100% clear from your comment.

Steve

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Arthur (80.190.xx.xx) on Tue 25 Apr 2006 at 19:39
[ View Weblogs ]
OUCH! You're absolutely correct; my head was in a warm, dark, and quiet place where it doesn't belong. Maybe I need more sleep or more caffeine. Or maybe I just need to think before and during typing.

Head hung low in shame, I'm going to go find that caffeine I'm blaming this on.

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by simonw (84.45.xx.xx) on Tue 25 Apr 2006 at 23:49
[ View Weblogs ]
I'm curious has anyone benchmarked the cost of hostnamelookup?

I thought in Apache2 it was asynchronous, and as we use a separate DNS server, it is the cost of having the forked process open for the time it takes to do a DNS lookup (usually less than 2ms as it is usually cached).

I can see how a slashdotting might be a problem, but I'm curious to know exactly what the cost of this is. For most of my stuff it is enabled, as prompt reporting is a definite plus in my book, and I don't want to be shipping log files off to other machines for simple reporting.

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Anonymous (81.40.xx.xx) on Fri 22 Dec 2006 at 20:36
# dpkg --purge ucspi-tcp
it seems to work for me (dns-cache is ok after mta installation)

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Bob (62.58.xx.xx) on Wed 26 Apr 2006 at 03:06
There seems to be a small but quite important typo in the first mention of the configuration file of pdnsd, although later the correct name is given:

Once installed the software is configured via the file /etc/dnsd.conf.

should of course read:

Once installed the software is configured via the file /etc/pdnsd.conf.

(extra p in front of dnsd.conf)

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Steve (82.41.xx.xx) on Wed 26 Apr 2006 at 08:02
[ View Steve's Scratchpad | View Weblogs ]

Thanks, I've corrected this now.

Steve

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by rdunne (147.252.xx.xx) on Tue 20 Jun 2006 at 09:49
[ View Weblogs ]
I have done every here but the local host will not resolve for me.. its pointing to a dns in my bulding with i can dig but nothing when i use local host..

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by rdunne (147.252.xx.xx) on Tue 20 Jun 2006 at 12:14
[ View Weblogs ]
ahh i can get 2 lookup rte.ie and dit.ie but i wont do any others ... why why , please help

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Anonymous (66.198.xx.xx) on Mon 3 Jul 2006 at 17:14
I have done the other steps but i dont know how do this 2 steps.

how to configure the software you must do two things:

* Configure pdnsd so that it will forward requests it doesn't know about to a real DNS server, letting it cache those results.
* Update your system so that DNS lookups against the newly installed cache, or proxy.

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Steve (62.30.xx.xx) on Tue 4 Jul 2006 at 06:47
[ View Steve's Scratchpad | View Weblogs ]

Your two points are addressed in the article:

  • " Specify the IP address of the real DNS server to query against here:"
  • "To tell your server to make DNS queries against the freshly installed server you would update that file to read:"

Steve

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Utumno (60.248.xx.xx) on Wed 23 Jan 2008 at 05:38
[ View Utumno's Scratchpad | View Weblogs ]

Very simple package. 1 minute and everything is working.

One more advice for ppp/dsl users: you have to comment out option 'usepeerdns' in /etc/ppp/peers/[provider] so that your pppd will not overwrite /etc/resolv.conf with list of DNS servers it gets during the negotiation.

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Anonymous (63.88.xx.xx) on Fri 13 Mar 2009 at 21:48
As of 3/13/2009 under Debian Lenny the pdnsd.conf file looks nothing like what is described here and starts off with this Note:

/* Note: this file is overriden by automatic config files when
_ /etc/default/pdnsd AUTO_MODE is set and that
/usr/share/pdnsd/pdnsd-$AUTO_MODE.conf exists
*/

[ Parent | Reply to this comment ]

Re: Speedup DNS requests with a local cache
Posted by Anonymous (93.163.xx.xx) on Thu 4 Mar 2010 at 21:47

I have written a basic howto for Debian Lenny (inspired by this article) it's here: http://wiki.zibet.net/wiki.pl/Speedup_DNS_requests_with_a_local_c ache

[ Parent | Reply to this comment ]