Internet advertising has become big business, and we see the effects of it in almost every page we view. The ad content typically comes from one or more well known ad servers, and some folks have come up with some clever ways (e.g., hosts files, DNS integration, etc.) to minimize the “ad effect” in the content we view. I have been using Mike’s host file for quiet some time, but for some reason OS X (actually lookupd) doesn’t handle large hosts files real well. Since OS X would get bogged down during DNS resolution, I decided to merge all of the ad domains into DNS to centrally fix the problem for the clients I support.
This was super easy to do, and only required two steps (assuming you are already running bind). The first step is to add one “zone” statement to named.conf for each ad domain you want to nix. The following example shows the named.conf entry you would add for the ad domain adservers.com:
zone "adservers.com"
{
type master; notify no; file "master/null.zone";
};
You can get a comprehensive list of the well known ad server domains from the ad blocking website. Once you retrieve the list, you can merge the domains into the named.conf using a combination of shell utilities, or you can download the Perl script (updateads.pl) I wrote to automate this process. The Perl script grabs the latest host file from the ad blocking website, formats the data, and spits out several lines that can be appended to named.conf:
$ updateads.pl |more
// *** Added domains on Thu May 25 13:53:34 2006 *** //
zone "ad1.com" { type master; notify no; file "master/null.zone"; };
zone "ad2.com" { type master; notify no; file "master/null.zone"; };
zone "ad3.com" { type master; notify no; file "master/null.zone"; };
[ ..... ]
Once you add all of the domains to named.conf, you need to create a zone file with one wildcard A record (this record is what is used to remove the ad servers, since the wildcard record will translate all entries in a given domain to 127.0.0.1). I am currently using the following zone file (with different domain names) to implement my ad blocking solution:
; File: null.zone
; Last modified: 07-10-2005
$TTL 86400 ; one day
@ IN SOA ns.mydomain.com hostmaster.mydomain.com. (
2005071005 ; serial number YYYYMMDDNN
28800 ; refresh 8 hours
7200 ; retry 2 hours
864000 ; expire 10 days
86400 ) ; min ttl 1 day
NS ns.mydomain.com.
A 127.0.0.1
* IN A 127.0.0.1
I have found that using this technique speeds up the time it takes to render a page, enhances privacy, and will also cut down on the amount of traffic consumed by your site. Tis good stuff!
May 28th, 2006 at 7:54 am
Uhm, dude. Nice script and all, but it’s already possible to view the list as an include file for your BIND config:
- http://pgl.yoyo.org/adservers/serverlist.php?hostformat=bindconfig
cheers,
Peter
May 28th, 2006 at 2:28 pm
Hi Peter,
Your site has so much cool information, I must have missed the link to the named.conf generator. Since I want to generate a complete named.conf given a base config, I plan to continue using the Perl script to ease my life. Thanks for the awesome work!
- Ryan
May 29th, 2006 at 12:21 am
A lot of people miss it for some reason - I’ll have to redesign the site one of these days, it’s too messy to find anything. Anyway, I’m glad you found it useful!
cheers,
- Peter
June 9th, 2006 at 5:09 am
Just set up Privoxy on some host on your network, make it listen on the appropriate IP and port 8080 (or its standard 8118, whichever) and then just use it as a normal HTTP proxy in the web browser.
Cuts virtually every ad and tons of other nasty webbugs, javascripts, etc etc.
I have it installed on my workstation and use it just for myself, which is also a possibility (there are win32 binaries with an installer no less, totally simple to take into use.) No need to tinker with esoteric DNS settings when there is software out there that does it much better and is specialized to the task.
It’s at http://www.privoxy.org and there are binaries for many OS’s and of course source for compilation.
To further tighten up ad filtering, visit http://www.neilvandyke.org/privoxy-rules/ and add that actions file as well; optional but quite useful.
May 30th, 2007 at 8:09 am
Awesome docs, btw.
Here is a tip for something that I implemented. Instead of redirecting traffic to 127.0.0.1, setup a web server that simply hosts ‘Blocked’ as the page. This keeps the web browser from trying multiple times to connect to a non-existent server and makes the pages look a little more clean if there are multiple ads on one site. In my case, ads get redirected to another internal address on a virtual apache server.
Additionally, this helps my wife who is not computer savy. If she is trying to visit a ‘bad’ site that I have auto-blocked, she will get the standard ‘Blocked’ message instead of a browser error page. She then knows not to bother me about a page that won’t load because the browser appears to be ‘broken’.