Bind-DLZ with MySQL

September 1st, 2010

DNS management with Bind has traditionally been flat files and slave/master configurations. Bind also has a feature/extension called DLZ — dynamically loaded zones. This feature can be very useful when designing applications that use databases or directories for storage rather than having to design your application to address a filesystem to create resource records or zone files.

In this article, I will explain how to set this up for a configuration where there are thousands of name-based virtual hosts hosted on the same VIP/email infrastructure using the same resource record on a CentOS 5.X system using MySQL to store records. The Bind version is 9.6.0-P1.

The first step is to install any pre-requisites:

yum install openssl-devel mysql-devel openldap-devel unixODBC-devel gcc

Note that you’ll want to uninstall gcc after this process is complete to reduce the likelihood of an attacker compiling an exploit on this box if they were to gain unprivileged access.

Next, download and extract the Bind sources:

pushd /tmp/
curl -C - -L -O 'http://ftp.isc.org/isc/bind9/9.6.0-P1/bind-9.6.0-P1.tar.gz'
tar xzvf bind-9.6.0-P1.tar.gz
pushd bind-9.6.0-P1

If compiling on a 64 bit system, you might have to setup some variables so that the appropriate mysql libraries are found:

export CPPFLAGS="-I/usr/lib64/mysql $CPPFLAGS"
export LDFLAGS="-L/usr/lib64/mysql $LDFLAGS"
export LD_LIBRARY_PATH="/usr/lib64/mysql"

The next step is to run configure — this example uses mysql only:

./configure  \
  --prefix=/usr/local/bind  \
  --disable-openssl-version-check \
  --with-dlz-mysql=yes

Once successful with configure, the next step is to install:

make && sudo make install

Be sure to add a user and group, as well as setup some basic directories for data:

groupadd -r -g 25 named
useradd -r -u 25 -s /bin/nologin -d /usr/local/named -g named named
mkdir /var/cache/bind
chown named:named /var/cache/bind

Now that the easy part is finished, the next step is to setup MySQL to store the appropriate DNS records.

In this example, data is populated in MySQL via a stored procedure in SQL Server via a linked server to a MySQL master (ODBC). A python script then creates the necessary entries in the postfix database to allow mail routing to occur. One of the tables populated here is the postfix.domains table. This is simply a list of all domains that are hosted at this site. I take advantage of this to replicate only this table to each of my DNS servers running MySQL and Bind-DLZ locally. This explanation will help the reader understand the next portion where I setup tables and views and populate them with data.

The next step is to create the database which will store the records. I use a database called postfix since my setup is tightly coupled with email routing and replication from the email database. (Login to MySQL to perform the following actions or script as appropriate.)

mysql> create database postfix;

Next, I create a template of resource records that will apply to all zones hosted within MySQL. (Note that this is a site which hosts thousands of domains on the same VIP/email architecture.)

DROP TABLE IF EXISTS dns_values;
CREATE TABLE dns_values (
  host VARCHAR(255) DEFAULT '' NOT NULL,
  type ENUM('SOA','NS','MX','A','CNAME','TXT','HINFO','PTR') NOT NULL DEFAULT 'SOA',
  data VARCHAR(255),
  ttl INT(11) DEFAULT 300 NOT NULL,
  mx_priority VARCHAR(10),
  refresh INT(11) DEFAULT 0 NOT NULL,
  retry INT(11) DEFAULT 0 NOT NULL,
  expire INT(11) DEFAULT 0 NOT NULL,
  minimum INT(11) DEFAULT 0 NOT NULL,
  serial BIGINT(20) DEFAULT 0 NOT NULL,
  resp_person VARCHAR(255),
  primary_ns VARCHAR(255),
  key host_index (host),
  key type_index (type)
) ENGINE=MyISAM;

The next step is to populate the basic set:

mysql> LOCK TABLES `dns_values` WRITE;
/*!40000 ALTER TABLE `dns_values` DISABLE KEYS */;
INSERT INTO `dns_values` VALUES
('@','SOA','root.mail.example.com.',300,NULL,10800,900,604800,600,2009020401,'root.mail.example.com.','ns1.example.com.'),
('@','NS','ns1.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','NS','ns2.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','A','xxx.xxx.xxx.xxx',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('images','A','xxx.xxx.xxx.xxx',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('mail','A','xxx.xxx.xxx.xxx',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('*','A','xxx.xxx.xxx.xxx',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('imap','CNAME','mail.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('smtp','CNAME','mail.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','TXT','v=spf2.0/pra mx ip4:xxx.xxx.xxx.0/24 -all',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','TXT','v=spf1 mx ip4:xxx.xxx.xxx.0/24 -all',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','MX','mail.example.com.',300,'10',10800,900,604800,600,2009020401,NULL,NULL),
('webmail','CNAME','mail.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL);
/*!40000 ALTER TABLE `dns_values` ENABLE KEYS */;
UNLOCK TABLES;

Create the postfix.domains table:

mysql> CREATE TABLE domains (
  domain varchar(128) NOT NULL default '',
  PRIMARY KEY  (domain)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Go ahead and populate the domains table with some values. Note that I replicate data from another table but you can just as well enter any values manually.

mysql> insert into domains (domain) values('example.com');

The next step is to create a view that will combine the dns_values table with the domains table to present all records as one table:

mysql>CREATE VIEW dns_records AS
SELECT
  d.domain        as zone
  ,dv.host        as host
  ,dv.type        as type
  ,dv.data        as data
  ,dv.ttl         as ttl
  ,dv.mx_priority as mx_priority
  ,dv.refresh     as refresh
  ,dv.retry       as retry
  ,dv.expire      as expire
  ,dv.minimum     as minimum
  ,dv.serial      as serial
  ,dv.resp_person as resp_person
  ,dv.primary_ns  as primary_ns
FROM domains d, dns_values dv ;

Next, setup grants on MySQL to allow the user who is accessing MySQL from Bind access:

mysql> GRANT USAGE,SELECT ON postfix.* TO binddlz@localhost identified by 'trickypassword';
FLUSH PRIVILEGES;

I started with a pretty basic named.conf file:

key rndc {
  algorithm hmac-md5 ;
  secret "longsecret";
};

controls {
  inet 127.0.0.1 allow { localhost; } keys { rndc; };
};

include "/usr/local/bind/etc/named.conf.options";

// prime the server with knowledge of the root servers
zone "." {
  type hint;
  file "/usr/local/bind/etc/db.root";
};

// setup local zones
zone "localhost" {
  type master;
  file "/usr/local/bind/etc/db.local";
};

zone "127.in-addr.arpa" {
  type master;
  file "/usr/local/bind/etc/db.127";
};

zone "0.in-addr.arpa" {
  type master;
  file "/usr/local/bind/etc/db.0";
};

zone "255.in-addr.arpa" {
  type master;
  file "/usr/local/bind/etc/db.255";
};

include "/usr/local/bind/etc/named.custom.zones";
include "/usr/local/bind/etc/named.dlz.zones";

As far as named.conf.options, it is also pretty basic:

options {
  directory "/var/cache/bind";
  allow-transfer { xxx.xxx.xxx.xxx; xxx.xxx.xxx.xxx; };
  zone-statistics yes;
  statistics-file "/usr/local/bind/var/stats/named-stats.out";
  recursion no;
};

As you can see, I simply included the following configuration portion as named.dlz.zones.

dlz "mysql zone" {
  database "mysql
  {host=localhost dbname=postfix user=binddlz pass=trickypassword ssl=false}
  {select zone from dns_records where zone = '%zone%'}
  {select ttl, type, mx_priority, case
      when lower(type)='txt' then concat('\"', data, '\"')
      when lower(type) = 'soa' then concat_ws(' ', data, resp_person, serial, refresh, retry, expire, minimum)
    else data end from dns_records_view where zone = '%zone%' and host = '%record%'}";
};

Now start Bind with the following command and test:

/usr/local/bind/sbin/named -c /usr/local/bind/etc/named.conf -f -g -u named

Useful tips:
* do not include both ns and contact in SOA record, use only respo_contact in either data or resp_contact fields
* make sure you restart Bind every time you restart MySQL or you will lose your connection(s)
* if you would like to create the dns_records table without a view, simply use the dns_values table and add the zone as the first column

Bash Tip! for loop on directory listing

August 24th, 2010

One very common task when scripting with bash is to use a for loop to iterate over the contents of a directory or directory tree. There are two primary methods of accomplishing this task; using ls and using find. We’ll not consider the manual method as that would be completely unworthy of our attention.

I find it easy to start with ls when I don’t need to recurse into a directory tree as that is a command that I use often. This often turns into a process such as this:

for dir in $(ls)
do
  echo ${dir}
done

Now the above method typically does not work for me. I have an alias setup to print out pretty colors when I issue the ls command and that will cause each command which operates on the variable $dir to fail with a “No such file or directory” error. I always have to remember this and re-write the command with the flag to disable color formatting:

for dir in $(ls --color=never)
do
  echo ${dir}
done

The above script will work every time.

The next option is using find. find is awesome and all powerful. Learn and use find. The most common issue when using find is that you may have to filter out the current and/or parent directories when processing the results. Take this example:

for dir in $(find . -maxdepth 1 -type d)
do
  echo ${dir}
done

This loop will print out the current directory, as well as all other directories in the current working directory. If you are running some sort of processing within this loop, you may end up re-processing everything unless you discard the current working directory (noted by the dot).

This example will not process the current working directory:

for dir in $(find . -maxdepth 1 -type d)
do
  if [ ${dir} == "." ]
  then
    continue
  fi
  echo ${dir}
  while pushd ${dir}
  do
    echo ${dir}
  done
  popd
done

Bash for loops are incredibly useful and easy to work with. Use the above tips and make bash work for you.

CIFS over SSH – Extending the network

August 6th, 2010

I recently had an issue where a file copy from a celerra NAS to a server outside the network was failing and I couldn’t figure out why. The file copy was a pull from the outside server which needed access inside the network. The BGP route had somehow changed to go over Integra’s network rather than Verizon and I couldn’t get anyone to fess up to blocking ports 445 and 139. To solve this issue, I turned to SSH tunnelling.

To setup a tunnel from inside a protected network to expose a resource to an external client, you can use the following format:

$ sudo ssh -N -R 445:cifsNAS:445 outsideserver.com

I then created a hosts file entry on the outside server to map cifsNAS to 127.0.0.1.

#/etc/hosts
127.0.0.1  cifsNAS

What this does is SSH to outsideserver.com and open up port 445 on that host, which will then tunnel all traffic from outsideserver1:445 to cifsNAS:445. This solved my temporary issue and I was able to copy the needed files over.

Awstats with Apache and mod_rewrite

July 15th, 2010

I recently setup a new Awstats install and used mod_rewrite to make it easier to view web stats. Using the following configuration within a virtual host declaration, you can simply make requests in the following format:


http://awstats/$CONFIG/$YEAR/$MONTH/$DAY/

This is assuming that you run daily rollups.

  ServerAlias awstats

  RewriteCond %{REQUEST_URI} !^/awstats/awstats.pl [NC]
  RewriteCond %{REQUEST_URI} !^/icon               [NC]
  RewriteRule ^/(.*)/(.*)/(.*)/(.*)/ http://%{HTTP_HOST}/awstats/awstats.pl?databasebreak=day&day=$4&month=$3&year=$2&config=$1 [L,NE]

Bash tip: Print begin and end timestamp on apache logs…

July 9th, 2010

This morning I needed to audit some log files that I had recently processed through AWstats and received a report that there was a discrepancy in the data. The complaint was that one day was missing. I used the following bash script to print out the start and end timestamp of each log file:

for file in $( ls -tr *.gz ) ;
do
  BEGIN=$(zcat ${file} | head -n 1 | awk '{print $4}');
  END=$(zcat ${file} | tail -n 1 | awk '{print $4}');
  echo "${file} - ${BEGIN} - ${END}";
done

Note that each log file was named uniquely by web server and logrotate number, eg webserver1.access_log.XX.gz.

System Administrator Technical Interviews

July 8th, 2010

I have had the opportunity to interview many candidates over the past few months and have a few tips:

  • When indicating that you have VMware experience, clearly indicate which features you have experience with. I have interviewed many candidates who claim to be experts on VI3/vSphere and yet have never used clustering or shared storage.
  • When asked about rating yourself from 1-5 or 1-10, make sure you understand which side is the proficient side and give an example of what you think is proficient in a particular area.

My methodology is to ask the interviewee to rate themselves and then ask them what that rating means to them. If they rate themselves a 4 out of 5 with general Linux system administration, I then ask them to give me a few examples of what somebody who has a 4/5 rating would be able to do. I then ask them questions based on that assessment. If you can’t win on those terms, you typically can’t win.

It is not my desire to stump somebody in an interview, I would prefer to ask them questions about what they have done in the past and get into a good dialogue about things they are familiar with.  Do your interviewer a favor and be very clear on the resume and during the interview process.

Apache 2.2 – Return 200 OK for missing images

June 18th, 2010

I recently faced a problem where I needed to configure Apache to return a 200 OK when it received a request for an image that was missing, along with a custom 404 ErrorDocument which was an image. The reason for this requirement is that when Outlook 2003/2007 displays an HTML page where an image request returns a ‘404 Not Found’, it displays a broken image link icon, which is a little red ‘x’.

The solution that I ended up using was to configure mod_rewrite to look for any requests that were not valid files, links, or directories and return the custom ErrorDocument if these conditions were true. This results in a 200 OK for all requests — even on missing images. Note that this results in Apache never using the ErrorDocument 404 configured.

This configuration must be set at the directory level and not the virtual host level as this references the filesystem which requires a rewritebase to be set (which cannot be done at the virtual host level).

  RewriteEngine On
  RewriteBase /var/www/html
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-l
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteCond %{REQUEST_URI} !^/server-status [NC]
  RewriteRule .* /missing.jpg [L]

Reporting search referrals

May 4th, 2010

Here is a quick awk command that will parse apache web logs and print a simple virtual host/date/referral csv report that only includes referrals from google, bing, or yahoo:

awk 'tolower($11) ~ "google|bing|yahoo" {print $2 "," $4 "," $11}' ${input_file} >> report.csv

Linux guest hangs at “starting udev” (VMware vSphere)

March 23rd, 2010

Having recently upgraded the Virtual Infrastructure at work to vSphere, I have encountered many scenarios with CentOS 5.3 guests not booting or taking a long time to boot. The last message on the console typically indicates that it’s hanging while starting udev.

The fix for this issue is to ensure proper time keeping practices have been adhered to in accordance with the VMware Timekeeping KB.

The basic steps are:
1. Modify the kernel line of /boot/grub/grub.conf to include the following line:

clocksource=acpi_pm notsc divider=10

clocksource=acpi_pm – uses the Power Management Timer (PMTMR) available in some southbridges as primary timing source
notsc – disable the timestamp counter
divider=10 – reduces the frequency of timer interrupts by 10 (from 1000/second to 100/second)

2. Disable time sync through VMware tools (note that this will continue to happen on bootup, pause, resume, etc..):

vmware-guestd --cmd "vmx.set_option synctime 1 0"

3. Setup time sync through NTP:
a. Setup your /etc/ntp.conf to point to a good NTP server pool.
b. Set NTP to start and persist across reboots.

# yum -y install ntp
# chkconfig ntpd on
# /etc/init.d/ntpd start

Expanding an ext3 filesystem online

March 4th, 2010

One common scenario that I face in my daily work is to add disk to various filesystems. Setting up systems correctly so that this is possible will save time and frustration. One of the easiest cases is adding disk to a virtual machine when the guest is using LVM and ext3.

As always, please be sure to backup your data before trying any filesystem or disk manipulation.

After adding the virtual hard disk using the VI client, provision the space from within the virtual machine using the following steps:

1. re-scan storage

echo "- - -" > /sys/class/scsi_host/host0/scan

2. Create physical volume from new device (Note: check with your SAN admin to see if you need to create a partition and align appropriately.)

pvcreate /dev/sdb

3. Extend the volume group to the new PV (physical volume):

vgextend vg01 /dev/sdb

3. Extend the LV (logical volume) to the desired size:

lvextend -L +2G /dev/vg01/lvol05

4. Resize the filesystem to cover the newly extended LV:

resize2fs /dev/vg01/lvol05

Your newly resized filesystem should now be available.

I have not yet tried expanding existing VMDK files on the fly with vSphere but I plan to test that out next.