HTTP Caching in yum…

April 1st, 2011

yum is great to work with when it works and a pain in the ass when it does not. I recently had a problem where I would get the dreaded metadata does not match checksum warning while trying to update a CentOS 5.3 system I was working on.

filelists.xml.gz: [Errno -1] Metadata file does not match checksum

The problem here is that the repomd.xml which lists the file name and the SHA1 checksum that it should calculate on the file or the file itself is not being updated properly due to some level of HTTP caching between the yum client and the repository server.

Usually you can resolve this error through a metadata clean:

yum clean metadata

…or at the very least, clean all:

yum clean all

In this particular case, nothing seemed to work. I even removed the cache manually:

rm -rf /var/cache/yum/$OFFENDINGREPOSITORY

I also tried to copy the cache from another host which did not have the same problem:

$ another host> rsync -av /var/cache/yum/$OFFENDINGREPOSITORY /var/cache/yum/$OFFENDINGREPOSITORY

Something else was causing the problem — some caching server between me and the repository.

To resolve this issue, I relied on a tip from another site, to disable http caching at the yum.conf level:

#/etc/yum.conf
...
http_caching=none

This immediately resolved the problem and yum worked again.

I do not recommend continuing with this flag set in the config as caching is highly useful and makes things work faster. At minimum, it would be best to cache packages with the following option:

http_caching=packages

After going through this experience, I found another option that might have worked better, ‘yum clean expire-cache’. Next time this happens, I’d like to try this option out to see if it solves the problem.

Using PHP to check SSHD availability

March 17th, 2011

I’m working on a project where I need to spin up some EC2 instances and deploy code to them over SSH which requires that I know when SSH is available. To do this, I’ve written a little PHP snippet that will connect to the server via TCP port 22 and loop until the service is available:

        $address   = "10.3.1.2" ;
        $port         = 22 ;
        $sleeptime = 1 ;

        $resource = socket_create( AF_INET, SOCK_STREAM, SOL_TCP ) ;
        while ( ! @socket_connect( $resource, $address, $port ) ) {
                print "not ready\n" ;
                sleep( $sleeptime )  ;
        }

        print "ready\n" ;

Note the @ sign in front of the socket_connect function call to suppress PHP Warnings when it fails. This is because I expect it to fail 99% of the time.

Apache Memory Consumption

February 22nd, 2011

One quick way to determine how much memory is being consumed by all apache processes is to use the following command:

ps -eo user,pid,pcpu,cmd,rss | egrep httpd | awk '{SUM += $5} END {print "Total memory used: " SUM}'

Zend Studio / Zend Server Integration

February 17th, 2011

I have recently had the opportunity to work with Zend Server and Zend Studio as part of performance testing / bench-marking a large application. I had an interesting, intermittent problem getting Zend Server and Zend Studio to work together as part of the debugging/profiling process with the following error showing up frequently:

Failed to communicate with Zend Studio. Make sure Zend Studio is running on 'xxx.xxx.xxx.xxx,127.0.0.1' according to the settings in Server Setup | Monitor

This error message is mis-leading. The IP listed (xxx.xxx.xxx.xxx) is the IP address of my client desktop machine passed in through the browser.

The error actually means (in my case) that an instance of Zend Server was not configured to allow my client debugger access and was failing when attempting to open a debug session with this Zend Server instance, which was not on the Zend Server where I was reviewing monitoring events from. (How many Zend Servers are involved here?)

Here is the process that occurs when you review a monitored event and attempt to debug this event (assuming Zend Server and Zend Studio are setup to integrate):
1. Clicking on the “Debug Event” button sends an HTTP request to Zend Server where the event was captured.
2. The Zend Server where the request was sent sends an HTTP request to the host where the application is hosted and attempts to open a debug session with that Zend Server to your client Zend Studio.
3. Zend Server communicates with your client over port(s) 20080/10137 to start a debug session in Zend Studio.

Here is the process that I was experiencing with the error above:
1. Click “Debug Event” in Zend Server console, which triggers an HTTP request to Zend Server where the event was captured.
2. The Zend Server performed a DNS lookup on the domain where the application was hosted, which was a production server and not the performance environment where I was working.
3. The Zend Server where the application was hosted in production did not allow remote debug access from my client so the debug session was never initiated.
4. The Zend Server where I attempted to open a debug session from reported the error at the top of this post indicating that my client was not listening from Zend Studio – mis-leading error.

The end result was that I needed to ensure a host entry was in the hosts file (could also be done properly via DNS) for each site that I was troubleshooting/debugging/profiling on the Zend Server where I was monitoring the testing that was being performed. This would create a scenario where the Zend Server where the monitoring was being performed at would also initiate the debugging session with my Zend Studio client and things would work as I expect them to work.

Copy and Paste (clipboard) Fedora 14 to Windows 7 rdesktop Session

February 4th, 2011

EDIT: It turns out that this was a mistake – I had long ago acquired the habit of stopping the rdpclip.exe process on any Windows client that I was using rdesktop to work on whenever copy/paste stopped working and re-connect to solve the problem. I had done this at the same time where I changed the connection settings which made it appear to solve the problem. The real answer when copy/paste stops working through rdesktop is to kill the rdpclip.exe process and disconnect/reconnect.

I recently was upgraded to a Windows 7 desktop for Windows usage at work. I quickly noted that I was not able to copy and paste from my Linux desktop to my rdesktop session that I was running to interact with Windows 7 which makes life much harder to work with.

After some research and comparison against Windows XP, Windows Server 2003 R2, and Windows Server 2008 R2, it appeared to be isolated to Windows 7 alone so I called on one of my Windows Sysadmin buddies for help.

Unfortunately, I don’t have any Windows arcane knowledge after this experience but he quickly found that the flag I needed to pass to rdesktop was the -r clipboard:PRIMARYCLIPBOARD flag to allow clipboard redirection through rdesktop. This is currently an un-documented feature in rdesktop.

This is undocumented because it’s not in the man page. I realize that many folks are moving away from man pages of late but there is no excuse for incomplete man pages if there is one at all.

Using strace to attach to a multi-threaded process (like a JVM/Java)…

December 10th, 2010

When using strace to attach to a process that is running many threads, use the following format:

strace -f -c -p PID  -o /tmp/outfile.strace

This will capture system calls from all threads within the process.

Tomcat troubleshooting – Thread Stack Trace

December 2nd, 2010

One of the best troubleshooting tools when using tomcat is to cause the JVM to print a thread stack trace for each current thread. This will provide insight into what all threads are currently doing.

Note that if you wait until there is a problem to take the first stack trace, it will be very difficult to solve any problems. Take a look now and see what the current behavior is so that it will be easier to troubleshoot later.

To get a thread stack trace, use the following command:

sudo -u tomcat jstack PID > /tmp/$(date +%Y%m%d)-jvm-thread-stack.txt

Windows Server Uptime

November 30th, 2010

One valuable piece of information when troubleshooting any server issue is how long has the server been up. This is something that is not as easy to find on Windows as Linux, but I needed this today.

To see the uptime of a Windows Server 2003 server, use the following command:

net statistics server

The date shown is the last time the server booted.

Bind-DLZ with MySQL

September 1st, 2010

DNS management with Bind has traditionally been flat files and slave/master configurations. Bind also has a feature/extension called DLZ — dynamically loaded zones. This feature can be very useful when designing applications that use databases or directories for storage rather than having to design your application to address a filesystem to create resource records or zone files.

In this article, I will explain how to set this up for a configuration where there are thousands of name-based virtual hosts hosted on the same VIP/email infrastructure using the same resource record on a CentOS 5.X system using MySQL to store records. The Bind version is 9.6.0-P1.

The first step is to install any pre-requisites:

yum install openssl-devel mysql-devel openldap-devel unixODBC-devel gcc

Note that you’ll want to uninstall gcc after this process is complete to reduce the likelihood of an attacker compiling an exploit on this box if they were to gain unprivileged access.

Next, download and extract the Bind sources:

pushd /tmp/
curl -C - -L -O 'http://ftp.isc.org/isc/bind9/9.6.0-P1/bind-9.6.0-P1.tar.gz'
tar xzvf bind-9.6.0-P1.tar.gz
pushd bind-9.6.0-P1

If compiling on a 64 bit system, you might have to setup some variables so that the appropriate mysql libraries are found:

export CPPFLAGS="-I/usr/lib64/mysql $CPPFLAGS"
export LDFLAGS="-L/usr/lib64/mysql $LDFLAGS"
export LD_LIBRARY_PATH="/usr/lib64/mysql"

The next step is to run configure — this example uses mysql only:

./configure  \
  --prefix=/usr/local/bind  \
  --disable-openssl-version-check \
  --with-dlz-mysql=yes

Once successful with configure, the next step is to install:

make && sudo make install

Be sure to add a user and group, as well as setup some basic directories for data:

groupadd -r -g 25 named
useradd -r -u 25 -s /bin/nologin -d /usr/local/named -g named named
mkdir /var/cache/bind
chown named:named /var/cache/bind

Now that the easy part is finished, the next step is to setup MySQL to store the appropriate DNS records.

In this example, data is populated in MySQL via a stored procedure in SQL Server via a linked server to a MySQL master (ODBC). A python script then creates the necessary entries in the postfix database to allow mail routing to occur. One of the tables populated here is the postfix.domains table. This is simply a list of all domains that are hosted at this site. I take advantage of this to replicate only this table to each of my DNS servers running MySQL and Bind-DLZ locally. This explanation will help the reader understand the next portion where I setup tables and views and populate them with data.

The next step is to create the database which will store the records. I use a database called postfix since my setup is tightly coupled with email routing and replication from the email database. (Login to MySQL to perform the following actions or script as appropriate.)

mysql> create database postfix;

Next, I create a template of resource records that will apply to all zones hosted within MySQL. (Note that this is a site which hosts thousands of domains on the same VIP/email architecture.)

DROP TABLE IF EXISTS dns_values;
CREATE TABLE dns_values (
  host VARCHAR(255) DEFAULT '' NOT NULL,
  type ENUM('SOA','NS','MX','A','CNAME','TXT','HINFO','PTR') NOT NULL DEFAULT 'SOA',
  data VARCHAR(255),
  ttl INT(11) DEFAULT 300 NOT NULL,
  mx_priority VARCHAR(10),
  refresh INT(11) DEFAULT 0 NOT NULL,
  retry INT(11) DEFAULT 0 NOT NULL,
  expire INT(11) DEFAULT 0 NOT NULL,
  minimum INT(11) DEFAULT 0 NOT NULL,
  serial BIGINT(20) DEFAULT 0 NOT NULL,
  resp_person VARCHAR(255),
  primary_ns VARCHAR(255),
  key host_index (host),
  key type_index (type)
) ENGINE=MyISAM;

The next step is to populate the basic set:

mysql> LOCK TABLES `dns_values` WRITE;
/*!40000 ALTER TABLE `dns_values` DISABLE KEYS */;
INSERT INTO `dns_values` VALUES
('@','SOA','root.mail.example.com.',300,NULL,10800,900,604800,600,2009020401,'root.mail.example.com.','ns1.example.com.'),
('@','NS','ns1.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','NS','ns2.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','A','xxx.xxx.xxx.xxx',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('images','A','xxx.xxx.xxx.xxx',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('mail','A','xxx.xxx.xxx.xxx',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('*','A','xxx.xxx.xxx.xxx',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('imap','CNAME','mail.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('smtp','CNAME','mail.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','TXT','v=spf2.0/pra mx ip4:xxx.xxx.xxx.0/24 -all',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','TXT','v=spf1 mx ip4:xxx.xxx.xxx.0/24 -all',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','MX','mail.example.com.',300,'10',10800,900,604800,600,2009020401,NULL,NULL),
('webmail','CNAME','mail.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL);
/*!40000 ALTER TABLE `dns_values` ENABLE KEYS */;
UNLOCK TABLES;

Create the postfix.domains table:

mysql> CREATE TABLE domains (
  domain varchar(128) NOT NULL default '',
  PRIMARY KEY  (domain)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Go ahead and populate the domains table with some values. Note that I replicate data from another table but you can just as well enter any values manually.

mysql> insert into domains (domain) values('example.com');

The next step is to create a view that will combine the dns_values table with the domains table to present all records as one table:

mysql>CREATE VIEW dns_records AS
SELECT
  d.domain        as zone
  ,dv.host        as host
  ,dv.type        as type
  ,dv.data        as data
  ,dv.ttl         as ttl
  ,dv.mx_priority as mx_priority
  ,dv.refresh     as refresh
  ,dv.retry       as retry
  ,dv.expire      as expire
  ,dv.minimum     as minimum
  ,dv.serial      as serial
  ,dv.resp_person as resp_person
  ,dv.primary_ns  as primary_ns
FROM domains d, dns_values dv ;

Next, setup grants on MySQL to allow the user who is accessing MySQL from Bind access:

mysql> GRANT USAGE,SELECT ON postfix.* TO binddlz@localhost identified by 'trickypassword';
FLUSH PRIVILEGES;

I started with a pretty basic named.conf file:

key rndc {
  algorithm hmac-md5 ;
  secret "longsecret";
};

controls {
  inet 127.0.0.1 allow { localhost; } keys { rndc; };
};

include "/usr/local/bind/etc/named.conf.options";

// prime the server with knowledge of the root servers
zone "." {
  type hint;
  file "/usr/local/bind/etc/db.root";
};

// setup local zones
zone "localhost" {
  type master;
  file "/usr/local/bind/etc/db.local";
};

zone "127.in-addr.arpa" {
  type master;
  file "/usr/local/bind/etc/db.127";
};

zone "0.in-addr.arpa" {
  type master;
  file "/usr/local/bind/etc/db.0";
};

zone "255.in-addr.arpa" {
  type master;
  file "/usr/local/bind/etc/db.255";
};

include "/usr/local/bind/etc/named.custom.zones";
include "/usr/local/bind/etc/named.dlz.zones";

As far as named.conf.options, it is also pretty basic:

options {
  directory "/var/cache/bind";
  allow-transfer { xxx.xxx.xxx.xxx; xxx.xxx.xxx.xxx; };
  zone-statistics yes;
  statistics-file "/usr/local/bind/var/stats/named-stats.out";
  recursion no;
};

As you can see, I simply included the following configuration portion as named.dlz.zones.

dlz "mysql zone" {
  database "mysql
  {host=localhost dbname=postfix user=binddlz pass=trickypassword ssl=false}
  {select zone from dns_records where zone = '%zone%'}
  {select ttl, type, mx_priority, case
      when lower(type)='txt' then concat('\"', data, '\"')
      when lower(type) = 'soa' then concat_ws(' ', data, resp_person, serial, refresh, retry, expire, minimum)
    else data end from dns_records_view where zone = '%zone%' and host = '%record%'}";
};

Now start Bind with the following command and test:

/usr/local/bind/sbin/named -c /usr/local/bind/etc/named.conf -f -g -u named

Useful tips:
* do not include both ns and contact in SOA record, use only respo_contact in either data or resp_contact fields
* make sure you restart Bind every time you restart MySQL or you will lose your connection(s)
* if you would like to create the dns_records table without a view, simply use the dns_values table and add the zone as the first column

Bash Tip! for loop on directory listing

August 24th, 2010

One very common task when scripting with bash is to use a for loop to iterate over the contents of a directory or directory tree. There are two primary methods of accomplishing this task; using ls and using find. We’ll not consider the manual method as that would be completely unworthy of our attention.

I find it easy to start with ls when I don’t need to recurse into a directory tree as that is a command that I use often. This often turns into a process such as this:

for dir in $(ls)
do
  echo ${dir}
done

Now the above method typically does not work for me. I have an alias setup to print out pretty colors when I issue the ls command and that will cause each command which operates on the variable $dir to fail with a “No such file or directory” error. I always have to remember this and re-write the command with the flag to disable color formatting:

for dir in $(ls --color=never)
do
  echo ${dir}
done

The above script will work every time.

The next option is using find. find is awesome and all powerful. Learn and use find. The most common issue when using find is that you may have to filter out the current and/or parent directories when processing the results. Take this example:

for dir in $(find . -maxdepth 1 -type d)
do
  echo ${dir}
done

This loop will print out the current directory, as well as all other directories in the current working directory. If you are running some sort of processing within this loop, you may end up re-processing everything unless you discard the current working directory (noted by the dot).

This example will not process the current working directory:

for dir in $(find . -maxdepth 1 -type d)
do
  if [ ${dir} == "." ]
  then
    continue
  fi
  echo ${dir}
  while pushd ${dir}
  do
    echo ${dir}
  done
  popd
done

Bash for loops are incredibly useful and easy to work with. Use the above tips and make bash work for you.