One quick way to determine how much memory is being consumed by all apache processes is to use the following command:
ps -eo user,pid,pcpu,cmd,rss | egrep httpd | awk '{SUM += $5} END {print "Total memory used: " SUM}'
One quick way to determine how much memory is being consumed by all apache processes is to use the following command:
ps -eo user,pid,pcpu,cmd,rss | egrep httpd | awk '{SUM += $5} END {print "Total memory used: " SUM}'
I have recently had the opportunity to work with Zend Server and Zend Studio as part of performance testing / bench-marking a large application. I had an interesting, intermittent problem getting Zend Server and Zend Studio to work together as part of the debugging/profiling process with the following error showing up frequently:
Failed to communicate with Zend Studio. Make sure Zend Studio is running on 'xxx.xxx.xxx.xxx,127.0.0.1' according to the settings in Server Setup | Monitor
This error message is mis-leading. The IP listed (xxx.xxx.xxx.xxx) is the IP address of my client desktop machine passed in through the browser.
The error actually means (in my case) that an instance of Zend Server was not configured to allow my client debugger access and was failing when attempting to open a debug session with this Zend Server instance, which was not on the Zend Server where I was reviewing monitoring events from. (How many Zend Servers are involved here?)
Here is the process that occurs when you review a monitored event and attempt to debug this event (assuming Zend Server and Zend Studio are setup to integrate):
1. Clicking on the “Debug Event” button sends an HTTP request to Zend Server where the event was captured.
2. The Zend Server where the request was sent sends an HTTP request to the host where the application is hosted and attempts to open a debug session with that Zend Server to your client Zend Studio.
3. Zend Server communicates with your client over port(s) 20080/10137 to start a debug session in Zend Studio.
Here is the process that I was experiencing with the error above:
1. Click “Debug Event” in Zend Server console, which triggers an HTTP request to Zend Server where the event was captured.
2. The Zend Server performed a DNS lookup on the domain where the application was hosted, which was a production server and not the performance environment where I was working.
3. The Zend Server where the application was hosted in production did not allow remote debug access from my client so the debug session was never initiated.
4. The Zend Server where I attempted to open a debug session from reported the error at the top of this post indicating that my client was not listening from Zend Studio – mis-leading error.
The end result was that I needed to ensure a host entry was in the hosts file (could also be done properly via DNS) for each site that I was troubleshooting/debugging/profiling on the Zend Server where I was monitoring the testing that was being performed. This would create a scenario where the Zend Server where the monitoring was being performed at would also initiate the debugging session with my Zend Studio client and things would work as I expect them to work.
EDIT: It turns out that this was a mistake – I had long ago acquired the habit of stopping the rdpclip.exe process on any Windows client that I was using rdesktop to work on whenever copy/paste stopped working and re-connect to solve the problem. I had done this at the same time where I changed the connection settings which made it appear to solve the problem. The real answer when copy/paste stops working through rdesktop is to kill the rdpclip.exe process and disconnect/reconnect.
I recently was upgraded to a Windows 7 desktop for Windows usage at work. I quickly noted that I was not able to copy and paste from my Linux desktop to my rdesktop session that I was running to interact with Windows 7 which makes life much harder to work with.
After some research and comparison against Windows XP, Windows Server 2003 R2, and Windows Server 2008 R2, it appeared to be isolated to Windows 7 alone so I called on one of my Windows Sysadmin buddies for help.
Unfortunately, I don’t have any Windows arcane knowledge after this experience but he quickly found that the flag I needed to pass to rdesktop was the -r clipboard:PRIMARYCLIPBOARD flag to allow clipboard redirection through rdesktop. This is currently an un-documented feature in rdesktop.
This is undocumented because it’s not in the man page. I realize that many folks are moving away from man pages of late but there is no excuse for incomplete man pages if there is one at all.
When using strace to attach to a process that is running many threads, use the following format:
strace -f -c -p PID -o /tmp/outfile.strace
This will capture system calls from all threads within the process.
One of the best troubleshooting tools when using tomcat is to cause the JVM to print a thread stack trace for each current thread. This will provide insight into what all threads are currently doing.
Note that if you wait until there is a problem to take the first stack trace, it will be very difficult to solve any problems. Take a look now and see what the current behavior is so that it will be easier to troubleshoot later.
To get a thread stack trace, use the following command:
sudo -u tomcat jstack PID > /tmp/$(date +%Y%m%d)-jvm-thread-stack.txt
One valuable piece of information when troubleshooting any server issue is how long has the server been up. This is something that is not as easy to find on Windows as Linux, but I needed this today.
To see the uptime of a Windows Server 2003 server, use the following command:
net statistics server
The date shown is the last time the server booted.
DNS management with Bind has traditionally been flat files and slave/master configurations. Bind also has a feature/extension called DLZ — dynamically loaded zones. This feature can be very useful when designing applications that use databases or directories for storage rather than having to design your application to address a filesystem to create resource records or zone files.
In this article, I will explain how to set this up for a configuration where there are thousands of name-based virtual hosts hosted on the same VIP/email infrastructure using the same resource record on a CentOS 5.X system using MySQL to store records. The Bind version is 9.6.0-P1.
The first step is to install any pre-requisites:
yum install openssl-devel mysql-devel openldap-devel unixODBC-devel gcc
Note that you’ll want to uninstall gcc after this process is complete to reduce the likelihood of an attacker compiling an exploit on this box if they were to gain unprivileged access.
Next, download and extract the Bind sources:
pushd /tmp/ curl -C - -L -O 'http://ftp.isc.org/isc/bind9/9.6.0-P1/bind-9.6.0-P1.tar.gz' tar xzvf bind-9.6.0-P1.tar.gz pushd bind-9.6.0-P1
If compiling on a 64 bit system, you might have to setup some variables so that the appropriate mysql libraries are found:
export CPPFLAGS="-I/usr/lib64/mysql $CPPFLAGS"
export LDFLAGS="-L/usr/lib64/mysql $LDFLAGS"
export LD_LIBRARY_PATH="/usr/lib64/mysql"
The next step is to run configure — this example uses mysql only:
./configure \ --prefix=/usr/local/bind \ --disable-openssl-version-check \ --with-dlz-mysql=yes
Once successful with configure, the next step is to install:
make && sudo make install
Be sure to add a user and group, as well as setup some basic directories for data:
groupadd -r -g 25 named
useradd -r -u 25 -s /bin/nologin -d /usr/local/named -g named named
mkdir /var/cache/bind
chown named:named /var/cache/bind
Now that the easy part is finished, the next step is to setup MySQL to store the appropriate DNS records.
In this example, data is populated in MySQL via a stored procedure in SQL Server via a linked server to a MySQL master (ODBC). A python script then creates the necessary entries in the postfix database to allow mail routing to occur. One of the tables populated here is the postfix.domains table. This is simply a list of all domains that are hosted at this site. I take advantage of this to replicate only this table to each of my DNS servers running MySQL and Bind-DLZ locally. This explanation will help the reader understand the next portion where I setup tables and views and populate them with data.
The next step is to create the database which will store the records. I use a database called postfix since my setup is tightly coupled with email routing and replication from the email database. (Login to MySQL to perform the following actions or script as appropriate.)
mysql> create database postfix;
Next, I create a template of resource records that will apply to all zones hosted within MySQL. (Note that this is a site which hosts thousands of domains on the same VIP/email architecture.)
DROP TABLE IF EXISTS dns_values;
CREATE TABLE dns_values (
host VARCHAR(255) DEFAULT '' NOT NULL,
type ENUM('SOA','NS','MX','A','CNAME','TXT','HINFO','PTR') NOT NULL DEFAULT 'SOA',
data VARCHAR(255),
ttl INT(11) DEFAULT 300 NOT NULL,
mx_priority VARCHAR(10),
refresh INT(11) DEFAULT 0 NOT NULL,
retry INT(11) DEFAULT 0 NOT NULL,
expire INT(11) DEFAULT 0 NOT NULL,
minimum INT(11) DEFAULT 0 NOT NULL,
serial BIGINT(20) DEFAULT 0 NOT NULL,
resp_person VARCHAR(255),
primary_ns VARCHAR(255),
key host_index (host),
key type_index (type)
) ENGINE=MyISAM;
The next step is to populate the basic set:
mysql> LOCK TABLES `dns_values` WRITE;
/*!40000 ALTER TABLE `dns_values` DISABLE KEYS */;
INSERT INTO `dns_values` VALUES
('@','SOA','root.mail.example.com.',300,NULL,10800,900,604800,600,2009020401,'root.mail.example.com.','ns1.example.com.'),
('@','NS','ns1.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','NS','ns2.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','A','xxx.xxx.xxx.xxx',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('images','A','xxx.xxx.xxx.xxx',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('mail','A','xxx.xxx.xxx.xxx',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('*','A','xxx.xxx.xxx.xxx',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('imap','CNAME','mail.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('smtp','CNAME','mail.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','TXT','v=spf2.0/pra mx ip4:xxx.xxx.xxx.0/24 -all',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','TXT','v=spf1 mx ip4:xxx.xxx.xxx.0/24 -all',300,NULL,10800,900,604800,600,2009020401,NULL,NULL),
('@','MX','mail.example.com.',300,'10',10800,900,604800,600,2009020401,NULL,NULL),
('webmail','CNAME','mail.example.com.',300,NULL,10800,900,604800,600,2009020401,NULL,NULL);
/*!40000 ALTER TABLE `dns_values` ENABLE KEYS */;
UNLOCK TABLES;
Create the postfix.domains table:
mysql> CREATE TABLE domains ( domain varchar(128) NOT NULL default '', PRIMARY KEY (domain) ) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Go ahead and populate the domains table with some values. Note that I replicate data from another table but you can just as well enter any values manually.
mysql> insert into domains (domain) values('example.com');
The next step is to create a view that will combine the dns_values table with the domains table to present all records as one table:
mysql>CREATE VIEW dns_records AS SELECT d.domain as zone ,dv.host as host ,dv.type as type ,dv.data as data ,dv.ttl as ttl ,dv.mx_priority as mx_priority ,dv.refresh as refresh ,dv.retry as retry ,dv.expire as expire ,dv.minimum as minimum ,dv.serial as serial ,dv.resp_person as resp_person ,dv.primary_ns as primary_ns FROM domains d, dns_values dv ;
Next, setup grants on MySQL to allow the user who is accessing MySQL from Bind access:
mysql> GRANT USAGE,SELECT ON postfix.* TO binddlz@localhost identified by 'trickypassword';
FLUSH PRIVILEGES;
I started with a pretty basic named.conf file:
key rndc {
algorithm hmac-md5 ;
secret "longsecret";
};
controls {
inet 127.0.0.1 allow { localhost; } keys { rndc; };
};
include "/usr/local/bind/etc/named.conf.options";
// prime the server with knowledge of the root servers
zone "." {
type hint;
file "/usr/local/bind/etc/db.root";
};
// setup local zones
zone "localhost" {
type master;
file "/usr/local/bind/etc/db.local";
};
zone "127.in-addr.arpa" {
type master;
file "/usr/local/bind/etc/db.127";
};
zone "0.in-addr.arpa" {
type master;
file "/usr/local/bind/etc/db.0";
};
zone "255.in-addr.arpa" {
type master;
file "/usr/local/bind/etc/db.255";
};
include "/usr/local/bind/etc/named.custom.zones";
include "/usr/local/bind/etc/named.dlz.zones";
As far as named.conf.options, it is also pretty basic:
options {
directory "/var/cache/bind";
allow-transfer { xxx.xxx.xxx.xxx; xxx.xxx.xxx.xxx; };
zone-statistics yes;
statistics-file "/usr/local/bind/var/stats/named-stats.out";
recursion no;
};
As you can see, I simply included the following configuration portion as named.dlz.zones.
dlz "mysql zone" {
database "mysql
{host=localhost dbname=postfix user=binddlz pass=trickypassword ssl=false}
{select zone from dns_records where zone = '%zone%'}
{select ttl, type, mx_priority, case
when lower(type)='txt' then concat('\"', data, '\"')
when lower(type) = 'soa' then concat_ws(' ', data, resp_person, serial, refresh, retry, expire, minimum)
else data end from dns_records_view where zone = '%zone%' and host = '%record%'}";
};
Now start Bind with the following command and test:
/usr/local/bind/sbin/named -c /usr/local/bind/etc/named.conf -f -g -u named
Useful tips:
* do not include both ns and contact in SOA record, use only respo_contact in either data or resp_contact fields
* make sure you restart Bind every time you restart MySQL or you will lose your connection(s)
* if you would like to create the dns_records table without a view, simply use the dns_values table and add the zone as the first column
One very common task when scripting with bash is to use a for loop to iterate over the contents of a directory or directory tree. There are two primary methods of accomplishing this task; using ls and using find. We’ll not consider the manual method as that would be completely unworthy of our attention.
I find it easy to start with ls when I don’t need to recurse into a directory tree as that is a command that I use often. This often turns into a process such as this:
for dir in $(ls)
do
echo ${dir}
done
Now the above method typically does not work for me. I have an alias setup to print out pretty colors when I issue the ls command and that will cause each command which operates on the variable $dir to fail with a “No such file or directory” error. I always have to remember this and re-write the command with the flag to disable color formatting:
for dir in $(ls --color=never)
do
echo ${dir}
done
The above script will work every time.
The next option is using find. find is awesome and all powerful. Learn and use find. The most common issue when using find is that you may have to filter out the current and/or parent directories when processing the results. Take this example:
for dir in $(find . -maxdepth 1 -type d)
do
echo ${dir}
done
This loop will print out the current directory, as well as all other directories in the current working directory. If you are running some sort of processing within this loop, you may end up re-processing everything unless you discard the current working directory (noted by the dot).
This example will not process the current working directory:
for dir in $(find . -maxdepth 1 -type d)
do
if [ ${dir} == "." ]
then
continue
fi
echo ${dir}
while pushd ${dir}
do
echo ${dir}
done
popd
done
Bash for loops are incredibly useful and easy to work with. Use the above tips and make bash work for you.
I recently had an issue where a file copy from a celerra NAS to a server outside the network was failing and I couldn’t figure out why. The file copy was a pull from the outside server which needed access inside the network. The BGP route had somehow changed to go over Integra’s network rather than Verizon and I couldn’t get anyone to fess up to blocking ports 445 and 139. To solve this issue, I turned to SSH tunnelling.
To setup a tunnel from inside a protected network to expose a resource to an external client, you can use the following format:
$ sudo ssh -N -R 445:cifsNAS:445 outsideserver.com
I then created a hosts file entry on the outside server to map cifsNAS to 127.0.0.1.
#/etc/hosts 127.0.0.1 cifsNAS
What this does is SSH to outsideserver.com and open up port 445 on that host, which will then tunnel all traffic from outsideserver1:445 to cifsNAS:445. This solved my temporary issue and I was able to copy the needed files over.
I recently setup a new Awstats install and used mod_rewrite to make it easier to view web stats. Using the following configuration within a virtual host declaration, you can simply make requests in the following format:
http://awstats/$CONFIG/$YEAR/$MONTH/$DAY/
This is assuming that you run daily rollups.
ServerAlias awstats
RewriteCond %{REQUEST_URI} !^/awstats/awstats.pl [NC]
RewriteCond %{REQUEST_URI} !^/icon [NC]
RewriteRule ^/(.*)/(.*)/(.*)/(.*)/ http://%{HTTP_HOST}/awstats/awstats.pl?databasebreak=day&day=$4&month=$3&year=$2&config=$1 [L,NE]