Command line replace with perl

February 5th, 2010

I often run into issues on the command line where I’d like to perform a non-greedy search and replace. This is not possible with sed, grep, or egrep, AFAIK, so I must resort to perl. Here is how it’s done:

perl -pe 's|"http://(.*?)/.*$|$1|g' 

The above example will take a list of requested URIs or search domains from an apache log and print out only the domain. I’m using pipes instead of slashes in the regex to eliminate the need to escape the slashes.

Note the question mark used to make a non-greedy match.

Apache – Getting rid of (internal dummy connection) in the logs.

February 4th, 2010

On busy sites the (internal dummy connection) message in apache logs can be a major annoyance as it fills the logs. A simple way to filter this out is to use the following log declarations (assuming IPV6 capable host):

SetEnvIf Remote_Addr "::1" dontlog

CustomLog /var/log/httpd/access_log combined env=!dontlog

RE: http://wiki.apache.org/httpd/InternalDummyConnection

Decode base64 text using perl from the command line

February 3rd, 2010

I recently ran into an issue where I needed to decode some base64 text from the command line and used perl to manage the task:

perl -MMIME::Base64 -ne 'print decode_base64($_)' < /tmp/input.txt

RE: http://perldoc.perl.org/MIME/Base64.html

Removing memberUid from OpenLDAP group

February 2nd, 2010

The following syntax can be used to remove a user from an OpenLDAP group:

$ ldapmodify -x -D "cn=manager,dc=example,dc=com" -W <<EOF
> dn: cn=sshusers,ou=groups,dc=example,dc=com
> changetype: modify
> delete: memberUid
> memberUid: previousMember
> EOF

VIM Control Characters – search and replace

January 25th, 2010

Control characters require an escape sequence prior to using in a search and replace operation in VIM. In Linux, the escape character is Ctl-v. Windows requires a Ctl-q.

Remove all ^M characters from a file on Linux:

%s/^V^M//g

Remove all ESCAPES (\27, ^[) from a file on Windows:

%s/^q^[//g

Bash Tip! Renaming files using Bash string operations

January 6th, 2010

To rename all html files in a particular directory to shtml files, use the following loop:

for file in *.html
do
  mv ${file} ${file%%.html}.shtml
done

This uses the ${variable%%match} format which strips the longest match from the end of the variable.

Samba 3.4 Changes idmap backend!

November 9th, 2009

I recently upgraded some hosts to Fedora 11 which has Samba 3.4 included. I configure most of the hosts I control to be integrated with Active Directory for authentication and this upgrade broke that authentication.

The problem was that the winbind daemon was not able to query the LDAP server which was used as the idmap directory due to the way I had configured the idmap backend in smb.conf. This caused winbind and samba to restart successfully and I could enumerate groups and users perfectly well in bulk. I was not able, however, to enumerate a single user using ‘wbinfo -i ‘ or groups and I was not able to login.

This problem was caused by this config value:

idmap backend = "ldap:ldap://ldap1,ldap://ldap2"

After a bit of troubleshooting, I discovered that winbind was not able to query the LDAP server successfully. I fixed this issue by changing the above config value to:

idmap backend = ldap:ldap://ldap1 ldap://ldap2

I then restarted winbind and tested failover by enumerating a few users and then stopping the primary ldap server (ldap1) and enumerating a few more users.

error while loading shared libraries: : cannot open shared object file: No such file or directory

June 1st, 2009

A fairly typical scenario when installing software that does not come fro the distribution package manager is to install an application and find that it cannot find a library necessary to run, although the library is definitely installed.  I recently ran into this issue when compiling spine on an old Debian Sarge system.

$ ./spine
./spine: error while loading shared libraries: libmysqlclient_r.so.15: cannot open shared object file: No such file or directory

I verified that this file existed with locate.

$ locate libmysqlclient_r.so.15
/usr/local/mysql-5.0.45-linux-i686/lib/libmysqlclient_r.so.15.0.0
/usr/local/mysql-5.0.45-linux-i686/lib/libmysqlclient_r.so.15

The problem was that I did not have the mysql library directory included in /etc/ld.so.conf or /etc/ld.so.conf.d.  To fix the problem, I added the mysql library path to /etc/ld.so.conf and re-ran ldconfig using sudo.

#/etc/ld.so.conf/usr/local/lib/sasl2
/usr/local/lib
/lib
/usr/lib
/usr/lib/atlas
/usr/local/mysql-5.0.45-linux-i686/lib

include /etc/ld.so.conf.d/*.conf

Then run ldconfig:

$ sudo ldconfig

I was then able to run spine without any issues.

History with time stamps!

May 21st, 2009

When reviewing the history file in bash, it’s terrible not knowing when a command was executed.  Using the HISTTIMEFORMAT variable in a .bashrc file, the timestamp can be added to all commands.

# ~/.bashrc
HISTTIMEFORMAT=”%m/%d/%y %I:%M:%S %p ”
Sample output:

525  05/21/09 07:56:46 PM tail -f /var/log/messages  /var/log/secure

As you can see, the command is preceeded by the line number and a timestamp.

sudo PATH and bash

May 21st, 2009

I just spent a bit of time troubleshooting a path issue with sudo on a CentOS server.  This is a note to self to preserve this bit of knowledge.

When dealing with sudo, the only time the PATH environment variable will be preserved is if it’s in .bashrc and NOT in .bash_profile.