Archive for March, 2010

Linux guest hangs at “starting udev” (VMware vSphere)

Tuesday, March 23rd, 2010

Having recently upgraded the Virtual Infrastructure at work to vSphere, I have encountered many scenarios with CentOS 5.3 guests not booting or taking a long time to boot. The last message on the console typically indicates that it’s hanging while starting udev.

The fix for this issue is to ensure proper time keeping practices have been adhered to in accordance with the VMware Timekeeping KB.

The basic steps are:
1. Modify the kernel line of /boot/grub/grub.conf to include the following line:

clocksource=acpi_pm notsc divider=10

clocksource=acpi_pm – uses the Power Management Timer (PMTMR) available in some southbridges as primary timing source
notsc – disable the timestamp counter
divider=10 – reduces the frequency of timer interrupts by 10 (from 1000/second to 100/second)

2. Disable time sync through VMware tools (note that this will continue to happen on bootup, pause, resume, etc..):

vmware-guestd --cmd "vmx.set_option synctime 1 0"

3. Setup time sync through NTP:
a. Setup your /etc/ntp.conf to point to a good NTP server pool.
b. Set NTP to start and persist across reboots.

# yum -y install ntp
# chkconfig ntpd on
# /etc/init.d/ntpd start

Expanding an ext3 filesystem online

Thursday, March 4th, 2010

One common scenario that I face in my daily work is to add disk to various filesystems. Setting up systems correctly so that this is possible will save time and frustration. One of the easiest cases is adding disk to a virtual machine when the guest is using LVM and ext3.

As always, please be sure to backup your data before trying any filesystem or disk manipulation.

After adding the virtual hard disk using the VI client, provision the space from within the virtual machine using the following steps:

1. re-scan storage

echo "- - -" > /sys/class/scsi_host/host0/scan

2. Create physical volume from new device (Note: check with your SAN admin to see if you need to create a partition and align appropriately.)

pvcreate /dev/sdb

3. Extend the volume group to the new PV (physical volume):

vgextend vg01 /dev/sdb

3. Extend the LV (logical volume) to the desired size:

lvextend -L +2G /dev/vg01/lvol05

4. Resize the filesystem to cover the newly extended LV:

resize2fs /dev/vg01/lvol05

Your newly resized filesystem should now be available.

I have not yet tried expanding existing VMDK files on the fly with vSphere but I plan to test that out next.