Cloud, DevOps, Evangelism

Manually Aligning a VM the Hard Way

Some of you may know that the EMC VNX VSA was the virtualized storage used on most of the VMware Hands On Labs.  Its was a great chance to really help out our VMware brethren.

However, there was a problem, one that we didn't catch.  Vendors are made up of humans, and none of us is perfect, including EMC.  Our friends over at NetApp noted (via Nick Howell's blog) that they had discovered that the VNX VSA was not a properly aligned VM (e.g. it wasn't aligned on a sector boundary).  Nick mentioned that he was going to clarify how they determined it (Nick?  Maybe you can write that post? Hint! Hint!), but in the meantime, I'd like to thank them (NetApp) for helping us out and letting us know.  While we may be competitors, when it comes to the VMworld HoL, we can (and do!) put aside our differences and just help out.

So I went about fixing that oversight, because VM alignment is important and should be fixed, especially when running hundreds of them like at the Labs.  I would have used Nick Weaver's awesome UBERAlign tool, but it doesn't support LVM volumes quite yet, which the VSA uses.  So I had to go after it manually.   I also couldn't do live (for obvious reasons - you can't realign while powered on).  Here was the process I undertook:

Examine the File System - The first order of business was to look at the configuration.  The easiest way to do this was to attach the VM's disk to an existing Linux VM as a secondary disk:

  1. This way, I can make changes to the disk without having to do it live.  In this case, I noted that the single disk on the VM was broken into 2 partitions using the 'fdisk' command:</p>

    There was a single partition marked as bootable (/dev/sdc1 in this case) and a second marked as lvm (/dev/sdc2).   Note that neither starts on a 512-byte sector boundary - they are unaligned.  This is a pretty common Linux structure, and usually means that the boot partition contains basically grub and the kernel (usually ext2 or ext3), and the second partition contains one or more LVM logical volumes.  As a first step, I mounted /dev/sdc1 (the boot partition) at /tmp/boot so that I could back up its contents using the mount command:

    mount /dev/sdc1 /tmp/boot

    Then I investigated the LVM volume.  I ran 'pvs', which looks for LVM Physical Volumes (its short for pvscan).  I saw a single Physical Volume (as expected):

    Then, I ran vgscan to search for volume groups. Because this is a pretty simple VM, there is only one volume group (VolGroup00):

    Further, ontop of that volume group, there are 2 Logical Volume on top of that Logical Volume:

    Lets figure out what those are (just try mounting them):

    OK, now we know that one of them is the root filesystem (ext3) and the other is the swapspace. Pretty Cool!</li>

  2. Copy Off Data - While it can be possible to do this in-situ (UBERAlign can do this!), I find it easier and clearer to do this against a new disk.  By doing it this way, we also reduce the fragmentation in the filesystem and increase the free/zero space, making the VM as small as it can be for best performance.  So, now that we have both the boot and root filesystems mounted, we are just going to copy off the data.  We are going to use rsync for this, because it gives nice status updates, is fast and supports sparse files efficiently (which are used in the VSA).  We copy the data like this:
    rsync -avS /tmp/boot/* /backup/boot/
    rsync -avS /tmp/lv0/* /backup/root/

    In my case, this took about 10 minutes, but it depends on how fast your storage is, and how much data you have.</li>

  3. Recreate Partition Table - We now need to recreate the partition table in such a fashion that its aligned.  In short, we need to make sure that partitions start on a 512 byte boundary, which fdisk doesn't do by default (ugh!).  So lets use GNU parted to modify this partition table:You'll get a number of errors about informing the kernel, but this is normal.  At this point, its best to reboot your helper VM, just so we can clear out any of the LVM state (its possible to do this live, but its a pain).How did I choose the starting values of 4 and 120?  Well, I chose them because I used parted's ability to set the units:units MiB

    This ensures that all future numerical values are in Mibibytes, and will therefore be aligned on a 512 byte boundary.  It also makes it easier to think in Mibibytes.  This is the most important step, so get this right!</li>

  4. Create LVM Setup and Filesystems- Now we need to recreate the setup we had before.  First, we create the physical volume, volume group, and logical volumes as they existed before:The '-l 100%FREE' technique tells LVM to create the Logical Volume with whatever space remains in the volume group.  In other words 'Use the rest'.
  5. Copy Data Back - Now we copy all the data back, using the same method as above (rsync):
    rsync -avS  /backup/boot/* /tmp/boot/
    rsync -avS  /backup/root/* /tmp/root/
  6. Fix the Bootloader- Now that we have all the data copied, you'd think we were done, but we have one more step.  Because we moved everything around, the bootloader, called grub, will be confused about where its boot files are (because it uses disk offsets).  We need to reinstall the boot loader.  Fortunately, this is a simple process.  We start the grub bootloader utility, then tell it where to consider its grub root (NOT the same as the root filesystem).  After that we have it setup grub on the drive (in this case hd2, because its the third drive in this system):
  7. Disconnect - From here, just shutdown your helper VM, disconnect the VM's HD from the helper, and boot your old VM.  It should boot up just fine!
  8. </ol>
    Phew!  That was a lot of work, and now you know how to do it!   Just use UBERAlign instead, if you can!