From Physical Disk to Virtual Machine and Back Again2017-08-042018-06-29Richard Ketcham

Backing up a CentOS Physical Machine

and Cloning it as a Virtual Machine

The CentOS 7 version of the NMR Console software configuration is currently installed in “production” mode on host named AV301.chem.washington.edu.

We want to capture a backup of the current, production configuration, in a format that will serve as a backup in the conventional sense, but also can be used to construct virtual machine clones that take off from the current state of our production machine, and are safely hackable in a virtual-hosted development environment.

AV301 is the target physical machine in this example.

We are working in a VirtualBox virtual host enviroment.

Backup

Examining the disk structure, we have a physical boot partition followed by a physical partition containing an LVM “physical volume”.

[root@av301 ~]# lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT
NAME FSTYPE SIZE MOUNTPOINT
sda 931.5G
├─sda1 ext3 1012M /boot
└─sda2 LVM2_member 929.2G
├─vg_01-homevol ext4 200G /home
├─vg_01-swapvol swap 10G [SWAP]
├─vg_01-rootvol ext4 20G /
└─vg_01-optvol ext4 600G /opt
sdb 74.5G
├─sdb1 ext3 69.9G /export/av301
└─sdb2 swap 1G

Saving the physical disk partition layout:
[root@av301 ~]# sfdisk -d >av301.sfdisk
# partition table of /dev/sda
unit: sectors

/dev/sda1 : start= 2048, size= 2072576, Id=83, bootable
/dev/sda2 : start= 2074624, size=1948594176, Id=83
/dev/sda3 : start= 0, size= 0, Id= 0
/dev/sda4 : start= 0, size= 0, Id= 0
# partition table of /dev/sdb
unit: sectors

/dev/sdb1 : start= 63, size=146480607, Id=83
/dev/sdb2 : start=146480670, size= 2104515, Id=82
/dev/sdb3 : start= 0, size= 0, Id= 0
/dev/sdb4 : start= 0, size= 0, Id= 0

Raw Image Backup of the Boot Block + /boot

I will make a raw disk backup from the begining of the hard drive to the end of the boot partition. I prefer this method because I will only need to raw-copy (i.e., dd copy) this back to my target disk and be able to immediately boot up to kernel load on my new disk without needing to tinker with the boot system.

First I zero-fill the remaining space of the filesystem.

[root@av301 BU]# cat /dev/zero >/boot/zeros
cat: write error: No space left on device
[root@av301 BU]# rm /boot/zeros
[root@av301 BU]# dd if=/dev/sda count=2074623 of=sda.dd.0-1
2074623+0 records in
2074623+0 records out
1062206976 bytes (1.1 GB) copied, 5.40773 s, 196 MB/s
[root@av301 BU]#

Because I zero filled the partition, the empty disk spaces compresses to a negligible size:

[root@av301 BU]# df -h /boot
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 973M 247M 675M 27% /boot
[root@av301 BU]# ls -l sda.dd.0-1
-rwxr-xr-x 1 ketcham user 1062206976 Aug 1 13:48 sda.dd.0-1
[root@av301 BU]# gzip sda.dd.0-1
[root@av301 BU]# ls -l sda.dd.0-1.gz
-rwxr-xr-x 1 ketcham user 246300486 Aug 1 13:48 sda.dd.0-1.gz

Backup of System (Root) Partition

I save the LVM metadata.

[root@av301 ~]# vgcfgbackup
Volume group "vg_01" successfully backed up.
[root@av301 ~]# ls -l /etc/lvm/backup
total 4
-rw------- 1 root root 3167 Aug 4 09:30 vg_01

If I need to recover the disk, I can use vgcfgrestore to replicate the LVM structures from this saved metadata, but only to a raw disk of the same size.

I use the LVM snapshot capability to freeze the system root partition, so that it will not be changing while I am backing it up.

[root@av301 backup]# lvcreate -L5G -s /dev/mapper/vg_01-rootvol -n rootsnapshot
Using default stripesize 64.00 KiB.
Logical volume "rootsnapshot" created.

[root@av301 backup]#mkdir /mnt/rootsnapshot

[root@av301 backup]# mount /dev/mapper/vg_01-rootsnapshot /mnt/rootsnapshot

To reduce the size of the backup, I will exclude some non-essential directories and files.

[root@av301 backup]# tar --one-file-system -czf av301_root.tgz \
--exclude './tmp/*' \
--exclude './root/*' \
--exclude './var/cache/*' \
--exclude ./var/log/* \
-C /mnt/rootsnapshot/ .

I will, however, want to re-create the directory tree structure of the directories I excluded so:
[root@av301 backup]# MYTAR=$(pwd)/av301_misc.tar
[root@av301 backup]# pushd /
[root@av301 backup]# find ./var/cache ./var/log -type d -print0 | \
    tar -cf $MYTAR --null --files-from - --no-recursion
[root@av301 backup]# popd

I am done with my LVM snapshot now, so remove it:
[root@av301 backup]# lvremove /dev/vg_01/rootsnapshot
Do you really want to remove active logical volume vg_01/rootsnapshot? [y/n]: y
Logical volume "rootsnapshot" successfully removed

So now I have the essentials that I need to reconstruct a bootable system: the grub boot block, the boot partition and the system root partition.

Backup Other NMR Stuff

/opt and /home are mounted as separate filesystems on Av301. The user home directories do not need to be backed up. I will only back up the Bruker installation in /opt/topspin and a couple of other small things.

tar -czf av301-nmr-sw.tgz \
/opt/topspin \
/opt/topsense.sh \
/home/nmrsu/protonstd \
--exclude '/opt/topspin/prog/curdir/*'

From Backup to Virtual Machine

With this system backup in hand, I believe I have captured everything I need to completely reconstruct the original system, as in disaster recovery.

However, my immediate objective is to recreate the nmr user environment in  a virtual host test bed, so I want to create a minimal VM that extracts only the essential features I need.  I do not need to recreate the LVM filesystem.

I will greatly simplify my task by starting with a previously saved virtual av301 ancestor which I have in stock (I will use the last av301 VM image in the lineage, the one I converted to the present production physical host).  My saved VM has a simpler disk layout:

bash-4.2# lsblk -o NAME,FSTYPE,MOUNTPOINT,SIZE
NAME FSTYPE MOUNTPOINT SIZE
sda 12G
├─sda1 ext4 /boot 1012M
├─sda2 ext4 / 10G
└─sda3 1G

I will leave the boot partition intact as well as the grub installation (the space between the first sector and the the first partiton).

I start by making a clone my av301 VM, which I will name vv301. I first boot vv301 and update it to match the kernel version of my av301 backup. This is important so that the kernel drivers in the new system will match the booted kernel, which boots off the recycled boot partition.  Then it is convenient to reboot my vv301 VM from a (virtual) live boot media, or network boot, so that the system root partition is not in use while I untar the snapshot backup to the system partition.  I will user clonezilla-live, which I have set up to boot from my boot server.

While pxebooted to clonzilla-live, I mounted the vv301 virtual disk’s system partition and untarred my backup onto it:

bash-4.2# mkdir /mnt/sda2
bash-4.2# mount /dev/sda2 /mnt/sda2
bash-4.2# tar xzf av301_root.tgz --newer -C /dev/sda2 .

I took a shortcut here which needs explanation.  Considering that my target filesystem is a clone of the VM parent of the system in my tar archive,  I am effectively updating that VM to match the current (production) system, so in principle I could “rsync” the differences, if I had both the physical and the virtual filesystems mounted simultaneously.  From a tar archive, I can similarly overwrite only files that have been modified or are missing by using tar’s –newer switch, which is much faster than untarring to an empty filesystem.  However, unlike rsync, files not in the archive will not get deleted from the virtual disk. I did this (tar –newer) with good results, but it probably is a bad idea in the long run.  Since we will be using this method to go from production, physical disk to developmental, virtual machine, and back to physical again, the remnant files would accumulate with each cycle, making a mess, even if not having any functional effect.

So in the future I should first empty the filesystem, then do a complete tar:

bash-4.2# mount /dev/sda2 /mnt/sda2
bash-4.2# rm -rf /mnt/sda2/*
bash-4.2# tar xzf av301_root.tgz -C /dev/sda2

Or, to be cleaner, reformat instead of rm -rf, but then I also have to recreate the partition UUID

bash-4.2# myUUID=`lsblk --noheadings -o UUID /dev/sda2`
bash-4.2# mke2fs -U $myUUID /dev/sda2

because, by default, CentOS uses the UUID to identify partitions in grub.conf, fstab, and initrd.

I will need to modify some configuration files before I boot my developmental VM, vv301.
I will only need a basic network configuration to access kerberos and ldap for authentication and authorization, so I will first undo much of the production network configuration.   A single interface, DHCP client will do. A NAT network is  preferable to avoid conflict with the real, production machine, but for now I have configured my VM with a bridged interface, as it will run on a different subnet.

First I will save the existing config files to RCS so that it will be simple to restore them when I eventually convert back to the production machine. For example /etc/sysconfig/network-scripts/ifcfg-eth0, before I edit it  to change BOOTPROTO=dhcp:

bash-4.2# chroot /mnt/sda2
bash-4.2# cd /etc/sysconfig/network-scripts/
bash-4.2# ci -l ifcfg-eth0
RCS/ifcfg-eth0,v > production ip address of av301
done
bash-4.2# ci ifcfg-eth1
RCS/ifcfg-eth1,v > interface to console for production config
done
bash-4.2# vi ifcfg-eth0

And the other configuration files:

bash-4.2# echo "vv301.chem.washington.edu > /etc/hostname
bash-4.2# vi /etc/hosts
bash-4.2# vi /etc/sysconfig/iptables
bash-4.2# cd /etc/skel
bash-4.2# mv .topspin-av301 .topspin-vv301

[I doubt the following will work from the chroot environment.
Probably easiest to boot up vv301 for final network configuration,
and then reboot]

bash-4.2# systemctl disable openvpn

Also, I will comment out lines in /etc/fstab so as not to mount non-essential partitions so that I will not need bother to restore them.

bash-4.2# vi /etc/fstab

Now I am ready to boot my virtual clone of av301, named vv301. Upon reboot, I with check my network configuration, and then log in with my UW NetID to confirm that the authn & authz systems are working.

 

Variations on Conversion to Virtual

Thoroughly Reconstitute the Physical Machine in a VM
(this section is unfinished)

To perform a complete restoral to a raw disk, first I need to uncompress my raw image and then I will loop mount it

[root@av301 BU]# gunzip sda.dd.0-1.gz
[root@av301 BU]# file sda.dd.0-1
sda.dd.0-1: x86 boot sector; partition 1: ID=0x83, active, starthead 32, startsector 2048, 2072576 sectors; partition 2: ID=0x83, starthead 35, startsector 2074624, 1948594176 sectors, code offset 0x63

Convert to VDI

[rlk@NIXIE av301]$ vbox-img convert --srcfilename sda.dd.0-1 --dstfilename av301.r3.vdi --srcformat RAW --dstformat VDI
Converting image "sda.dd.0-1" with size 1062206976 bytes (1013MB)...

I must resize the virtual disk to make room to restore the system partition onto it.

[rlk@NIXIE av301]$ vboxmanage modifymedium --resize 953870 av301.r3.vdi
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
[rlk@NIXIE av301]$ ls -l av301.r3.vdi
-rw-------. 1 rlk rlk 441450496 Aug 1 15:40 av301.r3.vdi

Now we have a virtual disk with a bootable boot partition. I need to work from within VirtualBox to create an empty partition and filesystem and untar my system partition backup into it. For this I will boot from a “live-cd” medium or network boot. (My VirtualBox development environment is already provisioned with various PXE bootable live media such as GParted, Clonezilla, etc.)

In the first variation, I will proceed without recreating the LVM partitions as on the source disk, but just a simple DOS primary partition formatted with an ext4 filesystem.

After I untar the system partition, I will edit /etc/fstab to reflect my altered partition layout.  Also I will edit the network configuration to run in my VirtualBox development environment

Because I have made changes to the disk structure, I will need to update /boot/initramfs before I can boot

(this section is unfinished…)

 

And Back Again

A general method for converting a VirtualBox image to a raw (physical disk image) is

vbox-img convert --srcfilename vv301.r3.vdi --dstfilename vv301.r3.raw --srcformat VDI --dstformat RAW

And then raw-copy to an  unmounted physical disk (sdg) :

dd if=vv301.r3.raw of=/dev/sdg

But for the present purpose, I will use a shortcut, the inverse of how I cloned the physical disk to the VM image.  I take a tar backup of the system partition and then untar it onto the system partition, or system LVM volume, of the target physical disk, reusing the pre-existing boot sector and boot partition of the target disk.  Again, take care that the same kernel version is installed, updating if needed, and make sure that the partition UUIDs match up.  To create the tar archive, I boot a utility OS on Virtual Box and then mount the source partition, so that it is not in use as I am creating the archive, and I do the same with the target physical disk when I untar.