CentOS 7 ‘Bare Metal’ recovery with Avamar

Overview

This post will detail how to do a ‘Bare Metal’ recovery of a CentOS 7 server with the following caveats:

• Virtual Machine hosted in vSphere
• Managed in Foreman
• Backup with Avamar

The steps were used successfully on CentOS 7 and RHEL 8 with minor differences as noted below. The process should not be limited to a Virtual Machine, vSphere or Foreman and should be able to be modified to suit other variations.

Assumptions

• Level of knowledge is Red Hat Certified Systems Administrator or equivalent
• Some experience in the following
  • Avamar Administrator
  • Foreman
  • VMware vSphere
• Download suitable boot ISO for your OS to boot to Rescue mode
  • Example: CentOS-7-x86_64-NetInstall-2009.iso 

Recovery Scenario

Customer backend storage to vSphere had failed and Virtual Machine Disks had been lost.

Foreman was not affected and all servers were still present in their correct Host Groups with assigned Puppet classes and parameters.

Inspecting a host (server) in Foreman showed that the compute (Virtual Machine tab) had been lost. To correct we first ‘Disassociate’ the host then restore with hammer as follows

# hammer host update --name server1.example.com --compute-resource-id 2 --compute-profile-id 6 --volume datastore=DATASTORE_NAME --compute-attributes cluster=CLUSTER_NAME,cpus=16,memory_mb=32768 --puppet-proxy puppet.example.com

The compute IDs can be discovered by inspecting the listed URLs from the following locations in Foreman to find the ID:

https://foreman.example.com/compute_resources
https://foreman.example.com/compute_profiles

Now you are ready to rebuild your server in Foreman.

Important note: The software snapshot (Foreman Composite Content View) used to rebuild the server must match the backup regard kernel versions. If not there will be a mismatch between the /boot/initramfs used to boot the server and the kernel modules restored by the backup. This is not a showstopper, but will require some manual intervention to create the initramfs or restoration of /boot files from backup.

Avamar Backup Label Number

Launch Avamar Administrator and locate the backup that you wish to restore from. The Label Number highlighted in the example below is 885.

Check all disks are present

It may be that there were disks added after the server was originally built in Foreman. These will be missing from the rebuilt server.

We were able to use historical Zabbix data regard the mounts but you can use Avamar.

SSH to the server and re-activate to Avamar in the same domain. In the example below the /mount1 disk had been added after the server was first built in Foreman and therefore wasn’t restored during the rebuild.

# /usr/local/avamar/bin/avtar --avamaronly --showlog mounts --server=avamar.example.com --id=username --path=/domain/group/subgroup/server1.example.com --labelnumber=885

mount_decision: reason="default_backup" fstype="xfs" path="/mount1"

mount: status="default_backup" path="/mount1" hdev="/dev/mapper/volgroup-logvol" kind="xfs" blksize=4096 freeblks=25654768 maxblks=26200577 freefiles=52425992 maxfiles=52426752 dev=64773

Calculate the disk size in GB:

• (blksize x maxblks) / (1024 x 1024 x 1024)
• (4096 x 26200577) / (1024 x 1024 x 1024) = 99.94 (100GB)

Now add the disk to the Virtual Machine in vSphere and mount to /mount1.

Configure the Recovery Disk

Calculate the disk size needed to restore all mounts. In example let’s say the OS disk was 50GB and the additional disks (/mount1 above) are 100GB. Then in vSphere add an extra 150GB disk to the server. Configure the disk in the OS and mount

# rescan-scsi-bus.sh -a

## assuming new disk is /dev/sdc
# mkfs.xfs /dev/sdc
# blkid /dev/sdc
# cp -a /etc/fstab /etc/fstab_backup-`date +%F`
# echo 'UUID=3a5c0851...  /restore  xfs  defaults  0 0' >> /etc/fstab
# mkdir /restore
# mount -a

Create Avamar Files for Restore

Create avtar command file.

# vim /restore/avtar.cmd

--bindir=/mnt/sysimage/usr/local/avamar/bin
--sysdir=/mnt/sysimage/usr/local/avamar/etc
--vardir=/mnt/sysimage/usr/local/avamar/var
--server=avamar.example.com
--account=/domain/group/subgroup/server1.example.com
--id=username
--ap=password
--target=.

Create restore script.

# vim /restore/avtar-command

#!/bin/bash
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/lib64:/usr/lib:/usr/lib64:/mnt/sysimage/lib:/mnt/sysimage/lib64:/mnt/sysimage/usr/local/avamar/lib
/mnt/sysimage/usr/local/avamar/bin/avtar.bin -x --flagfile=avtar.cmd --labelnumber=885 --exclude=./boot --exclude=./tmp

# chmod +x /restore/avtar-command

Copy configuration files for restore later.

# cp -a /etc/fstab /restore/
# cp -a /etc/sysconfig/network-scripts/ifcfg-eth* /restore/

Restore Server

• Shutdown server
• Configure server in vSphere to boot to rescue mode
  • Add virtual machine boot delay of 5000ms
  • Add optical drive
    • Local client
    • CentOS-7-x86_64-NetInstall-2009.iso (example)
• Launch vSphere Console
• Start server and follow prompts to boot to ISO
• Select ‘Troubleshooting’
• Select ‘Rescue a CentOS system’

You will now be logged into the Console via vSphere with no ability to copy and paste so enjoy the typing to follow and remember tab complete is your friend.

Configure networking.

## CentOS7/RHEL7
# cp -a /mnt/sysimage/etc/hosts /etc/
# cp -a /mnt/sysimage/etc/sysconfig/network /etc/sysconfig/
# cp -a /mnt/sysimage/etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/
# service network restart
# cp -a /mnt/sysimage/etc/resolv.conf /etc/

## CentOS8/RHEL8
# ip addr add 10.x.x.x/XX dev eth0
# route add default gw 10.x.x.x
# cp -a /mnt/sysimage/etc/resolv.conf /etc/

Recover data from backup to the /restore mount. This may take several hours dependent upon size of data to be restored.

# cd /mnt/sysimage/restore
# ./avtar-command

Copy from /restore to mount points to be restored.

:etc
# rm -rf /mnt/sysimage/etc
# cp -a /mnt/sysimage/restore/etc /mnt/sysimage/

:home
# rm -rf /mnt/sysimage/home
# cp -a /mnt/sysimage/restore/home /mnt/sysimage/

:opt
# rm -rf /mnt/sysimage/opt
# cp -a /mnt/sysimage/restore/opt /mnt/sysimage/

:usr
# rm -rf /mnt/sysimage/usr
# cp -a /mnt/sysimage/restore/usr /mnt/sysimage/

:var
# rm -rf /mnt/sysimage/var
# cp -a /mnt/sysimage/restore/var /mnt/sysimage/

:any additional data directories
# rm -rf /mnt/sysimage/mount1
# cp -a /mnt/sysimage/restore/mount1 /mnt/sysimage/

Restore configuration files.

## compare the current and restored /etc/fstab and correct as required
## any UUID mounts for the current /etc/fstab should most likely be kept
# diff /mnt/sysimage/restore/fstab /mnt/sysimage/etc/

## restore network scripts
# cp -a /mnt/sysimage/restore/ifcfg-eth* /mnt/sysimage/etc/sysconfig/network-scripts/

Exit and server will reboot. SELinux may relabel and reboot again.

Issues

By default Avamar doesn’t backup the /var/spool directory so this won’t be restored causing some of the issues listed below.

## CentOS7/RHEL7
## sendmail won't start as /var/spool/mqueue doesn't exist
# yum reinstall sendmail

## anacron can't chdir to /var/spool/anacron
# yum reinstall cronie-anacron


## CentOS8/RHEL8
## postfix won't start as /var/spool/postfix doesn't exist
# dnf reinstall postfix

## anacron can't chdir to /var/spool/anacron
# dnf reinstall cronie-anacron


## All OS
## yum: HTTP error (410 - Gone): Unit abdfca82... has been deleted
## re-subscribe to Foreman
# subscription-manager register --org="my_org" --name="server1.example.com" --activationkey="my-activation-key" --force

## if domain joined run checks and rejoin if required

Tidy Up

• Remove /restore disk and tidy up /etc/fstab
• Remove optical disk drive
• Remove boot delay