Monitoring the ESXi Upgrade Process

When doing manual host upgrades, either through the direct method or via a locally placed upgrade bundle, there is a distinct lack of progress information available after running the esxcli command.

Thankfully the ESXi host provides a running logfile of the upgrade process, which makes it much easier to keep track of what is going on and that the upgrade is indeed being performed.

The esxupdate.log is located in /var/log, and by issuing the following command in a terminal window you can have a rolling log showing you the upgrade status and progress:

[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
~ # tail -f /var/log/esxupdate.log
[/cc]

By running the following command in one terminal window (this uses the VMware offline bundle to upgrade from ESXi 5.1 to 5.1 Update 1):

[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
~ # esxcli software vib update -d /vmfs/volumes/[lunID]/update-from-esxi5.1-5.1_update01.zip
[/cc]

you get output like this in the secondary terminal window where the log file is being monitored:

[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
2013-06-26T10:24:51Z esxupdate: HostImage: INFO: Attempting to download VIB tools-light
2013-06-26T10:25:07Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = ‘/sbin/backup.sh 0 /altbootbank’, outfile = ‘None’, returnoutput = ‘True’, timeout = ‘0.0’.
2013-06-26T10:25:08Z esxupdate: BootBankInstaller.pyc: INFO: boot config of ‘/altbootbank’ is being updated, 5
2013-06-26T10:25:08Z esxupdate: HostImage: DEBUG: Host is remediated by installer: locker, boot
2013-06-26T10:25:08Z esxupdate: root: DEBUG: Finished execution of command = vib.install
2013-06-26T10:25:08Z esxupdate: root: DEBUG: Completed esxcli output, going to exit esxcli-software
[/cc]

That sure beats waiting “blindly” for an upgrade/installation to finish, and in many ways this is also much better than a non-sensical progressbar.

Backing up vCenter DB with Veeam B&R 6.5

It´s a well known problem that with Veeam Backup & Recovery Replication 6.5, and earlier, backing up the SQL server that hosts the vCenter DB poses a problem. KB1051 VSS for vCenter outlines the issue well, and provides a workaround.

If you experience this problem, you will see entries like this in your Veeam B&R backup logs:

Veeam vCenterDB Backup Error
Veeam vCenterDB Backup Error

The workaround provided by Veeam is to create host VM-Host Affinity Rules, effectively pinning a VM to a given host, and then perform the VM backup through the host rather than through the vCenter.

While this might be a way to work around the failed backup scenario, it effectively limits some of the flexibility you have in a virtualized environment. I want to have my VMs roaming around my datacenter utilizing DRS, and by pinning the VM to a given host that flexibility is reduced for the VMs in question.

I can appreciate why this issue appears, and why the workaround works, but frankly there has to be a better way of doing this. Hopefully this issue will be sorted in the next v7 version of Veeam B&R, as there certainly are ways for Veeam to work around the issue and make this a more seamless experience for the backup administrator.

Proposal

Why not create a new option in the backup job, where you define that this particular job should run through a host, instead of the vCenter?

If selected, Veeam B&R would query the vCenter for which host the given VM resides on, then connect to the host itself and perform the backup through that?

That way we could work without having to set VM-host Affinity Rules, yet still keep vCenter out of the loop when the actual backup is performed.

Doing such a query is pretty easy,  below is an example using PowerCLI:

[cc lang=”powershell” width=”95%” theme=”blackboard” nowrap=”0″]
Get-VMHost -VM dbserv | fl -Property Name
Name : esx02
[/cc]

 

This simple query returns the host that the given VM resides on at the given time, why not do something like this inside of Veeam B&R to make sure that vCenter DB backups work, without having to resort to VM-Host Affinity Rules?

Preserve your Veeam B&R Backups Jobs when Moving vCenter

This week I´m working at a client site, upgrading their entire existing vSphere 4.1 infrastructure to vSphere 5.1. The customer engagement also includes upgrading Veeam Backup and Replication 6.0 to 6.5, and usually an isolated upgrade of Veeam B&R is a no-brainer next, next, next, done install.

To complicate things in this particular environment, I also had to migrate the vCenter SQL DB from a local MS SQL Server 2005 Express instance to a full-fledged MS SQL Server 2008 R2 instance.

Of course, it was also decided to move the vCenter installation from a physical server, to a VM in the same operation.

To be able to have an exit path, until the vSphere 4.1 hosts management agents, were reconfigured, a new vCenter VM was created with a new IP-adress and server name.

After the migration from vCenter 4.1 to 5.1, and from physical to virtual was completed, the existing Veeam B&R server was upgraded to the latest 6.5 release.

Now, this is where things started to get a bit interesting with regards to the existing Veeam B&R backup jobs.  As far as it was concerned, the new vCenter was unknown and the old one was no longer anywhere to be found on the network.

If you add the new vCenter to the Veeam B&R server, you will need to either redefine all the existing backup jobs by adding the same VM from the new vCenter to the existing job, and keep the old one around until your retention period has passed (and have it fail on the existing VM object until it´s removed) or you will have to recreate all the jobs pointing to the new vCenter instance and start new “first time backups” for all the VMs.

For some reason you simply can not rename/redefine your vCenter connection inside of the Veeam B&R GUI.

Thankfully there is an easy way to work around this issue, without having to mess about with the Veeam B&R database: Create a DNS alias!

That´s right, the solution was that simple. I created a DNS CNAME  alias pointing the old vCenter network name to the new vCenter network name. After doing that, I had to re-enter the credentials for the vCenter connection in Veeam B&R to force a reconnect, and all of a sudden all existing backups jobs where present again and working as intended.

The reason this works, is that when you change the vCenter name and/or IP address (or even move it to a new server) it does not change the VM identification number (vmid) or Managed Object Reference (MoRef), in essence the VMs stay the same and Veeam B&R can continue managing them as before.

Now, can we please get an option to update the registered vCenter name in Veeam Backup & Replication v7.0?

Update:

Check out Veeam vCenter Migration Utility for a small utility to help you move Veeam B&R jobs after replacing your vCenter.