Monitoring the ESXi Upgrade Process

When doing manual host upgrades, either through the direct method or via a locally placed upgrade bundle, there is a distinct lack of progress information available after running the esxcli command.

Thankfully the ESXi host provides a running logfile of the upgrade process, which makes it much easier to keep track of what is going on and that the upgrade is indeed being performed.

The esxupdate.log is located in /var/log, and by issuing the following command in a terminal window you can have a rolling log showing you the upgrade status and progress:

[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
~ # tail -f /var/log/esxupdate.log

By running the following command in one terminal window (this uses the VMware offline bundle to upgrade from ESXi 5.1 to 5.1 Update 1):

[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
~ # esxcli software vib update -d /vmfs/volumes/[lunID]/

you get output like this in the secondary terminal window where the log file is being monitored:

[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
2013-06-26T10:24:51Z esxupdate: HostImage: INFO: Attempting to download VIB tools-light
2013-06-26T10:25:07Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = ‘/sbin/ 0 /altbootbank’, outfile = ‘None’, returnoutput = ‘True’, timeout = ‘0.0’.
2013-06-26T10:25:08Z esxupdate: BootBankInstaller.pyc: INFO: boot config of ‘/altbootbank’ is being updated, 5
2013-06-26T10:25:08Z esxupdate: HostImage: DEBUG: Host is remediated by installer: locker, boot
2013-06-26T10:25:08Z esxupdate: root: DEBUG: Finished execution of command = vib.install
2013-06-26T10:25:08Z esxupdate: root: DEBUG: Completed esxcli output, going to exit esxcli-software

That sure beats waiting “blindly” for an upgrade/installation to finish, and in many ways this is also much better than a non-sensical progressbar.

Installing and configuring VMware vCenter Operations

VMware vCenter Operations was released to the general public a week or so ago and is available for download right now. As usual you can download a 60 day trial, and get started immediately.

Like other recent management utilities from VMware, vCenter Operations comes in the form of a .OVF template (like vCMA/vMA).

Installing VMware vCenter Operations

Download VMware vCenter Operations and import by starting vCenter Client, navigate to the “File” menu and select “Deploy OVF template…

Browse to the download location, and find the “VMware-vcops-” file. Select it and click open.

Click on “Next” and review the details.

Hit “Next” once more, and click on “Accept” to accept the VMware EULA and enable the “Next” button.

Specify the name and location of the VMware vCenter Operations VM, and click “Next” to continue.

Now we get to select which host or cluster the VM should be deployed to. Make your choices, and click on “Next

Select your preferred resource pool, if you have any, and once again click “Next

Now select your preferred datastore, and guess what? We get to click “Next” one more time!

Decide if you want a thin or thick provisioned VM, the default is thick but I went with thin provisioned in this particular setup.

The last configuration item, for now, is to map the networks. Select your network mappings and click on “Next”.

Review the final setup screen, and once you are satisfied that your settings are correct, click on “Finish” to start the OVF template import.

The import starts, and after a few minutes it should be ready to go!


Time to start it up!

Configuring VMware vCenter Operations

After the vCenter Operations VM has finished booting, it displays a little information screen showing the IP address and other tidbits of information. The most important piece of information right now is the DHCP assigned IP address. Make a note of that IP for later.

To make sure we don’t run into problems with time synchronization we need to make sure that the vCenter Operations VM time is synchronized with the ESX host time. To do so, right click on the VMware vCenter Operations VM inside of the vCenter Client, select “Edit Settings”.

Select the “Options” tab, and find the VMware Tools section.

Find the “Synchronize guest time with host” option, and select it.

Open the vCenter Operations web page in a browser, and log in. The default username/password for vCenter Operations is admin/admin

Log in, and follow the directions on screen to change the default username/password.

The new password must be at least 8 characters, and at least one digit and one character.
Note: This also changes the root account password for the vCenter Operations VM.

Next up is configuring the vCenter Operations connection to the vCenter.

Fill out the vCenter Server information form, with information pertinent to your infrastructure.
Note that the registration credentials needs to have administrator privileges on the vCenter Server. You can use the same credentials for both registration and collection, or you can differentiate them if required in your environment.

Click on “Save“, and a test is performed to make sure that the information provided is correct.

If registration is successful, a new popup appears explaining that you need to use the vSphere Client to assign the vCenter Operations licenses.

Click on “Ok” and the vCenter Operations setup dashboard appears in your browser.

Go back to your vCenter Client and navigate to the “Home” screen.

You should now see the new “vCenter Operations” icon under “Solutions and Applications”. If it does not appear immediately, close the vCenter Client and restart it to have it pick up the installation.
To install the vCenter Operations license, go to “Home” and find the “Licensing” icon.

Click on it, and change the “View by:” option to “Asset
Right click on “vCenter Operations” and select “Change License Key

Select “Assign new license key to this solution”, click on “Enter Key…” and enter your license key and optionally a label for the key. Click on “OK” to return to the “Assign License” window, and click on “OK” again to install the license.

Your license should now be installed and active.

Go back to the vCenter Client “Home” screen and find the vCenter Operations icon under “Solutions and Applications”. Click on it, and vCenter Operations should already be active monitoring your infrastructure.

Thats it! You now have VMware vCenter Operations running in your environment. For details on how it works and reports your operations refer to the VMware vCenter Operations official documentation. Eric Sloof has also posted a couple of great videos in his VMware vCenter Operations – Troubleshooting Workflow post that gives an in-depth overview of what VMware vCenter Operations is capable of doing.

Virtual Admins – Virtual Pirates?

When Rich Brambley posted “A Pirate Invented Server Virtualization” today, it reminded me of a little story from my own production environment.

This story is a couple of years old, but sadly it’s still valid. A very specialized application that we run, requires a SQL Server Express instance, a proxy/licensing server and client installation with license files to work. This application isn’t very advanced, nor very resource intensive so by nature it’s prime for running in a virtualized environment. None of the application developers customers had ever attempted setting it up in a virtualized environment, and so we offered to be a pilot customer and test it live in our environment. The testing confirmed what we thought, it worked perfectly when virtualized! We were happy, and so were the developers. At least, that’s what we thought.

In fact, what happened next is one of the more bizarre fall outs I’ve personally seen when it comes to implementing a “virtualize first” strategy.

When we got the final version of the software, out of the testing phase, I got about my business of installing and configuring it. All was fine, until I got to the part where I was supposed to install the proxy/licensing service. It turns out that the application developers had put checks in place to prevent installation on virtual machines!

After we tested and verified the solution in our environment, the developer turned around and blocked us from implementing it as we wished. In fact, we still run this service on one of the few physical servers we still have in place in our data center. If it were up to me, I would have kicked the application right out of our environment, but sadly core parts of our business relies on this application and we are stuck with it.

The developers reasoning for doing this was that it was way to easy to duplicate the proxy/licensing service in a virtual environment, and by doing that we could potentially bypass the concurrent user license model they had put in place. Their checks are based on a hardware id generated by the physical hardware thus it could potentially be the same if we duplicated the VMs. They could of course work around that, by using the server DNS name as one of the hardware id checks or even the NIC MAC address, but sadly they opted to completely block the installation and operation of that particular part of their infrastructure if you run virtualized.

Bad call? Very much so, and I have made this very clear to them to no avail.

So, perhaps Dilbert is right? Was virtualization indeed created by pirates? I would rather be a Ninja than a Pirate any day.