Alerting on “Bootbank cannot be found at path ‘/bootbank’” in vRealize Operations

If you boot your ESXi hosts from SD-cards or USB you might have run into this issue. Suddenly your host(s) displays the following under events:
“Bootbank cannot be found at path ‘/bootbank’.”

Usually this means that the boot device has been corrupted somehow, either due to a device failure or other issues. Normally the host continues to run, until it’s rebooted that is…

For some reason, vRealize Operations doesn’t pick this up as a host issue that it alerts on, so if your alerting regime is based on vROps alerts, you might not get alerted immediately. Thankfully there is a way to remedy this, and have vROps and vRealize Log Insight work together at the same time.

On order for this to work, you need to have configured the vRealize Log Insight Integration with vRealize Operations first.

Log in to Log Insight and search for “Bootbank cannot be found at path ‘/bootbank’.”. If you want to restrict it even more, use two filters. One for vc_event_type = exists, and one for the text search itself.

Click on the little red bell icon and select “Create Alert from Query”.

This will bring up the “Edit Alert” window, where you can define your information.

Create a proper Description and Recommendation for the alert, and enable “Send to vRealize Operations Manager”. You also need to specify a Fallback Option. The Fallback Option is basically which object Log Insight should attach the alert to, if the originating object isn’t found in vRealize Operations.

And that’s it really, as long as the vLI and vROps integration is configured and working, it’s easy add your own custom alerts in vRLI, and have them pop up in vROps.

If you want to copy my Description and Recommendations, here they are:

Description:

“The device containing the VMware ESXi bootbank can not be found. This may be because of a boot device failure. Specific details should be available in the symptom details. For more information, check the Tasks & Events pane for the host in the vSphere Web Client”

Recommendation:

“Change or replace the boot device, if necessary. Contact the hardware vendor for assistance. After the problem is resolved, the alert will be canceled when the sensor that reported the problem indicates that the problem no longer exists.”

 

Cross vCenter VM Mobility Fling – macOS?

The VMware Cross vCenter VM Mobility – CLI was recently updated so I decided to try it out. In short, this little Java based application allows you to easily move or clone VMs between disparate vCenter environments.

The Fling is listed with the following requirements:

  • JDK 1.7 or above
  • Two vCenter instances with ESX 6.0
  • Windows : Windows Server 2003 or above
  • Linux : RHEL 7.x or above, Ubuntu 11.04 or above

There is no mention of macOS there, but I decided to give it a go any way, and it turns out that it works just fine on macOS as well!
Just make sure you have the Java JDK installed locally. When I ran it the first time, I got the following error, since the JAVA_HOME environment variable was not set.

[cc lang=”bash”]
~/Downloads/xvc-mobility-cli_1.2$ sh xvc-mobility.sh
set JAVA_HOME to continue the operation
[/cc]

This is very easy to fix, just run the following command in your terminal of choice, and xvc-mobility.sh should work just fine on your Mac.

[cc lang=”bash”]
export JAVA_HOME=$(/usr/libexec/java_home)
[/cc]

Next up is running the fling with the correct parameters (this is a clone operation, not a relocate):

[cc lang=”bash”]
~/Downloads/xvc-mobility-cli_1.2$ sh xvc-mobility.sh -svc [source-vcenter] -su [source-vcenter-username]
-dvc [destination-vcenter] -du [destination-vcenter-username]
-vms [vm-name] -dh [destination-host]
-dds [destination-datastore] -op clone -cln [destination-vm-name]

13:41:40.591 [main] INFO com.vmware.sdkclient.vim.Task – CloneVM_Task | State = SUCCESS | Error = null | Result = [email protected]
13:41:40.597 [main] INFO com.vmware.sdkclient.vim.Task – Monitor task end
13:41:40.597 [main] INFO com.vmware.sdkclient.vim.Task – CloneVM_Task took : 0:51:33.728
13:41:40.603 [main] INFO c.v.s.helpers.CrossVcProvHelper – Successfully cloned the vm:[destination-vm-name]
[/cc]

I was able to clone a VM from my lab in Bergen to my lab in Oslo, without any problems what-so-ever. Not only is that a Cross vCenter vMotion, but also a Cross Country one, awesome!

Now this is just an example, please check the official documentation for all the parameters, and what the tool expects.

VMware vSphere 6.5 PSOD: GP Exception 13

While at a customer site, migrating an old vSphere 5.5 environment to 6.5, several hosts suddenly crashed with a PSOD during the migration. Long story short, we got hit by this: VMware KB 2147958: ESXi 6.5 host fails with PSOD: GP Exception 13 in multiple VMM world at VmAnon_AllocVmmPages (2147958)

It turned out that a bunch of the VMs we were vMotioning from the old environment had the cpuid.corePerSocket advanced setting set in the .vmx file, and this can cause ESXi 6.5 to enter a state of panic, and in our case it certainly did.

Upgrading the hosts to 6.5a, like the knowledgebase article states, alleviated the issue and we did not experience PSOD’s again while migrating the 100+ VMs from the old environment to the new one.