Upgrading to vCenter 8 Update 1: Invalid Type, expected String, instead got NoneType

vCenter 8.0 Update 1 was released on April 18th, and I quickly jumped to upgrading my existing home lab vCenter, but ran into an issue that prevented the upgrade from completing, namely the error message “Invalid Type, expected String, instead got NoneType”.

Obviously something was wrong with my current vCenter installation, but finding out what was required to correct it wasn’t straight forward. Thanks to my partner in crime, and notorious human log file parser, Espen Ødegaard, I found the following in /var/log/vmware/applmgmt/PatchRunner.log:

2023-04-19T09:10:39.560Z vpxd:Expand ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vpxd:Expand' failed.
(...)
Traceback (most recent call last):
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 40, column 7

Note: The log output has been truncated for legibility.

Clearly something is not conformant in the vpxd.cfg file on my vCenter, on line 40.

Opening /etc/vmware-vpx/vpxd.cfg showed the following:

 38   <vcls>
 39     <clusters>
 40       <78ed4e3c-4f28-4d27-a0d5-01182e957269>
 41         <enabled>true</enabled>
 42       </78ed4e3c-4f28-4d27-a0d5-01182e957269>
 43     </clusters>
 44   </vcls>

This is not correct, it should be referencing <domain-id> and not an UUID on line 40/42. It seems I have messed up the procedure for cleaning up vCLS VMs at some point, putting in the wrong configuration as a vCenter Advanced setting.

I tried cleaning this up via the vCenter Web Client, but could not find the config.vcls.clusters.domain-c(number).enabled configuration parameter at all, even when searching for only cls.

Manually cleaning up vpxd.cfg did the trick though, correcting the configuration to the correct settings:

 38   <vcls>
 39     <clusters>
 40       <domain-c59002>
 41         <enabled>true</enabled>
 42       </domain-c59002>
 43     </clusters>
 44   </vcls>

Once this was fixed, I restarted the vCenter appliance and restarted the upgrade process, but since there was a failed upgrade attempt already I had to clean up the failed upgrade state by running the following via SSH on the vCenter appliance:

service-control --stop applmgmt
rm -rf /storage/core/software-update/updates/*
rm -rf /storage/core/software-update/stage/*
rm -rf /storage/db/patching.db
mv /storage/core/software-packages/staged-configuration.json /storage/core
mv /etc/applmgmt/appliance/software_update_state.conf /storage/core
service-control --start applmgmt

Ref: vCenter Server update fails with error “Installation failed - Install in progress - You have reached maximum number of retries to resume the patching. Please restore the vCenter using the backup.” (87238).

Once that step was performed, a new upgrade attempt was performed, this time with a successful result!

Obviously putting wrong configuration settings in either via vCenter Advanced Parameters or directly via editing vpxd.cfg can get you into trouble, and should be avoided if possible in production environments — Especially if you mess them up.

Related Posts

• Beware Of The Rogue VMs! — Published 11 Nov, 2024
• VMware by Broadcom Promises Free Security Updates for vSphere — Published 16 Apr, 2024
• VMware vCenter CVE-2022-31697 — Published 20 Dec, 2022
• ESXi5.5 to 6.0 Upgrade From Local HTTP Daemon — Published 26 Aug, 2015
• Automatically Name Datastores in vSphere? — Published 18 Feb, 2014