ESXi 7.0 SD Card/USB Drive Issue Temporary Workaround

by Espen Ødegaard · Read in about 4 min (725 words)

Guest Post #

Info

Espen Ødegaard

This is a guest post by Espen Ødegaard, Senior Systems Consultant for Proact.

You can find him on Twitter and LinkedIn. Espen is usually found in vmkernel.log, esxtop, sexigraf or vSAN Observer. Or eating, he eats a lot.

As VMware has not released a fix yet (regarding issues with SD card and USB drive), I’m still experiencing issues with ESXi 7.0 U2a Potentially Killing USB and SD drives, running from USB or SD card installs. As previous workaround (copying VMware Tools to RAMDISK with option ToolsRamdisk) only worked for 8 days (in my case), I needed something more “permanent”, to get the ESXi-hosts more “stable” (e.g. host being able to enter maintenance mode, move VMs around, snapshots/backup, doing CLI-stuff/commands, etc.).

Stopping the “stale IOs” against vmhba32 makes ESXi happy again #

As I previously just janked out my USB (which also works, by the way), I needed something more remote-friendly. Mentioned other places, a combination of esxcfg-rescan -d vmhba32, and restarting services/processes currently using the device (vmhba32), frees up the “stale/stuck IOs”, and ESXi is “happy again” (most things seems to be working fine, as VMware ESXi runs fine in RAM).

That said, any “permanent configuration changes” to ESXi, etc. will not work, as all IOs against the device (which stores the changes) are not working.. This includes trying to patch the host. The device is in other words marked as failed, with APD/PDL (which I’m guessing is why the host is somewhat working again; not trying any IOs against the vmhba32 device equals no timeouts. No timeouts equals working processes etc. (wild guess). A quick reboot seems to make the drive working again, luckily. But only until the issue resurfaces (hours or days, in my experience so far).

Checking the possible options for esxcfg-rescan #

[root@esx-14:~] esxcfg-rescan -h
esxcfg-rescan <options> <adapter>
   -a|--add       Scan for only newly added devices.
   -d|--delete    Scan for only deleted devices.
   -A|--all       Scan all adapters.
   -u|--update    Scan existing paths only and update their state.
   -h|--help      Display this message.

Running esxcfg-rescan -d on the device that has issues #

In my case, it’s vmhba32

[root@esx-14:~] esxcfg-rescan -d vmhba32
Rescan complete, however some dead paths were not removed because they were in use by the system. Please use the 'storage core device world list' command to see the VMkernel worlds still using these paths.

Check for any process (worlds) currently using the device #

[root@esx-14:~] localcli storage core device world list|egrep -ie '(device|mpx)'
Device                  World ID  Open Count  World Name
mpx.vmhba32:C0:T0:L0    1051918           1   hostd
mpx.vmhba32:C0:T0:L0    1424916           1   localcli

Here we see that device mpx.vmhba32:C0:T0:L0 is being used by hostd (with PID 1051918) Tip: You may also just run the command localcli storage core device world list, and check the output. I simply added a filter on device & mpx only, to limit output.

Restart hostd, if needed (or any other processes locking the device) #

[root@esx-14:~] /etc/init.d/hostd restart
watchdog-hostd: Terminating watchdog process with PID 1051906 1051182
hostd stopped.
/usr/lib/vmware/hostd/bin/create-statsstore.py:30: DeprecationWarning: pyvsilib is replaced by vmware.vsi
  import pyvsilib as vsi
hostd started.

Re-check if any process is still using the device #

[root@esx-14:~] localcli storage core device world list|egrep -ie '(device|mpx)'
Device                                                                    World ID  Open Count  World Name

Note: After restarting the hostd process in my case, I needed to wait another 2-3 minutes, sometimes, before the world was actually stopped, and the process was no longer using vmhba32 (guessing another timeout).

Results #

Commands like df -h etc. should now work, and you may set the host in maintenance mode, completing vMotion & evacuating VMs as usual (which was stuck before), or do “CLI stuff”. Other procedures which might have failed before, now may start working again. So after vmhba32 is “flagged as failed”, you may

  • Enter maintenance mode, if needed.
  • Evacuate VMs/vMotion, etc, as usual.
  • Take snapshots of VMs.
  • Pre-checks (scripts) work.
  • Do CLI commands (which previouly got stuck).
  • Reboot host.

Also: After VMware releases a fix for this, I simply plan to reboot the host first (makes the device working again) , then apply the patch, etc. Hopefully it won’t be that long, until a fix is released. For now, I’ll apply this “workaround” in my environment, which seems to be better than stale IOs againt the ESXi, including the repercussion (failing processes), and possible mutiple reboots needed, etc.


This is a post in the Guest Post series. Posts in this series:


Post last updated on June 1, 2021: Update ESXi-7.0-SD-Card-Issue-Temporary-Workaround.md

About the author

Christian Mohn works as a Chief Technologist SDDC for Proact in Norway.

See his About page for more details, or find him on Twitter.

Sponsors