I recently had a case of “go with your gut” when we added some new NVMe disks to an existing VMware vSAN solution at a customer.
Normally I’m very cautious and will put hosts into maintenance mode, no matter how small the hardware change I’m doing is, but against my better judgement this time I decided to hot add some disks (which of course is supported). However, I fumbled and managed to insert it and quickly remove it again before inserting it again, and ended up with a dreaded Purple Screen of Death (PSOD) on the host.
Naturally this freaked me out and I was eager to figure out what the problem was. Searching through the KBs at VMware didn’t give me any clues, but a quick Google search took me to the
ESXi 7.0u2c Release Notes.
In the ongoing saga of SD-Card/USB-boot device support in ESXi, VMware has just published a new KB article named Persistent storage warnings when booting ESXi from SD-Card/USB devices (85615) outlining that from 7.0u3 (at the time of writing, not released yet) onward support for such boot devices is deprecated.
They also list the workarounds for this as either install to a non SD/USB device, or to add a persistent storage device.
Note: As of 03. September 2021 the Persistent storage warnings when booting ESXi from SD-Card/USB devices. (85615) KB article has been removed from kb.vmware.com for unknown reasons. Hopefully it will be back soon.
Additional information can also be found in Boot device guidance for low endurance media(vSphere and vSAN) (82515), which somewhat contradictory states that upgrades will still be supported in 7.x.
Just like everything else, plan accordingly. I do applaud VMware for being transparent about this, and even publishing the new KB article before ESXi 7.0u3 is released. Everyone still running production on SD-Card/USB devices needs to take action as soon as possible to ensure support going forward.
Well, not directly related to the new ESXi 7 U2c build, BUT if you’re one of the lucky ones (like me), you’ve been experiencing big issues with SD/USB device since ESXi U1/U2, hence the vCenter <> ESXi (at least the configurations, tokens, etc.) has not been working for a while. I mean, last time vCenter talked successfully with my host was back in May 2021, when the USB dropped. That’s like four (4!) months ago. Today, VMware finally released the ESXi 7 U2c, which has included an updated module for the vmkusb. Hopefully this will fix the previous experienced SD/USB device issues, but before that - let’s talk patching!
As timings will have it, two podcasts I’ve been a guest on has been released this week. This time it is On The Line with Cohesity — Episode 42: Back to Travel and VMworld 2021 with Christian Mohn.
I’ve recently had the opportunity to join the The On-Premise IT Roundtable Podcast for a discussion about storage and bottlenecks, titled It’s Time to Embrace the Bottlenecks in Storage.
Next week is Storage Field Day 22, and for the first time since Tech Field Day 6 10 years ago I am a delegate! I am really looking forward to hearing from the presenting sponsors, and being part of what looks like a really great group of delegates.
This is a guest post by Espen Ødegaard: As VMware has not released a fix yet (regarding issues with SD card and USB drive), I’m still experiencing issues with ESXi 7.0 U2a Potentially Killing USB and SD drives, running from USB or SD card installs. As previous workaround (copying VMware Tools to RAMDISK with option
ToolsRamdisk) only worked for 8 days (in my case), I needed something more permanent, to get the ESXi-hosts more stable (e.g. host being able to enter maintenance mode, move VMs around, snapshots/backup, doing CLI-stuff/commands, etc.).
This is a guest post by Espen Ødegaard: As searching and filtering for events in vCenter Server trough vSphere Client somewhat limited (OK, it really sucks, to be honest), it’s usually much faster using PowerCLI, to retrieve, filter & searching events.
This is a guest post by Espen Ødegaard: After upgrading my 4-node vSAN-cluster (homelab) to ESXi 7.0 build 17867351 U2a, I detected that ESXi had issues talking to the USB device, where the ESXi was installed. I found a related KB from VMware, outlining issues with the new VMFS-L, which started my baseline for troubleshooting VMFS-L Locker partition corruption on SD cards in ESXi 7.0 (83376) In short, it says that the VMFS-L partition may have become corrupt, and a re-install is needed. There is no resolution for the SD card corruption as of the time this article was published