Story
Another vCenter Patch, Another problem 😀
This seems to be a reoccurring story these last couple posts…
Error on Host
This time after updating again a host in the cluster had the error message.
Troubleshooting
Un like the last time this happened, the event log wasn’t as blatant (flooded) complaining about the /tmp being full. and checking the host with
vdf -h
which showed only 90% full, which was still pretty high, which might have explained the one log event that I did see about it:
The ramdisk 'tmp' is full. As a result, the file /tmp/img-stg/data/vmware_f.v00 could not be written
Which was in the log right after this event of attempting to install a base ESXi image?
Installing image profile '(Updated) HPE-ESXi-Image' with acceptance level checking disabled
This seemed a bit weird but I could find any info other than what’s usuallly a very Microsoft type answer of “you can just ignore it” or “usually this is not an issue, just it says vCenter saying it is connecting to esxi host and installing it’s agent”
OK I guess… moving on… the very next error event was:
Could not stage image profile '(Updated) HPE-ESXi-Image': ('VMware_bootbank_vmware-fdm_7.0.2-18455184', '[Errno 28] No space left on device')
Huh, Now note this host was installed running the official VMware Image provided by HPE for this exact hardware supported by the VMware HCL. So there should be no funny business. However I feel maybe there’s a bit of the known HPE bug as mentioned the last time this happened. It just hasn’t fully flooded /tmp just yet.
Lil Side Trail
So couple things to note here, first the ESXi image is installed on a USB/SD Card style setup as such it should be well know to define the persistent log location, as well as the scratch location. However, not many source specify changing the system swap location.
- Persistent Log; VMware KB; Tech Blogger
(Most standard ESXi Log info) - Scratch Log: VMware KB; Tech Blogger 1; Tech Blogger 2
(Crash Logs, Support log creations) - Swap Location: VMware Doc 1 (Configure), VMware Doc2 (About), Tech Blogger Who seem to regurgitate the exact about page from VMware.
However, researching this even more lots of posts on reddit mentioned the swap file for VM’s being on their VM directories, so if using a shared datastore they will reside there, and I shouldn’t see issues around swap usage at all at the host level.
Which if you look on the vCenter Web UI on a ESXi hosts there are two options available: VM – Swap, and System Swap.
The VMware docs doesn’t seem to describe accurately the difference between these two options.
Lookup up the error about not being able to stage the file I found this one blog post which of course mentioned changing the swap location to get past the error…
The main thing mentioned by the blogger is “The problem is caused by ESXi not having enough free space available to extract the installation packages.” but failed to specify where that exactly is, and the event log didn’t specify that either. Now since his solution was to adjust the system swap location, it begs the question. Is the package extraction location the System Swap location?
Since the host settings seem to be only specified with the alternative option checkboxes as:
Can use host cache
Can use datastore specified by host for swap files
It’s still not fully clear to me where the swap is actually located with these, assumed default settings. Or if extraction of the image actually using swap, or why the same imagine already on the ESXi host is being re-applied when your upgrade vCenter?
Resolution
So many question, so little answers, so unfortunately I’m going to go on a bit of a whim, and simply try exactly what I did before, clear the file from the /tmp location that was takin up a lot of it’s space, install the HPE patch for the known bug, in hopes it resolves the issue….
Sure enough the exact same thing happened, as in my initial post it just seems it wasn’t fully full. So the symptoms were just a bit different.
- vMotion all VMs to another host in the cluster (amazing vMotion works without issue)
- Ignore the HA warning on the VMs migrated
- Place Host into Maintenance mode (This clears the HA warnings on the VMs and cluster)
- Verify /tmp has room. Update any ESXi packages from the hardware vendor if applicable.
- Reboot the host.
- Exit Maintenance mode.
Hope this helps someone who might see the same type of error events in their ESXi event logs.
