Datastore out of Space

So, when I blogged about updating or migrating Veeam, Migrating/Restoring Veeam – Zewwy’s Info Tech Talks I forgot my snapshot… whoops…

I just caught it before I wanted to upgrade to Veeam 13, I hadn’t even connected the disk when I went to verify the disk and datastore, sure enough the datastore was right on the cusp of running out of space. 1.5 Gigs remaining.

I figured leave everything as is and just migrate it to a larger datastore… but what I failed to take into account is that the VM was still running, and any disk writes are still growing the delta file, so in the middle of the move the vMotion stopped stating. “The operation cannot be allowed at the current time because the virtual machine has a question pending: ‘msg.hbacommon.outofspace: There is no more space for virtual disk ‘Veeam_19-000001.vmdk’. You might be able to continue this session by free…” I clicked cancel which simply failed the vMotion job…

During this operation, if the machine had been powered off the transfer would have completed in a shorter amount of time (compared to the more random I/O of the snapshot cleanup) and the process from a source SSD to a target SSD was providing throughput of 300MB/s for the first 2 minutes of the process before dropping to roughly 160 MB/s just before the failure occurred. With roughly 12ms delay response from the target SSD and 35ms delayed response from the target.

I decided to Hard Stop the VM, then selected to delete all snapshots. I don’t know how vCenter never warned me of a large Snapshot size. With the large snapshot size, and a cheap, CHEAP SSD (TeamSSD  – TEAM_T253256GB), and the fact that the backup storage (which you normally shouldn’t do this in production but I do have it setup this way in my lab) that the backup VMDK on a slow spindle 2TB drive was also part of the snapshot, the snapshot deletion took FOREVER! over 3 hours.

Then shutting it shown was bad cause I forgot that I have a vTPM on the machine so after trying to vmotion it again after the snapshot removals… it errored out with “Permission to perform this operation was denied. NoPermission.message.format”, then ”
Unlock encrypted virtual machine
Veeam
The object or item referred to could not be found. The provider ‘NativeKP’ does not exist. Failed to decrypt the key”

I added both known NPKs I backed up but the error then said ”
Reconfigure virtual machine
Veeam
An encryption key is required.”

Eventually doing the following:

• SSH into ESXi.
• Go to the VM folder:

  cd /vmfs/volumes/<datastore>/<vmname>
  

• Make a backup:

  cp <vmname>.vmx <vmname>.vmx.bak
  

• Edit the VMX:

  vi <vmname>.vmx
  

  Remove any lines containing:

• encryption.keySafe
• encryption.data
• migrate.encryptionMode = “required”
• ftcpt.ftEncryptionMode = “ftEncryptionRequired”

Then re-registered the VM.

Then vmotion it… “Relocate virtual machine
Veeam
Completed
Migrating Virtual Machine active state”

Oh finally, man I hate adding vTPMs to VM on ESXi hosts.. it always seems to come back to bite me. and my datastore usage is finally back under control and no other VMs were affected… what a roller coaster.