Blog

Posted on

Careful Cloning ESXi Hosts

I’ll keep this post short. I was doing some ESXi host deployments in my home lab, and I noticed that when I would install on a 120GB SSD, the install would go smoothly, but I wasn’t able to use any of the storage as a Datastore. However, if I took a fresh install copy of ESXi from installing onto an 8GB USB Stick and DD’d it to the 120GB SSD I got several advantages from this:

  1. When done via a USB3 Pipe of Linux live holding a copy of my base image to deploy I could get speeds in excess of 100 MB/s, and with only 8GB of data to transfer, the “install” would complete in a mere 90 seconds.
  2. The IP address and root password are preconfigured to what I already now, and I can simply change the IP address from the DCUI and call it a day.

Using this method I could have a host up in less than 5 minutes (2 min to boot linux live, 90 seconds to install the base ESXi OS image, and 2 more to boot ESXi). This was of course on machine without ECC and all the server hardware firmware jazz… in those cases install times are always longer. anyway…

This was an amazing option, until I noticed that when I connect one machine in I just deployed and changed the IP address, and (since I’m super anal about networking during this type of project/operations) I noticed my ping from one machine (a completely different IP address) started to drop when the new device came up… and after a while the ping responses would come back but drop from the new host, and vice versa, flip and flop it goes. I’m used to this usually if there’s an IP conflict and two devices have the same IP address. In this case they were different IP addresses… after enough symptom gathering and logical deduction of because I had to assume that the MAC address just be the same and this is the same problem in reverse (different IP’s but same MAC) and as such experiencing the same symptoms.

To validate this I simply deployed my image to a new machine, then I went on the hunt to figure out how to see the MAC address, since I couldn’t plug in the NIC and get to the web based MGMT interface I had to figure out how to do that via the console CLI directly… mhmm after enough googling on my phone I found this spiceworks thread with my answer:

vim-cmd hostsvc/net/info | grep “mac =”

I then checked this against the ESXi host that I saw the flipping flopping with, and sure enough they matched…  After doing a fresh install I noticed that the first 3 sections match the physical MAC, but in my DD deployed ones they retain the MAC of the system from which it was installed and those when I ran the command above, I could tell which ones were deployed via my method. This was further mentioned in this reddit thread by a commenter who goes by the name of sryan2K1:

“The physical NIC MACs are never used. vmk ports, along with VMs themselves will all use VMWare’s OUI as the first half of the address on the wire.”

OK, now maybe I can still salvage my deployment method by simply deleting and recreating the VMK after deployment, but I’d guess it best be done via the DCUI or direct console… I found one KB by VMware/Broadcom but it gave a 404 but Luckly there was a wayback machine link for it here.

Which states the following:

“During Initial Installation and DCUI, ESXi management interface (default vmk0) is created during installation.

The MAC address assigned will be the primary active physical NIC (pnic) associated.

If the associated vmnic is modified with the management interface vmkernel will once again assign MAC address of the associated physical NIC.

To create a VMkernel port and attach it to a portgroup on a Standard vSwitch, run these commands:

esxcli network ip interface add --interface-name=vmkX --portgroup-name=portgroup
esxcli network ip interface ipv4 set --interface-name=vmkX --ipv4=ipaddress --netmask=netmask --type=static"

Alternatively, you can also use esxcli to create the management interface vmkernel on the VDS.

Creation of the management interface with the ‘esxcli network’ will generate a VMware Universally Unique address instead of the pnic MAC address.

It is recommended to use the esxcli network IP interface method to create the management interface and not use DCUI.

Workarounds:               None

Additional Information:
Using DCUI to remove vmnic binding from management vmkernel or any modification will apply change at vSwitch level. Management interface is associated with propagating the change to any port groups within the vSwtich level.

Impact/Risks:                None.”

I”m assuming it means if you use the DCUI to reconfigure the MGMT interface settings the MAC will automatically be reconfigured to match what I found during initial clean install and mentioned in the reddit thread of using the first 3 sections to derive the MAC of the VMK.

But what if you don’t have any additional interfaces to use to make the section change in the DCUI to have that actually happen? cause what I’ve noticed changing the IP address and disabling IPv6 and rebooting did not change the VMK’s MAC address. Oh there’s in option in the DCUI “Reset Network Settings” within there there’s several options, I simply picked reset to factory defaults. Said success, checked the MAC via the first command stated above and bam the VMK nic changed to what it should be! Sweet my deployment method is still viable.

Hope this helps someone.

Posted on

The virtual machine must be encrypted

Sooo I lost a VM in my fray of re-organizing my server farm. Like a lost pup I figured I just rely on my good old Veeam backup sets. Recover VM, alright here we goo….

What.. what does that mean…. Oh wait is this cause of when I blogged about adding vTPMs to VMs?

Re-checked the linked video from VMware… 2 min in … “Failure to save your key backup will result in unrecoverable data loss”…. mhmmm, OK I thought all I did was add a TPM device to my VM and enabled secure boot, that’s the deal here?

Somewhere I read that the VM config files get encrypted, but I don’t think that’s the case here either.  Even checking the Pre-reqs from VMware I can’t see anything nothing this:

Prerequisites

Ensure that your vSphere environment is configured with a key provider. See the following for more information:
Configuring vSphere Trust Authority
Configuring and Managing a Standard Key Provider
Configuring and Managing vSphere Native Key Provider
Ensure that host encryption mode is enabled. See Enable Host Encryption Mode Explicitly.
The guest OS you use can be Windows Server 2008 and later, Windows 7 and later, or Linux.
The ESXi hosts running in your environment must be ESXi 6.7 or later (Windows guest OS), or 7.0 Update 2 (Linux guest OS).
The virtual machine must use EFI firmware.
Verify that you have the required privileges:
Cryptographic operations.Clone
Cryptographic operations.Encrypt
Cryptographic operations.Encrypt new
Cryptographic operations.Migrate
Cryptographic operations.Register VM

What I think is happening here is my NKP that IS a Prerequisite went poof (the vCenter server that was used to create it is shutdown and not being used), and another temp vCenter is being used.

My first thought was maybe I could just add a new NKP and go as I figured the TPM physical module that’s installed simply needs this, and I think it’s this hardware that’s faulting the boot.

I didn’t want to muck the with original I just recovered so I tried to clone it, but the clone failed too complaining about encryption before adding a TPM, further validating my assumption. What I don’t understand it how the VM was allowed to be created from backup in the first place if I can’t even clone it…?

Any since I know recovery is possible (since I just did it), I guess maybe I can just remove it? Or I could also create a new VM and use vmkfstools to clone the hdd… let’s try that first…

Go to boot VM, well got past that error but the Machine was bitlocked, I was hoping it wasn’t going to be.. go to AD server, open ADUC… no bitlocker tab… ughhhh…

ADUC Missing BitLocker Recovery Tab in 1809 – Microsoft Community

Right but where is that in on a server, oh in server manager it moved…

Yay there’s the bitlocker tab and… it’s empty.. man give me a fucking break… so now I have a bunch of backups that are useless cause I lost the bitlocker key… shiiiiiiit

Well I don’t have anything to follow up on here but a lesson learnt to backup your bitlocker key (I don’t know why it wasn’t save to the AD computer object).

Posted on

Clearing up Space on an Exchange Server

I wanted to migrate an old exchange server, but the size was way more then I ever expected… so I dusted out my old script…

GitHub – Zewwy/Manage-ExchangeLogs: Script to help ease in managing Exchange logs

inetpub logs ended up being over 50 gigs. etl nearly 5 Gigs, another 5 On the next. all together cut out over 60 Gigs of data space. But lots remained.

I then found a OWA prem folder with lots of data and found a usual blogger covering it’s data removal here: Remove old Exchange OWA files to free up disk space – ALI TAJRAN

“To know which Exchange build number folders we can remove, we have to find which Exchange versions are running in the organization.

$ExchangeServers = Get-ExchangeServer | Sort-Object Name
ForEach ($Server in $ExchangeServers) {
Invoke-Command -ComputerName $Server.Name -ScriptBlock { Get-Command Exsetup.exe | ForEach-Object { $_.FileversionInfo } }
}

After running the above script in Exchange Management Shell, look at the Exchange build number.”

Pretty much run that command and delete all the folders expect the one the system is currently on.

Just like this blogger saved another 10 Gigs off that folder alone, but still heafty, I checked the Exchange DB folder path and found enless log files (guess I have to extend my script. Ended up writing a one liner to grab all the log files and clear then, this was almost another 50 gigs in space saved. We are well over 100 Gigs in space saved/deleted. but there’s still some heft, checking the Exchange client mailbox DB is only 6 gigs (and I”ll see if I can save space there but over all it is peanuts to all the space being used.

Next I found WinSXS folder taking up space. I followed the steps from this blog: What to Do If the Windows Folder Is Too Big on Windows 10? (diskpart.com)

I had already ran diskclean including system files. I ran the DISM commands specified as well but that only brough the 15 gigs of space it was using to roughly 7 Gigs, half is not bad but I was hoping it could do better.

Man wonder if I can just delete it? No! Says Microsoft:

“Don’t delete the WinSxS folder, you can instead reduce the size of the WinSxS folder using tools built into Windows. For more information about the WinSxS folder, see Manage the Component Store.”

We following that MS KB pretty runs the same commands as the blog.

I’ll live with that for now, I logged into the mailbox and deleted everything I could, I should have no other mailboxes… wonder what’s got the Exchange DB up to 6 gigs of used space?

I followed another one of Ali’s Blogs: Get mailbox database size and white space – ALI TAJRAN

But that just told me what I had already found out looking at the base file system. He links to another one of this post to use a a script he had written. I checked the script… Clean use of case statement… Well done bud 🙂

Anyway I pipe get mailbox selection into a stats comdlet and then Format the whole list:

(Get-mailbox -resultsize unlimited) | Get-MailboxStatstics | Select *Name,*Count,*Size | FL

This shows All mail items size, and all message and attachment sizes. For me all mailboxes were tiny, and I can get rid of several of them as well. but I don’t think that will change the size on disk…

Got rid of everything down to 2 mailboxes and with 2 gig quotes each no way it could be over 4 gigs, yet it’s still the 6-7 gigs noted eariler.. mhmmm…

So in his blog he basically create a new mailbox DB via the EAC (or Powershell) moves the mailboxes and deletes the old DB. OK I can do that…

So old DB as seen here:

Create new folder path for new DB file (or use a new disk whatever ya want):


Create new DB:

New-MailboxDatabase -Name "DB02" -Server "EX01" -EdbFilePath "C:\DBPath\DB02.edb" -LogFolderPath "C:\LogPath\DB02""

Restart the services.. wait is my new DB and files? Oh I forgot to mount it:

Alright… time to move the mailboxes…

Huh, I was hoping for a process display but I guess it makes sense to throw the job in the background to not be interrupted by signing out or closing the console. Checking resource monitor… Starts chanting… Oh ya Go Beanus, Go Beanus!!

Looks like I/O settled… and…

Get-MoveRequest

Completed nice… size? less then 300 Megs baby.

K, just need to delete the unmount and delete the old DB. What a dink he knew it would fail but I followed his other blog post here:

Cannot delete mailbox database in Exchange Server – ALI TAJRAN

and after running the remove-mailboxdatabase cmdlet it still told me I had to manually delete the file, so I did… I finally got the server to roughly 30 gigs… not bad but I really don’t like that 7 gig SXS folder…

I even cleaned of the SoftwareDistrobution (Windows update Cache folder)

Hope this helps someone, time to hole punch this vmdk and migrate this server.

*Update* The hole punch didn’t work, Why? cause I forgot to run sdelete.
*WARNING* I tried to run sdelete on the VM while it was thin provisioned on a datastore that didn’t have enough storage to will the whole drive, as such the VM errored out with there is no more free space on disk.

It’s like the old adamant goes, things gotta get dirty before they get clean. In this case the drive has to be completed filled (with zeros) before it can be hole punched. Make sure the VM resides on a Datastore with enough actual space if the drive were to be completely filled.

*Update #2* Seems I went down a further rabbit hole then I wanted this to go, unlike my post about hole punching a Linux VM, which was pretty easy and straight forward. This one had a couple extra problems up it’s sleeve.

Problem #1 – Clearing Up Storage Space When Extending Is Not Possible.

This is literally what this whole blog post was about, read it and clear whatever space you can. If you have a physical Exchange server you’re probably done, and all your backups would probably be agent based at this point.

However if you’re virtualized (specifically with VMware/ESXi), then you have some more steps to do, the “hole punching”

Problem #2 – Hole Punched Size Doesn’t Match OS Disk Usage

This is were I want to extend on this blog post cause while the solution seems simple and straight forward. Each step has it’s own caveats and issue to overcome. Let’s discuss these now…

  1. You have to “zero the empty space” before VMware’s tools and properly complete the hole punch. This is only an issue if you happen to be over provisioning the datastore. If so:
  2.  At this point its assumed you cleared as much space as possible, and 2 you have defragged the HDD using the Windows defrag tool, and you have the VM overprovisioned. Simply shrink the partition down to a size that IS available on the datastore, or migrate to a datastore with enough storage. In my case I opted for the first choice to shrink the partition when I hit YET ANOTHER PROBLEM:
  3.  Even though I knew I had cleared the used space down to roughly 30 GBs of space, running the shrink wizard in diskmgmt tool stated it could only shrink the disk to 200GB since “There was a system file preventing further shrinkage”. WTF man, we ran disk cleanup, we cleared SXS folder, we cleared old logs, lots, we cleared the actual Exchange Database files, we disabled, and shrunk the pagefile then re-enabled… What is possibly preventing the shrinkage?

I found this post: windows 7 – Can’t shrink partition due to mystery file – Super User after I looked in the event log for event 259 showing the file in question preventing the shrinkage is “$LogFile::$DATA”… Da Fuck does that mean…

In short.. It’s an NTFS journaling file using “Alternate Data Streams“, or as quoted by Andrew Lambert “The $LogFile file is a part of the NTFS filesystem metadata, it’s used as part of journaling. The ::$DATA part of the name indicates the default $DATA stream of the file. I don’t know why it’s causing a problem, though.”

Bunch of comments about System Restore points, but I checked and there were none. Many other comments mentioning the use of 3rd party tools (no thanks). I can’t seem to locate it but I pretty sure I remember reading a comment somewhere that other NTFS aware applications have the ability to move and correct such things. So here’s my plan of action:

  1. Create a snapshot so I don’t have to recover the whole VM is something goes wrong. (On a slower I/O datastore but one with enough space for whole disk just to be safe).
  2. Boot The Exchange VM but with GParted Live disc connected.
  3. Use GParted to shrink the partition.
  4. Clone the VM. (This is what I don’t get the cloned VM still shows a disk size usage of 70 GBs…. AHhhhhhhhh!!)

Here’s another interesting note, as I stated in point 1 I had this VM on a Datastore based on spindle disk, shown on the ESXi host as “Non-SSD”, and cloned it to an SSD Datastore where it now states to use 70 Gig when the OS boots only having a partitioned disk of 46 Gigs, with 12 Gigs free. Opening the defrag application states defrag not possible cause it’s an SSD, guess let’s run sdelete see what happens?!

sdelete -z c:
sdelete -c c:

The backend VMDK grew from 70 Gigs to 80 Gigs… man wtf is going on… Hole punch it:

vmkfstools -K /vmfs/volumes/SSD/VM/VM.vmdk

You’re tellin me.. the SSD can handle ripping the drive at over 250 MB/s but holepunching causes I/O errors?

Good ol technology never ceasing to piss me off… fine I’ll destroy this VM, move the main spindle drive into a new ESXi host. Which will have an SSD Datastore with more storage and hopefully not on the way out (if it actually is an I/O error, storage/drive failure on the SSD). One sec…

So yeah… even with a larger SSD the copy worked, the hole punch “succeeded” but the drive was still 80 gigs. I made a clone and the vmdk came down to 60 gigs, I still can’t make sense of the roughly 30 gigs of discrepancy. Since the whole idea is to move this to my wireless ESXi host, I’ll see what exporting it as it is now results in the final OVA file size and then update this blog post.

 

Posted on

Hole Punching a Linux VM on ESXi

I covered this in the past here:

Reclaim unused space from VMDK – Zewwy’s Info Tech Talks

But this time I wanna covered it a bit differently. Things I noticed:

  1. A proper VM with VMtools installed, and thin provisioned will automatically shrink the overall size being shown and used on disk on the ESXi browser.

Yet for some reason after I used the SCP method to move a VM from one host to another (frowned upon as it secretly converts it from thin to thick). Yet even after migrating to a new datastore specifying thin, it still show as full disk usage on the host.

I know is less checking the VM itself via it’s console/terminal whatever:

In my old blog post I mentioned “using DD” but not showing for stating how at all, Googling this I found this thread with an interesting answer:

“The /zero-Tag is actually a file name. The command just copies zeros from the virtual File /dev/zero (infinite number of zeros) into /mnt/hdb/zero until the disk is full, or some other error occurs.

This is why you have to remove the file /mnt/hdb/zero after that in order to regain the unused space.

However, a better way to fill free space with zeros (on ext2,3,4 file systems) is to use a tool called zerofree.”

Oooo zerofree?

Huh, something created (a tool) for exactly the task (job) at hand. Great!

Error, how classic, complained path is mounted RW, like yeah and?

Ughhh, google? virtualbox – zerofree on ubuntu 18.04 – Ask Ubuntu

Step 1) Reboot into ubuntu recovery console, hold down [Perfect ESC keystroke]

K how do I get into Advanced Grub boot? Holding Shift did nothing, if I mash ESC I get grub> (notices tiny flicker of Grub menu).. great I have to Mash ESC only once perfectly in a less then 1 second boot window… man give me a break… Once in I see advanced option as stated by the answer.

Step 2) advanced options -> recovery mode -> root console

Step 3) find the root directory

mount | grep "sda"

Step 4) run zerofree

echo "u" > /proc/sysrq-trigger
mount /dev/mapper / -o remount,ro
zerofree -v /dev/sda1

Step 5) reboot

Checking the ESXi host… what it went up 2 gigs, da fuq man…

Step 6) Compress Disk.

In my previous post this was to vMotion to datastore or use holepunch option which from what I can tell this the -K option, which I dont’ think can be used on a live VM, and a storage vMotion can’t be done. without vCenter. Since I’m temp working without a vCenter server, let’s try the holepunch option, this will require shutting down the VM, but since the zerofree required it anyway down time was already in play.

On the ESXi host:

[root@Art-ESXi:~] vmkfstools -K /vmfs/volumes/Art-Team/VM1/VM1.vmdk
vmfsDisk: 1, rdmDisk: 0, blockSize: 1048576
Hole Punching: 100% done.

Oh noooo I’ve tainted my results… checking the web UI the space has gone back done to stating 20 Gigs like the first snippet… but doing a du -h on the flat file shows it is only the 10 Gigs as expected it to be:

Well I don’t know what to make of this discrepancy…

Huh, I found this post of someone doing the exact same thing around the time I wrote my original post but simply used the command:

dd bs=1M count=8192 if=/dev/zero of=zero

I have no clue how output file of zero means the disk… guess you can try that too, then holepunch the VMDK just like I did.

There.. I shutdown the VM again and this time did my ol trick to vMotion a VM without vCenter, and after re-registering the vMotion (between two datastores) it finally showed up with the correct space in the UI. 🙂

Hope this post helps someone, it’s been covered many times in the past.

Posted on

How to vMotion a VM without vCenter WITHOUT Shared Storage

While I have covered this in the past here:
How to vMotion a VM without vCenter – Zewwy’s Info Tech Talks

This was using shared network storage between hosts…. what If you have no vCenter AND no shared storage? In my previous post I suggested to go check out VMware arena’s post but that just covers how to copy files from one host to another, and what I’ve noticed is while it does work if you let it complete, the vmdk is no longer thin and takes up the full space as specified by its defined size.  This is also mentioned as such in this serverfault thread “Solutions like rsync or scp will be rate-limited and have no knowledge of the content (e.g. sparse VMDK files, thin-provisioned volumes, etc.)”

So options provided there are:

  1. Export the VM as an OVF file, move to a local system, then reimport the OVF to your ESXi destination.I attempted this but on the host I could only export vmdk. While attempting to do so I got network issues (browser asking to download multiple files, but I must not have noticed and time and timed out? not sure). This also requires an intermediary device and double down/up on the network, I’m hoping for a way between hosts directly.
  2. Use vSphere and perform a host/storage migration.This post is how to do it without. Also note I attempted this but in my case I’m using my abomination ESXi host I created in my previous blog post, and vCenter fails the task with errors. (Again SCP succeeds but doesn’t retain thin provisioning). Not sure why SCP succeeds but vCenter fails seems to be more redundant to poor connection and keeps going, which happens when the WiFi NICs underload in those situations.
  3. Leverage one of Veeam’s free products to handle the ad hoc move.

I love Veeam, but in this case I’m limited in resources, lets see if we can do it via native ESXi here.

So that exhausts all those options. What else we got…

Move VMware ESXi VM to new datastore – preserve thin-provisioning – Server Fault

Oh someone figured out what I did in my intital post all the way back in 2013… wonder how I missed that one.. oh well, same answer as my initial post though required shared storage… moving on…

LOL no way… Willam Lam all the way back from over 14 years ago! Answering the question I had about compression of the files. and saying the OVF export is still the best option.. mhmmm…

I don’t want to stick to just scp, man did it suck getting to 97% done on a 60 Gig provisioned VMDK, that’s only taking up roughly 20 gigs, to have to not work cause I put my machine to sleep thinking it was a remote connection (SSH) to the machine and the machine is doing the actual transfer… just to wake my machine the next morning to have a “corrupt” vmdk that fails to boot or svmotion to get thin. I have machines with fast local storage but poor network, it’s a problem from back in the day with poor slow internet speeds. So what do we have? We got gzip and tar, what’s the diff?

In conclusion, GZIP is used to compress individual files, whereas TAR is used to combine numerous files and directories into a single archive. They are frequently used together to create compressed archive files, often with the “.tar.gz” extension.

Also answered here.

“If you come from a Windows background, you may be familiar with the zip and rar formats. These are archives of multiple files compressed together.

In Unix and Unix-like systems (like Ubuntu), archiving and compression are separate.

tar puts multiple files into a single (tar) file.
gzip compresses one file (only).
So, to get a compressed archive, you combine the two, first use tar or pax to get all files into a single file (archive.tar), then gzip it (archive.tar.gz).

If you have only one file, you need to compress (notes.txt): there’s no need for tar, so you just do gzip notes.txt which will result in notes.txt.gz. There are other types of compression, such as compress, bzip2 and xz which work in the same manner as gzip (apart from using different types of compression of course).”

OK, so from this it would seem like a lot of wasted I/O to create a tar file of the main VDMK flat file, but we could gain from compressing it. Let’s just do a test of simple compression and monitor the host performance while doing so.

Another thing I noticed that I didn’t seem to cover in my previous post in doing this trick was the -ctk.vmdk files. Which are change block tracking files, as noted from here:

“Version 3 added support for persistent changed block tracking (CBT), and is set when CBT is enabled for a virtual disk. This version first appeared in ESX/ESXi 4.0 and continues unchanged in recent ESXi releases. When CBT is enabled, the version number is incremented, and decremented when CBT is disabled. If you look at the .vmdk descriptor file for a version 3 virtual disk, you can see a pointer to its *-ctk.vmdk ancillary file. For example: version=3

# Change Tracking File
changeTrackPath=”Windows-2008R2x64-2-ctk.vmdk”
The changeTrackPath setting references a file that describes changed areas on the virtual disk.
If you want to back up the changed area information, then your software should copy the *-ctk.vmdk file and preserve the “Change Tracking File” line in the .vmdk descriptor file. If you do not want to back up the changed area information, then you can discard the ancillary file, remove the “Change Tracking File” line, read the VMDK file data as if it were version 1, and roll back the version number on restore.

I’ll have to consider this when running some of the commands coming up. Now we still don’t know how much, if any, space we’ll save from compression alone and the time it’ll take to create the compressed file… from my research I found this resource pretty helpful:

Which Linux/UNIX compression algorithm is best? (privex.io)

Since we want to keep it native doing quick tests via the command line shows ESXi to have both gzip and xz but not lx4 or lbzip2, which kind of sucks as they showed to have the best performance in terms of compression speeds… as quoted by the article “As mentioned at the start of the article, every compression algorithm/tool has it’s tradeoffs, and xz’s high compression is paid for by very slow decompression, while lz4 decompresses even faster than it compressed.” Which is exactly what I want to see in the end result, if we save no space, then the process will burn I/O and expected life of the drive being used or pretty much zero gains.

Highest overall compression ratio: XZ If we gonna do this this is what we want, but how long it takes and how much resources (CPU cycles, and thus overall WATTS) trade off will come into question (though I’m not actually taking measurements and doing calculations, I’m looking at it at points and time and making assumed guessed at overall returns).

Time to find out what we can get from this (I’m so glad I looked up xz examples cause it def is not intuitive (no input then output parameters, read this to know what I mean) :

xz -c /vmfs/volumes/SourceDatastore/VM/vm-flat.vmdk > /vmfs/volumes/TargetDatastore/whereever/vmvmdk.xz

Mhmmm no progress… crap didn’t read far enough along and I should have specified the -v flag, not sure why that wouldn’t be defaulted, having no response of the console kind of sucks… but checking the host resources via the web GUI shows CPU being used, and write speed….. sad….

CPU usage:

and Disk I/O:

Yeah… maybe 4 MB/s and this is against a SSD storage on a SATA bus, there’s no way the storage drive or the controller is at fault here… this is not going to be worth it…

Kill command, check compressed file less than 300 MB in size, OI, that def not going to pay off here…

I decided to try taring everyting into one file without compression hoping to simply get it to to one file roughly 20gigs in size with max I/O. As mentioned here:

“When I try the same without compression, then I seem to get the full speed of my drive. ”

However to my dismay (maybe it ripped the SSDs cache too hard?) I unno I’d get I/O error, even though the charts showed insane throughput, I decided to switch to another datastore a spindle drive on the ESXi host and you can see the performance just sucks compared to the SSD itself.

Which now again stuck waiting cause instead of amazing throughput its stuck going only 20 MB/s apparently… uggghhhh.

To add to this frustration, I figured I’d try the OVF export option again, but I guess cause the tar operation has a read on the file, I’m assume a file lock, when attempting the OVF export it just spits an web response “File Not Found”. So, I can’t even have a race knowing full well the SSD could read much faster than what it’s currently operating at. I don’t really know what the bottleneck is at this point…

Even at this rate it’s feeling almost pointless, but man just to keep a vmdk thin, why, oh WHY SCP can’t you just copy the file as the size it is… mhmmm there has to be a way other than all this crap….

I don’t think this guy had any idea he went from thin too thick on the VM….

I thought about SSHFS, but it’s not available on ESXi server….

Forgot about Willams project GhettoVCB Great if I actually wanted more of a backup solution… considered for future blog, but over kill to just move a VM.

The deeper I go here the more the simply export to OVF template and import is seeming reaaaaaalll appeasing.

Awww man this tar operation looks like its takin more size then the source. doing a du -h on the source shows 19.7 Gigs… tar file has now surpassed 19.8 Gigs in size… with no sign of slowing down or stopping lol. Fuck man I think tar is also completely unaware of thin disk and I think it’ll make the whole tar file what ever the provisioned size was (aka thick). Shiiiiiiiiiiiiit!

Trying the Export VM option looked so promising,

until the usual like always… ERROR!!

FFS man!!! Can’t you just copy the files via SSH between hosts? Yeah but only if you’re willing to copy the whole disk and if you’re lucky holepunch it back to thin at the destination… can’t you do it with the actual size on disk… NO!

Try the basic answer on almost all posts about this, just export as template and import… Browser download ERROR… like Fuck!!!

Firefox… nope same problem… Fuck…. Google what ya got for me? well seems like almost the same as my initial move of using SCP but use WinSCP via my client machine and uttering in a middle man in the process, but I guess using the web interface  to download/upload was already a man in the middle process anyway… fine let’s see if I can do that… my gawd is this ever getting ridiculous… what a joke… Export VM from ESXi embedded host client Failed – Network Error or network interruption – Server Fault

And of course when I connect via Win SCP it see the hard drive as being 60 Gigs, so even though trafser speed are good to taking way more space then needs and thus waste data over the bus… FUCK MAN!!!!!

If only there was a way to change that, oh wait there is, I blogged about it before here: How to Shrink a VMDK – Zewwy’s Info Tech Talks

OK Make a clone just to be safe (you should always have real backups but this will do. and amazing this operation on the SSD was fast and didn’t fail.

Woo almost 300 MB/s and finished in under 4 minutes. Now let’s edit the size.

Well I tried the edit size, but only after doing a vmkfstools convertion of the vmdk would it show the new size in WinSCP, even then transferred the files and it was still corrupted in the end..

ESXi 6.5 standalone host help export big VM ? | MangoLassi

Mhmmm another link to Willams site, covering the exact same thing, but this time using a tool ovftool….

and wait a second… He also said there’s a way to use the ovftool on the ESXi server itself, in this post here….. mhmmmm If I install the Linux OVF tool on the ESXi host, I should be able to transfer the VM while keeping the thin disk all “native” on ESXi… close enough anyway…

Step 1) Download the OFV tool, Linux Zip

Step 2) Upload Zip file via Web GUI to Datastore. (Source ESXi)

Step 3) unzip tool (unzip ovfool.zip), then delete zip.

Step 4) Open outbound 443 on Source ESXi server. Otherwise you get error on tool.

Step 5) run command to clone VM, get error that managed by ESXi host.

Step 6) remove hosts from ESXi and run command again… fail cause network error (Much like OVF export error, seem that happens over port 443/HTTPS)

Man Fuck I can’t fucking win here!!!

I think I’m gonna have to do it the old fashioned way… doing it via “seeding”. plug in a Drive into the source ESXi, and physically move it to the target.

Tooo beeeee continued……

I grabbed the OVFtool for windows (the machine I was doing all the mgmt work on anyway, yet it too failed with network issues.

I decided to reboot the mgmt services on the host:

Then gave it one last shot…

Holy efff man the first ever success yet… don’t know if this would havefixed all my other issues, the export failing for https, and all the others? And the resulting OVA was only about 8 Gigs. Time to see if I can deploy it now on the target host.

I deployed the OVA to the target via the WebGUI without issue.

I also tested the ESXi webGUI export VM option and this time it also succeeded without failure, checking the host resources CPU is fairly high on both ovftool export or the webGUI export option. Using esxtop showed hostd process taking up most of the CPU usage during the processes. Further making me believe restarting that service is what fixed my issues…

Posted on

Wireless ESXi Host

The Story

So, the other day I pondered an idea. I wanted to start making some special art pieces made from old motherboards, and then I also started to wonder could I actually make such an art piece… and have it functional?

I took an apart my old build that was a 1U server I made from an old PA-500 and a motherboard I repurposed from a colleague who gifted me their old broken system. Since it was a 1U system, I had purchased 2 special pieces to make it work, a special CPU heatsink (complete solid copper, with a side blower fan, and a 300 watt 1U PSU. both of which made lots of noise.

I also have another project going called “Operation Shut the fuck up” in which all the noisy servers I run will be either shutdown or modified to make zero noise. I hope with the project to also reduce my overall power consumption.

So I started by simply benching the Mobo and working off that, which spurred a whole interest into open case computer designs. I managed to find some projects on Thingiverse for 2020 extrusions and corner braces, cable ties… the works. The build was coming along swimmingly. There was just one thing that kept bugging me about the build… The wires…

Now I know the power cable will be required reguardless, but my hope was to have/install an outlet at the level the art piece was going to be placed at and have it nicely nested behind the art piece to hide it. Now there were a couple ways to resolve this.

  1. Use an Ethernet over Power (Powerline) adapter to use the existing copper power lines already installed in the house. (Not to be confused with PoE).
    There was just one problem with this, my existing Powerline kit died right when I wanted to use it for the purpose. (Looking inside looks like the, soldered to the board, fuse blew, might be as simple as replacing that but it could be a component behind the fuse failed and replacing it would simply blow the new fuse).
    *This is still a very solid option as the default physical port can be used and no other software/configuration/hackery needs to be done, (Plug n Play).
  2.  The next best option would be to use one of these RJ45 to Wireless adapters:
    Wireless Portable WiFi Repeater/Bridge/AP Modes, VONETS VAP11G-300.
    VONETS VAP11G-500S Industrial 2.4GHz Mini WiFi Bridge Wireless Repeater/Router Ethernet to WiFi Adapter
    This option is not as good as the signal quality over wireless is not has good as physical even when using Powerline adapters. However, this option much like the Powerline option, again allows the use of the default NIC, and only the device itself would need to be preconfigured using another system but otherwise again no software/configuration/hackery needs to be done.
  3.  Straight up use a WiFi Adapter on the ESXi host.

Now if you look up this option you’ll see many different responses from:

  1. It can’t be done at all. But USB NICs have community drivers.
    This is true and I’ve used it for ESXi hosts that didn’t have enough NICs for the different Networks that were available (And VLAN was not a viable option for the network design). But I digress here, that’s not what were are after, Wifi ESXi, yes?
  2.  It can’t be done. But option 1, powerline is mentioned, as well as option 2 to use a WiFi bridge to connect to the physical port.
  3.  Can’t be done, use a bridge. Option 2 specified above. and finally…
  4.  Yeah, ESXi doesn’t support Wifi (as mentioned many times) but….. If you pass the WiFi hardware to a VM, then use the vSwitching on the host.. Maybe…

As directly quoted by.. “deleted” – “I mean….if you can find a wifi card that capable, or you make a VM such as pfsense that has a wifi card passed through and that has drivers and then you router all traffic through some internal NIC thats connected to pfsense….”

It was this guys comment that I ran with this crazy idea to see if it could be done…. Spoiler alert, yes that’s why I’m writing this blog post.

The Tasks

The Caveats

While going through this project I was hit with one pretty big hiccup which really sucks but I was able to work past it. That is… It won’t be possible to Bridge the WAN/LAN network segments in OPNsense/PFsense with this setup. Which really sucked that I had to find this out the hard way… as mentioned by pfsense parent company here:

“BSS and IBSS wireless and Bridging

Due to the way wireless works in BSS mode (Basic Service Set, client mode) and IBSS mode (Independent Basic Service Set, Ad-Hoc mode), and the way bridging works, a wireless interface cannot be bridged in BSS or IBSS mode. Every device connected to a wireless card in BSS or IBSS mode must present the same MAC address. With bridging, the MAC address passed is the actual MAC of the connected device. This is normally a desirable facet of how bridging works. With wireless, the only way this can function is if all the devices behind that wireless card present the same MAC address on the wireless network. This is explained in depth by noted wireless expert Jim Thompson in a mailing list post.

As one example, when VMware Player, Workstation, or Server is configured to bridge to a wireless interface, it automatically translates the MAC address to that of the wireless card. Because there is no way to translate a MAC address in FreeBSD, and because of the way bridging in FreeBSD works, it is difficult to provide any workarounds similar to what VMware offers. At some point pfSense® software may support this, but it is not currently on the roadmap.”

Cool what does that mean? It means that if you are running a flat /24 network, as most people in home networks run a private subnet of 192.168.0.0/24, that this device will not be able to communicate in the layer 2 broadcast domain. The good news is ESXi doesn’t needs to work, or utilizes features of broadcast domains. It does however mean that we will need to manage routes as communications to the host using this method will have to be on it’s own dedicated subnet and be routed accordingly based on your network infrastructure. If you have no idea what I’m talking about here then it’s probably best not to continue on with this blog post.

Let’s get started. Oh another thing, at the time of this writing a physical port is still required to get this setup as lots of initial configurations still need to take place on the ESXi host via the Web GUI which can initially only be accessible via the physical port, maybe when I’m done I can make a mirco image of the ESXi hdd with the required VM, but even then the passthrough would have to be configured… ignore this rambling I’m just thinking stupid things…

Step 1) Have a ESXi host with a PCI-e based WiFi card.

I’ve tested this with both desktop Mobo with a PCI-e Wifi card, and a laptop with a built in Wifi Card, in both cases this process worked.

As you can see here I have a very basic ESXi server with some old hardware but otherwise still perfectly useable. For this setup it will be ESXi on USB stick, and for fun I made a Datastore on the remaining space on the USB stick since it was a 64 Gig stick. This is generally a bad idea, again for the same reasons mentioned above that USB sticks are not good at HIGH random I/O, and persistent I/O on top of that, but since this whole blog post is getting an ESXi host managed via WiFi which is also frowned upon why not just go the extra mile and really piss everyone off.

Again I could have done everything on the existing SATA based SSD and avoid so much potential future issue…. but here I am… anyway…

You may also note that at this time in the post I am connecting to a physical adapter on the ESXi host as noted by the IP addresses… once complete these IP addresses will not be used but remain bound the physical NIC.

Step 2) Create VM to manage the WiFi.

Again I’m choosing to use OPNsense cause they are awesome in my opinion.

I found I was able to get away with 1 GB of memory (even though min stated is 2) and 16 GB HDD, if I tried 8 GB the OPNsense installer would fail even though it states to be able to install on 4 GB SD Cards.

Also note I manually change boot from BIOS to EFI which has long been supported. At this stage also check off boot into EFI menu, this allows the VMRC tool to connect to ISO images from my desktop machine that I’m using to manage the ESXi host at this time.

Installing OPNsense

Now this would be much faster had I simply used the SSD, but since I’m doing everything the dumbest way possible, the max speed here will be roughly 8 MB/s… I know this from the extensive testing I’ve done on these USB drives from the ESXi install. (The install caused me so much grief hahah).

Wow 22 MB/s amazing, just remember though that this will be the HDD for just the OPNsense server that won’t need storage I/O, it’ll simply boot and manage the traffic over the WiFi card.

And much like how ESXi installed on the exact same USB drive, we are going to configure OPNsense to not burn out the drive. By following the suggestions in this thread.

Configuring  OPNsense

Much like the ESXi host itself at this point I have this VM connected to the same VMPG that connects to my flat 192.168 network. This will allow us to gain access to the web interface to configure the OPNsense server exactly in the same manner we are currently configuring the ESXi host. However, for some reason the main interface while it will default assign to LAN it won’t be configured for DHCP and assumes 192.168.1.1/24 IP… cool, so log into the console and configure the LAN IP address to be reachable per your config, in my case I’m going to give it an IP address in my 192.168.0.0/24 network.

Again this IP will be temporary to configure the VM via the Web GUI. Technically the next couple steps can be done via the CLI but this is just a preference for me at this time, if you know what you are doing feel free to configure these steps as you see fit.

I’m in! At this point I configure SSH access and allow root and password login. Since this it a WiFi bridged VM and not one acting as a firewall between my private network and the public facing internet this is fine for me and allows more management access. Change these how you see fit.

At this point, I skip the GUI wizard.  Then configured the settings per the link above.

Even with only 1 GB of memory defined for the VM, I wonder if this will cause any issues, reboot, system seems to have come up fine… moving on.

Holy crap we finally have the pre-reqs in place. All we have to do now is configure the WiFi card for PCI passthrough, give it to the VM, and reconfigure the network stacks. Let’s go!

Locate WiFi card and Configure Passthrough

So back on the ESXi web interface go to … Host -> Manage -> Hardware and configure the device for pasththrough until, you find all devices are greyed out? What the… I’ve done this 3 times what happed….

All PCI Passthrough devices grayed out on ESXi 6.7U3 : r/vmware (reddit.com)

FFS, OK I should have mentioned this in the pre-reqs but I guess in all my previous builds test this setting must have been enabled and available on the boards I was using… I hope I’m not hooped here yet again in this dang project…

Great went into the BIOS could find nothing specific for VT-d or VT-x (kind of amazed VM were working on this thing the whole time. I found one option  called XD bit or something, it was enabled, I changed it to disabled, and it caused the system to go into a boot loop. It would start the ESXi boot up and then half way in randomly reboot, I changed the setting back and it works just fine again.

I’m trying super hard right now not to get angry cause everything I have tried to get this server up and running while not having to use the physical NIC has failed… even though I know it’s possible cause I did this 2 other times successfully and now I’m hung cause of another STUPID ****ING technicality.

K I have one other dumb idea up my ass… I have a USB based WiFi NIC, maybe just maybe I can pass that to OPNsense…

VMware seems to possibly allow it: Add USB Devices from an ESXi Host to a Virtual Machine (vmware.com)

OPNsense… Maybe? compatible USB Wifi (opnsense.org)

Here goes my last and final attempt at this hardware….

Attempting USB WiFi Passthrough

Add device, USB Controller 2.0.

Add Device, Find USB device on host from drop down menu.

Boot VM….. (my hearts racing right now, cause I’m in a HAB (Heightened Anger Baseline) and I have no idea if this final work around is going to work or not).

Damn it doesn’t seem to be showing under interfaces… checking dmesg on the shell…

I mean it there’s it has the same name as the PCI-e based WiFi card I was trying to use, but that is 1) pulled from the machine, and 2) we couldn’t pass it through, and dmesg shows it’s on the usbus1… that has to be it… but why can’t I see it in the OPNsense GUI?

OMG… I think this worked… I went to Interfaces wireless, then added the run0 I saw in dmesg….

I then added as an available interface….

For some weird reason it gave it a weird assignment as WifIBridge… I went back into the console and selected option 2 to assign interfaces:

Yay now I can see an assignable interface to WAN. I pick run0

Now back into OPNsense GUI… OMG… there we go I think we can move forward!

Once you see this we can FINALLY start to configure the wireless connection that will drive this whole design! Time for a quick break.

Configuring WiFi on OPNsense

No matter if you did PCI-e passthrough or USB passthrough you should now have an accessible OPNsense via LAN, and assigned the WiFi device interface to WAN. Now we need to get WAN connected to the actual WiFi.

So… Step 1) remove all blocking options to prevent any network issues, again this is an internal bridge/router, and not a Edge Firewall/NAT.

Uncheck Block Private Networks (Since we will be assigning the WAN interface a Private IP), and uncheck Block bogon networks.

Step 2) Define your IP info. In my case I’m going to be providing it a Static IP. I want to give it the one that is currently being used to access it that is bound to the vNIC, but since it’s alread bound and in use we’ll give it another IP in the same subnet and move the IP once it’s released from the other interface. For now we will also leave it as a slash 32 to prevent a network overlap of the interface bound on LAN thats configured for a /24.

No IPv6.

Step 3) Define SSID to connect to and Password.

I did this and clicked apply and to my dismay.. I couldn’t get a ping response… I ssh’d into the device by the current VMX nic IP and even the device itself couldn’t ping it (interface is down, something is wrong).

Checking the OPNsense GUI under INterface Assignments I noticed 2 WiFI interfaces (somehow I guess from me creating it above, and then running the wizard on the console?).

Dang I wanted to grab a snip, but from picking the main one (the other one was called a clone), it has now been removed from the dropdown, and after picking that one the pings started working!

Not sure what to say here, but now at this point you should have a OPNsnese server accessible by LAN (192.168.0.x) and WAN (192.168.0.x). The next thing is we need to make the Web interface accessible by the WAN (Wireless) interface.

Basically, something as horrendous as this drawing here:

Anyway… the first goal is to see if the WiFi hold up, to test this I simply unplug the physical cable from the beaitful diagram above, and make sure the pings to the WAN interface stay up… and they both went down….

This happened to me on my first go around on testing this setup… I know I fixed it.. I just can’t remember how… maybe a reboot of the VM, replug in physical cable. Before I reboot this device I’ll configure a gateway as well.

Interesting, so yup that fixed the WiFi issue, OPNsense now came up clean and WiFi still ping response even when physical nic is removed from the ESXi host… we are gonna make it!

interesting the LAN IP did not come up and disappeared. But that’s OK cause I can access the Web GUI via the WAN IP (Wirelessly).

finally OK, we finally have our wireless connection, now we just need to create a new vSwitch and MGMT network on the ESXi host that we will connect to the OPNsense on the VMX0 side (LAN) that you can see is free to reconfigure. This also free’d the IP address I wanted to use for the WAN, but since I’ve had so many issues… I’m just going to keep the one I got working and move on.

Configure the Special Managment network.

I’m going to go on record and say I’m doing it this way simply cause I got this way to work, if you can make it work by using the existing vSwitch and MGMT interfaces, by all means giver! I’m keeping my existing IPs and MGMT interfaces on the default switch0 and creating a new one for the wireless connection simply so that if I want to physically connect to the existing connection.. I simply plug in the cable.

Having said that on the ESXi host it’s time to create a new vSwitch:

Now create the new VMK, the IP given here is the in the new subnet that will be routed behind the OPNsense WAN. In my example I created a new subnet 192.168.68.0/24 this will be routed to the WAN IP address given to OPNsense in my example here that will be 192.168.0.33. (Outside the scope of this blog post I have created routes for this on my devices gateway devices, also since my machine is in the same subnet at the OPNsense WAN IP, but the OPNsense WAN IP address is not my subnets gateway IP this can cause what is known as asymetric routing, to resolve this you simply have to add the same route I just mentioned to the machine managing the devices. You have been warned, design your stuff better than I’m doing here… this is all simply for educational purposes… don’t do this ever in production)

Now we need to create a VMPG for the VM to connect the VMX0 IP into the new vSwitch to provide it the gateway IP for that new subnet (192.168.68.1/24)

Now we can finally configure the vNIC on the OPNsense VM to this new VMPG:

Before we configure the OPNsense box to have this new IP address let’s configure the ESXi gateway to be that:

OK finally back on the OPNsense side let’s configure the IP address…

Now to validate this it should simply be making sure the ESXi host can ping this IP…

All I should have to do now is configure the route on my machine doing all this work and I should also be able to ping it…

More success… final step.. unplug physical nic to pings stay up?? OMG and they do!!! hahaha:

As you can see the physical NIC IP drops but the new secret MGMT IPs behind the WiFi stay up! There’s one final thing we need to do though.

Configure Auto Start of OPNsense

This is a critical step in the design setup as the OPNsense needs to come up automatically in order to be able to manage the ESXi host if there is ever a reboot of the host.

Then simply configure the auto start setting for this VM:

I also go in and change the auto start delay to 30 seconds.

Summary

And there you have it… and ESXi host completely managed via WiFi….

There are a ton of limitations:

  1. No Bridging so you can’t keep a flat layer 2 broadcast domain. Thus:
  2. Requires dedicated routes and complex networking.
  3. All VM traffic is best handled directly on internal vSwitch otherwise all other VM traffic will share the same WiFi gateway providing a terrible experince.
  4. The Web interface will become sluggish when the network interface is under load.
  5.  However it is overall actually possible.
  6. * Using PCI-e passthrough disallows snapshots/vMotions of the OPNsense VM but USB does allow it, when doing a storage vMotion the VM crashed on me, for some reason auto start disabled too had to manually start the VM back up. (I did this by re-IPing the ESXi server via console and plugging in a phsyical cable)
  7. With USB WiFi Nic connections can be connected/disconnected from the host, but with PCI-e Passthrough these options are disabled.
  8. With USB NIC you can add more vNICs to OPNsense and configure them, it just brings down the network overall for about 4-5 min, but be patient it does work.Here’s a Speedtest from a Windows Virtual Machine on the ESXi host.

Hope you all enjoyed this blog post. See ya all next time!

*UPDATE* Remember when I stated I wanted to keep those VMKs in place incase I ever wanted to plug the physical cable back in? Yeah that burnt me pretty hard. If you want a backup physical IP make it something different then you existing network subets and write it down on the NIC…

For some really strange reason HTTPS would work but all other connections such as SSH would timeout very similar to an asymmetric routing issue, and it actually cause it kind was. I’m kinda shocked that HTTPS even managed to work… huh…

Here’s a conversation I had with other on VMware IRC channel trying to troubleshoot the issue. Man I felt so dumb when I finally figured out what was going on.

*Update 2* I notice that the CPU usage on the OPNsense VM would be very high when traffic through it was taking place (and not even high bandwidth here either) AND with the pffilter service disabled, meaning it working it pure routing mode.

High CPU load with 600Mbit (opnsense.org)

Poor speeds and high CPU usage when going through OPNsense?

“Furthermore, set the CPU to 1 core and 4 sockets. Make sure you use VirtIO nics and set Multiqueue to 4 or 8. There is some debate going on if it should be 4 or 8. By my understanding, setting it to 4 will force the amount of queues to 4, which in this case matches your amount of CPU cores. Setting it to 8 will make OPNsense/FreeBSD select the correct amount.” Says Mars

“In this case this is also comparing a linux-based router to a BSD based one. Linux will be able to scale throughput much easily with less CPU power required when compared to the available BSD-based routers. Hopefully with FreeBSD 13 we’ll see more optimization in this regard and maybe close the gap a bit compared to what Linux can do.” Says opnfwb

Mhmmm ok I guess first thing I can try is upping the CPU core count. But this VM also hosts the connection I need to manage it… Seems others have hit this problem too…

Can you add CPU cores to VM at next restart? : r/vmware (reddit.com)

while the script is decent, the comment by cowherd is exactly what I was thinking I was going to do here: “Could you clone the firewall, add cores to the clone, then start it powering up and immediately hard power off the original?”

I’ll test this out when time permits and hopefully provide some charts and stats.

Posted on

Using Fake PMem to Strach That Itch

Sooooo… let’s say you have a ESXi server setup, with lots of memory, you have local storage and you could simply install ESXi on that and be done with it… but let’s just say you wanna be a hard ass and use the old USB install method. VMware’s ESXi will allow such an install but also warn you that no scratch (USB drives are not meant for heavy sustained I/O and fail due to this, thus they disable scratch here) and no core dump locations are configured (no way it could write the data fast enough, or be reliable with there’s a system crash).

So you wanna stick to your guns and have ESXi installed on USB, and you already disabled the annoying alert for core dumps cause this is a lab host and you simply don’t care…. alright….

Even then the pestering “System logs on host localhost.localdomain are stored on non-persistent storage. Consult product documentation to configure a syslog server or a scratch partition.” shows up, in my previous post I simply added a note…

Note* Option 2 was still required to get rid of another message: System logs are stored on non-persistent storage (2032823) (vmware.com)

That being “Option 2 – Set Syslog.global.logDir”, for this special beast of a lab setup though while we DO have an SSD planned to be a datastore, and again we could simply have installed ESXi here and moved on with my life. I want this to be the most comlpex ESXi host ever, and instead have full separate of the SSD and the ESXi host SO install, in a way the SSD can be fully unplugged up the system would still boot fine (given the USB drive is still alive).

So now I discovered you can create a datastore in memory, what a perfect place for a scratch partition (I know it won’t actually persist as noted by Willam himself and it won’t mean anything cause any core dumps won’t exist if the system experiences a power failure, but in my use case I simply don’t care I just need scratch, and what better place then memory…. Yes I’m aware scratch is suppose to be when memory is low, but if its reserved it’ll be there even when the host thinks there’s no memory to flow over to)… Anyway…

Where the fuck was I… oh Yeah PMem… Let’s see if this works in ESXi 7…

How to simulate Persistent Memory (PMem) in vSphere 6.7 for educational purposes?  (williamlam.com)

“Disclaimer: This is not officially supported by VMware. Unlike a real physical PMem device where your data will be persisted upon a reboot, the simulated method will NOT persist your data. Please use this at your own risk and do not place important or critical VMs using this method.

In ESXi 6.7, there is an advanced boot option which enables you to simulate or “fake” PMem by consuming a percentage of your physical ESXi hosts memory and allocating that to form a PMem datastore. You can append this boot option during the ESXi boot up process (e.g. Control+O) or you can easily manage it using ESXCLI which is my preferred method of choice.

Run the following command and replace the value with the desired percentage for PMem allocation:”

esxcli system settings kernel set -s fakePmemPct -v 5

“Note: To disable fake PMem, simply set the value to 0

You can also verify whether fake PMem is enabled or its current configured value is by running the following command:

esxcli system settings kernel list -o fakePmemPct

For the changes to go into affect, you obviously will need to reboot your ESXi host.”

Man I fought with getting my install to work for hours cause of a faulty USB drive.. I really should just install on the SSD… nah! I want the difficult way!

… Reboot and…. finally! it shows 🙂

I went to change the scratch location, but providing the path with the escape character didn’t work…. but specifying it directly the GUI took it.

as you can see the old path, but after reboot I got the error again, looks like it didn’t like the path…

That’s great…. the datastore name changes, and so does it’s mounted path, which explains why the error popped up again.. OI

So much for that idea

 

Posted on

PVE Hosts Won’t Boot, Missing Drive

This is pretty dumb… every other hypervisor I’ve ever played with, if the boot drive is fine… the OS boots… period….

Yet the other day I tried to boot my PVE host and it just won’t boot it would get stuck stating that a datastore (nothing that’s a dependency for the OS to actually boot) was causing the OS not to boot….

I found this PVE thread that was more recent with a comment that worked for the OP.

“if you created the partition via the gui, we create a systemd-mount unit under /etc/systemd/system

(e.g. mnt-datastore-foo.mount)

you can disable that unit with ‘systemctl disable <unit>’

or delete the file

we’ll improve the docs in the near future and have planned to make the gui disk management a bit easier in regards to removing/reusing”

This was posted in 2021, yet when I checked that path (booting into Recovery mode) it didn’t contain any file with a ending of .mount… So not sure what this is about. I did however find this thread which was exactly my problem… and funny enough the OP literally posted their own answer (which is the answer here as well) and no other comments made on the post, which was created in  2018…

[SOLVED] – Reboot Proxmox Host now will not boot | Proxmox Support Forum

“So I had upgraded some packages and the proxmox host recommended rebooting the system. After rebooting the system hangs at the screen showing [DEPEND] in yellow for 3 lines:
Dependency failed for /mnt/Media
Dependency failed for local file system
Dependency failed for File system check on /dev/Data-Storage/Media

I tried running control-D to continue but it does not continue.

I’m guessing I need to clean up the entries how can i do that? I’m assuming I just need to boot into emergency and edit /etc/fstab and remove the entries?

OK yes removing those from /etc/fstab fixed it and now it boots.”

This is exactly what I did as well… I saw the offending entry which was a BTRFS storage I had configured in the past and that storage unit had been shutdown. (I thought I blogged this, but I only blogged about using LVM over iscsi.. Configuring shared LVM over iSCSI on Proxmox – Zewwy’s Info Tech Talks)

Anyway, removing the entry from fstab and rebooting.. bam PVE host came right up.

Constructive criticism to PVE, while yes any knowledgeable Linux sysadmin will figure out how to fix this, as I just did here. However, how about NOT having the boot process fail simply cause a configured storage is not available… like all other hypervisors… BOOT the host and show the storage as failed in the management UI to clean it up that way…. Just.. food for thought….

Posted on

PA VM in bazaar state… by Design

So today I had some weird stuff happening (Fedora Download was downloading slow, 300 KB/s)… I thought it was the mirror, but no matter what mirror I picked I had the same results, I asked a buddy to verify my findings and they could download Fedora with speed… Long story short, I thought maybe it was my firewall, and my colleague mentioned the same. Since this is a Lab setup it would be nice to get a perpetual license for learning purposes, but PAN clearly don’t work like. I was pretty sure my license had expired, so decided to first quick finds out what happens when a license expires: What Happens When Licenses Expire? (paloaltonetworks.com)…

Threat Prevention
Alerts appear in the System Log indicating that the license has expired.
You can still:
  • Use signatures that were installed at the time the license expired, unless you install a new Applications-only content update either manually or as part of an automatic schedule. If you do, the update will delete your existing threat signatures and you will no longer receive protection against them.
  • Use and modify Custom App-ID™ and threat signatures.
You can no longer:
  • Install new signatures.
  • Roll signatures back to previous versions.

Good to know, nothing that would cause the issue I’m experiencing….

DNS Security
You can still:
  • Use local DNS signatures if you have an active Threat Prevention license.
You can no longer:
  • Get new DNS signatures.

nope… and…

Advanced URL Filtering / URL Filtering
You can still:
  • Enforce policy using custom URL categories.
You can no longer:
  • Get updates to cached PAN-DB categories.
  • Connect to the PAN-DB URL filtering database.
  • Get PAN-DB URL categories.
  • Analyze URL requests in real-time using advanced URL filtering.
WildFire
You can still:
  • Forward PEs for analysis.
  • Get signature updates every 24-48 hours if you have an active Threat Prevention subscription.
You can no longer:
  • Get five-minute updates through the WildFire public and private clouds.
  • Forward advanced file types such as APKs, Flash files, PDFs, Microsoft Office files, Java Applets, Java files (.jar and .class), and HTTP/HTTPS email links contained in SMTP and POP3 email messages.
AutoFocus
You can still:
  • Use an external dynamic list with AutoFocus data for a grace period of three months.
You can no longer:
  • Access the AutoFocus portal.
Cortex Data Lake
You can still:
  • Store log data for a 30-day grace period, after which it is deleted.
  • Forward logs to Cortex Data Lake until the end of the 30-day grace period.
GlobalProtect
You can still:
  • Use the app for endpoints running Windows and macOS.
  • Configure single or multiple internal/external gateways.
You can no longer:
  • Access the Linux OS app and mobile app for iOS, Android, Chrome OS, and Windows 10 UWP.
  • Use IPv6 for external gateways.
  • Run HIP checks.
  • Enforce split tunneling based on destination domain, client process, and video streaming application.

All a bunch of nope…

VM-Series
Support
You can no longer:
  • Receive software updates.
  • Download VM images.
  • Benefit from technical support.

This is a VM series yes… so what does that link mean….

VM-Series
You can still:
You can continue to configure and use the firewall you deployed prior to the license expiring with no change in session capacity. The firewall won’t reboot automatically and cause a disruption in traffic.
However, if the firewall reboots for any reason, the firewall enters an unlicensed state. While unlicensed, a firewall supports a maximum of 1,200 sessions. No other management plane features or configuration options are restricted.

OK… Maybe… but I’m sure a download of a single file doesn’t take over 1,200 sessions… while I did reboot the unit (cloned, power off OG, power on clone, etc)

All other things are the same as posted above… Then I noticed some really weird things….

  1. Checking for updates doesn’t state anything about license status, just tries and quietly fails.
  2. Checking support status shows “Device not found on this update server”
  3. Dynamic Updates do not show a “currently installed” version.
    1. The current version installed with Review Policies, and review apps under action.
    2. The previous installed one will have the same plus a revert action.
    3. Downloaded one will have an install action.
    4. All others seen since last communication to PAN will have download
  4. Retrieving licenses from licenses server returns “Failed to install features. The device is not found.
  5. Finally the smoking gun… Serial Number on the Dashboard will be listed as unknown.

So, I ended Googling this and found not one, but TWO KB’s!!!

Serial number becomes “unknown” after changing the instance typ… – Knowledge Base – Palo Alto Networks

and

Serial number becomes “unknown” upon rebooting PA-VM – Knowledge Base – Palo Alto Networks

After reading these, it all made sense… and it’s all rather dumb… to paraphrase it simply….

It’s due to DRM, how the DRM works is it derives the serial number from two ID’s CPUID and UUID… and when you migrate a PAN VM the CPU is different cause of the different host it resides… this in turn breaks the licensing.

*Standing Ovation*

What’s PAN solution… Open a support ticket… that’s right.. instead of coming up with a technical solution to make DRM work while still retaining the ability to migrate the VM (The most important and valuable reason why you want to run it as a VM anyway)….

Instead of having a way to edit the CPUID and UUID in the PAN portal to fix this yourself…..

No they want you to waste their tech support personals time….

This ….. IS……. DUMB!!!!!

Posted on

Rebuild FreeNAS/TrueNAS

So, the boot drive died in my FreeNAS server that was running 11.1u1. Good build. I figured since it died now would be a time to try TrueNAS Core. My SAN didn’t seem to be booting the installer, I tried different USB sticks and eventually just ended up using my IODD device, but all of which wouldn’t show the installer (albet on xterm/serial connection since the SAN is headless).

I decided to run the installer on a laptop and installing on …a….USB stick… wait a second….. when FreeNAS was first failing I tried moving the system log files off there since USB drives aren’t really meant for high R/W operations. These are set by the System -> System Dataset. What I think happed was the drive probably could’ve lasted a while longer but since the log files was still configured for the default “boot-pool” aka on the USB boot stick, and I had configured iSCSI with proxmox and at the end I discovered the log flooding problem… I bet it was this log flooding along with writing and log rotating on the USB boot device must have killed it…. Just wow man….

Anyway, I installed on a USB stick on a laptop picking the USB drive as a souce and picking BIOS boot option. I plugged the device into my SAN and boot up, now watching the boot operation (after verifying the boot order was correct) via the serial console window, after the system info the console gets stuck at “Press any key to continue”. I looked at the USB drive and the light indicated that something was happening, also looking at the NIC lights seemed something was going on, but also seem a bit pattern like and I wasn’t sure if something or anything was working. I decided to quickly check my DHCP server for any new entries and sure enough there it was….

I navigated to the address on my browser and sure enough, it booted!

There it is the dataset we need to change, now I don’t have any other pools defined so I can’t change it just yet, but lets make this our first priority so that if another bug from proxmox creeps in it won’t break my SAN. To do that I’m going to need to import the ZFS pools I had on this SAN when FreeNAS died.

Storage -> Pools -> Add

What a clean look. Import, No encryption. Looking for pools, this took a while.

There’s my pools! Sweet:

Next… exciting stuff…

Click Import.

Waiting….. waiting…. Let’s go!

What’s this notification…

I’ll leave that for now… can I change my system dataset? Oh no way it did it automatically….

well… even though I managed to import the pools successfully, since I can’t remember how the extents were made, I think they were file based, but no matter how I make it file based or device based the ESXi hosts see the drive but they aren’t automatically importing the datastore, probably cause they can’t see the FS that’s suppose to be on the iSCSI disk that’s being presented (the extents). I should have simply kept them device based and used the root pool. To add icing to the cake the one server I wanted to recover was the only one I didn’t run the backup copy job on after my USB drive that was hosting those files went belly up and I had to rerun all the jobs, AND the Veeam servers main backup repo was on one of the extents I can’t recover… so even though I had 3-2-1 rule in place I still ended up losing this server….

this is too much for me today, I’m goin to bed…

A new day, but end of it, so won’t finish this yet today either but there is hope.

I found this, which lead to this, which lead to this, and there’s sign of hope seeing the same thing in my own vmkernel log file.

So there it is, and ran the commands as specified in the threads and KB:

tail /var/log/vmkernel.log

Which showed me the volume not mounting due to snapshot.

esxcli storage vmfs snapshot list

Which showed me my old Datastore information.

esxcli storage vmfs snapshot resignature -l "iOSlow"

Which mounted my datastore under a new name:

Sure enough connected the backup drive to my Veeam server. And restored my missing VM. Now I just need to move the rest of the data and create a clean datastore. Or I guess I could simply rename it, but it didn’t auto mount on the other esxi server.

Legal Information Proudly powered by WordPress