Category: Hypervisors

All posts related to Hypervisors

Posted on

New vCenter Same Veeam

The Story

The Niche Situation

Now I know the title might sounds strange, but this is to cover a niche issue which may randomly arise out in the industry. vCenter died, there was no backup, a new vCenter was spun up in its place with all the same hostname, IP address and everything, and the hosts re-added, and you happen to use Veeam as your backup solution. Now I have been down this rabbit hole in the past, and I have blogged about an unsupported method to fix the Veeam jobs in the situation. But it’s technically unsupported, so I asked what the “supported method” would be on the Veeam forms.

The short answer, “Oh just use the VM-Migrator tool”, as referenced here.

“Veeam Backup & Replication tracks VMs in jobs using Managed Object Reference IDs (MORef-IDs), which change after migration or recreation of vCenter, causing MORef-ID misalignment.

Veeam VM Migrator utility is integrated into Veeam Backup PowerShell module, and it allows you to resolve MORef-ID misalignment. As a result, your backup incremental chains will remain intact after an inventory change in vCenter.

The utility consists of the following cmdlets:

  • Set-VBRVmBiosUuid — this cmdlet updates the BIOS UUIDs of existing VM entries within the Veeam Backup & Replication configuration database based on information from the old vCenter.
  • Set-VBRVCenterName — this cmdlet modifies vCenter name by adding the _old suffix to its name.
  • Generate-VBRViMigrationSpecificationFile — this cmdlet generates a migration task file which contains the list of mapping tasks.
  • Start-VBRViVMMigration — this cmdlet starts MORef-IDs update.”

So, this tool is supposed to do what I did via the backend but this is a supported frontend tool to do it, but I case is generally different than what the tool wants in that my old and new vCenter are the same, and not simply two unique instances of vCenter with unique names both running live in parallel. Mines simply been directly rebuilt in place.

Step 1) Realize your vCenter is toast.

However, you realize this, will be random and situational, in my case my trial expired, and all ESXi hosts show disconnected. I’m gonna treat this as a full loss, by simply shutting down and nuking all the VM files… it’s simply dead and gone…. and I have no configuration backup available.

This is why this is considered a niche situation, as I’d hope that you always have a configuration backup file of your critical infrastructure server. But… what if (and here we are, in that what if, again)…

Step 2)  Rebuild vCenter with same name.

Yay, extra 20 min cause of a typo, but an interesting lesson learnt.

Renaming vCenter SSO Domain – Zewwy’s Info Tech Talks

Let’s quickly rebuild our cheap cluster,  configure retreat mode and add our hosts back in…

OK so now we’ve set our stage and we have a broken Veeam instance, if we try to scan it it will be no good cause the certificate has changed, from the center changing… so David says “So in your case, if you can restore Veeam’s configuration database to before you made these changes, instead of your step 4 there, you will begin the migration procedure and use the Set-VBRVCenterName cmdlet on the existing vCenter in Veeam, re-add your newly rebuilt vCenter to Veeam, and then perform the migration.”

Step 3) run “Set-VBRvCenterName”.

So far, so good.. now..

Step 4) Add new vCenter to Veeam.

Step 5) Generate Migration File.

Now I’m back to assuming, cause instructions are unclear in Veeams provided guidance. I’m assuming I have to run the generate command before I run the start migration command….

Checking out the generated file, its a plain text file with a really weird syntax choice, but the VM-IDs are clearly as I was doing manually in my old blog post.

Step 6) Start the Migration.

I have no clue what that warning is about… I mean the new vCenter was added to Veeam, the VM IDs matched what I see in the URL when navigating them, like my old blog… I guess I’ll just check on VBR console…

I did a recalculate on the VM inside the backup job and it calculated, so looks like it worked. Let’s run a backup job and check the chain as well…

The Job ran just fine…  and the chains still intact. Looks like it worked, this was the supported way, and it did feel easier, especially if scaled out to hundreds of VMs.

Hope this helps someone.

Posted on

Renaming vCenter SSO Domain

Whoopsie, I made boo boo, How me fix?

Source:  Repointing vCenter Server to another SSO Domain – VMware Cloud Foundation (VCF) Blog

In this example I will be repointing a single vCenter Server (version: 7.0.3) in the SSO Domain “csphere.local” to an entire new SSO Domain named “vsphere.local“. Since there is no other vCenter Server’s in the vsphere.local SSO Domain, a pre-check is not required, thus we will not have any conflicts to resolve.

In my lab I am repointing vCenter.zewwy.ca which is my vCenter Server. Notice that my Single Sign-On domain is csphere.local currently.

From the appliance shell we can run cmsso-util to review our command syntax. Here we can also see the other functions of the cmsso-util command such as unregister, reconfigure, repoint (for repointing a vCenter Server to another SSO Site), and domain-repoint. We will be using the domain-repoint argument to point our vCenter Server to a new SSO Domain.

Since we are not migrating this vCenter Server into an existing SSO Domain, the is no need to do a pre-check to review any possible data conflicts between the Source and Destination domains. We begin repointing with the following command:

 cmsso-util domain-repoint -m execute --src-emb-admin Administrator --dest-domain-name vsphere.local

NOTE: The SSO Administrator (Administrator@csphere.local) credentials ARE REQUIRED. Also, the Destination domain name (–dest-domain-name) equals the name of the new SSO Domain you are pointing the Source vCenter Server to.

Yay it worked I can finally successfully sign into vsphere… dang one typo costed about 20 min  of time, crazy, but I guess it’s slightly better than having to do a whole rebuild which takes a bit more time, but not far off sheesh. Hope this post helps someone.

Posted on

Adding a Hyper-V host to Veeam

Before You Begin – Veeam Backup & Replication User Guide for Microsoft Hyper-V

Before you add a Microsoft Hyper-V server to the backup infrastructure, check the following prerequisites:

  • Check permissions required to add the server. For more information, see Permissions.
    • Admin permissions based account got it…
  • [For SCVMM] SCVMM Admin UI must be installed on the backup server. Otherwise, you will not be able to add SCVMM servers to the backup infrastructure.
  • SCVMM console version must match the management server version.
  • Make sure that you do not add to the backup infrastructure Hyper-V hosts or clusters managed by an SCVMM server if this SCVMM server is already added to the backup infrastructure.
    • Nope just a stand alone host
  • File and printer sharing must be enabled in network connection settings of the added Microsoft Hyper-V host. Otherwise, Veeam Backup & Replication will fail to deploy required components.
    • Uhhh wut?
  • Make sure that the NETBIOS name of the Microsoft Hyper-V Server is successfully resolved.
    • Uhhh wut?
  • If you get the “Invalid Credentials” error when adding a Hyper-V host using a local account, see this Veeam KB article.

This is gonna suck..

Unable to add a single Hyper-V host to Veeam. : r/Veeam

i am unable to add Hyper V hosts to Veeam | Veeam Community Resource Hub

Why?…

When you add a Hyper‑V host to Veeam Backup & Replication, the product deploys its transport service and integration components remotely using Windows’ built‑in administrative shares (ADMIN$, C$). That’s why File and Printer Sharing must be enabled on the NIC: without those hidden shares, Veeam cannot copy files or install its agents. By default, only the built‑in Administrator or domain admin accounts can access these shares remotely, because User Account Control (UAC) strips remote admin rights from other local accounts. This often surprises people who harden their hosts by disabling the Administrator account or removing shares, since Veeam’s deployment model depends on them being present.

On standalone Hyper‑V hosts, this creates a security trade‑off. You can either leave the built‑in Administrator enabled (simpler, but harder to audit), or disable UAC remote restrictions so named local admin accounts can access the shares (more auditable, but technically weaker security posture because all local admins gain remote rights). In practice, many administrators prefer creating a dedicated service account for Veeam and a separate account for human administration, then disabling the built‑in Administrator. This way, activity is traceable and controlled, while still allowing Veeam to function. The nuance is that Veeam chose the “lowest common denominator” approach — SMB admin shares — which works everywhere but clashes with modern hardening practices, so standalone hosts require careful balancing of convenience, auditability, and exposure.

Step 1) Enable SMB

Install-WindowsFeature -Name FS-FileServer -IncludeManagementTools

Edit your firewall rules as required as this will create 3 new ones and open them up (135 DCOM, 445 SMB, and dynmic ports one), in my case I disabled them and only enabled the SMB restrictive rule.

Check off Microsoft file and print sharing service under the NIC settings for which will be used to add Hyper-V to Veeam.

Maybe we can enable it only during deployment then disable it, lets find out. On Hyper-V lets create a dedicated Veeam admin account, then disable remote UAC while adding the host to Veeam. Done, adding host to Veeam…

Option 1) Specify the local administrator account. (Usually disabled on hardened servers)

OR

Option 2) edit registry to allow remote uac, so the built in admin shares can be accessible by admin account that is named and not the built in administrator account.

Why Veeam does allow for the ability to prepare a Hyper-V host via these install packages manually without exposing the post to these additional attack surfaces is honestly beyond me. I usually love Veeam but this one is kind of dumb.

Step 2) Disable Remote UAC restrictions

I’ll stick with option 2: User Account Control and remote restrictions – Windows Server | Microsoft Learn

To disable UAC remote restrictions, follow these steps:

  1. Click Start, click Run, type regedit, and then press ENTER.
  2. Locate and then click the following registry subkey: 
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System
  3. If the LocalAccountTokenFilterPolicy registry entry doesn’t exist, follow these steps:
    1. On the Edit menu, point to New, and then select DWORD Value.
    2. Type LocalAccountTokenFilterPolicy, and then press ENTER.
  4. Right-click LocalAccountTokenFilterPolicy, and then select Modify.
  5. In the Value data box, type 1, and then select OK.
  6. Exit Registry Editor.

Now open a file explorer window in Veeam Server, and point to \\IPofHyper-V\admin$ it should prompt you for creds, you should be able to provide the creds of the named admin account and it should connect.

Well I got past the error…

Sigh n’ groan…. ughhhh.. too be continued, time to make a Server 2025 image…

Lets try again..

And this time success…

Restore Storage Theory

🖥️ Scenario

  • Source: Veeam is running inside a VM on ESXi.
  • Repository: Local storage attached to that VM (so Veeam sees it as a local NTFS/ReFS volume).
  • Target: A standalone Hyper‑V host with only local storage (no SMB shares, no clustered SOFS).

🔧 How Veeam Writes the VM HDD Files

  1. Restore job starts
    • You pick the Hyper‑V host as the restore target.
    • Veeam knows it must deliver VHDX files + VM configuration to that host’s storage path (e.g., D:\VMs\MyVM\).
  2. Transport service on Hyper‑V host
    • Veeam deploys or uses its Veeam Data Mover Service (part of the Veeam transport service) on the Hyper‑V host.
    • This service is responsible for receiving blocks of data and writing them to disk.
  3. Data transfer
    • The Veeam server (on ESXi) reads blocks from the backup file in its local repository.
    • Those blocks are sent over the network to the Hyper‑V host using Veeam’s own transport protocol (TCP/IP).
    • Important: This is not SMB — it’s Veeam’s proprietary data mover channel.
  4. File creation on Hyper‑V host
    • The transport service on the Hyper‑V host opens a file handle on the local filesystem (NTFS/ReFS).
    • It creates the target VHDX file and writes the incoming blocks directly using standard Windows file I/O APIs (CreateFile, WriteFile, etc.).
    • VM configuration files (.vmcx, .vmrs) are also written directly to the host’s local storage.
  5. Completion
    • Once all blocks are written, Hyper‑V sees the restored VM files in its local storage.
    • Veeam registers the VM with Hyper‑V Manager if you chose a full VM restore.

✅ Key Points

  • No SMB is used here.
  • Veeam uses its own transport service to push data over TCP/IP to the Hyper‑V host, which then writes the files directly to local disk.
  • SMB only comes into play if the repository or Hyper‑V storage is on a remote file server (like a NAS or SOFS cluster).

Retore to Hyper-V

Here a whole video on the process, cause I wasn’t sure how to do it as when I selected restore entire VM to new location, only my ESXi hosts were selected, AI said it not possible, Googling said that Instant Restore was the only option… mhmm that video showed the same thing…

I won’t lie I felt so dumb at first cause the restore prompt said “waiting on user input” and there was an open console link at the bottom of the instant restore wizard, so I clicked that and it kept asking for creds (I thought the hyper-v ones) and it kept failing… till I realized you just have the VM already running (or not based on your selection) but it’s already registered to the host, you have to finish an instant restore by clicking migrate to production option.

I tell ya… that made me feel really…. reallllly dumb…..

anyway I hope this posts helps someone.

 

Posted on

Hyper-V Networking

Install Hyper-V, and let’s learn it’s Networking

Hyper-V Networking

On VMware, when you install the OS (ESXi) on the hardware in the DCUI you simply set the VLAN tag right on the network interface you configure, and bind it to a physical NIC in the menu area, simple.

We leave the network connected to a non tagged, internet allowed connection. Now we can configure our networks.

1. Create one external switch

  • Bind it to your physical NIC (configured as a trunk on the upstream switch).
  • Check “Allow management OS to share this network adapter” once — this creates the first host vNIC.

2. Add more host vNICs via PowerShell:

# Add another vNIC for the host bound to the same external switch
Add-VMNetworkAdapter -ManagementOS -Name "HostVLAN20" -SwitchName "ExternalSwitch"
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "HostVLAN20" -Access -VlanID 6

3. Verify in Windows

  • You’ll now see multiple vEthernet adapters in the host OS (e.g., “vEthernet (HostVLAN20)”, “vEthernet (HostVLAN30)”).
  • Assign IP addresses or enable DHCP as needed.

*Note* this is for doing it on a single network server (for my lab use) usually you’d have dedicated NICs for other networks designated on those servers… anyway.. where was I… right now that we got additional network on the server we can work on a shared storage, you know how I like my iSCSI.

So, when I setup Hyper-V, and connect a VM to the “default” virtual switch, the Hyper-V server acts as a DHCP provider and NATs the connections from that vSwitch to whatever other network the host is connected to?

Great question — this trips up a lot of people when they first use Hyper‑V! Let’s break it down clearly:

🖧 Hyper‑V Default Switch Behavior

  • Introduced in Windows 10 (1709): The “Default Switch” is automatically created by Hyper‑V and cannot be deleted or modified.
  • NAT (Network Address Translation): Yes — the host machine provides NAT services. This means your VM’s traffic is translated and routed through the host’s network connection, regardless of whether that’s Wi‑Fi, Ethernet, or even VPN.
  • DHCP Services: The host also runs a lightweight DHCP service for the Default Switch. This is what hands out IP addresses to your VMs automatically (usually in the 172.16.x.x range).
  • Connectivity: VMs connected to the Default Switch can reach the internet and other devices on the host’s network, but inbound connections from the LAN to the VM are limited. It’s designed for convenience, not for complex networking setups.

⚖️ Comparison with Other Switch Types

Switch Type Provides DHCP? Provides NAT? Use Case
Default Switch ✅ Yes (host DHCP) ✅ Yes (host NAT) Quick setup, internet access without config
External Switch ❌ No (relies on external network DHCP) ❌ No VM directly on physical network, full LAN visibility
Internal Switch ❌ No ❌ No Communication between host and VMs only
Private Switch ❌ No ❌ No Communication between VMs only

✅ So to answer directly:

Yes — when you connect a VM to the Default Switch, Hyper‑V’s host acts as a DHCP provider and NATs the VM’s traffic to whatever network the host is connected to. It’s essentially a convenience bridge so you don’t have to manually configure networking.

“That’s what I did configured a external virtual switch so the hosts NIC would be VLAN tagged in a certain network, I was figuring the default switch would act more like a … well switch at layer2 of the hosts Nic, so VMs on it would get DHCP from the main network, as if it was connected to a switch in the base network, not layer3 and then 4 NAT like c’mon…”

Exactly — you’ve nailed the distinction. The Default Switch in Hyper‑V is deliberately not a true Layer‑2 bridge to your host NIC. It’s a convenience construct that Microsoft built to “just work” for internet access without requiring you to mess with VLANs, external switches, or DHCP servers on your LAN.

Here’s the breakdown:

🔌 Why the Default Switch behaves like NAT

  • It’s implemented using Windows Internet Connection Sharing (ICS) under the hood.
  • ICS is a Layer‑3 service: it provides a private subnet (usually 172.16.0.0/12), runs a DHCP service, and NATs traffic out through whatever adapter the host is using.
  • That’s why VMs on the Default Switch don’t see your LAN’s DHCP server — they only see the host’s mini DHCP/NAT service.
  • It’s intentionally isolated so you don’t accidentally expose VMs directly to your corporate or home LAN without realizing it.

⚖️ Why it’s not Layer‑2 bridging

  • A true Layer‑2 bridge would forward broadcasts (like DHCP Discover) from the VM to your LAN. That’s what an External Virtual Switch does.
  • The Default Switch instead terminates those broadcasts at the host and answers them itself. That’s why you only ever get the host‑provided DHCP.
  • VLAN tagging, trunking, or other advanced networking features aren’t supported on the Default Switch — it’s locked down.

✅ When to use each

  • Default Switch (NAT + DHCP): Quick internet access, testing, dev boxes, laptops where you don’t want to reconfigure NICs.
  • External Switch (Layer‑2 bridge): When you want VMs to behave like physical machines on your LAN, get DHCP from the real network, or use VLAN tagging.
  • Internal/Private Switches: For host‑VM communication or VM‑VM only networks.

So your expectation — that the Default Switch would act like a transparent Layer‑2 bridge — makes sense, but Microsoft intentionally designed it as a Layer‑3 NAT gateway instead. If you want “real switch” behavior, the External Switch is the only way to go.

“So, whatever my host machine is allowed to do, the guest VMs in the default switch are also allowed to do? like access the internet, and the firewall that gave the host the ability to access the internet would never see the private NATed IP address the Hyper-V host is dishing out to those VMs?”

You’ve got it — that’s exactly how the Default Switch works in Hyper‑V. Let me spell it out clearly:

🌐 How traffic flows with the Default Switch

  • VMs get private IPs: Each VM connected to the Default Switch is assigned an IP from the host’s internal DHCP service (usually 172.16.x.x).
  • Host does NAT: The Hyper‑V host translates those private IPs into its own IP before sending traffic out to the physical network.
  • Firewall visibility: From the perspective of your upstream firewall or router, all traffic from those VMs looks like it’s coming from the host’s IP. The firewall never sees the VM’s private NATed addresses.
  • Permissions: Whatever the host is allowed to do (e.g., access the internet, reach certain networks), the VMs can also do — because their traffic is piggybacking on the host’s connection.

⚖️ Important implications

  • Outbound access: VMs can reach the internet or LAN resources as long as the host can. They inherit the host’s connectivity.
  • Inbound access: External devices cannot directly initiate connections to those VMs, because NAT blocks unsolicited inbound traffic unless you set up port forwarding (which the Default Switch doesn’t expose for configuration).
  • Firewall rules: The upstream firewall only enforces rules against the host’s IP. If you need per‑VM visibility or control, you’ll need an External Switch so each VM gets its own LAN IP.

✅ So yes: the Default Switch makes your VMs “ride along” with the host’s network permissions. They’re hidden behind NAT, which is why your firewall only ever sees the host’s traffic.

Additional Information

If you want to apply a VLAN tag to a physical NIC that’s not a Hyper-V host on Windows:

Since I didn’t see anything in the usual Network  IPv4 configuration window in Windows. I decided to see how useful this AI stuff is.. so I asked it…

“how do I VLAN tag the IP on windows 11 interface?”

Quick Answer: On Windows 11, VLAN tagging isn’t enabled by default. You need a NIC that supports IEEE 802.1Q VLAN tagging, then configure the VLAN ID either through the NIC’s Device Manager properties (Advanced tab → VLAN ID) or by using Intel PROSet/ANS tools or PowerShell (Set-NetAdapter -VlanID) if your driver supports it.

🖥️ Steps to Configure VLAN Tagging on Windows 11

1. Check NIC Compatibility

  • VLAN tagging requires a network card (NIC) that supports IEEE 802.1Q.
  • Not all NICs expose VLAN options in Windows. Intel and Broadcom NICs often support it, but you may need vendor-specific drivers.

2. Configure via Device Manager

  • Press Win + R, type devmgmt.msc, and hit Enter.
  • Expand Network adapters, right-click your NIC → Properties.
  • Go to the Advanced tab.
  • Look for VLAN ID or Priority & VLAN.
  • Enter the VLAN ID (e.g., 100) and click OK.

3. Configure via Intel PROSet/ANS (if supported)

  • Install Intel PROSet/Advanced Network Services (ANS).
  • In the NIC properties, Intel’s tab allows creating multiple VLAN interfaces with different IDs.
  • Each VLAN interface can then be assigned its own IP address.

4. Configure via PowerShell

  • Open PowerShell as Administrator.
  • Run: 
    Set-NetAdapter -Name "Ethernet" -VlanID 100
  • Replace "Ethernet" with your adapter name and 100 with your VLAN ID.
  • Note: This only works if your NIC driver supports VLAN tagging. Otherwise, you’ll get an error like “Requested operation not supported on adapter”.

5. Assign IP Address

  • Once VLAN tagging is enabled, go to Network & Internet settings → Change adapter options.
  • Right-click the VLAN-enabled adapter → Properties → Internet Protocol Version 4 (TCP/IPv4).
  • Assign the IP address, subnet mask, and gateway for that VLAN.

⚠️ Important Notes

  • Driver Support is Key: If your NIC driver doesn’t expose VLAN options, you cannot tag VLANs directly in Windows without Hyper-V or third-party tools.
  • Hyper-V Alternative: You can create VLAN-tagged virtual NICs using Hyper-V’s Virtual Switch Manager if your hardware doesn’t support native tagging.
  • Testing: Use ping or tracert to verify connectivity once VLAN tagging is applied.

✅ Bottom Line: To VLAN tag an IP on Windows 11, you must enable VLAN tagging on the NIC (via Device Manager, Intel PROSet/ANS, or PowerShell if supported), then assign the IP to that VLAN interface. If your NIC doesn’t support VLAN tagging, you’ll need Hyper-V or vendor-specific utilities.

My response “I’ve got an intle 82579LM Gig NIC, I see it says VLAN enabled in the advanced tab, but I don’t see an area to configure it, I’m sure the NIC supports it since I was using ESXi with vlan tagging just fine.”

That’s a great observation — the Intel 82579LM definitely supports VLAN tagging at the hardware level, but here’s the catch: Windows drivers for that NIC don’t expose VLAN configuration in the same way ESXi or Linux does.

🔎 Why You Don’t See VLAN Config in Windows

  • The Advanced tab in Device Manager shows “Priority & VLAN” or “VLAN Enabled,” but that option only toggles whether the NIC can process VLAN-tagged frames.
  • Unlike ESXi, Windows doesn’t let you directly assign VLAN IDs to a physical NIC unless the driver provides a VLAN interface creation utility (like Intel PROSet/ANS).
  • For the 82579LM, Intel stopped including VLAN configuration in their Windows drivers after Windows 8/10. On Windows 11, the driver exposes VLAN capability but no UI to configure VLAN IDs.

✅ Workarounds

Here are the practical options:

1. Intel PROSet/ANS (if available)

  • Older Intel drivers came with Advanced Network Services (ANS), which let you create VLAN sub-interfaces.
  • On Windows 11, Intel no longer bundles ANS with drivers for the 82579LM, so you won’t see the VLAN tab.

2. PowerShell Command

  • Windows has a Set-NetAdapter -VlanID cmdlet, but it only works if the NIC driver supports VLAN tagging through the OS.
  • On the 82579LM with current Windows 11 drivers, this usually fails with “Requested operation not supported”.

3. Hyper-V Virtual Switch

  • If you enable Hyper-V, you can create a virtual NIC and assign VLAN IDs there.
  • This is the most reliable way to do VLAN tagging on Windows when the physical NIC driver doesn’t expose VLAN config.

4. Third-Party NIC Utilities

  • Some OEMs (Dell, HP) ship custom drivers/utilities that re-enable VLAN tagging. If your system vendor has one, that’s worth checking.

⚠️ Bottom Line

Your Intel 82579LM can tag VLANs, but on Windows 11 the Intel driver doesn’t expose VLAN configuration. That’s why you don’t see a place to set the VLAN ID.

  • On ESXi/Linux → VLAN tagging works natively.
  • On Windows → You need Intel PROSet/ANS (no longer supported for this NIC), Hyper-V, or vendor-specific drivers.

Well ok I guess once we install Hyper-V hopefully it’ll be more like ESXi in a sense and we can tag.

*UPDATE* ExternalSwitch, WiFi NIC

I was trying to use a Hyper-V server with an External Switch, bound to a WiFi NIC. and was getting unexpected results based on the above information. but my results were NOT as I had expected, I was assuming Default Switch behavior. You may be asking, “Why would you expect that behavior if you’re creating an External Switch?”  Now, if you read my Wireless ESXi host blog, you probably are well aware of the VLAN limitations of WiFi, and can never truly be used as a trunk port (Limitation of the 802 standard, not of OS or software).

So how could a ExternalSwitch work, via WiFi if the layer 2 broadcast doesn’t work and can’t “speak” with the rest of the layer 2 stack? Yet I create a VM and it DOES get a DHCP lease address from my local subent?! What the heck is going on here…

So I had to ask AI, what was going on here, it says, yeah… that’s expected… here’s the deets… get ready.. it’s a long one….

BAH-LETED, there was way tooooo much BS from the AI response to justify keeping this data in here… long story short… Local subnet VMs work fine (it does ARP Masquerading), VLANs will never work per the usual BS I’ve talked about in my Wireless ESXi host blog.

Posted on

Upgrade vCenter from 7 to 8

Upgrading vCenter

Let me start by having a firm base, a working vCenter 7 with a proper connected Veeam server. Since my server is dead I’m going to start from scratch.

Pre-req Step (Base vCenter 7 with Veeam)

Other Pre-reqs:
DNS server (or local host records)

Step 1) Install vCenter 7

20 minutes later, complete. Now I made one mistake along the way (to replicate what I think happened the first go around) and that was to give the VM name vCenter.zewwy.ca, while when configuring the network FQDN I gave it vcenter.zewwy.ca. After install, I was able to replicate the findings I had from my first go around which I now know will cause and upgrade problem for the upgrade to vCenter 8. When this is done, the install wizard has no issues with this and the install will complete successfully. However now the hostname with have a uppercase, while the PNID will have a lowercase:

It’s this case while will cause the upgrade to have hiccups which I’ll cover below. However, I need to add my hosts which were connected to my previous upgrade, let’s see if it’ll take my hosts…

Step 2) I followed my blog post to fix up my Veeam jobs again. 

Step 3) Upgrade to vCenter 8.

Stage 1 Error

Remember when I showed above the mistake I made by entering the FQDN without a capital when configuring the network when I ran the vCenter7 installer. Putting lower case here, provided the above error, to get past it had to use the case sensitive hostname with a capital.

Stage 2 Error

Now pick your host and you have another error to bypass at Stage 2.

I had another weird issue where even though the VM deployed, it was not reachable over the network and the installer timed out. To resolve this I simply changed the VM network VMPG, saved, and then changed it back to the proper one saved, and it was pingable. To get back to stage 2 simly navigate to the VAMI web page on port 5480. When you get to the stage to connect to the source you enter the details and get this error at the pre-upgrade checks:

changing the vcenter source entry again to a capital as we did before will not work the same error pops up showing to go to that blog post on how to change the FQDN. Since my FQDN already looked correct (with a capital), but the command was showing (the PNID) was with a lower case, Instead of changing the FQDN to be lowercase, and going through all the steps in that blog (there are a lot) I simply set the PNID to have a capital in it

Get:

/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

Set:

/usr/lib/vmware-vmafd/bin/vmafd-cli set-pnid --server-name localhost --pnid vCenter.mydomain.com

Yay no more error:

Now pick your content to migrate, and at this point you should stop using vCenter for the duration of the migration.

At this point the Pings dropped (roughly 12 no reponse at about 1-2 minute down time…. then they came back up as the new vCenter IP changed over. At which point step 2 began.

I went to play a bit of party animals, it had timed out on the old vami IP, which the installer may have auto switched over, but logging into VAMI on the new server showed everything was green.

Logging in, everything looks good, checking Veeam yup, rescan of vcenter worked without issue, check re-calculating VM on a backup job, yup it works.

Success. This post doesn’t cover additional steps (applying new license keys, checking using migrations (vsphere.local), remoting loging services or connections (rsyslog)) all those you’d have to verify after completing these steps. Now to upgrade the ESXi hosts, both esxi hosts remained on 7.0.3x versions for this post.

Posted on

VMware Changes Update URLs

If you run a home lab, or manage systems for companies you may have noticed updates not working in VAMI… something like…. Ohhh I dunno.. this:

Check the URL and try again.

Unable to patch the vCenter via VAMI as it fails to download the updates from Broadcom public repositories

Cause

Public facing repository URLs and authentication mechanisms are changing. Download URLs are no longer common but unique for each customer therefore will require to be re-configured.

Well… wow thank you Broadcom for being so… amazing.

If you want to be overly confused about the whole thing you can this this KB: Authenticated Download Configuration Update Script

As the original link I shared above all you have to do is login to the Broadcom support portal, and get a token, and edit the URL…. but….

Notes:

    • The custom URL is not preserved post migration upgrade, FBBR restore and VCHA failover
    • If there is a proxy device configured between vCenter and the internet, ensure it is configured to allow communications to the new URL
    • Further patches automatically update this URL. For example, if 8.0.3.00400 is patched to 8.0.3.00500, the default URL will change to end in 8.0.3.00500.

Looks like this was enforced just a couple days ago … Sooooo, happy patching?   ¯\_(ツ)_/¯

Posted on

Permission to perform this operation was denied. NoPermission.message.format

For anyone who may use my site as a source of informational references, I do apologies, for the following:

  1. My Site Cert expiring. ACME is great, I’m just a bit upset they refuse to announce their HTTP auth sources so I can’t create a security rule for it. Right now it would be restricted to App Type. While not bad.. not good enough, so I manually have to allow the traffic for the cert to be renewed.

    No… I have no interest in allowing ACME access to my DNS for DNS auth.

  2. Site was down for 24 hours. If anyone noticed at all, yes my site was down for over 24 hours. This was due to a power outage that lasted over 12 hours after a storm hit. No UPS could have saved me from this. Though one is in the works even after project “STFU” has completed.

    No, I have no interest in clouding my site.

I have a couple blog post ideas roaming around, I’m just having a hard time finding the motivation.

Anyway, if you get “Permission to perform this operation was denied. NoPermission.message.format” while attempting to move a ESXi host into a vCenter cluster. Chances are you may have a orphaned vCLS VM.

If so, log into VAMI and restart the ESX Agent Manager (EAM) service.

After restarting that service everything should hunky dory…

Cheers.

Posted on

Careful Cloning ESXi Hosts

I’ll keep this post short. I was doing some ESXi host deployments in my home lab, and I noticed that when I would install on a 120GB SSD, the install would go smoothly, but I wasn’t able to use any of the storage as a Datastore. However, if I took a fresh install copy of ESXi from installing onto an 8GB USB Stick and DD’d it to the 120GB SSD I got several advantages from this:

  1. When done via a USB3 Pipe of Linux live holding a copy of my base image to deploy I could get speeds in excess of 100 MB/s, and with only 8GB of data to transfer, the “install” would complete in a mere 90 seconds.
  2. The IP address and root password are preconfigured to what I already now, and I can simply change the IP address from the DCUI and call it a day.

Using this method I could have a host up in less than 5 minutes (2 min to boot linux live, 90 seconds to install the base ESXi OS image, and 2 more to boot ESXi). This was of course on machine without ECC and all the server hardware firmware jazz… in those cases install times are always longer. anyway…

This was an amazing option, until I noticed that when I connect one machine in I just deployed and changed the IP address, and (since I’m super anal about networking during this type of project/operations) I noticed my ping from one machine (a completely different IP address) started to drop when the new device came up… and after a while the ping responses would come back but drop from the new host, and vice versa, flip and flop it goes. I’m used to this usually if there’s an IP conflict and two devices have the same IP address. In this case they were different IP addresses… after enough symptom gathering and logical deduction of because I had to assume that the MAC address just be the same and this is the same problem in reverse (different IP’s but same MAC) and as such experiencing the same symptoms.

To validate this I simply deployed my image to a new machine, then I went on the hunt to figure out how to see the MAC address, since I couldn’t plug in the NIC and get to the web based MGMT interface I had to figure out how to do that via the console CLI directly… mhmm after enough googling on my phone I found this spiceworks thread with my answer:

vim-cmd hostsvc/net/info | grep “mac =”

I then checked this against the ESXi host that I saw the flipping flopping with, and sure enough they matched…  After doing a fresh install I noticed that the first 3 sections match the physical MAC, but in my DD deployed ones they retain the MAC of the system from which it was installed and those when I ran the command above, I could tell which ones were deployed via my method. This was further mentioned in this reddit thread by a commenter who goes by the name of sryan2K1:

“The physical NIC MACs are never used. vmk ports, along with VMs themselves will all use VMWare’s OUI as the first half of the address on the wire.”

OK, now maybe I can still salvage my deployment method by simply deleting and recreating the VMK after deployment, but I’d guess it best be done via the DCUI or direct console… I found one KB by VMware/Broadcom but it gave a 404 but Luckly there was a wayback machine link for it here.

Which states the following:

“During Initial Installation and DCUI, ESXi management interface (default vmk0) is created during installation.

The MAC address assigned will be the primary active physical NIC (pnic) associated.

If the associated vmnic is modified with the management interface vmkernel will once again assign MAC address of the associated physical NIC.

To create a VMkernel port and attach it to a portgroup on a Standard vSwitch, run these commands:

esxcli network ip interface add --interface-name=vmkX --portgroup-name=portgroup
esxcli network ip interface ipv4 set --interface-name=vmkX --ipv4=ipaddress --netmask=netmask --type=static"

Alternatively, you can also use esxcli to create the management interface vmkernel on the VDS.

Creation of the management interface with the ‘esxcli network’ will generate a VMware Universally Unique address instead of the pnic MAC address.

It is recommended to use the esxcli network IP interface method to create the management interface and not use DCUI.

Workarounds:               None

Additional Information:
Using DCUI to remove vmnic binding from management vmkernel or any modification will apply change at vSwitch level. Management interface is associated with propagating the change to any port groups within the vSwtich level.

Impact/Risks:                None.”

I”m assuming it means if you use the DCUI to reconfigure the MGMT interface settings the MAC will automatically be reconfigured to match what I found during initial clean install and mentioned in the reddit thread of using the first 3 sections to derive the MAC of the VMK.

But what if you don’t have any additional interfaces to use to make the section change in the DCUI to have that actually happen? cause what I’ve noticed changing the IP address and disabling IPv6 and rebooting did not change the VMK’s MAC address. Oh there’s in option in the DCUI “Reset Network Settings” within there there’s several options, I simply picked reset to factory defaults. Said success, checked the MAC via the first command stated above and bam the VMK nic changed to what it should be! Sweet my deployment method is still viable.

Hope this helps someone.

Posted on

Hole Punching a Linux VM on ESXi

I covered this in the past here:

Reclaim unused space from VMDK – Zewwy’s Info Tech Talks

But this time I wanna covered it a bit differently. Things I noticed:

  1. A proper VM with VMtools installed, and thin provisioned will automatically shrink the overall size being shown and used on disk on the ESXi browser.

Yet for some reason after I used the SCP method to move a VM from one host to another (frowned upon as it secretly converts it from thin to thick). Yet even after migrating to a new datastore specifying thin, it still show as full disk usage on the host.

I know is less checking the VM itself via it’s console/terminal whatever:

In my old blog post I mentioned “using DD” but not showing for stating how at all, Googling this I found this thread with an interesting answer:

“The /zero-Tag is actually a file name. The command just copies zeros from the virtual File /dev/zero (infinite number of zeros) into /mnt/hdb/zero until the disk is full, or some other error occurs.

This is why you have to remove the file /mnt/hdb/zero after that in order to regain the unused space.

However, a better way to fill free space with zeros (on ext2,3,4 file systems) is to use a tool called zerofree.”

Oooo zerofree?

Huh, something created (a tool) for exactly the task (job) at hand. Great!

Error, how classic, complained path is mounted RW, like yeah and?

Ughhh, google? virtualbox – zerofree on ubuntu 18.04 – Ask Ubuntu

Step 1) Reboot into ubuntu recovery console, hold down [Perfect ESC keystroke]

K how do I get into Advanced Grub boot? Holding Shift did nothing, if I mash ESC I get grub> (notices tiny flicker of Grub menu).. great I have to Mash ESC only once perfectly in a less then 1 second boot window… man give me a break… Once in I see advanced option as stated by the answer.

Step 2) advanced options -> recovery mode -> root console

Step 3) find the root directory

mount | grep "sda"

Step 4) run zerofree

echo "u" > /proc/sysrq-trigger
mount /dev/mapper / -o remount,ro
zerofree -v /dev/sda1

Step 5) reboot

Checking the ESXi host… what it went up 2 gigs, da fuq man…

Step 6) Compress Disk.

In my previous post this was to vMotion to datastore or use holepunch option which from what I can tell this the -K option, which I dont’ think can be used on a live VM, and a storage vMotion can’t be done. without vCenter. Since I’m temp working without a vCenter server, let’s try the holepunch option, this will require shutting down the VM, but since the zerofree required it anyway down time was already in play.

On the ESXi host:

[root@Art-ESXi:~] vmkfstools -K /vmfs/volumes/Art-Team/VM1/VM1.vmdk
vmfsDisk: 1, rdmDisk: 0, blockSize: 1048576
Hole Punching: 100% done.

Oh noooo I’ve tainted my results… checking the web UI the space has gone back done to stating 20 Gigs like the first snippet… but doing a du -h on the flat file shows it is only the 10 Gigs as expected it to be:

Well I don’t know what to make of this discrepancy…

Huh, I found this post of someone doing the exact same thing around the time I wrote my original post but simply used the command:

dd bs=1M count=8192 if=/dev/zero of=zero

I have no clue how output file of zero means the disk… guess you can try that too, then holepunch the VMDK just like I did.

There.. I shutdown the VM again and this time did my ol trick to vMotion a VM without vCenter, and after re-registering the vMotion (between two datastores) it finally showed up with the correct space in the UI. 🙂

Hope this post helps someone, it’s been covered many times in the past.

Posted on

How to vMotion a VM without vCenter WITHOUT Shared Storage

While I have covered this in the past here:
How to vMotion a VM without vCenter – Zewwy’s Info Tech Talks

This was using shared network storage between hosts…. what If you have no vCenter AND no shared storage? In my previous post I suggested to go check out VMware arena’s post but that just covers how to copy files from one host to another, and what I’ve noticed is while it does work if you let it complete, the vmdk is no longer thin and takes up the full space as specified by its defined size.  This is also mentioned as such in this serverfault thread “Solutions like rsync or scp will be rate-limited and have no knowledge of the content (e.g. sparse VMDK files, thin-provisioned volumes, etc.)”

So options provided there are:

  1. Export the VM as an OVF file, move to a local system, then reimport the OVF to your ESXi destination.I attempted this but on the host I could only export vmdk. While attempting to do so I got network issues (browser asking to download multiple files, but I must not have noticed and time and timed out? not sure). This also requires an intermediary device and double down/up on the network, I’m hoping for a way between hosts directly.
  2. Use vSphere and perform a host/storage migration.This post is how to do it without. Also note I attempted this but in my case I’m using my abomination ESXi host I created in my previous blog post, and vCenter fails the task with errors. (Again SCP succeeds but doesn’t retain thin provisioning). Not sure why SCP succeeds but vCenter fails seems to be more redundant to poor connection and keeps going, which happens when the WiFi NICs underload in those situations.
  3. Leverage one of Veeam’s free products to handle the ad hoc move.

I love Veeam, but in this case I’m limited in resources, lets see if we can do it via native ESXi here.

So that exhausts all those options. What else we got…

Move VMware ESXi VM to new datastore – preserve thin-provisioning – Server Fault

Oh someone figured out what I did in my intital post all the way back in 2013… wonder how I missed that one.. oh well, same answer as my initial post though required shared storage… moving on…

LOL no way… Willam Lam all the way back from over 14 years ago! Answering the question I had about compression of the files. and saying the OVF export is still the best option.. mhmmm…

I don’t want to stick to just scp, man did it suck getting to 97% done on a 60 Gig provisioned VMDK, that’s only taking up roughly 20 gigs, to have to not work cause I put my machine to sleep thinking it was a remote connection (SSH) to the machine and the machine is doing the actual transfer… just to wake my machine the next morning to have a “corrupt” vmdk that fails to boot or svmotion to get thin. I have machines with fast local storage but poor network, it’s a problem from back in the day with poor slow internet speeds. So what do we have? We got gzip and tar, what’s the diff?

In conclusion, GZIP is used to compress individual files, whereas TAR is used to combine numerous files and directories into a single archive. They are frequently used together to create compressed archive files, often with the “.tar.gz” extension.

Also answered here.

“If you come from a Windows background, you may be familiar with the zip and rar formats. These are archives of multiple files compressed together.

In Unix and Unix-like systems (like Ubuntu), archiving and compression are separate.

tar puts multiple files into a single (tar) file.
gzip compresses one file (only).
So, to get a compressed archive, you combine the two, first use tar or pax to get all files into a single file (archive.tar), then gzip it (archive.tar.gz).

If you have only one file, you need to compress (notes.txt): there’s no need for tar, so you just do gzip notes.txt which will result in notes.txt.gz. There are other types of compression, such as compress, bzip2 and xz which work in the same manner as gzip (apart from using different types of compression of course).”

OK, so from this it would seem like a lot of wasted I/O to create a tar file of the main VDMK flat file, but we could gain from compressing it. Let’s just do a test of simple compression and monitor the host performance while doing so.

Another thing I noticed that I didn’t seem to cover in my previous post in doing this trick was the -ctk.vmdk files. Which are change block tracking files, as noted from here:

“Version 3 added support for persistent changed block tracking (CBT), and is set when CBT is enabled for a virtual disk. This version first appeared in ESX/ESXi 4.0 and continues unchanged in recent ESXi releases. When CBT is enabled, the version number is incremented, and decremented when CBT is disabled. If you look at the .vmdk descriptor file for a version 3 virtual disk, you can see a pointer to its *-ctk.vmdk ancillary file. For example: version=3

# Change Tracking File
changeTrackPath=”Windows-2008R2x64-2-ctk.vmdk”
The changeTrackPath setting references a file that describes changed areas on the virtual disk.
If you want to back up the changed area information, then your software should copy the *-ctk.vmdk file and preserve the “Change Tracking File” line in the .vmdk descriptor file. If you do not want to back up the changed area information, then you can discard the ancillary file, remove the “Change Tracking File” line, read the VMDK file data as if it were version 1, and roll back the version number on restore.

I’ll have to consider this when running some of the commands coming up. Now we still don’t know how much, if any, space we’ll save from compression alone and the time it’ll take to create the compressed file… from my research I found this resource pretty helpful:

Which Linux/UNIX compression algorithm is best? (privex.io)

Since we want to keep it native doing quick tests via the command line shows ESXi to have both gzip and xz but not lx4 or lbzip2, which kind of sucks as they showed to have the best performance in terms of compression speeds… as quoted by the article “As mentioned at the start of the article, every compression algorithm/tool has it’s tradeoffs, and xz’s high compression is paid for by very slow decompression, while lz4 decompresses even faster than it compressed.” Which is exactly what I want to see in the end result, if we save no space, then the process will burn I/O and expected life of the drive being used or pretty much zero gains.

Highest overall compression ratio: XZ If we gonna do this this is what we want, but how long it takes and how much resources (CPU cycles, and thus overall WATTS) trade off will come into question (though I’m not actually taking measurements and doing calculations, I’m looking at it at points and time and making assumed guessed at overall returns).

Time to find out what we can get from this (I’m so glad I looked up xz examples cause it def is not intuitive (no input then output parameters, read this to know what I mean) :

xz -c /vmfs/volumes/SourceDatastore/VM/vm-flat.vmdk > /vmfs/volumes/TargetDatastore/whereever/vmvmdk.xz

Mhmmm no progress… crap didn’t read far enough along and I should have specified the -v flag, not sure why that wouldn’t be defaulted, having no response of the console kind of sucks… but checking the host resources via the web GUI shows CPU being used, and write speed….. sad….

CPU usage:

and Disk I/O:

Yeah… maybe 4 MB/s and this is against a SSD storage on a SATA bus, there’s no way the storage drive or the controller is at fault here… this is not going to be worth it…

Kill command, check compressed file less than 300 MB in size, OI, that def not going to pay off here…

I decided to try taring everyting into one file without compression hoping to simply get it to to one file roughly 20gigs in size with max I/O. As mentioned here:

“When I try the same without compression, then I seem to get the full speed of my drive. ”

However to my dismay (maybe it ripped the SSDs cache too hard?) I unno I’d get I/O error, even though the charts showed insane throughput, I decided to switch to another datastore a spindle drive on the ESXi host and you can see the performance just sucks compared to the SSD itself.

Which now again stuck waiting cause instead of amazing throughput its stuck going only 20 MB/s apparently… uggghhhh.

To add to this frustration, I figured I’d try the OVF export option again, but I guess cause the tar operation has a read on the file, I’m assume a file lock, when attempting the OVF export it just spits an web response “File Not Found”. So, I can’t even have a race knowing full well the SSD could read much faster than what it’s currently operating at. I don’t really know what the bottleneck is at this point…

Even at this rate it’s feeling almost pointless, but man just to keep a vmdk thin, why, oh WHY SCP can’t you just copy the file as the size it is… mhmmm there has to be a way other than all this crap….

I don’t think this guy had any idea he went from thin too thick on the VM….

I thought about SSHFS, but it’s not available on ESXi server….

Forgot about Willams project GhettoVCB Great if I actually wanted more of a backup solution… considered for future blog, but over kill to just move a VM.

The deeper I go here the more the simply export to OVF template and import is seeming reaaaaaalll appeasing.

Awww man this tar operation looks like its takin more size then the source. doing a du -h on the source shows 19.7 Gigs… tar file has now surpassed 19.8 Gigs in size… with no sign of slowing down or stopping lol. Fuck man I think tar is also completely unaware of thin disk and I think it’ll make the whole tar file what ever the provisioned size was (aka thick). Shiiiiiiiiiiiiit!

Trying the Export VM option looked so promising,

until the usual like always… ERROR!!

FFS man!!! Can’t you just copy the files via SSH between hosts? Yeah but only if you’re willing to copy the whole disk and if you’re lucky holepunch it back to thin at the destination… can’t you do it with the actual size on disk… NO!

Try the basic answer on almost all posts about this, just export as template and import… Browser download ERROR… like Fuck!!!

Firefox… nope same problem… Fuck…. Google what ya got for me? well seems like almost the same as my initial move of using SCP but use WinSCP via my client machine and uttering in a middle man in the process, but I guess using the web interface  to download/upload was already a man in the middle process anyway… fine let’s see if I can do that… my gawd is this ever getting ridiculous… what a joke… Export VM from ESXi embedded host client Failed – Network Error or network interruption – Server Fault

And of course when I connect via Win SCP it see the hard drive as being 60 Gigs, so even though trafser speed are good to taking way more space then needs and thus waste data over the bus… FUCK MAN!!!!!

If only there was a way to change that, oh wait there is, I blogged about it before here: How to Shrink a VMDK – Zewwy’s Info Tech Talks

OK Make a clone just to be safe (you should always have real backups but this will do. and amazing this operation on the SSD was fast and didn’t fail.

Woo almost 300 MB/s and finished in under 4 minutes. Now let’s edit the size.

Well I tried the edit size, but only after doing a vmkfstools convertion of the vmdk would it show the new size in WinSCP, even then transferred the files and it was still corrupted in the end..

ESXi 6.5 standalone host help export big VM ? | MangoLassi

Mhmmm another link to Willams site, covering the exact same thing, but this time using a tool ovftool….

and wait a second… He also said there’s a way to use the ovftool on the ESXi server itself, in this post here….. mhmmmm If I install the Linux OVF tool on the ESXi host, I should be able to transfer the VM while keeping the thin disk all “native” on ESXi… close enough anyway…

Step 1) Download the OFV tool, Linux Zip

Step 2) Upload Zip file via Web GUI to Datastore. (Source ESXi)

Step 3) unzip tool (unzip ovfool.zip), then delete zip.

Step 4) Open outbound 443 on Source ESXi server. Otherwise you get error on tool.

Step 5) run command to clone VM, get error that managed by ESXi host.

Step 6) remove hosts from ESXi and run command again… fail cause network error (Much like OVF export error, seem that happens over port 443/HTTPS)

Man Fuck I can’t fucking win here!!!

I think I’m gonna have to do it the old fashioned way… doing it via “seeding”. plug in a Drive into the source ESXi, and physically move it to the target.

Tooo beeeee continued……

I grabbed the OVFtool for windows (the machine I was doing all the mgmt work on anyway, yet it too failed with network issues.

I decided to reboot the mgmt services on the host:

Then gave it one last shot…

Holy efff man the first ever success yet… don’t know if this would havefixed all my other issues, the export failing for https, and all the others? And the resulting OVA was only about 8 Gigs. Time to see if I can deploy it now on the target host.

I deployed the OVA to the target via the WebGUI without issue.

I also tested the ESXi webGUI export VM option and this time it also succeeded without failure, checking the host resources CPU is fairly high on both ovftool export or the webGUI export option. Using esxtop showed hostd process taking up most of the CPU usage during the processes. Further making me believe restarting that service is what fixed my issues…

*Dec 2025 update*

Everything I mentioned above it doing a move of a VM between two different ESXi hosts, if you simply want to move a VM between datastores on the ESXi host itself then.

      • ESXi Web GUI (Datastore Browser): Power off VM → Unregister VM → Copy/move VM folder to target datastore → Re‑register VM.
      • CLI/Shell scripts: Use vmkfstools or scp to copy VMDK files, then update the VMX file and re‑register.

A typical shell script might:

  1. Power off the VM.
  2. Use vmkfstools -i to clone the VMDK to the new datastore. 
    vmkfstools -i /vmfs/volumes/source_datastore/vm/vm.vmdk \
           /vmfs/volumes/target_datastore/vm/vm.vmdk -d thin
  3. Copy the VMX and other config files with cp or scp.
  4. Register the VM on the new datastore using the ESXi host client.

This ensures the VM is properly cloned/moved while preserving thin/thick provisioning.

⚠️ Risks & Considerations

  • Downtime: Without Storage vMotion, the VM must be powered off.
  • Provisioning changes: Some copy methods may convert thin disks to thick unless you specify -d thin.
  • Networking/Devices: Detach virtual CDs/USBs before unregistering, or the VM may fail to boot.
  • Unsupported scripts: Community scripts work but aren’t officially supported by VMware. Always test in a lab before production.
  • Risks specifically with doing a move using the WebUI
    When you use the “Move” option in the datastore browser, ESXi is essentially doing a cold file copy at the hypervisor level. A few things explain what you’re seeing:

    • Performance:
      • The copy runs through the host’s management plane, not optimized like Storage vMotion.
      • It’s single‑threaded and doesn’t leverage advanced I/O scheduling, so speeds can be much slower than expected.
      • If your datastore is local spinning disks or consumer‑grade SSDs, throughput can be especially poor compared to enterprise storage.
    • Resource impact:
      • You’ll see spikes in CPU and memory usage because the host is handling the copy itself.
      • Other VMs can feel sluggish if the datastore I/O queue is saturated.
    • No cancel option:
      • Unfortunately, once the task is launched in the Host Client, there’s no supported way to stop it mid‑stream.
      • Killing the process or rebooting the host risks corrupting the VM files.
      • VMware only provides cancel/rollback with Storage vMotion (via vCenter), not with manual datastore moves.

    So in short use vmkfstools -i from the ESXi shell to clone/move disks. It’s more efficient. I learnt this hard way by trying it out, and watching the system resources to see it flying at a blistering 20-30 MB/s, so now my ESXi is in a degraded state while I have to wait for the process to complete which will take hours at this rate. Oiii…

Legal Information Proudly powered by WordPress