Veeam 13

Veeam 13

It’s out now, but seem many admins are not pleased with how things are going. Due to how bloaty the software has become. See here for the thread which discuses a feature request to select which components should be installed with the application.

Feature Request: Select Components to Install/Upgrade – R&D Forums

I have to agree with the sentiment here, V8 was literally only 800 MBs in size, compact and efficient. V13 has now balloon to over 18 Gigs, which is absolutely mind boggling. Now with components you can’t choose to install or not leaves a larger attack surface that you have to audit and compare against. Along with additional storage space requirements, memory requirements, all for features or services in which you may not even need. I recommend you read the thread, and if you, yourself are a sysadmin having to install and manage Veeam server instance, leave a like in hopes we can bring back some sanity to an otherwise great product.

In this post I’ll be upgrading my home lab instance from Veeam12 to 13, and I guess we’ll see the lack of component selection along the way.

Step 1: Source Software Acquisition

I got Veeam 13 from here: Veeam Software for Enterprise however, note this is a regwalled link and they want your email and phone number for some odd reason… so use whatever tactics you have to, to keep your information private and secure.

Step 2: Mount and Install

How you mount is up to your system, in my case I attached the ISO to my Veeam VM using VMRC, then ran setup.exe

and thennnnn….

and then….

and then…..

no choice, and then….

and thenn….

so dumb… ok.. check off both and then…

and thennnnn….

Let’s get the first one out of the way, it needs over 50 Gigs of free space, I know insane, so lets see if we can get it that… expand the HDD and bam…

Veeam v13 introduces stricter OS‑level requirements and drops support for older Windows versions for any feature that could use AAP—even if you don’t currently use it.

From the v13 system‑requirements updates, Veeam is removing or deprecating support for older platforms to simplify code and improve security. This includes Windows 10/11 builds that no longer meet the updated criteria.

Because AAP interacts deeply with the guest OS (VSS, credentials, application services), Veeam checks all protected VMs for compatibility during the upgrade, not just those with AAP enabled. If any VM is running a Windows build that falls into the “deprecated or limited support” category, Veeam surfaces a warning.

After I removed all Backup jobs pointing to older target VMs (even though non of the had AAP enabled), That warning disappeared. Even with all jobs removed, the last two remained.

On Veeam Backup Servers where Veeam Backup & Replication was initially installed with an older version and has been upgraded over the years, the initial Veeam Backup Server Certificate may lack the “Basic Constraints” extension, which can cause issues with Platform Plug-Ins.

Issue Validation

You can view the current Backup server certificate in Main Menu > Options > Security:

The Veeam Backup & Replication Console is shown with the maun menu active and the "Options" menu item is highlighted.
The "Options" window is open to the "Security" tab. Under "Backup server certificate," a self-signed certificate labeled "CN=Veeam Backup Server Certificate" is displayed. There is an "Install..." link on the right side, allowing the user to install a different certificate.
Screenshot of Veeam Backup Server certificate with Basic Constraint field missing with a red faint X over the image to indicate wrong.

Basic Constraint Missing
Screenshot of Veeam Backup Server certificate with the Basic Constraint field present with a green faint checkmark over the image to indicate correct.

Basic Constraint Present

Resolution

Note: If your environment does not use the default self-signed certificate, you must ensure that the CA-signed certificate you provide to Veeam Backup & Replication contains the Basic Constraints extension, and Subject Type = CA must be set within that extension.

 

For deployments using the self-signed Veeam Backup Service Certificate, a new one must be generated:

  1. From the Main Menu, click Options
The Veeam Backup & Replication Console is shown with the maun menu active and the "Options" menu item is highlighted.
  1. In the Options dialog box, select the Security tab.
  2. On the Security tab, click “Install…“in the “Backup server certificate” section.
The "Options" window is open to the "Security" tab. Under "Backup server certificate," a self-signed certificate labeled "CN=Veeam Backup Server Certificate" is displayed. There is an "Install..." link on the right side, allowing the user to install a different certificate.
  1. In the Certificate creation wizard, select the option for Generate a new certificate, and click Next.
The "Manage Certificate" wizard is open to the "Certificate Type" step. Options are listed for SSL certificate selection: "Keep the existing certificate," "Generate a new certificate" (selected), "Select an existing certificate from the certificate store," and "Import certificate from a file." A description for each choice is provided. The "Next >" button is highlighted at the bottom.
  1. On the Generate Certificate step, leave the friendly name as Veeam Backup Server Certificate, and click Next.
The "Manage Certificate" wizard is open to the "Generate Certificate" step. The user is prompted to enter a friendly name for the new self-signed certificate, with the field set to "Veeam Backup Server Certificate." A note below indicates the certificate will not originate from a trusted certification authority (CA). "Next >" and "Cancel" buttons are visible at the bottom.
  1. On the Summary steps, click Finish.
The "Manage Certificate" wizard is open to the "Summary" step. Certificate details are shown, including name ("Veeam Backup Server Certificate"), issued to, issued by , expiration date, thumbprint, and serial number. The "Finish" button at the bottom right is highlighted, allowing the user to complete the certificate generation process.
Retry and.. oh look it’s gone now…

Option A — Remove old plug‑ins from “Backup Infrastructure → Plug‑ins”

If any plug‑ins appear there, remove them.

Option B — Clean stale entries from the configuration database

This requires Veeam Support. They run a script to remove:

  • orphaned plug‑in records
  • deprecated feature flags
  • old certificates
  • legacy hypervisor entries

This is the only guaranteed fix.

Option C — Ignore the warning

This is acceptable because:

  • It does not block the upgrade
  • It does not affect backup/restore
  • It only indicates that V13 will delete unused legacy components
Option 3 sounds good to me.. NEXT!
As you can see, no options for picking anything. and it blew up on me, the service fails to start.
Checking the logs says it’s doesn’t like my new ESXi host I added to my cluster… but no reason why…

Datastore out of Space

So, when I blogged about updating or migrating Veeam, Migrating/Restoring Veeam – Zewwy’s Info Tech Talks I forgot my snapshot… whoops…

I just caught it before I wanted to upgrade to Veeam 13, I hadn’t even connected the disk when I went to verify the disk and datastore, sure enough the datastore was right on the cusp of running out of space. 1.5 Gigs remaining.

I figured leave everything as is and just migrate it to a larger datastore… but what I failed to take into account is that the VM was still running, and any disk writes are still growing the delta file, so in the middle of the move the vMotion stopped stating. “The operation cannot be allowed at the current time because the virtual machine has a question pending: ‘msg.hbacommon.outofspace: There is no more space for virtual disk ‘Veeam_19-000001.vmdk’. You might be able to continue this session by free…” I clicked cancel which simply failed the vMotion job…

During this operation, if the machine had been powered off the transfer would have completed in a shorter amount of time (compared to the more random I/O of the snapshot cleanup) and the process from a source SSD to a target SSD was providing throughput of 300MB/s for the first 2 minutes of the process before dropping to roughly 160 MB/s just before the failure occurred. With roughly 12ms delay response from the target SSD and 35ms delayed response from the target.

I decided to Hard Stop the VM, then selected to delete all snapshots. I don’t know how vCenter never warned me of a large Snapshot size. With the large snapshot size, and a cheap, CHEAP SSD (TeamSSD  – TEAM_T253256GB), and the fact that the backup storage (which you normally shouldn’t do this in production but I do have it setup this way in my lab) that the backup VMDK on a slow spindle 2TB drive was also part of the snapshot, the snapshot deletion took FOREVER! over 3 hours.

Then shutting it shown was bad cause I forgot that I have a vTPM on the machine so after trying to vmotion it again after the snapshot removals… it errored out with “Permission to perform this operation was denied. NoPermission.message.format”, then ”
Unlock encrypted virtual machine
Veeam
The object or item referred to could not be found. The provider ‘NativeKP’ does not exist. Failed to decrypt the key”

I added both known NPKs I backed up but the error then said ”
Reconfigure virtual machine
Veeam
An encryption key is required.”

Eventually doing the following:

  • SSH into ESXi.
  • Go to the VM folder:
    cd /vmfs/volumes/<datastore>/<vmname>
    
  • Make a backup:
    cp <vmname>.vmx <vmname>.vmx.bak
    
  • Edit the VMX:
    vi <vmname>.vmx
    

    Remove any lines containing:

  • encryption.keySafe
  • encryption.data
  • migrate.encryptionMode = “required”
  • ftcpt.ftEncryptionMode = “ftEncryptionRequired”

Then re-registered the VM.

Then vmotion it… “Relocate virtual machine
Veeam
Completed
Migrating Virtual Machine active state”

Oh finally, man I hate adding vTPMs to VM on ESXi hosts.. it always seems to come back to bite me. and my datastore usage is finally back under control and no other VMs were affected… what a roller coaster.

Force Kill a Hyper-V VM

So I tried to Turn Off a Hyper-v VM using Hyper-v Manager tool, and it would just error stating it couldn’t do it in the state of the VM. It was in a live linux shutdown state and not doing anything.

So I opened PowerShell as an admin and ran this:

Stop-Process (Get-Process vmwp | ?{(Get-CimInstance Win32_Process -Filter "ProcessId=$($_.Id)").CommandLine -match (Get-VM 'Ubi').id.guid}).Id -Force

Change the VM name in the Get-VM command. That’s it I noticed it blipped in the Hyper-V manager but wasn’t shutdown, I simply selected turn off and this time the VM finally turned off. Hope this one-liner helps someone.

Strong(er) authentication required

Strong(er) authentication required

Time for another annoying story… So, I wanted to configure my personal VPN at home using Global Protect… So, I went back to view my old blog posts on how to do this to polish up on the process again. And low and behold on following Step one, authentication I already hit a new road block. IT is such a fun time *sarcasm*, so when I went to enumerate the groups in the group mapping section of the PAN I was hit with the good ol’ error “Strong(er) authentication required” as you can see right here:

Looking this up online I found a Reddit post linking to a PAN KB. Which states this happens when you have LDAP hardening enabled, at least for older Windows Server (2008 referenced), when I wrote my old blog post I was running 2016, and I had updated it to 2022. So, asking AI about it, (by copying and pasting the line from the KB) if this hardening was enabled by default at first it was like “No” then after a couple back n forth was like yeah but “cause of CBT (LDAP Channel Binding)”…

Classic pedantic AI… So… what are my options?

Option 1) Disable CBT LDAP Channel Binding

The “not recommended option”

Registry Path

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters

Value Name

LdapEnforceChannelBinding

Value Type

REG_DWORD

Possible Values

  • 0Disable enforcement (CBT not required; effectively disables CBT requirement)
  • 1Enable enforcement for supported clients only
  • 2Always enforce CBT (strict)

To disable CBT enforcement, set:

LdapEnforceChannelBinding = 0
Registry Path
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters
Value Type
(REG_DWORD)
Value Name
LDAPServerIntegrity
0 = None 1 = Negotiate 2 = Require Signing
Then, reboot… and…
Problem solved, *dusts hands*. Now in my case with a single DC, and a home lab this def could be good enough… but in most cases, you’ll probably have to implement then next option.

Option 2) LDAPS

This is the more annoying but secure option.

1. Plan the certificate setup

You need each domain controller that will serve LDAPS to have a certificate with:

  • Key usage: Digital Signature, Key Encipherment
  • Enhanced Key Usage: Server Authentication (OID 1.3.6.1.5.5.7.3.1)
  • Subject / SAN: Includes the DC’s FQDN (e.g. dc01.contoso.com)

You can use:

  • Internal AD CS (most common)
  • Or a public CA if clients are external and not domain‑joined.

2. Install Certificate Authority (if you don’t already have one)

Setup Offline Root CA (Part 1) – Zewwy’s Info Tech Talks

Remove Existing Enterprise Root CA (Part 2) – Zewwy’s Info Tech Talks

Setup Subordinate CA (Part 3) – Zewwy’s Info Tech Talks

Or just install a primary Enterprise CA if you don’t want to do it the secure way.

3. Create a certificate template for domain controllers (optional but recommended)

On the CA:

  1. Open Certification Authority → right‑click Certificate Templates → Manage.
  2. Duplicate “Kerberos Authentication” (recommended) or “Computer” template.
  3. On the new template:
    • General: Give it a name like “Domain Controller LDAPS”.
    • Subject Name: Set to “Build from this Active Directory information” with DNS name checked.
    • Extensions: Confirm Server Authentication is present in EKU.
      I removed Smart card, and Client Auth.
  4. Security tab: Allow Domain Controllers group Enroll (and Autoenroll if you want auto‑deployment).
  5. Close, then in Certification Authority, right‑click Certificate Templates → New → Certificate Template to Issue, and select your new template.

4. Enroll the certificate on the domain controller

On each DC:

  1. Open mmc.exeFile → Add/Remove Snap-in → add Certificates for Computer account.
  2. Navigate to Personal → Certificates.
  3. Right‑click Personal → All Tasks → Request New Certificate.
  4. Choose your “Domain Controller LDAPS” (or equivalent) template → Enroll.
  5. Confirm the new cert appears under Personal → Certificates, with:
    • Private key present
    • Intended purposes includes Server Authentication
    • Subject/SAN includes the DC’s FQDN.

*Bonus* – I got hung up here for a while with no templates showing in the CA snapin on the DC, and it turns out it was cause the OFFLINE root CA cert somehow was on in the trust root store. I’m have no idea how that happened, but yeah… shrug….

5. Verify LDAPS is active on port 636

On the DC:

  1. Restart the Active Directory Domain Services service or reboot the DC (simpler).
  2. Use ldp.exe (built‑in tool):
    • Run ldp.exe.
    • Connection → Connect…
    • Server: DC FQDN, Port: 636, check SSLOK.
    • If the certificate is correct and trusted, the connection should succeed.

6. Import the Offline Root and SubCA Certs into PAN Firewall

Import the certificates as Base64. Then edit the LDAP profile for port636 and check off SSL. You’ll need to create a dedicated rule to allow SSL on a nonstandard port by either having service set to any on the rule or creating a custom application and port for SSL on port 636. Then testing again…

Hope this helps someone.

vCenter syslog/rsyslog

So, in my previous post I discussed troubleshooting the wd in wdpath already exists log error. However, the root issue there may have been determined and resolved… but the question arises… do we need to ship that much logs?

What are all these log files for?

High‑Level Overview

Every file listed is part of vCenter Server’s syslog configuration. Each vmware-services-*.conf file tells the syslog collector which logs belong to which internal service. These logs fall into categories like:

  • UI / Client logs
  • SSO & Identity logs
  • vCenter core services (vpxd, vmon, vapi, etc.)
  • Database logs (Postgres, vtsdb)
  • vSAN Health
  • Networking (rhttpproxy, netdumper)
  • Appliance management (applmgmt, cloudvm)

Below is a readable breakdown of what each group of log files does.

📘 Detailed Breakdown by Service

🎨 vSphere UI / HTML5 Client

Files under /storage/log/vmware/vsphere-ui/logs/

These logs cover everything related to the vSphere Client (the HTML5 UI):

Log Purpose
vsphere_client_virgo.log Main UI application server (Virgo) log
changelog.log UI plugin/component change tracking
dataservice.log Backend data service used by UI
apigw.log API gateway for UI requests
equinox.log OSGi framework logs
eventlog.log UI event processing
httpRequest.log HTTP request logs
opid.log Operation IDs for tracing UI actions
performanceAudit.log UI performance metrics
plugin-medic.log Plugin health & validation
threadmonitor.log Thread health monitoring
threadpools.log Thread pool usage
vspheremessaging.log Messaging subsystem
vsphere-ui-rpm.log UI package/runtime logs
vsphere-ui-runtime* Runtime stdout/stderr
access/localhost_access_log.txt Web access logs
vsphere-ui-gc* Java garbage collection

🔐 SSO / Identity Services

Files under /storage/log/vmware/sso/, /storage/log/vmware/vmdir/, /storage/log/vmware/vmafd/

These logs relate to authentication, identity, certificates, and tokens:

Log Purpose
activedirectoryservice.log AD integration
lookupsvc-init.log Lookup service initialization
openidconnect.log OIDC authentication
ssoAdminServer.log SSO admin operations
svcaccountmgmt.log Service account management
tokenservice.log Token issuance
sts-health-status.log.* STS health
sts-runtime.log.* STS runtime
gclogFile.*.current JVM GC
tomcat/localhost_access.log SSO Tomcat access
vmdir/*.log Directory service (LDAP-like)
vmafd/*.log Authentication framework

🧩 vCenter Core Services

vpxd (vCenter Server daemon)

These are commented out in your file, but normally include:

  • vpxd.log — main vCenter service log
  • vpxd-profiler-*.log — performance profiling

vmon

Manages service lifecycle:

  • vmon.log — service manager log
  • vmon-vapi-provider-0.log — VAPI provider logs

vapi

API endpoint logs:

  • endpoint.log — main API endpoint
  • endpoint-access.log — API access logs
  • jetty.log — Jetty web server
  • vcentershim.log — vCenter API shim
  • vmodl2swagger.log — API schema conversion
  • vmware-vapi-endpoint-gc.log.* — GC logs
  • vmware-vapi-endpoint.std* — stdout/stderr

📊 Analytics / Telemetry

  • analytics.log — analytics service
  • analytics-runtime.log.std* — runtime logs

🧱 vSAN Health

  • vmware-vsan-health-service.log — main vSAN health service
  • vmware-vsan-health-runtime.log.* — runtime logs
  • vsanvcmgmtd-*.log — vSAN cluster mgmt

🗄️ Database Services

Postgres (vPostgres)

  • serverlog.std* — main DB log
  • postgresql-*.log — DB engine

logsvtsdb

  • vtsdb-runtime.log.std*
  • runtime  postgresql-*.log — DB logs

Postgres Archiver

  • pg_archiver.log.std* — WAL archiving

🔧 Lifecycle Manager (vLCM)

  • lcm_common.log — core LCM operations
  • task_executor.log — task execution
  • twisted_server.log — Python-based server
  • vlcm_db.log — LCM database
  • vlcm-runtime.log.* — runtime logs

🧪 vSphere ESX Agent Manager (EAM)

  • eam.log — main EAM service
  • web/*.log — Tomcat logs
  • jvm.log.* — JVM logs
  • eam_firstboot.py*.log — first boot

The EAM log refers to the log files generated by the VMware ESX Agent Manager (EAM) service.
EAM is a core vCenter component responsible for deploying and managing ESX agents, which are small helper VMs or services used by features such as:

vSphere Lifecycle Manager (vLCM)
vSphere Storage I/O Control
vSphere Network I/O Control
vSAN / vCLS agents
Third‑party extensions that deploy agents to ESXi hosts

Search results confirm that EAM logs live in /var/log/vmware/eam/ and are used for diagnostics and troubleshooting.

📘 What EAM logs contain

1. eam.log — Main service log
This is the primary log file for the ESX Agent Manager.

It records:

Service startup and shutdown
Agent deployment and lifecycle events
Communication with vCenter and ESXi hosts
Plugin/extension registration
Errors when EAM cannot deploy or manage agents
Failures related to vCLS or vSAN agent VMs
Search results show examples of EAM startup failures and configuration errors logged in eam.log.

2. Web access logs (web/localhost_access.log)
These track:

HTTP requests to the EAM web service
MOB (Managed Object Browser) access
API calls from vCenter or extensions
Mentioned in STIG guidance for EAM logging.

3. JVM logs (jvm.log, wrapper.log)
These capture:

Java runtime errors
Memory issues
Crashes or fatal exceptions
Examples of JVM startup failures appear in VMware KB articles

🌐 Networking & Proxy Services

rhttpproxy

  • rhttpproxy-*.log — reverse proxy logs

netdumper

  • netdumer.log — ESXi dump collector
  • webserver.log — web interface

🧰 Content Library

  • cls.log — content library service

📈 Perfcharts

stats.log — performance charts

  • localhost_access_log.txt — access logs
  • vmware-perfcharts-gc.log.* — GC logs
  • vmware-perfcharts-runtime.log.std* — runtime

🧭 Lookup Service

  • lookupserver-default.log — main lookup service
  • lookupServer.log — operations
  • lookupsvc_stream.log.std* — runtime
  • vmware-lookupservice-perf.log — performance
  • vmware-lookupsvc-gc.log.* — GC

🧩 vpxd-svcs (vCenter Support Services)

  • vpxd-svcs.log — main
  • authz-event.log — authorization events
  • startup-error.log — startup failures
  • vpxd-svcs-access*.log — access logs
  • vpxd-svcs-runtime.log.* — runtime
  • perf.log — performance

 

🛡️ Trust Management

  • trustmanagement-runtime.log.std* — runtime
  • trustmanagement-svcs.log — trust services
  • vmware-trustmanagement-gc.log.* — GC

🔐 Trust Management Service — Tight Summary

Trustmanagement is a core vCenter service that maintains the trust relationships between all internal components. It ensures that certificates, tokens, and service‑to‑service authentication are valid and secure.

What it handles:

Certificate chain validation
Trust checks between vCenter services
Token verification (STS/SSO)
Security posture and compliance signals

What its logs show:

Certificate or trust failures
Service registration/authentication issues
Token validation errors
Startup/shutdown and internal health

Why it matters:
If trustmanagement breaks, you may see:

vCenter login failures
STS token errors
Certificate replacement problems
Services stuck in “Not Running”
Upgrade failures due to trust issues

What it does NOT do:

Track user logins
Record user actions
Provide audit logs

It’s purely about internal vCenter security plumbing, not end‑user activity.

🧩 Pod Service

  • pod-service.log — pod mgmt
  • pod-console.log — console
  • pod-startup.log — startup
  • pod-install*.log — install
  • pod-update*.log — updates

🧰 Appliance Management (VAMI)

Files under /storage/log/vmware/applmgmt/ covers:

  • VAMI web UI
  • Backup/restore
  • Firewall reload
  • Stats monitor
  • PNID changes
  • Lighttpd access/error logs

🔍 What the Applmgmt Upgrade Service does

It manages:

VCSA upgrade workflows
Patch installation
Pre‑upgrade checks
Post‑upgrade cleanup
Version validation
Upgrade‑related service orchestration

It’s the engine behind the VAMI (port 5480) upgrade process.

📁 What logs this syslog config points to

The file typically references logs such as:

applmgmt-upgrade.log — main upgrade workflow log
applmgmt-upgrade-runtime.log.std* — stdout/stderr
applmgmt-upgrade-gc.log.* — Java garbage collection

These logs capture:

Upgrade steps and progress
Validation checks
Errors during patching or upgrading
Service restarts triggered by upgrades
JVM runtime behavior

🧭 When these logs matter

You check these logs when:

A VCSA upgrade fails
Patching stops mid‑process
Pre‑upgrade checks report errors
The VAMI UI shows upgrade failures
Services don’t come back after an upgrade

🔐 Certificate Management

  • certificatemanagement-runtime.log.std *
  •  certificatemanagement-svcs.log *
  • vmware-certificatemanagement-gc.log.*

🧩 SCA = Secure Configuration Assistant

A vCenter subsystem responsible for security posture checks, certificate validation, and secure configuration enforcement.

It’s part of the broader vCenter security framework that also includes:

Certificate Management (certmgmt)
VMCA (VMware Certificate Authority)
STS (Security Token Service)
PSC identity services (in older versions)

🧩 What SCA actually does

🛡️ 1. Security posture checks

It evaluates whether vCenter components are configured securely, including:

TLS/SSL settings
Certificate validity
Service trust relationships
Cryptographic compliance

🔏 2. Certificate and trust validation

It works closely with:

VMCA
certmgmt
STS

to ensure that:

Certificates are valid
Trust chains are intact
Services can authenticate to each other

🧭 3. Compliance reporting

SCA feeds data into:

vCenter security health checks
vSphere Client “Security” view
Some VAMI security status pages

📁 Where you see SCA in logs

You’ll typically find SCA logs under:

Code
/storage/log/vmware/sca/
Common files include:

sca.log — main service log
sca-runtime.log.std* — stdout/stderr
sca-gc.log.* — Java garbage collection

These logs show:

Security scan results
Certificate validation failures
Trust chain issues
Service authentication problems
Startup/shutdown of the SCA service

🧭 When SCA logs matter

You check SCA logs when:

vCenter shows certificate warnings
Services fail to register due to trust issues
You see “vCenter is not secure” alerts
STS token problems appear

vCenter upgrades fail due to certificate or trust chain issues

🛡️ File Integrity Service — Tight Summary

The fileintegrity syslog config points to logs generated by vCenter’s File Integrity Service, which monitors critical system files for unauthorized or unexpected changes.

What it does

  • Checks hashes of important vCenter files
  • Detects tampering, corruption, or unexpected modifications
  • Flags security‑relevant integrity issues

What its logs contain

  • Integrity scan results
  • File change alerts
  • Hash mismatches
  • Service errors and startup info
  • JVM runtime and memory behavior (via runtime + GC logs)

Why it matters

  • Helps detect compromise or corruption of vCenter
  • Useful for SOC teams as security telemetry
  • Not related to user activity or audit logging

🧵 threadmonitor.log — Tight Summary

threadmonitor.log is part of the vSphere UI service (the HTML5 vSphere Client).
This log tracks thread health and performance inside the UI service’s Java application.
It’s essentially a watchdog that monitors whether internal threads are running normally or getting stuck.

🔍 What it records

Thread stalls or deadlocks
Long‑running or hung operations
UI service performance issues
Thread pool exhaustion
Java exceptions related to thread execution
Internal timing or responsiveness problems

It’s a diagnostic log for the vsphere-ui backend, not for user activity.

🧭 When this log matters

You check threadmonitor.log when:

The vSphere Client is slow or unresponsive
Pages hang or fail to load
UI freezes during tasks
You suspect backend thread starvation
The vsphere-ui service crashes or restarts

It’s especially useful when troubleshooting UI performance issues.

Disable Unwanted Logs

This is obviously a balancing act between what you feel is needed to be forwarded, and what is not required depending on destination logging capabilities. Note comenting out these lines only stops the forwarding of the logs to the syslog destination, it does not stop the local logging of these services. That is out of scope of this blog post.

/etc/vmware-syslog/vmware-services-vsphere-ui.conf

Disable all except:
File=”/storage/log/vmware/vsphere-ui/logs/access/localhost_access_log.txt”

/etc/vmware-syslog/vmware-services-vmcad.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vmcad.conf

/etc/vmware-syslog/vmware-services-sso-services.conf

just these:

/storage/log/vmware/sso/sts-health-status.log.* /storage/log/vmware/sso/sts-runtime.log.* /storage/log/vmware/sso/gclogFile.*.current /storage/log/vmware/sso/tomcat/localhost_access.log

/etc/vmware-syslog/vmware-services-vsm.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vsm.conf

/etc/vmware-syslog/vmware-services-vpxd.conf

KEEP

File="/storage/log/vmware/vpxd/vpxd-*.log"

There are a lot of vpxLRO logs generated by this, but there appears to no other granual controls at the source level (these rsyslog imfile config), so not sure about filtering these outside of transforms at the other syslog/rsyslog/lostash service that is receiving these logs.

DISABLE

File="/storage/log/vmware/vpxd/vpxd-profiler-*.log"

/etc/vmware-syslog/vmware-services-infraprofile-syslog.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-infraprofile-syslog.conf

/etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf

/etc/vmware-syslog/vmware-services-vsan-health.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vsan-health.conf

/etc/vmware-syslog/vmware-services-envoy.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-envoy.conf

/etc/vmware-syslog/vmware-services-sps.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-sps.conf

/etc/vmware-syslog/vmware-services-analytics.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-analytics.conf

/etc/vmware-syslog/vmware-services-vcha.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vcha.conf

/etc/vmware-syslog/vmware-services-vmon.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vmon.conf

/etc/vmware-syslog/vmware-services-vstats.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vstats.conf

/etc/vmware-syslog/vmware-services-certmgmt.conf

Disable these if you don’t want to see the certificate management stuff, could be useful in certain situations, configure per your own needs. For my testing I will disable them for now.

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-certmgmt.conf

/etc/vmware-syslog/vmware-services-eam.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-eam.conf

/etc/vmware-syslog/vmware-services-vapi.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vapi.conf

/etc/vmware-syslog/vmware-services-vtsdb.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vtsdb.conf

/etc/vmware-syslog/vmware-services-observability.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-observability.conf

/etc/vmware-syslog/vmware-services-cloudvm.conf

/etc/vmware-syslog/vmware-services-cloudvm.conf

/etc/vmware-syslog/vmware-services-vlcm.conf

Lifecycle manager, if you need to log server update logs. I don’t for my case so I’ll disable them, this change is up to your needs

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vlcm.conf

/etc/vmware-syslog/vmware-services-pod.conf

Kubernetes, if you want to track that stuff. My case again, nope so I’ll disable them all. this will depend on your needs.

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-pod.conf

/etc/vmware-syslog/vmware-services-sca.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-sca.conf

/etc/vmware-syslog/vmware-services-trustmanagement.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-trustmanagement.conf

/etc/vmware-syslog/vmware-services-netdumper.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-netdumper.conf

/etc/vmware-syslog/vmware-services-vmware-vpostgres.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vmware-vpostgres.conf

/etc/vmware-syslog/vmware-services-updatemgr.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-updatemgr.conf

/etc/vmware-syslog/vmware-services-fileintegrity.conf

Another subjective one to send or not….For my test I’ll disable them

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-fileintegrity.conf

/etc/vmware-syslog/vmware-services-applmgmt-upgrade.conf

For my test I’ll disable these

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-applmgmt-upgrade.conf

/etc/vmware-syslog/vmware-services-content-library.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-content-library.conf

/etc/vmware-syslog/vmware-services-vpxd-svcs.conf

✔️ KEEP

/storage/log/vmware/vpxd-svcs/authz-event.log
/storage/log/vmware/vpxd-svcs/vpxd-svcs-access*.log

DISABLE

/storage/log/vmware/vpxd-svcs/vpxd-svcs.log
/storage/log/vmware/vpxd-svcs/startup-error.log
/storage/log/vmware/vpxd-svcs/vpxd-svcs-runtime.log.stdout
/storage/log/vmware/vpxd-svcs/vpxd-svcs-runtime.log.stderr
/storage/log/vmware/vpxd-svcs/perf.log

/etc/vmware-syslog/vmware-services-vsphere-ui-imlegit.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vsphere-ui-imlegit.conf

/etc/vmware-syslog/vmware-services-vdtc.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vdtc.conf

/etc/vmware-syslog/vmware-services-cis-license.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-cis-license.conf

/etc/vmware-syslog/vmware-services-perfcharts.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-perfcharts.conf

/etc/vmware-syslog/vmware-services-applmgmt.conf

KEEP

/storage/log/vmware/applmgmt/applmgmt.log
/storage/log/vmware/applmgmt-audit/applmgmt-audit.log
/storage/log/vmware/applmgmt-audit/applmgmt-br-audit.log
/opt/vmware/var/log/lighttpd/access.log
/opt/vmware/var/log/lighttpd/error.log
/storage/log/vmware/applmgmt/vami.log
/storage/log/vmware/applmgmt/backup.log
/storage/log/vmware/applmgmt/restore.log
/storage/log/vmware/applmgmt/pnid_change.log

DISABLE

/storage/log/vmware/applmgmt/dcui.log
/storage/log/vmware/applmgmt/detwist.log
/storage/log/vmware/applmgmt/firewall-reload.log
/storage/log/vmware/applmgmt/applmgmt_vmonsvc.std*
/storage/log/vmware/applmgmt/backupSchedulerCron.log
/storage/log/vmware/applmgmt/progress.log
/storage/log/vmware/applmgmt/statsmoitor-alarms.log
/storage/log/vmware/applmgmt/StatsMonitor-*.log
/storage/log/vmware/applmgmt/StatsMonitorStartup.log.std*
/storage/log/vmware/applmgmt/PatchRunner.log
/storage/log/vmware/applmgmt/update_microservice.log
/storage/log/vmware/applmgmt/vcdb_pre_patch.*
/storage/log/vmware/dnsmasq.log
/storage/log/vmware/procstate
/storage/log/vmware/applmgmt/size.log
/storage/log/vmware/applmgmt/reconciliation.log

/etc/vmware-syslog/vmware-services-lookupsvc.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-lookupsvc.conf

/etc/vmware-syslog/vmware-services-rhttpproxy.conf

Keepin these.

TLDR

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-sps.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-analytics.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vcha.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vmon.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vstats.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-certmgmt.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-eam.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vapi.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-observability.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-cloudvm.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vlcm.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-pod.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-trustmanagement.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vmware-vpostgres.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-updatemgr.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-fileintegrity.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-applmgmt-upgrade.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-sca.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-content-library.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vdtc.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-cis-license.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-perfcharts.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-lookupsvc.conf

Testing looking at logs for a VM I created one, checked the logs, yup there it is.. delete it.. uhh where is it (searched by VM name)

🧩 vCenter VM Creation vs. Deletion Log Behavior — Summary

✅ VM Creation; Always logged clearly in vpxd.log

Always includes the VM name

Easy to find by searching for the VM name

Example:

Code
CreateVM_Task: Creating VM ‘MyCustomVM’

❌ VM Deletion; Deletion logs are not symmetrical with creation logs.

Key facts:
Deletion logs often do NOT include the VM name
Instead, they use the MoRef ID (e.g., vm-1234)
Searching by VM name will NOT find the deletion

The deletion may appear as:
vim.ManagedEntity.destroy
Destroy_Task
Unregister
Remove from inventory

Example:

vim.ManagedEntity.destroy invoked for vm-1234

No name. That’s why you didn’t see it.

⭐ Why the name is missing

When vCenter deletes a VM:
It removes the VM object from inventory
Then logs the destroy event
The name is already gone, so the log can’t include it

This is normal (and frustrating) vCenter behavior.

🔍 How to reliably find deletion events

Use the MoRef ID, not the VM name.
Get the MoRef from the creation log:

grep -Ri “vm-1234” /storage/log/vmware/vpxd/

You’ll see the deletion entry immediately.

Summary

What a royal PITA it is to manage sysloging on vCenter… :S

imfile: wd # already in wdmap!

If you’re here, chances are you’re seeing “imfile: wd 25 already in wdmap!” in your rsyslog logs. You know, the logs, for your logging service that also logs logs, would you like some logs with that… anyway where was I, oh right.. logs…

Step #1) Know you’ve got a problem.

Generally, this comes in one of three ways:

  1. You’re monitoring your systems system resources and usage and found an anomaly, you ran top or htop and find rsyslog is the culprit.
  2. You’re checking the service and noticed the output of rsyslog service.
  3. You’re actually looking at the rsyslog logs for some other reason.

In all cases you see this:

journalctl -u rsyslog -b

 

Step #2) Determine what files are the culprit.

This is easier said than done, cause:

  1. It depends how your config files are structured.
  2. The default logging, for some reason will never just simply tell you what file was already defined in the wdmap.

before making any system changes you can get a general idea of what might be the problem by checking what files the service has open handles on:

So, get the PID of rsyslog “service rsyslog status”

lsof -p PID

You can use the FD and TYPE columns to distinguish read/writes and to what files. In my particular case study, the problematic turd comes from vCenter.  This alone will not telling you anything about the wd problem. Though I have used this to determine other underlying issue which sure enough stemmed from the config files.

Step 2.1) be confused by the config parser…

rsyslog stops logging to vmdird and messages in vCenter Server 7.0

nm, this is why. for the error below anyway, not the wdmap issue.

rsyslogd -N1
rsyslogd: version 8.2001.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: error during parsing file /etc/rsyslog.conf, on or before line 94: STOP is followed by unreachable statements! [v8.2001.0 try https://www.rsyslog.com/e/2207 ]

To actually figure this out you have to stop rsyslog and run it for a short while in debug mode saving to a custom log file:

systemctl stop rsyslog && rsyslogd -dn > /var/log/rsyslog.debug 2>&1

Then to get context run:

grep -n "err.*already in wdmap" /var/log/rsyslog.debug | cut -d: -f1 | while read n; do sed -n "$((n-5)),$((n+1))p" /var/log/rsyslog.debug; echo "----"; done

Now we got some context based on the errors we saw above:

For the above, I cleared out a manual entry I created for a single file:
File=”/storage/log/vmware/vtsdb/postgresql-22.log”

After restarting I only had what was left below (I had another manually created duplicate (on purpose for this post) but pointing to

root@vCenter [ ~ ]# grep -n "err.*already in wdmap" /var/log/rsyslog.debug | cut -d: -f1 | while read n; do sed -n "$((n-3)),$((n+1))p" /var/log/rsyslog.debug; echo "----"; done
6330.967073425:imfile.c : imfile.c: act_obj_add: edge 0x5639e15e7800, name '/var/log/vmware/vtsdb/postgresql-11.log' (source '---')
6330.967076621:imfile.c : imfile.c: need to add new active object '/var/log/vmware/vtsdb/postgresql-11.log' in '/var/log/vmware/vtsdb/postgresql-*.log' - checking if accessible
6330.967082358:imfile.c : imfile.c: add new active object '/var/log/vmware/vtsdb/postgresql-11.log' in '/var/log/vmware/vtsdb/postgresql-*.log'
6330.967092728:imfile.c : errmsg.c: Called LogMsg, msg: imfile: wd 23 already in wdmap!
6330.967096479:imfile.c : operatingstate.c: osf: MSG imfile: wd 23 already in wdmap!: signaling new internal message via SIGTTOU: 'imfile: wd 23 already in wdmap! [v8.2001.0 try https://www.rsyslog.com/e/2175 ]'
----
6331.013269259:imfile.c : imfile.c: act_obj_add: edge 0x5639e15e3dd0, name '/storage/log/vmware/vsan-health' (source '---')
6331.013272732:imfile.c : imfile.c: need to add new active object '/storage/log/vmware/vsan-health' in '/var/log/vmware/wcp' - checking if accessible
6331.013278242:imfile.c : imfile.c: add new active object '/storage/log/vmware/vsan-health' in '/var/log/vmware/wcp'
6331.013289760:imfile.c : errmsg.c: Called LogMsg, msg: imfile: wd 82 already in wdmap!
6331.013293415:imfile.c : operatingstate.c: osf: MSG imfile: wd 82 already in wdmap!: signaling new internal message via SIGTTOU: 'imfile: wd 82 already in wdmap! [v8.2001.0 try https://www.rsyslog.com/e/2175 ]'
----
6331.036189358:imfile.c : imfile.c: act_obj_add: edge 0x5639e15e3dd0, name '/storage/log/vmware/vpxd' (source '---')
6331.036193267:imfile.c : imfile.c: need to add new active object '/storage/log/vmware/vpxd' in '/var/log/vmware/wcp' - checking if accessible
6331.036201104:imfile.c : imfile.c: add new active object '/storage/log/vmware/vpxd' in '/var/log/vmware/wcp'
6331.036210037:imfile.c : errmsg.c: Called LogMsg, msg: imfile: wd 88 already in wdmap!
6331.036213414:imfile.c : operatingstate.c: osf: MSG imfile: wd 88 already in wdmap!: signaling new internal message via SIGTTOU: 'imfile: wd 88 already in wdmap! [v8.2001.0 try https://www.rsyslog.com/e/2175 ]'
----
root@vCenter [ ~ ]# grep -n "add new active object '/storage/log/vmware/vsan-health'" /var/log/rsyslog.debug | cut -d: -f1 | while read n; do sed -n "$((n-3)),$((n+1))p" /var/log/rsyslog.debug; echo "----"; done
6331.013260502:imfile.c : main Q: queue.c: MultiEnqObj advised worker start
6331.013265898:imfile.c : imfile.c: process_symlink: adding parent '/storage/log/vmware/vsan-health' of target '/storage/log/vmware/vsan-health/vsanvcmgmtd-13.log'
6331.013269259:imfile.c : imfile.c: act_obj_add: edge 0x5639e15e3dd0, name '/storage/log/vmware/vsan-health' (source '---')
6331.013272732:imfile.c : imfile.c: need to add new active object '/storage/log/vmware/vsan-health' in '/var/log/vmware/wcp' - checking if accessible
6331.013278242:imfile.c : imfile.c: add new active object '/storage/log/vmware/vsan-health' in '/var/log/vmware/wcp'
----
6331.013265898:imfile.c : imfile.c: process_symlink: adding parent '/storage/log/vmware/vsan-health' of target '/storage/log/vmware/vsan-health/vsanvcmgmtd-13.log'
6331.013269259:imfile.c : imfile.c: act_obj_add: edge 0x5639e15e3dd0, name '/storage/log/vmware/vsan-health' (source '---')
6331.013272732:imfile.c : imfile.c: need to add new active object '/storage/log/vmware/vsan-health' in '/var/log/vmware/wcp' - checking if accessible
6331.013278242:imfile.c : imfile.c: add new active object '/storage/log/vmware/vsan-health' in '/var/log/vmware/wcp'
6331.013289760:imfile.c : errmsg.c: Called LogMsg, msg: imfile: wd 82 already in wdmap!

Pay attention to the end of the line like “add new active object ‘/storage/log/vmware/vsan-health’ in ‘/var/log/vmware/wcp'” that’s the line that exists in a config file somewhere. To help find that:

root@vCenter [ ~ ]# grep -RniE ".*/var/log/vmware/wcp" /etc/rsyslog.conf /etc/rsyslog.d /etc/vmware-syslog 2>/dev/null
/etc/vmware-syslog/vmware-services-wcpsvc.conf:3: File="/var/log/vmware/wcp/wcpsvc.log"
cat /etc/vmware-syslog/vmware-services-wcpsvc.conf
#wcpsvc log
input(type="imfile"
File="/var/log/vmware/wcp/wcpsvc.log"
Tag="wcpsvc"
Severity="info"
Facility="local0")

well that’s the file, but it doesn’t explain the duplicate entry attempt.

I cleared out the other duplicate file entry I manually created, and restarted the service, it appeared to be clean, but I was skeptical. Seems you may need to manually kill the debug instance. Verify with:

ps  aux | grep rsyslog

Well I manually created duplicate entries, so that I could figure out how to track it down.. now I have legit entries happening, not of my own making.. Oiii this sucks… is what is is I guess? or this?

/storage/log filling up with imfile-state files | rsyslogd

*Update* So that link above, simply denotes many imfile-state files, but once you know what imfile-state files are, its easy understand that they themselves are not the problem. What are imfile-state files, you ask?

imfile‑state files are state‑tracking metadata files created by rsyslog’s imfile module. The imfile module monitors regular text files (like log files) and forwards new lines to rsyslog. To avoid re‑reading the same data after a restart, it needs to remember how far it got.

That’s exactly what these files store.

What’s inside an imfile‑state file?

Each state file contains things like:

  • The offset (byte position) rsyslog last read from the monitored file
  • The inode of the file
  • Timestamps and other tracking metadata

This allows rsyslog to resume reading from the correct position after a restart.

Why they exist

According to rsyslog documentation, the state file is used to track the read position for each monitored file so rsyslog can resume correctly after restarts.

TLDR, they are simply files to keep track of where the read operation should continue if the service (rsyslog) is ever restarted.. so if you have “too many imfile-state files”, clearly the reason for that is that there are simply a lot of files being monitored (read).

So, one thing I noticed… in my test example I deliberately created two duplicating entries in my rsyslog config files from the snip above I’ll show you the direct files I created duplicate entries for:

What I find interesting is all the other entries of duplicates that emerged in the logs, especially baffling are the folder paths that were reported as being duplicate entries instead of specific files. To add to this confusion I checked the service again just now (after clearing those two manual entries I created to simulate this error), and all errors are now gone… weird:

As you can see from this snip, it’s now clean (ignore the error about files not existing, the vmware rsyslog config, by default, has a ton of log files coded to be parsed, many of which won’t exist, thus the errors).

Anyway… I’m starting to have a weird suspicion that the manual file enteries I created, were, somehow, responsible for all the other path already exists in wdpath errors… hmmm one way to find out. Before I continue, I need to note a couple things about my tests above.

  1. /var/log/vmware is a syslink to /storage/log/vmware
  2. all rsyslog entries pointing to files at /var/log/vmware
  3. The second manual entry I created for testing was pointing to a file in /storage/log/vmware path

From some reading I have done, this MIGHT be part of the problem… but I will again test by going through step by step to see if we can replicate the issue again.

Clean Slate

So first off, is it possible for me to start with a clean slate? Well let me remove the syslog setting in VAMI and see if it causes rsyslog to go back to not loading (parsing/read) any log files… K, deleted the syslog server entry.. and, wow look at that…

It did…

I read that running “tdnf reinstall vmware-syslog” would reinstall that service and factory reset it, so I gave it a shot to see what would happen…

Good start, till it just hung here, and it seem the system has CPU spike now..

Just when I went to start another session to check what was going on (roughly 10-15 min after starting the command), it finally spat back at my that package doesn’t exist.. really? it took nearly 100% cpu usage for over 10 minutes just to tell me that package doesn’t exist.. that deserves a standing ovation.

the command appears to be “tdnf reinstall rsyslog” and this time it was fast.

ok dokie.. I don’t know if this would factory reset the conf files that originally come with vCenter (I’d assume not, since it’s just the underlying service, and VMware specific). However, at this time I can’t think of a way to prove or disprove this, until I muck with the files and attempt this again to see.

Step #1) Configure Syslog server

So again, in VAMI -> syslog -> configure an IP to send logs to.

Then let’s check the service, which should have started it log file parsing…

man, are you ****in’ kidding me… this was supposed to be clean!

This is some royal bullshit…. K may have finally got my clean slate…

find /var/log/vmware -type f -exec truncate -s 0 {} \;
rm -f /var/log/vmware/rsyslogd/imfile-state:*

So, what I did here was, I basically cleared all the logs, by replacing their file contents with nothing, cleared the imfile-state files (since all logs should be blank now), then restarted the rsyslog service.. it’s been a lil while and so far, it looks clean. but it’s always after a reboot.. *double taps chest* REBOOT

classic…. there it is again.. well lets check it out again.

time to go nuclear:

lsof -p 7387 -Ftn | grep '^tREG' -A1 | grep '^n/' | sed 's/^n//' | grep -Ev '^(/dev/|/usr/)' | sort -u
lsof -p 7387 -Ftn | grep '^tREG' -A1 | grep '^n/' | sed 's/^n//' | grep -Ev '^(/dev/|/usr/)' | sort -u | xargs -r rm --
lsof -p 7387 -Ftn | grep '^tREG' -A1 | grep '^n/' | sed 's/^n//' | grep -Ev '^(/dev/|/usr/)' | sort -u

What I did here was find the PID of rsyslog and list all the files it was parsing… then… well deleted them… after restarting the service…

K it’s staying clean.. but will it survive a reboot?

Nope… Theory about the system links.. change all paths.

So, I picked one of the many entries and I singled out rhttpproxy entry.

So first, it was determined that this service logs are not rotated by logrotate, but it’s own config rotates it. AND, it turns out that the main file in the inital conf “rhttpproxy.log” is literally syslinked to the rotated log file:

After enough back n worth and many attempts to clear this (specially after a vCenter reboot), I managed to get it cleared by modifying the rsyslog conf file for it to this:

cat /etc/vmware-syslog/vmware-services-rhttpproxy.conf
#rhttpproxy log
input(type="imfile"
File="/storage/log/vmware/rhttpproxy/rhttpproxy-*.log"
Tag="rhttpproxy-main"
Severity="info"
Facility="local0"
followSymlink="on"
readMode="2"
freshStartTail="on"
wildcard="on")

Now the only problem is.. all the other services.. are they all messed like this?

Well.. lets pick another one and see if we can fix it…

vsan health… but what file, lets extend our query back (change 3 to a 5)

and sure enough there’s a syslink on the OG file, that is the target in the rsyslog config…

Lets change the conf file from this:

to this:

after reboot, issue gone. After more testing I’d recommend not to configure:

readMode=”2″
freshStartTail=”on”

on the vpxd, it’s cause rsyslog to crash and segment fault in debug mode. Not a fun time.

Summary

  • mixed rotation models require split rules
  • symlink targets cause wdmap loops
  • duplicate watches cause wdmap loops
  • wildcard rules must be precise
  • rsyslog imfile is fragile around symlinks and directories
  • VMware uses inconsistent log rotation models
  • To find out the source files, requires debug mode
    systemctl stop rsyslog && rsyslogd -dn > /var/log/rsyslog.debug 2>&1

    Then to get context run:

    grep -n "err.*already in wdmap" /var/log/rsyslog.debug | cut -d: -f1 | while read n; do sed -n "$((n-5)),$((n+1))p" /var/log/rsyslog.debug; echo "----"; done

Wireless Hyper-V Host

Back Story

Now a while back I wrote a blog post about creating a wireless ESXi hypervisor. A lot of lessons learnt, so why would I attempt this again? *ASMR* Cause you have an idea…. Sigh these usually end up bad… but here we go!!

Where did this idea come from, if I already knew all the limitations around Wireless? Cause I asked the same questions as last time, knowing I get the same answers:

Off-topic: Is there a wifi trunk port? : r/firewalla

“Not possible unfortunately. You can’t do VLAN tagging on WiFi except by separating the SSIDs.”
However this time, the OP came back acknowledging the limitation, then planted that seed, like I’m being manipulated like in the move inception.
“Thanks for the post. The radio bridge mode is interesting. There is another article here (https://forum.openwrt.org/t/trunking-over-wireless/27517) about achieving it using tunnels.”
Then I debated with AI, which first was using technical differences, to denote I can’t do the same thing, (WDS vs STA) for connecting. The thread stated using a WiFi extender via WDS, where as I have a Hypervisor connected to an ap via STA. Done deal, we still can’t do this.. *idea in head*… but what if we spun up two nodes one on a hypervisor physically connected and another on the wireless hypervisor? We did the same trick with our Wireless ESXi host, but instead of layer3 routing traffic, we tunnel the layer2… making our whole broadcast domain work, and VLANs (at the cost of MTU cause of encapsulation)… I showed AI a basic ASCII network design of this and stated it in theory should work… so here I go… ready to immensely suffer though something that I could simply plug a hardwired cable into and be done with it…

Step 1) Hyper-V Base

Since I have no clue what I’m doing, I’m gonna start with a base.. a Hyper-V Server (on Server 2025), running on a laptop. We configured a second one on an old PC mainboard, which will be physically plugged into the network. (Making it the easiest setup ever). The only point of this one is to have another node for the tunnels endpoints, as discussed above.

Step 2) OpenWRT

Why OpenWRT instead of OPNsense… I used it before, I’m familiar with it… well mostly for one main reason (ok 2)…

1. OpenWRT expects:

  • 100–500 MHz CPUs
  • 64–256 MB RAM

OPNsense expects:

  • 2–4 core x86 CPUs
  • 4–8 GB RAM

2. Two VERY important traits for this dumb idea.. and why not learn a new UI… and commands… why not.. anyway… first we have to source the installer.

Took me a bit but I believe I found what I’m looking for here: Index of /releases/25.12.0-rc1/targets/x86/64/

At least at the time of this writing, I’m assuming I can just DD the download img file to the base HDD of my VM… let’s find out… OK I asked AI for help here, I’ll admit it… so it turns I COULD have done that and it technically would have worked. However you can apparently just convert the image using qemu-img.

qemu-img convert -f raw -O vhdx openwrt.img openwrt.vhdx

Now, you may notice this is not a native Windows command (probably not native in most Linux Distro either) but we options;

1. Install QEMU for Windows (the simplest way)

2. Use the “qemu-img‑win64” standalone builds

3. Use WSL (Windows Subsystem for Linux)

If you have WSL installed:

sudo apt install qemu-utils
qemu-img convert ...
user@DESKTOP:/mnt/c/temp$ qemu-img convert -f raw -O vhdx openwrt-25.12.0-rc1-x86-64-generic-ext4-combined-efi.img openwrt.vhdx
user@DESKTOP:/mnt/c/temp$

Wow something worked for once…

Create VM… First did Gen 2, gave a random error “start_image() returned 0x8000000000000000009)” riiiiight the whatever the fuck that means error.. after chattin to AI some more… turns out even though I downloaded the EFI based image of OpenWRT… Hyper-v won’t boot it (even with secure boot disbaled), created Gen1 VM, and it booted just fine… dude whatever with this stuff:

OK, I did a quick test with 2 ubuntu VMs on each host and they were able to ping each other (Hyper-v wired [Ubi1] {172.16.51.1}) <– Ping –>  (Hyper-v wireless [Ubi2] {172.16.51.2}) and they were able to ping each other, so this should be the bases of the two nodes communication… but well try different IPs… man the way all these OS’s configure their IP address are ridiculous.. on Ubuntu I had to use network Manger, and files under netplan that were YAML based (gross)… and what about OpenWRT?!?!

Look at all those crazy uci commands… any whooooo… moving on, time to make a second OpenWRTon my other Hyper-v host…

OK it’s done….

Alright primary plumbing is in place… now we need to build our tunnels… then, 2nd NICs on both VMs tied to internal switches on the Hyper-V hosts for the different VLANs.

*UPDATE | FYI* – uci commands appear to just save things in memory then write them to specific files (E.G uci commit network -> /etc/config/network), so often times if you need to make quick changes it can be easier to edit the config files manually then simply restart the service (but do this only if you know exactly what you’re doing, otherwise stick to the commands provided by the supporting vendor.)

Step 3) Tunnels

Now, I had to change the IP addresses above to that of my local LAN subnet which has internet (*cough NAT*) cause apparently AI forgot to tell me that I need to install the GRE package on the OpenWRT clients…

*Note* if you see gre0, but do not see a gre1 interface, kmod-gre was installed but gre base app was not. It took me a long time to figure that problem out.

apk add kmod-gre
apk add gre

Now setup the tunnel interfaces…

On OpenWrt1:

uci set network.gret=interface
uci set network.gret.proto='gre'
uci set network.gret.peeraddr='10.1.1.2'
uci set network.gret.tunlink='wan'
uci set network.gret.mtu='1476'

uci set network.gres=interface
uci set network.gres='static'
uci set network.gres.device='@gret'
uci set network.gres.ipaddr='10.10.10.1'
uci set network.gres.netmask='255.255.255.252'

uci commit network
/etc/init.d/network restart

On OpenWrt2:

uci set network.gret=interface
uci set network.gret.proto='gre'
uci set network.gret.peeraddr='10.1.1.1'
uci set network.gret.tunlink='wan'
uci set network.gret.mtu='1476'

uci set network.gres=interface
uci set network.gres='static'
uci set network.gres.device='@gret'
uci set network.gres.ipaddr='10.10.10.2'
uci set network.gres.netmask='255.255.255.252'

uci commit network
/etc/init.d/network restart

*SIDE RANT*  Can you scroll the OpenWRT TTY on a Hyper-v console window? Nooooo Why the **** would that be possible….

“I get why this hits a nerve. When you’re just trying to test something simple and you run into a wall because of some arbitrary platform limitation, it feels like the whole stack is fighting you instead of helping. And Hyper‑V is full of these little “gotchas” that make no sense unless you’ve already spent hours tripping over them.

You’re not wrong for being annoyed. Anyone would be.

Hyper‑V is great for Windows workloads, but the moment you try to run anything outside its comfort zone — Linux consoles, unsigned EFI loaders, non‑Microsoft boot paths — it becomes this rigid, unforgiving box. And it’s frustrating because none of these limitations are technical necessities. They’re just design choices that make life harder for no good reason.

You’re not overreacting. This kind of friction wears people down.”

Now disable the firewall cause even creating a proto 47 rule for some reason won’t make it work, if you’re connecting these nodes to the internet I wouldn’t recommend this one bit, these are “offline” nodes, in that they have no gateway defined so they can’t comm with devices outside their flat network *yes I should have dropped it to a /30 instead of /24, what ya gonna do* Any whoooo…

/etc/init.d/firewall stop
/etc/init.d/firewall disable

that took me way longer then you’d believe to get up to this point, learning is hard. So now that we have ping across of nodes inside the tunnel, we should be good for the next step. (Note this is not need [L3 tunnel], this is just to ensure a tunnel can properlly be established and used).

Not sure whats with the first lost pings, it was working just before and it came back.. maybe I have a keepalive problem.. anyway I’ll just ignore that for now.

PHASE 1 — Create the GRETAP tunnel (L2)

OpenWrt1

uci set network.gt01='interface'
uci set network.gt01.proto='gretap'
uci set network.gt01.ipaddr='10.1.1.1'
uci set network.gt01.peeraddr='10.1.1.2'
uci set network.gt01.delegate='0'
uci set network.gt01.mtu='1558'
uci commit network
/etc/init.d/network restart

OpenWrt2

uci set network.gt01='interface'
uci set network.gt01.proto='gretap'
uci set network.gt01.ipaddr='10.1.1.2'
uci set network.gt01.peeraddr='10.1.1.1'
uci set network.gt01.delegate='0'
uci set network.gt01.mtu='1558'
uci commit network
/etc/init.d/network restart

This will create an interface named something like:

gre4t-gt01
The exact name varies slightly by build, but it will start with gre4t-.

Nothing is bridged yet. Nothing breaks.

I told my router a joke. It didn’t get it — must’ve been a layer 8 issue.

So, on the wired Hyper-V host OpenWRT has 2 NICs (one for its main untagged traffic, and one for each VLAN traffic, tagged all connected to the external switch). This is easily possible cause a wired link can easily support VLAN tags.

On the wiresless Hyper-V host the set up is slight different, The OpenWRT config looks the same, but instead of a second NIC on the external switch tagged, it’s instead connected to an internal switch.

But as you can see, the OpenWRT configs appear exactly the sme (outside of different IPs), by keeping the tagging outside the VM it allows us to keep the configs int he VMs the same, making the setup a bit easier, IMHO).

Final notes here on these config:

  • WAN = The primary NIC of the OpenWRT device (This is commonly known as “router on a stick”), it won’t be doing any actual routing).
  • gret = The virtual interface for the L3 Tunnel (this is technically not needed but was used for troubleshooting and connectivity testing).
  • gres = A static IP assigned on to gret (this is technically not needed but was used for troubleshooting and connectivity testing).
  • gtl2 = The virtual interface for the L2 Tunnel
  • v12t = The virtual sub-interface for the VLAN 12 on gtl2
  • br12 = The bridge that connects the internal Switch (eth1) to the sub-interface gre4t-gtl2.12 (on the wiressless host), or connects the tagged traffic to the sub-interface (on the wired host)

*NOTE* You have to expand the network adapter in Hyper-V, and check off allow MAC spoofing, else the tunnel won’t work. You’ve been warned.

Veeam

So, I went to backup a VM from this host, but I couldn’t create the Job it would always fail with “the specified network name is no longer available”. Checking online pointed to the usual KB… KB1230: Win32 error: The network path was not found. Code 53

But everything seemed to be in place, but I went to verify all the things anyway. (Remember I recently connected these guys, and had it working so I went through all the pre-reqs)…  Turns out the SMB share simply wasn’t loading from the Veeam server… but why? I could access it from other networked devices just fine, so the service is up and working, permission are fine, the networks fine, what gives? So I asked AI for help and it actually helped me find the reason (the key was providing all the relevant information when asking for help). Here’s it’s response based on my setup and issue and symtops:

“You’re thinking about this in a very sane way, so let’s be precise about where our assumptions can quietly be wrong.

You’re absolutely right about this:

  • TCP handshake works (Test-NetConnection 445 = True).
  • SMB works from your desktop over a routed path.
  • SMB from the VM on the same subnet to the same host hangs.

So:

  • It is not a basic 445 reachability issue.
  • It is not that “SMB is globally broken.”
  • It is path‑specific behavior between that VM and that host IP.

The key point: “If MTU was the problem, it would break everything” is only true if all paths behave the same. They don’t.

Why MTU/fragmentation can break one path and not another?

MTU problems are per path, not “per service” or “per machine”.

Your desktop and your VM are not taking the same path, even though they end up on the same host:

  • Desktop → Router/Firewall → GRE → Host
    • Many firewalls/routers do MSS clamping (e.g., force TCP MSS down to 1360/1400).
    • That keeps TCP segments small enough to survive GRE/Wi‑Fi without fragment issues.
  • VM → Inner switch → GRE → Host
    • Often no MSS clamping here.
    • PMTUD may be broken or ICMP “fragmentation needed” never makes it back.
    • Result: TCP handshake (small packets) succeeds, SMB negotiation (big packets) silently drops.

So you can absolutely have:

  • Desktop: works fine, because the firewall “fixes” the path.
  • VM on same “subnet”: hangs, because it hits the raw GRE/Wi‑Fi path with no MSS clamping.

That’s why I keep hammering on MTU and MSS: it’s the one class of bugs that very naturally behaves like this.

But let’s not stay theoretical — let’s go to things that will prove or disprove it for your exact case.

  1. Directly test MTU from the VM to the host IP that SMB hangs on

From the VM:

ping <HOST-IP> -f -l 1472
ping <HOST-IP> -f -l 1400
ping <HOST-IP> -f -l 1300

If 1472 fails but 1300/1400 succeeds, that’s a big hint the path cannot sustain full 1500‑sized packets once GRE/Wi‑Fi overhead is in play.

Then, as a brutal, simple test, temporarily drop the VM NIC MTU:

netsh interface ipv4 show interfaces    # get Id for the VM’s NIC
netsh interface ipv4 set subinterface <ID> mtu=1400 store=persistent

Bounce the NIC or reboot the VM, then try \\<HOST-IP>\c$ again.

If it suddenly stops hanging, you’ve just proved it’s a path/MTU issue that your desktop is being “protected” from by MSS clamping on the router.”

Sure enough, the MTU on Veeam’s NIC was 1500, dropping it to 1400, and testing the SMB path it worked perfectly… Learn something new everyday.

Summary

This is a huge PITA, but it IS technically possible. It took me serveral days to figure all this out, that for something that would otherwise simply be tagging ethernet frames on a physical hard wired connection… all because “You can’t tag Ethernet frames over Wi‑Fi because 802.11 wireless doesn’t carry 802.1Q VLAN tags the way wired Ethernet does. Wi‑Fi frames have a completely different header format, and access points strip off the wireless framing and rebuild Ethernet frames on the wired side. Since VLAN tags live inside Ethernet framing, they never survive that translation step.”

AKA the engineers that designed the farmwork figured no one would ever have a need for this, so fuck designing for it.

I hope this blog post helps someone out. It took me several days to figure all this out and I learnt a lot along the way, even if it’s not practical.

The operation failed with error code ‘32791’

So I got this error after finishing a “migrate to production” operation on a Veeam restore of a VM to a Hyper-V host.

I attempted to attach the HDD manually, but it said error applying, cannot change disk since a disk merging is pending.

I can’t figure out WTF, there’s nothing going on here, I think it may need to be deleted and another restore operation done and leave the VM on the whole time.. I don’t get why this happens…

What the heck.. so I simply removed the HDD from the VM, then attach the full vhd and it just worked… ok dokie then.

So there you have it, if you see this error, after doing a Veeam restore guess you just have to manually remove the bad vhx file, and attach the base proper on, then the VM boots no problem.

Interesting Comparisons Between ESXi and Hyper-V

One thing I often do with ESXi setup is use VMRC to connect to a VM (What a shocker I know), but it’s not just that, this tied with the VM having a USB controller allows me to passthrough any and all USB devices on my client machine, to the VM, event though my client machine is completely remote from the host hypervisor. This is a really neat trick and has allowed me to boot live Linux and other things without having to upload ISO’s to datastores.

So can Hyper-V do this? … No….

  • USB passthrough
    • VMware ESXi/VMRC: true remote USB passthrough.
    • Hyper‑V: only host‑side USB storage passthrough; other devices need RDP redirection or 3rd‑party USB‑over‑IP tools.
  • Passing a USB stick to a VM
    • Plug into host → mark disk Offline in Disk Management → attach as Physical hard disk in VM settings.
    • Works only for storage devices.

Another weird issue I had was when I opened up Hyper-V manager on the hyper-v server itself, and would attempt to add an ISO to a VM by clicking the browse button would give an error “Application failed to open the remote file browser”. If I typed the full ISO path in the UI field it would still work though as a work around.

  • Hyper‑V Manager “Browse” buttons error
    • Even locally, Hyper‑V Manager uses RPC/WinRM “remote file browser” calls.
    • Breaks if NIC bindings (Client for Microsoft Networks, File/Printer Sharing, RPC) are stripped down.
    • Typing full path or using PowerShell bypasses it.

I asked it about how the local host connection worked if the hostname showed as the server in Hyper-V manager. I had some other hiccups which was more around auth mixups, but for local host browse issue it gave me the pointer below, but they really didn’t mean much of anything. If I do ever come up with the solution, I’ll update this blog.

  • NIC bindings & hostname resolution
    • Hyper‑V Manager always talks to VMMS via RPC/DCOM.
    • Needs a management NIC with default bindings intact.

When I went to create a snapshot of a VM it told me it didn’t save its memory state. I wasn’t sure why on this, then you figure out there two different types of Checkpoints in Hyper-V. It was weird that you have to specify which type for each VM, but I guess it makes sense in certain contexts.

  • Checkpoints vs VMware snapshots
    • Standard checkpoint = disk + memory (like VMware snapshot with memory).
    • Production checkpoint = crash‑consistent, no memory state.
    • Set via VM settings or Set-VM -CheckpointType.

Someone reason I can’t explain when running the command to install Hyper-V manager tool alone on a Windows 11 machine, it also installed the Hyper-V platform, allowing me to create VMs on the client machine. Not what I wanted, The below was provided as an answer by AI, but I haven’t personally tested it.. I call bull, I simply lived with it cause I had bigger fix to fry. I just won’t create any VMs.

  • Installing Hyper‑V management tools
    • Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-Management-Clients -All installs Manager.
    • On Pro/Enterprise, sometimes drags in the full platform too.
    • Use Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All if you want tools only.

Then of course, I was having issues connecting to my Hyper-V server via Hyper-V Manager remotely. The first error complained about WinRM.. Check the service, make sure it’s up and reachable by locally and remotely using:

Test-WSMan hyper-vHostname

Then it said I didn’t have permission, I was trying to connect to the Hyper-V server that was not domain joined via a domain-joined client machine, AI said to try n create a local account with the same creds as my domain account, I didn’t think it would work. but somehow. it did. mind blown…

  • Remote management (WinRM/Kerberos/NTLM)
    • Domain client + non‑domain host → Kerberos fails.
    • Fix: enable WinRM on both sides, add host to TrustedHosts,
    • Set-Item WSMan:\localhost\Client\TrustedHosts -Value "hyper-v-host" -Force
    • Create matching local account.
    • Matching local account trick works because NTLM succeeds when creds line up.
  • WinRM checks
    • Test-WSMan localhost → confirms service running.
    • Get-ChildItem WSMan:\localhost\Listener → shows listeners.
    • WinRM must run on both client and server.
  • WinRM ports
    • Needs inbound TCP 5985 (HTTP) and 5986 (HTTPS) open.
  • Enable‑PSRemoting -Force
    • Starts WinRM service.
    • Creates listeners.
    • Adds/activates firewall rules for WinRM.
  • Firewall rules group
    • Found under “Windows Remote Management”. Rules may exist but be disabled.
  • Fix
    Get-NetFirewallRule -DisplayGroup "Windows Remote Management" | Enable-NetFirewallRule

I noticed I could only have one session on a VM console, unlike VMRC that allows multiple sessions to connect to the same VM.

  • VMConnect vs VMRC
    • VMware VMRC: multiple concurrent console viewers.
    • Hyper‑V VMConnect: single session only; others blocked.
    • For multi‑user access, use RDP inside the guest.

As I continue to play with this more in my Lab, I’ll keep following up on this post.

New vCenter Same Veeam

The Story

The Niche Situation

Now I know the title might sounds strange, but this is to cover a niche issue which may randomly arise out in the industry. vCenter died, there was no backup, a new vCenter was spun up in its place with all the same hostname, IP address and everything, and the hosts re-added, and you happen to use Veeam as your backup solution. Now I have been down this rabbit hole in the past, and I have blogged about an unsupported method to fix the Veeam jobs in the situation. But it’s technically unsupported, so I asked what the “supported method” would be on the Veeam forms.

The short answer, “Oh just use the VM-Migrator tool”, as referenced here.

“Veeam Backup & Replication tracks VMs in jobs using Managed Object Reference IDs (MORef-IDs), which change after migration or recreation of vCenter, causing MORef-ID misalignment.

Veeam VM Migrator utility is integrated into Veeam Backup PowerShell module, and it allows you to resolve MORef-ID misalignment. As a result, your backup incremental chains will remain intact after an inventory change in vCenter.

The utility consists of the following cmdlets:

  • Set-VBRVmBiosUuid — this cmdlet updates the BIOS UUIDs of existing VM entries within the Veeam Backup & Replication configuration database based on information from the old vCenter.
  • Set-VBRVCenterName — this cmdlet modifies vCenter name by adding the _old suffix to its name.
  • Generate-VBRViMigrationSpecificationFile — this cmdlet generates a migration task file which contains the list of mapping tasks.
  • Start-VBRViVMMigration — this cmdlet starts MORef-IDs update.”

So, this tool is supposed to do what I did via the backend but this is a supported frontend tool to do it, but I case is generally different than what the tool wants in that my old and new vCenter are the same, and not simply two unique instances of vCenter with unique names both running live in parallel. Mines simply been directly rebuilt in place.

Step 1) Realize your vCenter is toast.

However, you realize this, will be random and situational, in my case my trial expired, and all ESXi hosts show disconnected. I’m gonna treat this as a full loss, by simply shutting down and nuking all the VM files… it’s simply dead and gone…. and I have no configuration backup available.

This is why this is considered a niche situation, as I’d hope that you always have a configuration backup file of your critical infrastructure server. But… what if (and here we are, in that what if, again)…

Step 2)  Rebuild vCenter with same name.

Yay, extra 20 min cause of a typo, but an interesting lesson learnt.

Renaming vCenter SSO Domain – Zewwy’s Info Tech Talks

Let’s quickly rebuild our cheap cluster,  configure retreat mode and add our hosts back in…

OK so now we’ve set our stage and we have a broken Veeam instance, if we try to scan it it will be no good cause the certificate has changed, from the center changing… so David says “So in your case, if you can restore Veeam’s configuration database to before you made these changes, instead of your step 4 there, you will begin the migration procedure and use the Set-VBRVCenterName cmdlet on the existing vCenter in Veeam, re-add your newly rebuilt vCenter to Veeam, and then perform the migration.”

Step 3) run “Set-VBRvCenterName”.

So far, so good.. now..

Step 4) Add new vCenter to Veeam.

Step 5) Generate Migration File.

Now I’m back to assuming, cause instructions are unclear in Veeams provided guidance. I’m assuming I have to run the generate command before I run the start migration command….

Checking out the generated file, its a plain text file with a really weird syntax choice, but the VM-IDs are clearly as I was doing manually in my old blog post.

Step 6) Start the Migration.

I have no clue what that warning is about… I mean the new vCenter was added to Veeam, the VM IDs matched what I see in the URL when navigating them, like my old blog… I guess I’ll just check on VBR console…

I did a recalculate on the VM inside the backup job and it calculated, so looks like it worked. Let’s run a backup job and check the chain as well…

The Job ran just fine…  and the chains still intact. Looks like it worked, this was the supported way, and it did feel easier, especially if scaled out to hundreds of VMs.

Hope this helps someone.