vCenter syslog/rsyslog

So, in my previous post I discussed troubleshooting the wd in wdpath already exists log error. However, the root issue there may have been determined and resolved… but the question arises… do we need to ship that much logs?

What are all these log files for?

High‑Level Overview

Every file listed is part of vCenter Server’s syslog configuration. Each vmware-services-*.conf file tells the syslog collector which logs belong to which internal service. These logs fall into categories like:

  • UI / Client logs
  • SSO & Identity logs
  • vCenter core services (vpxd, vmon, vapi, etc.)
  • Database logs (Postgres, vtsdb)
  • vSAN Health
  • Networking (rhttpproxy, netdumper)
  • Appliance management (applmgmt, cloudvm)

Below is a readable breakdown of what each group of log files does.

📘 Detailed Breakdown by Service

🎨 vSphere UI / HTML5 Client

Files under /storage/log/vmware/vsphere-ui/logs/

These logs cover everything related to the vSphere Client (the HTML5 UI):

Log Purpose
vsphere_client_virgo.log Main UI application server (Virgo) log
changelog.log UI plugin/component change tracking
dataservice.log Backend data service used by UI
apigw.log API gateway for UI requests
equinox.log OSGi framework logs
eventlog.log UI event processing
httpRequest.log HTTP request logs
opid.log Operation IDs for tracing UI actions
performanceAudit.log UI performance metrics
plugin-medic.log Plugin health & validation
threadmonitor.log Thread health monitoring
threadpools.log Thread pool usage
vspheremessaging.log Messaging subsystem
vsphere-ui-rpm.log UI package/runtime logs
vsphere-ui-runtime* Runtime stdout/stderr
access/localhost_access_log.txt Web access logs
vsphere-ui-gc* Java garbage collection

🔐 SSO / Identity Services

Files under /storage/log/vmware/sso/, /storage/log/vmware/vmdir/, /storage/log/vmware/vmafd/

These logs relate to authentication, identity, certificates, and tokens:

Log Purpose
activedirectoryservice.log AD integration
lookupsvc-init.log Lookup service initialization
openidconnect.log OIDC authentication
ssoAdminServer.log SSO admin operations
svcaccountmgmt.log Service account management
tokenservice.log Token issuance
sts-health-status.log.* STS health
sts-runtime.log.* STS runtime
gclogFile.*.current JVM GC
tomcat/localhost_access.log SSO Tomcat access
vmdir/*.log Directory service (LDAP-like)
vmafd/*.log Authentication framework

🧩 vCenter Core Services

vpxd (vCenter Server daemon)

These are commented out in your file, but normally include:

  • vpxd.log — main vCenter service log
  • vpxd-profiler-*.log — performance profiling

vmon

Manages service lifecycle:

  • vmon.log — service manager log
  • vmon-vapi-provider-0.log — VAPI provider logs

vapi

API endpoint logs:

  • endpoint.log — main API endpoint
  • endpoint-access.log — API access logs
  • jetty.log — Jetty web server
  • vcentershim.log — vCenter API shim
  • vmodl2swagger.log — API schema conversion
  • vmware-vapi-endpoint-gc.log.* — GC logs
  • vmware-vapi-endpoint.std* — stdout/stderr

📊 Analytics / Telemetry

  • analytics.log — analytics service
  • analytics-runtime.log.std* — runtime logs

🧱 vSAN Health

  • vmware-vsan-health-service.log — main vSAN health service
  • vmware-vsan-health-runtime.log.* — runtime logs
  • vsanvcmgmtd-*.log — vSAN cluster mgmt

🗄️ Database Services

Postgres (vPostgres)

  • serverlog.std* — main DB log
  • postgresql-*.log — DB engine

logsvtsdb

  • vtsdb-runtime.log.std*
  • runtime  postgresql-*.log — DB logs

Postgres Archiver

  • pg_archiver.log.std* — WAL archiving

🔧 Lifecycle Manager (vLCM)

  • lcm_common.log — core LCM operations
  • task_executor.log — task execution
  • twisted_server.log — Python-based server
  • vlcm_db.log — LCM database
  • vlcm-runtime.log.* — runtime logs

🧪 vSphere ESX Agent Manager (EAM)

  • eam.log — main EAM service
  • web/*.log — Tomcat logs
  • jvm.log.* — JVM logs
  • eam_firstboot.py*.log — first boot

The EAM log refers to the log files generated by the VMware ESX Agent Manager (EAM) service.
EAM is a core vCenter component responsible for deploying and managing ESX agents, which are small helper VMs or services used by features such as:

vSphere Lifecycle Manager (vLCM)
vSphere Storage I/O Control
vSphere Network I/O Control
vSAN / vCLS agents
Third‑party extensions that deploy agents to ESXi hosts

Search results confirm that EAM logs live in /var/log/vmware/eam/ and are used for diagnostics and troubleshooting.

📘 What EAM logs contain

1. eam.log — Main service log
This is the primary log file for the ESX Agent Manager.

It records:

Service startup and shutdown
Agent deployment and lifecycle events
Communication with vCenter and ESXi hosts
Plugin/extension registration
Errors when EAM cannot deploy or manage agents
Failures related to vCLS or vSAN agent VMs
Search results show examples of EAM startup failures and configuration errors logged in eam.log.

2. Web access logs (web/localhost_access.log)
These track:

HTTP requests to the EAM web service
MOB (Managed Object Browser) access
API calls from vCenter or extensions
Mentioned in STIG guidance for EAM logging.

3. JVM logs (jvm.log, wrapper.log)
These capture:

Java runtime errors
Memory issues
Crashes or fatal exceptions
Examples of JVM startup failures appear in VMware KB articles

🌐 Networking & Proxy Services

rhttpproxy

  • rhttpproxy-*.log — reverse proxy logs

netdumper

  • netdumer.log — ESXi dump collector
  • webserver.log — web interface

🧰 Content Library

  • cls.log — content library service

📈 Perfcharts

stats.log — performance charts

  • localhost_access_log.txt — access logs
  • vmware-perfcharts-gc.log.* — GC logs
  • vmware-perfcharts-runtime.log.std* — runtime

🧭 Lookup Service

  • lookupserver-default.log — main lookup service
  • lookupServer.log — operations
  • lookupsvc_stream.log.std* — runtime
  • vmware-lookupservice-perf.log — performance
  • vmware-lookupsvc-gc.log.* — GC

🧩 vpxd-svcs (vCenter Support Services)

  • vpxd-svcs.log — main
  • authz-event.log — authorization events
  • startup-error.log — startup failures
  • vpxd-svcs-access*.log — access logs
  • vpxd-svcs-runtime.log.* — runtime
  • perf.log — performance

 

🛡️ Trust Management

  • trustmanagement-runtime.log.std* — runtime
  • trustmanagement-svcs.log — trust services
  • vmware-trustmanagement-gc.log.* — GC

🔐 Trust Management Service — Tight Summary

Trustmanagement is a core vCenter service that maintains the trust relationships between all internal components. It ensures that certificates, tokens, and service‑to‑service authentication are valid and secure.

What it handles:

Certificate chain validation
Trust checks between vCenter services
Token verification (STS/SSO)
Security posture and compliance signals

What its logs show:

Certificate or trust failures
Service registration/authentication issues
Token validation errors
Startup/shutdown and internal health

Why it matters:
If trustmanagement breaks, you may see:

vCenter login failures
STS token errors
Certificate replacement problems
Services stuck in “Not Running”
Upgrade failures due to trust issues

What it does NOT do:

Track user logins
Record user actions
Provide audit logs

It’s purely about internal vCenter security plumbing, not end‑user activity.

🧩 Pod Service

  • pod-service.log — pod mgmt
  • pod-console.log — console
  • pod-startup.log — startup
  • pod-install*.log — install
  • pod-update*.log — updates

🧰 Appliance Management (VAMI)

Files under /storage/log/vmware/applmgmt/ covers:

  • VAMI web UI
  • Backup/restore
  • Firewall reload
  • Stats monitor
  • PNID changes
  • Lighttpd access/error logs

🔍 What the Applmgmt Upgrade Service does

It manages:

VCSA upgrade workflows
Patch installation
Pre‑upgrade checks
Post‑upgrade cleanup
Version validation
Upgrade‑related service orchestration

It’s the engine behind the VAMI (port 5480) upgrade process.

📁 What logs this syslog config points to

The file typically references logs such as:

applmgmt-upgrade.log — main upgrade workflow log
applmgmt-upgrade-runtime.log.std* — stdout/stderr
applmgmt-upgrade-gc.log.* — Java garbage collection

These logs capture:

Upgrade steps and progress
Validation checks
Errors during patching or upgrading
Service restarts triggered by upgrades
JVM runtime behavior

🧭 When these logs matter

You check these logs when:

A VCSA upgrade fails
Patching stops mid‑process
Pre‑upgrade checks report errors
The VAMI UI shows upgrade failures
Services don’t come back after an upgrade

🔐 Certificate Management

  • certificatemanagement-runtime.log.std *
  •  certificatemanagement-svcs.log *
  • vmware-certificatemanagement-gc.log.*

🧩 SCA = Secure Configuration Assistant

A vCenter subsystem responsible for security posture checks, certificate validation, and secure configuration enforcement.

It’s part of the broader vCenter security framework that also includes:

Certificate Management (certmgmt)
VMCA (VMware Certificate Authority)
STS (Security Token Service)
PSC identity services (in older versions)

🧩 What SCA actually does

🛡️ 1. Security posture checks

It evaluates whether vCenter components are configured securely, including:

TLS/SSL settings
Certificate validity
Service trust relationships
Cryptographic compliance

🔏 2. Certificate and trust validation

It works closely with:

VMCA
certmgmt
STS

to ensure that:

Certificates are valid
Trust chains are intact
Services can authenticate to each other

🧭 3. Compliance reporting

SCA feeds data into:

vCenter security health checks
vSphere Client “Security” view
Some VAMI security status pages

📁 Where you see SCA in logs

You’ll typically find SCA logs under:

Code
/storage/log/vmware/sca/
Common files include:

sca.log — main service log
sca-runtime.log.std* — stdout/stderr
sca-gc.log.* — Java garbage collection

These logs show:

Security scan results
Certificate validation failures
Trust chain issues
Service authentication problems
Startup/shutdown of the SCA service

🧭 When SCA logs matter

You check SCA logs when:

vCenter shows certificate warnings
Services fail to register due to trust issues
You see “vCenter is not secure” alerts
STS token problems appear

vCenter upgrades fail due to certificate or trust chain issues

🛡️ File Integrity Service — Tight Summary

The fileintegrity syslog config points to logs generated by vCenter’s File Integrity Service, which monitors critical system files for unauthorized or unexpected changes.

What it does

  • Checks hashes of important vCenter files
  • Detects tampering, corruption, or unexpected modifications
  • Flags security‑relevant integrity issues

What its logs contain

  • Integrity scan results
  • File change alerts
  • Hash mismatches
  • Service errors and startup info
  • JVM runtime and memory behavior (via runtime + GC logs)

Why it matters

  • Helps detect compromise or corruption of vCenter
  • Useful for SOC teams as security telemetry
  • Not related to user activity or audit logging

🧵 threadmonitor.log — Tight Summary

threadmonitor.log is part of the vSphere UI service (the HTML5 vSphere Client).
This log tracks thread health and performance inside the UI service’s Java application.
It’s essentially a watchdog that monitors whether internal threads are running normally or getting stuck.

🔍 What it records

Thread stalls or deadlocks
Long‑running or hung operations
UI service performance issues
Thread pool exhaustion
Java exceptions related to thread execution
Internal timing or responsiveness problems

It’s a diagnostic log for the vsphere-ui backend, not for user activity.

🧭 When this log matters

You check threadmonitor.log when:

The vSphere Client is slow or unresponsive
Pages hang or fail to load
UI freezes during tasks
You suspect backend thread starvation
The vsphere-ui service crashes or restarts

It’s especially useful when troubleshooting UI performance issues.

Disable Unwanted Logs

This is obviously a balancing act between what you feel is needed to be forwarded, and what is not required depending on destination logging capabilities. Note comenting out these lines only stops the forwarding of the logs to the syslog destination, it does not stop the local logging of these services. That is out of scope of this blog post.

/etc/vmware-syslog/vmware-services-vsphere-ui.conf

Disable all except:
File=”/storage/log/vmware/vsphere-ui/logs/access/localhost_access_log.txt”

/etc/vmware-syslog/vmware-services-vmcad.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vmcad.conf

/etc/vmware-syslog/vmware-services-sso-services.conf

just these:

/storage/log/vmware/sso/sts-health-status.log.* /storage/log/vmware/sso/sts-runtime.log.* /storage/log/vmware/sso/gclogFile.*.current /storage/log/vmware/sso/tomcat/localhost_access.log

/etc/vmware-syslog/vmware-services-vsm.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vsm.conf

/etc/vmware-syslog/vmware-services-vpxd.conf

KEEP

File="/storage/log/vmware/vpxd/vpxd-*.log"

There are a lot of vpxLRO logs generated by this, but there appears to no other granual controls at the source level (these rsyslog imfile config), so not sure about filtering these outside of transforms at the other syslog/rsyslog/lostash service that is receiving these logs.

DISABLE

File="/storage/log/vmware/vpxd/vpxd-profiler-*.log"

/etc/vmware-syslog/vmware-services-infraprofile-syslog.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-infraprofile-syslog.conf

/etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf

/etc/vmware-syslog/vmware-services-vsan-health.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vsan-health.conf

/etc/vmware-syslog/vmware-services-envoy.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-envoy.conf

/etc/vmware-syslog/vmware-services-sps.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-sps.conf

/etc/vmware-syslog/vmware-services-analytics.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-analytics.conf

/etc/vmware-syslog/vmware-services-vcha.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vcha.conf

/etc/vmware-syslog/vmware-services-vmon.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vmon.conf

/etc/vmware-syslog/vmware-services-vstats.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vstats.conf

/etc/vmware-syslog/vmware-services-certmgmt.conf

Disable these if you don’t want to see the certificate management stuff, could be useful in certain situations, configure per your own needs. For my testing I will disable them for now.

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-certmgmt.conf

/etc/vmware-syslog/vmware-services-eam.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-eam.conf

/etc/vmware-syslog/vmware-services-vapi.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vapi.conf

/etc/vmware-syslog/vmware-services-vtsdb.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vtsdb.conf

/etc/vmware-syslog/vmware-services-observability.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-observability.conf

/etc/vmware-syslog/vmware-services-cloudvm.conf

/etc/vmware-syslog/vmware-services-cloudvm.conf

/etc/vmware-syslog/vmware-services-vlcm.conf

Lifecycle manager, if you need to log server update logs. I don’t for my case so I’ll disable them, this change is up to your needs

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vlcm.conf

/etc/vmware-syslog/vmware-services-pod.conf

Kubernetes, if you want to track that stuff. My case again, nope so I’ll disable them all. this will depend on your needs.

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-pod.conf

/etc/vmware-syslog/vmware-services-sca.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-sca.conf

/etc/vmware-syslog/vmware-services-trustmanagement.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-trustmanagement.conf

/etc/vmware-syslog/vmware-services-netdumper.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-netdumper.conf

/etc/vmware-syslog/vmware-services-vmware-vpostgres.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vmware-vpostgres.conf

/etc/vmware-syslog/vmware-services-updatemgr.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-updatemgr.conf

/etc/vmware-syslog/vmware-services-fileintegrity.conf

Another subjective one to send or not….For my test I’ll disable them

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-fileintegrity.conf

/etc/vmware-syslog/vmware-services-applmgmt-upgrade.conf

For my test I’ll disable these

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-applmgmt-upgrade.conf

/etc/vmware-syslog/vmware-services-content-library.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-content-library.conf

/etc/vmware-syslog/vmware-services-vpxd-svcs.conf

✔️ KEEP

/storage/log/vmware/vpxd-svcs/authz-event.log
/storage/log/vmware/vpxd-svcs/vpxd-svcs-access*.log

DISABLE

/storage/log/vmware/vpxd-svcs/vpxd-svcs.log
/storage/log/vmware/vpxd-svcs/startup-error.log
/storage/log/vmware/vpxd-svcs/vpxd-svcs-runtime.log.stdout
/storage/log/vmware/vpxd-svcs/vpxd-svcs-runtime.log.stderr
/storage/log/vmware/vpxd-svcs/perf.log

/etc/vmware-syslog/vmware-services-vsphere-ui-imlegit.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vsphere-ui-imlegit.conf

/etc/vmware-syslog/vmware-services-vdtc.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vdtc.conf

/etc/vmware-syslog/vmware-services-cis-license.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-cis-license.conf

/etc/vmware-syslog/vmware-services-perfcharts.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-perfcharts.conf

/etc/vmware-syslog/vmware-services-applmgmt.conf

KEEP

/storage/log/vmware/applmgmt/applmgmt.log
/storage/log/vmware/applmgmt-audit/applmgmt-audit.log
/storage/log/vmware/applmgmt-audit/applmgmt-br-audit.log
/opt/vmware/var/log/lighttpd/access.log
/opt/vmware/var/log/lighttpd/error.log
/storage/log/vmware/applmgmt/vami.log
/storage/log/vmware/applmgmt/backup.log
/storage/log/vmware/applmgmt/restore.log
/storage/log/vmware/applmgmt/pnid_change.log

DISABLE

/storage/log/vmware/applmgmt/dcui.log
/storage/log/vmware/applmgmt/detwist.log
/storage/log/vmware/applmgmt/firewall-reload.log
/storage/log/vmware/applmgmt/applmgmt_vmonsvc.std*
/storage/log/vmware/applmgmt/backupSchedulerCron.log
/storage/log/vmware/applmgmt/progress.log
/storage/log/vmware/applmgmt/statsmoitor-alarms.log
/storage/log/vmware/applmgmt/StatsMonitor-*.log
/storage/log/vmware/applmgmt/StatsMonitorStartup.log.std*
/storage/log/vmware/applmgmt/PatchRunner.log
/storage/log/vmware/applmgmt/update_microservice.log
/storage/log/vmware/applmgmt/vcdb_pre_patch.*
/storage/log/vmware/dnsmasq.log
/storage/log/vmware/procstate
/storage/log/vmware/applmgmt/size.log
/storage/log/vmware/applmgmt/reconciliation.log

/etc/vmware-syslog/vmware-services-lookupsvc.conf

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-lookupsvc.conf

/etc/vmware-syslog/vmware-services-rhttpproxy.conf

Keepin these.

TLDR

sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-sps.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-analytics.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vcha.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vmon.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vstats.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-certmgmt.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-eam.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vapi.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-observability.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-cloudvm.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vlcm.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-pod.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-trustmanagement.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vmware-vpostgres.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-updatemgr.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-fileintegrity.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-applmgmt-upgrade.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-sca.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-content-library.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-vdtc.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-cis-license.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-perfcharts.conf
sed -i 's/^/#/' /etc/vmware-syslog/vmware-services-lookupsvc.conf

Testing looking at logs for a VM I created one, checked the logs, yup there it is.. delete it.. uhh where is it (searched by VM name)

🧩 vCenter VM Creation vs. Deletion Log Behavior — Summary

✅ VM Creation; Always logged clearly in vpxd.log

Always includes the VM name

Easy to find by searching for the VM name

Example:

Code
CreateVM_Task: Creating VM ‘MyCustomVM’

❌ VM Deletion; Deletion logs are not symmetrical with creation logs.

Key facts:
Deletion logs often do NOT include the VM name
Instead, they use the MoRef ID (e.g., vm-1234)
Searching by VM name will NOT find the deletion

The deletion may appear as:
vim.ManagedEntity.destroy
Destroy_Task
Unregister
Remove from inventory

Example:

vim.ManagedEntity.destroy invoked for vm-1234

No name. That’s why you didn’t see it.

⭐ Why the name is missing

When vCenter deletes a VM:
It removes the VM object from inventory
Then logs the destroy event
The name is already gone, so the log can’t include it

This is normal (and frustrating) vCenter behavior.

🔍 How to reliably find deletion events

Use the MoRef ID, not the VM name.
Get the MoRef from the creation log:

grep -Ri “vm-1234” /storage/log/vmware/vpxd/

You’ll see the deletion entry immediately.

Summary

What a royal PITA it is to manage sysloging on vCenter… :S