Proxmox Cluster Findings

When I set up my new MSI PVE host, I decided to use the “Golden Blueprint” for storage: installing the base Proxmox OS on a cheap SATA SSD to shield it from heavy log writes, leaving my lightning-fast NVMe drive completely clean and unpartitioned as a dedicated LVM-Thin block storage pool for my VMs and containers.

But things got complicated when I tried to link my new host to my old host. Here is what I learned while wrestling with Proxmox cluster logic, VMware design differences, and hardcoded network traps.

1. The Populated Host Conundrum

My initial plan was straightforward: create a fresh cluster on my pristine new host, and then have my old host (which currently runs all my active workloads) join it. I quickly hit a hard wall. Unlike VMware vCenter—which effortlessly imports populated ESXi hosts and reindexes VM identifiers on the fly—Proxmox operates on a decentralized, shared-filesystem architecture (pmxcfs).

Because Proxmox uses hardcoded VMIDs (like 100, 101) to define physical storage paths and configuration maps, a host cannot join an existing cluster if it has any virtual machines on it. Joining a cluster completely overwrites the local /etc/pve directory, which would instantly orphan any existing virtual disks.

The Fix: I flipped the workflow. I created the cluster directly on my old host first. Because it initialized the cluster, its existing VMs remained perfectly safe.

2. Leaving a Cluster Requires the CLI

Before I figured out the correct order of operations, I had already initialized a test cluster on my new node. When I went to undo it, I discovered there is no “Delete Cluster” button in the Proxmox Web GUI. Because tearing down a cluster can cause severe data corruption if done incorrectly, Proxmox forces you into the shell.

To completely reset my new node back to standalone mode without reinstalling the entire OS, I had to open the terminal and force-clear the configuration database using these steps:

# Stop cluster synchronization services
systemctl stop pve-cluster
systemctl stop corosync

# Force the configuration filesystem into a temporary local mode
pmxcfs -l

# Permanently erase the old cluster registries
rm -f /etc/pve/corosync.conf
rm -rf /etc/corosync/*

# Kill the background lock process and restart standalone services
killall pmxcfs
systemctl start pve-cluster

3. The Hardcoded IP/VLAN Trap

While running pmxcfs -l, I noticed the terminal spat out an old IP address (192.168.0.68) that I had previously removed when migrating the host to a new VLAN-tagged management network.

I realized that Proxmox doesn’t query active network interfaces to resolve its own name; it relies entirely on a static registry. The old IP address was still hardcoded inside my /etc/hosts file. I used nano /etc/hosts to update the line to my new VLAN IP (172.16.21.20) so the host could resolve itself properly. (Blog post covering IP change updated)

4. Breaking the Handshake Hang

When I finally went to join my clean new node (172.16.21.20) to the old node’s cluster (172.16.21.60), the Web GUI installer completely hung on the message: “Request addition of this node.”

However, in classic homelab fashion, a simple web browser refresh cleared the stalled API cache, the handshake successfully completed on its own, and both nodes cleanly populated into a single sidebar.

5. Local Disks Show Up on the Wrong Nodes

Right after my two nodes successfully clustered, I noticed something deeply alarming in the sidebar UI: my old host’s massive local storage drive (mass-storage) was suddenly showing up underneath my shiny new msi-pve node.

I knew for a fact it wasn’t shared storage—there were no NFS, SMB, or iSCSI links connecting them. The drive was physically plugged into the old hardware. So why was my new host pretending it owned it?

The Cause: Global Cluster Definitions

This is one of Proxmox’s quirky design traits. Proxmox stores every single storage path inside a single, cluster-wide configuration file (/etc/pve/storage.cfg). The moment my new node joined the cluster, it downloaded this file and blindly copied the layout.

By default, Proxmox assumes any storage listed in that file is accessible by all nodes unless you explicitly state otherwise. It draws the icon under every host in the sidebar, creating a dangerous phantom placeholder. If I had tried to spin up a VM on my new node and targeted that ghost storage, the deployment would have crashed instantly with activation errors.

Luckily the fix, correcting this is incredibly simple and doesn’t require the command line:

1. I clicked on Datacenter at the very top of the sidebar.
2. I went to Storage, highlighted the phantom mass-storage pool, and clicked Edit.
3. I found the Nodes dropdown—which was completely blank (Proxmox-speak for “Allow All Nodes”).
4. I changed it to explicitly select only my old host (g9-pve) and saved it.

The second I clicked OK, the phantom icon vanished from underneath my new node, keeping my environment clean and preventing any catastrophic accidental deployments.

6. Migration Failure due to VMs bound virtual NIC

Coming from VMware, hitting a hard wall during a migration because of a network name mismatch feels entirely unnecessary. In VMware—even on standard vSwitches without distributed virtual switching—the migration wizard natively handles network remapping. If a target host doesn’t have a matching network, the wizard stops and lets you choose a new path on the fly.

This highlights a fundamental architectural difference in how these two hypervisors handle networking:

VMware uses Port Groups (VMPGs): This creates a layer of abstraction. The VM connects to a named Port Group, and that group handles the VLAN tagging down at the vSwitch level.
Proxmox uses Direct Bridging: By default, Proxmox expects you to point the VM’s virtual NIC directly at a specific host bridge (like vmbr1) and explicitly type the VLAN tag directly into the VM’s network device settings.

Because Proxmox binds the VM directly to a specific host bridge string rather than an abstract port group, its migration wizard is completely rigid. If the destination host doesn’t have an identically named bridge, the migration fails with an error instead of letting you remap it during the transfer. To make migrations seamless, you have no choice but to ensure your Linux bridge names match exactly across every single node in your cluster.

7. Migrating an LXC

With a container healthy, I attempted to clone it over to the new host’s NVMe drive. The Proxmox UI threw a sudden error:

Full clone of a running container is only possible from a snapshot (500)

Because containers share live kernel space, they cannot be live-cloned without a frozen snapshot boundary. I attempted to take a snapshot, only to hit a secondary brick wall:

The current guest configuration does not support taking new snapshots

This error points directly to the underlying physical storage. While advanced file systems like ZFS and LVM-Thin support snapshots out of the box, traditional file-based directory storages and thick LVM volume groups do not. I couldn’t take a snapshot, which meant I couldn’t live-clone. A cold migration was my only path forward, or so I thought…

Overriding the Migration Wizard with Manual Backup Streams

I shut the container down and clicked Migrate, but Proxmox aborted yet again:

ERROR: migration aborted: storage 'Mass-Storage' is not available on node 'msi-pve'

When dealing with local storage migration, the standard Proxmox wizard stubbornly expects the exact same storage pool name to exist on the target host. Because my old host’s 4TB drive was named Mass-Storage, and my new host only possessed an unmapped LVM-Thin layout, the handshake failed.

To bypass the rigid migration UI entirely, I resorted to a manual Secure Copy (SCP) backup stream. First, I shut down the source container on g9-pve and took a standard uncompressed cold backup file to my local directory. Then, because LVM block devices don’t show up via standard df -h folders, I targeted Proxmox’s universal root backup directory (/var/lib/vz/dump/) and pushed the data over the network via SCP:

bash

scp /mnt/pve/mass-storage/dump/vzdump-lxc-105-2026_09_21-19_57_35.tar.zst root@172.16.21.20:/var/lib/vz/dump/

The Final Handshake

Once the transfer finished, the backup file appeared beautifully under the new host’s local storage tab in the Web UI. I hit Restore, pointed the destination target directly into the brand-new 1TB NVMe LVM-Thin pool, and let it extract.

To guarantee no IP collisions occurred on my network while validating the data, I went into the original source container on g9-pve and checked the “Disconnected” flag on its network interface. With the original container safely blinded, I fired up the newly restored container on msi-pve. I adjusted its hardware bridge settings from vmbr1 to the new host’s active vmbr0 bridge, and the pings immediately started flowing.

The service came up perfectly healthy, allowing me to finally go back and permanently shut down the original source container on the old hardware.

8. Proxmox “Cluster Join Aborted – No Quorum” Errors

When adding a new node to an existing Proxmox VE cluster, you might encounter this frustrating terminal error:
An error occurred on the cluster node: cluster not ready - no quorum? TASK ERROR: Cluster join aborted!

Here is exactly why this happens and how to bypass it safely.

🔍 The Root Cause

Proxmox requires a strict majority (>50%) of voting nodes to be online and communicating via Corosync to allow administrative changes.

- If a node in your existing cluster is offline, powered down, or network-isolated, the remaining online nodes lose quorum.
- To prevent accidental data corruption, Proxmox drops into a protective, read-only state and blocks any new nodes from joining.

🛠️ The Fix: Bypassing Quorum

If you cannot easily boot the offline node back up, you can manually force the surviving node to lower its expected vote count.

Step 1: Force Expected Votes

SSH into your active cluster node and run the following command to temporarily lower the expected vote threshold to 1:

pvecm expected 1

This instantly restores cluster write-privileges to the remaining online node.

Step 2: Join the New Node

Return to your new node (via CLI or the Web GUI) and re-run the cluster join process. The join operation will now succeed.

Step 3: Boot the Offline Node

Once the new node has safely integrated into the cluster, power your original offline node back on. It will automatically check in with the rest of the network, detect the new node, and sync the updated cluster configuration seamlessly.

⚠️ Important Safety Checklist

Before running pvecm expected 1, always verify your storage layout:

Shared Storage (Ceph, NFS, iSCSI): Ensure the offline node is completely powered off. If it boots up isolated, it could cause a “split-brain” scenario and corrupt shared virtual disks.
Local Storage Only: If your cluster strictly uses local disks, the risk is near zero. The offline node cannot physically touch or corrupt data on the other hosts.

If you do Bring back your old host and didn’t remove it from the cluster, it may not be happy with you and remain in its own island cluster and not play nice with your old hosts…

9. Proxmox Cluster Split-Brain and Corosync Version Mismatch

When a Proxmox VE node is taken offline and changes are made to the remaining cluster (such as adding a new host), bringing that offline node back online can trigger a silent cluster failure. The node will refuse to rejoin, and the Proxmox Web UI will show the hosts as completely disconnected from one another.

Here is a breakdown of why this happens and how to safely force the node back into the cluster.

The Problem: The “Chicken and Egg” Loop

Proxmox relies on two core components to maintain a cluster:

1. Corosync: The underlying network messaging engine that handles cluster quorum and node handshakes.
2. pmxcfs (Proxmox Cluster File System): The distributed filesystem that mirrors configuration files (like /etc/pve/corosync.conf) across all nodes.

This creates a dependency loop: pmxcfs needs Corosync to sync files across the network, but Corosync needs to read pmxcfs to get its startup configuration.

If a node is offline when the active cluster undergoes an update, the cluster increments its configuration version (e.g., from config_version: 2 to 3). When the offline node boots back up, a security mechanism triggers:

1. Corosync reads its local file and sees Version 2.
2. It reaches across the network and hears Version 3 from the active cluster.
3. To prevent a “split-brain” scenario (where two isolated parts of a cluster might blindly overwrite each other’s storage configs), Corosync panics and immediately exits with the error:
  [CMAP] Received config version (3) is different than my config version (2)! Exiting

Because Corosync crashes, the network pipe never initializes. Because the pipe never initializes, the node can never automatically download the updated file from the healthy cluster.

The Fix: Trick Corosync to Open the Pipe

To resolve this, you must force the local node’s configuration filesystem into read/write mode, manually spoof the configuration version number to match the cluster, and restart the services.

Step 1: Kill Zombie Processes & Force Local Mode

Because the cluster is offline, Proxmox locks /etc/pve into a read-only state. Clear out any stuck systemd loops and force it to run locally:

# Kill any hung cluster processes holding filesystem locks
killall -9 pmxcfs corosync pve-cluster

# Stop the systemd tracking services
systemctl stop pve-cluster corosync

# Force the cluster filesystem to run in independent local mode
pmxcfs -l

Step 2: Manually Update the Config Version

With the filesystem unlocked, edit the configuration file to match the version expected by the active cluster:

nano /etc/pve/corosync.conf

- locate the config_version: line in the file.
- Change the number to match the cluster’s current version (e.g., change 2 to 3).
- Save and exit (Ctrl+O, Enter, Ctrl+X).

Step 3: Restart Services

Now that the version numbers match on paper, kill your manual filesystem override and let systemd start the cluster services normally:

# Clear the manual local mode instance
killall -9 pmxcfs

# Start the cluster services sequentially
systemctl start pve-cluster
systemctl start corosync

The Result: The True Config Wins

By manually matching the version numbers, Corosync is tricked into thinking the nodes are synchronized, allowing it to safely open the network tunnel.

The moment the connection stabilizes, Proxmox’s internal generation rules take over. It recognizes that the active cluster holds the true, updated configuration file. The cluster immediately pushes the complete configuration down to the returning node, gracefully overwriting your temporary edit and cleanly reintegrating the host.

You can verify the fix directly from the CLI:

pvecm status

If the above didn’t work cause your host was offline when you brought this one back online, and that host already has the same version number, but get connection issues in the GUI check out the next section.

10. Wiping Stale State and Bypassing SSL Mismatches

When a Proxmox VE cluster shifts structurally while a specific node is offline (such as adding new nodes or changing quorum dynamics), that offline node can fall too far out of sync. Even if configuration version numbers match, the node may completely fail to authenticate with the live cluster due to outdated cryptographic tokens or security states.

Here is how to safely wipe a node’s stale cluster data, preserve existing virtual machines, and force-join it back into production using an SSH fallback.

The Problem: Authentication Locks & The SSL 500 Error

When a node falls severely behind the cluster’s state, trying to rejoin it often leads to a dead end. To fix this, you must first reset the node back into a standalone host by purging its old configuration database files:

# Stop the stuck cluster loops
systemctl stop pve-cluster corosync
killall -9 pmxcfs corosync

# Force the configuration filesystem to open up locally
pmxcfs -l

# Purge the outdated corosync database files entirely
rm /etc/pve/corosync.conf
rm -rf /etc/corosync/*
rm -rf /var/lib/corosync/*

# Restart the local filesystem daemon as a clean standalone service
killall -9 pmxcfs
systemctl start pve-cluster

Once the node is wiped and returned to a standalone state, running a standard pvecm add [CLUSTER_IP] often triggers two scary security blocks:

The Guest Warning: * this host already contains virtual guests.
The SSL Failure: 500 Can't connect to [IP]:8006 (hostname verification failed).

Understanding the SSL 500 Error

By default, Proxmox attempts to execute cluster joins using its secure HTTPS API on port 8006. The target node serves an SSL certificate that matches its actual hostname (e.g., g9-pve), but the joining node is hitting it via a raw IP address. Because the IP address does not match the name listed on the security certificate, the API strictly rejects the connection and throws the 500 error.

The Solution: Forcing and Bypassing via SSH

To safely resolve both issues, use the following advanced join command:

pvecm add [IP of neigbouring host] --force --use_ssh

Why This Works:

The --force Flag: This tells Proxmox to intentionally ignore the virtual guest warning. Your VM’s raw virtual disks sitting on local storage are 100% safe and are never modified or deleted. The flag simply allows Proxmox to safely merge the node’s local configuration directory into the cluster’s unified database.
The --use_ssh Flag: This is the magic bullet for the certificate error. It instructs Proxmox to completely bypass the strict HTTPS API verification loop. Instead, it creates a secure tunnel over standard SSH to pull the fresh cluster schema, synchronize cryptographic keys, and launch Corosync cleanly.

The Result

The joining host will pull the clean, master configuration down, automatically register itself into the cluster map, and instantly turn green across your unified Proxmox dashboard.

Your existing VMs will reappear right where they belong under the node’s localized tree in the Web UI, fully integrated and ready to manage.