{"id":1859,"date":"2026-09-21T17:20:18","date_gmt":"2026-09-21T22:20:18","guid":{"rendered":"https:\/\/zewwy.ca\/?p=1859"},"modified":"2026-10-03T22:40:13","modified_gmt":"2026-10-04T03:40:13","slug":"proxmox-cluster-findings","status":"publish","type":"post","link":"https:\/\/zewwy.ca\/index.php\/2026\/09\/21\/proxmox-cluster-findings\/","title":{"rendered":"Proxmox Cluster Findings"},"content":{"rendered":"

<\/span>Proxmox Cluster Findings<\/span><\/h1>\n
When I set up my new MSI PVE host, I decided to use the “Golden Blueprint” for storage: installing the base Proxmox OS on a cheap SATA SSD to shield it from heavy log writes, leaving my lightning-fast NVMe drive completely clean and unpartitioned as a dedicated LVM-Thin block storage pool for my VMs and containers.<\/div>\n
But things got complicated when I tried to link my new host to my old host. Here is what I learned while wrestling with Proxmox cluster logic, VMware design differences, and hardcoded network traps.<\/div>\n
<\/div>\n

<\/span>1. The Populated Host Conundrum<\/span><\/h2>\n
My initial plan was straightforward: create a fresh cluster on my pristine new host, and then have my old host (which currently runs all my active workloads) join it. I quickly hit a hard wall. Unlike VMware vCenter\u2014which effortlessly imports populated ESXi hosts and reindexes VM identifiers on the fly\u2014Proxmox operates on a decentralized, shared-filesystem architecture (pmxcfs<\/code>).<\/div>\n
<\/div>\n
Because Proxmox uses hardcoded VMIDs (like 100<\/code>, 101<\/code>) to define physical storage paths and configuration maps, a host cannot join an existing cluster if it has any virtual machines on it. Joining a cluster completely overwrites the local \/etc\/pve<\/code> directory, which would instantly orphan any existing virtual disks.<\/div>\n
<\/div>\n
The Fix:<\/strong> I flipped the workflow. I created the cluster directly on my old<\/em> host first. Because it initialized the cluster, its existing VMs remained perfectly safe.<\/div>\n
<\/div>\n

<\/span>2. Leaving a Cluster Requires the CLI<\/span><\/h2>\n
Before I figured out the correct order of operations, I had already initialized a test cluster on my new node. When I went to undo it, I discovered there is no “Delete Cluster” button in the Proxmox Web GUI. Because tearing down a cluster can cause severe data corruption if done incorrectly, Proxmox forces you into the shell.<\/div>\n
<\/div>\n
To completely reset my new node back to standalone mode without reinstalling the entire OS, I had to open the terminal and force-clear the configuration database using these steps:<\/div>\n
<\/div>\n
\n
\n
\n
\n
<\/div>\n<\/div>\n
\n
# Stop cluster synchronization services<\/span>\r\nsystemctl stop pve-cluster\r\nsystemctl stop corosync\r\n\r\n<\/span># Force the configuration filesystem into a temporary local mode<\/span>\r\npmxcfs -l\r\n\r\n<\/span># Permanently erase the old cluster registries<\/span>\r\nrm -f \/etc\/pve\/corosync.conf\r\nrm -rf \/etc\/corosync\/*\r\n\r\n<\/span># Kill the background lock process and restart standalone services<\/span>\r\nkillall pmxcfs\r\nsystemctl start pve-cluster\r\n<\/span><\/code><\/pre>\n<\/div>\n<\/div>\n
\n
<\/span>3. The Hardcoded IP\/VLAN Trap<\/span><\/h2>\n<\/div>\n<\/div>\n<\/div>\n
While running pmxcfs -l<\/code>, I noticed the terminal spat out an old IP address (192.168.0.68<\/code>) that I had previously removed when migrating the host to a new VLAN-tagged management network.<\/div>\n
<\/div>\n
I realized that Proxmox doesn’t query active network interfaces to resolve its own name; it relies entirely on a static registry. The old IP address was still hardcoded inside my \/etc\/hosts<\/code> file. I used nano \/etc\/hosts<\/code> to update the line to my new VLAN IP (172.16.21.20<\/code>) so the host could resolve itself properly. (Blog post covering IP change updated)<\/div>\n
<\/div>\n
<\/span>4. Breaking the Handshake Hang<\/span><\/h2>\n
When I finally went to join my clean new node (172.16.21.20<\/code>) to the old node’s cluster (172.16.21.60<\/code>), the Web GUI installer completely hung on the message: \u201cRequest addition of this node.\u201d<\/em><\/div>\n
<\/div>\n
However, in classic homelab fashion, a simple web browser refresh cleared the stalled API cache, the handshake successfully completed on its own, and both nodes cleanly populated into a single sidebar.<\/div>\n
<\/div>\n
\n
<\/span>5. Local Disks Show Up on the Wrong Nodes<\/span><\/h2>\n
Right after my two nodes successfully clustered, I noticed something deeply alarming in the sidebar UI: my old host\u2019s massive local storage drive (mass-storage<\/code><\/strong>) was suddenly showing up underneath my shiny new msi-pve<\/code><\/strong> node.<\/div>\n
<\/div>\n
I knew for a fact it wasn\u2019t shared storage\u2014there were no NFS, SMB, or iSCSI links connecting them. The drive was physically plugged into the old hardware. So why was my new host pretending it owned it?<\/div>\n
<\/div>\n
The Cause: Global Cluster Definitions<\/div>\n
This is one of Proxmox\u2019s quirky design traits. Proxmox stores every single storage path inside a single, cluster-wide configuration file (\/etc\/pve\/storage.cfg<\/code>). The moment my new node joined the cluster, it downloaded this file and blindly copied the layout.<\/div>\n
<\/div>\n
By default, Proxmox assumes any storage listed in that file is accessible by all nodes<\/strong> unless you explicitly state otherwise. It draws the icon under every host in the sidebar, creating a dangerous phantom placeholder. If I had tried to spin up a VM on my new node and targeted that ghost storage, the deployment would have crashed instantly with activation errors.<\/div>\n
<\/div>\n
Luckily the fix, correcting this is incredibly simple and doesn’t require the command line:<\/div>\n
    \n
  1. \n
      \n
    1. I clicked on Datacenter<\/strong> at the very top of the sidebar.<\/span><\/li>\n
    2. I went to Storage<\/strong>, highlighted the phantom mass-storage <\/code><\/strong><\/span>pool, and clicked Edit<\/strong>.<\/span><\/li>\n
    3. I found the Nodes<\/strong> dropdown\u2014which was completely blank (Proxmox-speak for “Allow All Nodes”).<\/span><\/li>\n
    4. I changed it to explicitly select only my old host (g9-pve<\/code>)<\/strong> and saved it.<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
      The second I clicked OK, the phantom icon vanished from underneath my new node, keeping my environment clean and preventing any catastrophic accidental deployments.<\/div>\n<\/div>\n
      <\/span>6. Migration Failure due to VMs bound virtual NIC<\/span><\/h2>\n
      \n
      Coming from VMware, hitting a hard wall during a migration because of a network name mismatch feels entirely unnecessary. In VMware\u2014even on standard vSwitches without distributed virtual switching\u2014the migration wizard natively handles network remapping<\/strong>. If a target host doesn’t have a matching network, the wizard stops and lets you choose a new path on the fly.<\/div>\n
      This highlights a fundamental architectural difference in how these two hypervisors handle networking:<\/div>\n
      <\/div>\n