Blog

Posted on

The Poor Man’s VAAI for ZFS-over-iSCSI

The Poor Man’s VAAI: Orchestrating an Offline ZFS-over-iSCSI Storage Migration Manually

If you run a homelab or an enterprise environment utilizing Proxmox VE (PVE) backed by a TrueNAS storage array via a community ZFS-over-iSCSI plugin (like the popular TheGrandWazoo implementation), you have likely run into the limits of the API abstraction layer.
Try to perform a cross-pool storage migration (moving a virtual hard drive from one ZFS pool to another on the same TrueNAS machine), and you are likely to get slapped with this cryptic failure:
TASK ERROR: storage migration failed: TrueNAS API POST /iscsi/extent: 422 Unprocessable Entity — iscsi_extent_create.name: Extent name must be unique

Here is a deep technical breakdown of why this happens, how the hypervisor configuration misleads you, and how to execute a manual, bare-metal block migration—essentially building a poor man’s offline VAAI (vSphere Storage API for Array Integration) for ZFS over iSCSI.

The Architectural Root Cause

To understand the failure, you have to look at the intersection of three components: TrueNAS’s global iSCSI architecture, the Proxmox plugin layer, and the PVE VM configuration file.

1. TrueNAS Manage iSCSI Globally

TrueNAS maintains a centralized, system-wide configuration database for its iSCSI daemon (ctld or SCST). When an initiator requests a target, TrueNAS maps that connection globally. It enforces strict namespace uniqueness across the entire machine for Extents, Targets, and Associated Targets. TrueNAS does not care if you have two isolated ZFS pools; you cannot register the same extent name twice on the same machine.

2. The Proxmox Plugin Collision

When you define two separate TrueNAS-backed iSCSI datastores in Proxmox, they usually point to the same TrueNAS Host Portal IP.
The storage plugin hardcodes the iSCSI extent naming convention using the VM ID and slot number: vm-ID-disk-X.
When you initiate a storage migration to the other datastore via the Proxmox GUI, the plugin tries to provision the new storage before tearing down the old one. It fires off a POST request to the TrueNAS API to create an extent named vm-108-disk-1 on the destination pool. Because that exact name already exists in TrueNAS’s global registry (pointing to the source pool), TrueNAS immediately rejects it with a 422 Unprocessable Entity.

3. The PVE .conf File Illusion

In an iSCSI architecture, the storage configuration string in Proxmox is just a pointer. It tells Proxmox to talk to the storage daemon associated with that label and map a specific disk ID. Because both Proxmox datastore definitions map to the same target IP portal, Proxmox connects to the exact same network destination. TrueNAS sees the incoming request for vm-108-disk-1 and points the hypervisor right back to the original active iSCSI session on the old pool.

Poor Man’s Offline VAAI: The Manual Fix

In an enterprise environment with a SAN that natively supports VAAI block offloading, a storage migration issues a single command telling the array to clone blocks and re-map the LUN at the hardware layer—consuming zero hypervisor CPU or network bandwidth.
Since the Proxmox plugin lacks the logic to orchestrate this natively across multiple TrueNAS pools, we can act as the orchestration engine ourselves using raw ZFS and iSCSI manipulation.

Step 1: Detach the Disk in Proxmox

To safely move a block storage device without corrupting data, the file systems must be unmounted and the iSCSI lock freed.
  1. Shut down the target VM completely.
  2. In Proxmox, navigate to the VM ➔ Hardware ➔ Select the target drive (e.g., scsi0) ➔ Click Detach.
  3. The drive will drop down to the bottom of the list as an Unused Disk.

Step 2: Backend ZFS Block Migration (TrueNAS CLI)

Instead of forcing data across your network adapter via the hypervisor, leverage native ZFS snapshot streaming directly inside the TrueNAS kernel.
  1. SSH into TrueNAS or open the Web UI Shell.
  2. Escalate privileges to root using sudo.
  3. Take an atomic, read-only snapshot of the source Zvol: 
    sudo zfs snapshot BigBurtha/vm-108-disk-1@move

  4. Stream the raw data blocks directly into the destination pool (LexarPool) using a ZFS send/receive pipe. The -vR flags enable a progress bar and maintain replication properties: 
    sudo zfs send -vR BigBurtha/vm-108-disk-1@move | sudo zfs recv LexarPool/vm-108-disk-1

Step 3: Atomic LUN Mapping Flip (TrueNAS GUI)

Now that the data blocks reside on the new physical disks, you must rewrite the iSCSI pointer inside the TrueNAS database so the global network target routes to the new destination pool.
  1. In the TrueNAS Web UI, navigate to Shares ➔ Block Shares (iSCSI) ➔ Extents.
  2. Find the extent matching your disk (vm-108-disk-1) and click Edit.
  3. Locate the Source / Zvol Path dropdown menu. It will currently point to your old pool path: /dev/zvol/BigBurtha/vm-108-disk-1.
  4. Flip this dropdown to point to the new pool path: /dev/zvol/LexarPool/vm-108-disk-1.
  5. Save the configuration.

Step 4: Re-link and Boot

Go back to the Proxmox Web UI. Select the Unused Disk at the bottom of the VM hardware configuration panel, click Edit/Add, and reassign it back to its original slot (e.g., scsi0).
When you boot the VM, Proxmox will query the iSCSI target portal exactly as it did before. TrueNAS receives the request, references its updated global registry database, and serves the blocks directly out of LexarPool. Run a disk I/O benchmark utility inside the guest OS, and you will see the hardware statistics instantly register activity on the new storage array.

Step 5: Post-Migration Cleanup

Once the VM is verified stable and running on the new pool, the iSCSI service will release its execution locks on the old Zvol path, allowing you to cleanly reclaim your storage capacity.
From the TrueNAS Shell, execute the final destruction commands:
# Delete the migration snapshot from the original pool
sudo zfs destroy BigBurtha/vm-108-disk-1@move

# Wipe the old Zvol block device entirely
sudo zfs destroy BigBurtha/vm-108-disk-1

# Optional: Clean up the temporary snapshot on the new pool
sudo zfs destroy LexarPool/vm-108-disk-1@move

You might be wondering if utilizing zfs snapshot did I need to power down and detach the disk as step 1?
Yes, you are 100% correct. You absolutely could have done the ZFS snapshot and copy task completely live while the VM was running, reducing your actual downtime to just a few seconds.
Taking a ZFS snapshot is an atomic operation at the block layer; it takes less than a second to freeze the blocks, regardless of whether the VM is actively writing data or not.
By restructuring the order of operations, you can turn this into a near-live storage migration with minimal disruption.

The Live Migration Workflow (Zero-Downtime Copy)

Here is how you would optimize the steps for a blog post to achieve an almost seamless migration:

1. The Live ZFS Send/Recv (Zero Downtime)

While the VM is fully booted and handling active I/O, you run the initial snapshot and block transfer. ZFS will copy all the existing data blocks over to LexarPool.
sudo zfs snapshot BigBurtha/vm-108-disk-1@live-migrate
sudo zfs send -vR BigBurtha/vm-108-disk-1@live-migrate | sudo zfs recv LexarPool/vm-108-disk-1

(The VM stays online the entire time this data streams across your pools).

2. The Catch-Up Snapshot (Optional but Recommended)

If the block copy took a long time (e.g., a massive multi-terabyte drive), the VM will have written new data to BigBurtha while the copy was running. To minimize downtime even further, you take a second incremental snapshot to sync just the changes:
sudo zfs snapshot BigBurtha/vm-108-disk-1@sync
sudo zfs send -vR -i BigBurtha/vm-108-disk-1@live-migrate BigBurtha/vm-108-disk-1@sync | sudo zfs recv LexarPool/vm-108-disk-1

3. The “Micro-Downtime” Cutover

This is the only part where you have to stop the VM, because you cannot re-route an active iSCSI session while TrueNAS holds an open kernel lock on the running file system.
  1. Shut down the VM in Proxmox.
  2. Run a final incremental ZFS sync to catch the last few blocks written during the shutdown sequence: 
    sudo zfs snapshot BigBurtha/vm-108-disk-1@final
sudo zfs send -vR -i BigBurtha/vm-108-disk-1@sync BigBurtha/vm-108-disk-1@final | sudo zfs recv LexarPool/vm-108-disk-1

  3. In the TrueNAS Web UI (Shares ➔ Block Shares ➔ Extents), edit the vm-108-disk-1 extent and flip the dropdown path from BigBurtha to LexarPool.
  4. Update your Proxmox .conf file label if you want the storage names to look tidy (though as we discovered, this part is just cosmetic for the iSCSI portal routing).
  5. Boot the VM back up.

Why this is even closer to real VAAI

By doing the heavy lifting (the bulk data transfer) while the VM is fully operational, your total downtime drops from hours (waiting for network copies or local block transfers to finish while the VM is offline) to literally 30 seconds—just long enough to stop the VM, flip the TrueNAS iSCSI mapping dropdown, and hit start again.
This approach makes the “Poor Man’s VAAI” title fit even better, because it mirrors exactly how enterprise storage arrays handle asynchronous storage migration cutovers!

Summary

When software automation plugins fail to scale across architectural boundaries, understanding the underlying storage routing allows you to manipulate data directly at the block layer. By combining raw ZFS snapshot pipes with target remapping, you bypass API namespace limits completely—achieving zero-network-overhead backend replication.
Posted on

Virtualized TrueNAS Crashing

So, remember when I first build a test Frankenstein TrueNAS box to edge out some performance on otherwise shady consumer SSDs?  and then I ran with that Idea to test ZFS over iSCSI using TrueNAS?

Well since I was using a poormans setup, something wasn’t happy (PVE Virtio network stack? TrueNAS stack? I unno but I was using both VLAN segregation… Virtual NIC setup via Virtio, not sure but the host would become unresponsive to any network requests (pings or otherwise) but the host and VM would be otherwise fine (if I went to the console, I could issue commands without problem, but if I restarted the networking service only the host PVE would become responsive, TrueNAS would stay dead I had to reboot the whole host. AI assumes its the saturation, I have my doubts as I throttled the VLANs which didn’t help. But we’ll try anyway, I’ll move the iSCSI NICs to a PCI network card I apparently forgot I already had in this host. via hardware passthrough.. ok Let’s see how this goes…

I didn’t realize this one active VM is still on the ZFS over iSCSI datastore… move it and… host crash, so doing some testing interesting finds:

  1. Just as I stated above, restoring the network stack on the PVE host brings it back up fine, but
  2. Restarting the nework stack or even restarting the TrueNAS VM does not bring it back
  3. I have to reboot the PVE host as a whole
  4. The GUI task viewer, showing the task will stall on the transfer but the hosts stay perfectly stable and running fine, the transfer resumes where it left off after the storage PVE host and TrueNAS VM boot back up (takes roughly 20 seconds for PVE to boot, and 1 minute 30 seconds for TrueNAS) 2 min.
  5. Even when testing had a 6 minute outage window and the transfer resumed just fine when the backend storage came back. Is there a timeout out that checks actual transfer and if its stalled? I don’t know.
  6. The ZFS storage status in the left hand nav doesn’t reflect unknown status till roughly about 5 minutes of the storage becoming unavailable. But showing unknown back to good is almost instant.
The Solution: Hardware Isolation via PCIe Passthrough
The definitive fix is to physically separate the traffic by cutting Proxmox completely out of the storage loop. My motherboard has a spare PCIe slot, added a dedicated network card (a cheap, enterprise-stable multi-port Intel Gigabit adapter) and passed it directly to the storage VM.
Find the dedicated network card’s IOMMU group and add it as a Raw PCI Device to the TrueNAS VM hardware tab in Proxmox. Ensure All Functions and PCI-Express are checked.
This completely bypasses the Proxmox network stack, the virtual Linux bridge (vmbr0), and the VirtIO translation layers. TrueNAS now talks directly to bare-metal Intel hardware using native, highly optimized FreeBSD drivers.
After doing this I haven’t had the hosting PVE server or TrueNAS VM stop responding to pings, or services. It’s a lil less Frankensteiny now.
… or so I thought that separating the iSCSI traffic resolved it, everything far more stable, my usual instant test (Booting Windows VM against the SSD backed ZFS over iSCSI target) was working just fine… So, while I thought this resolved it, I jumped back to my AI chat window to state that this fix didn’t in fact work. I took a snip of my intuitional moment…
Which it then gave some examples of what it could be, I was kinda shocked when one of it’s hunches, was backed by hard evidence in the logs after the fact.. it stated and I quote..
“Clear out the e1000e Driver Bugs on your Management NIC

Your onboard management NIC (00:19.0) uses the Intel Linux e1000e driver, which is notorious for dropping off the network during heavy CPU or PCIe bus loads due to power management and interrupt coalescing.”

then told me to check the log….

journalctl -b -1 -n 200
and low and behold…
Oct 05 21:29:09 desktop-pve kernel: e1000e 0000:00:19.0 nic0: Detected Hardware Unit Hang:
TDH <9c>
TDT <c4>
next_to_use <c4>
next_to_clean <9a>
buffer_info[next_to_clean]:
time_stamp <100904184>
next_to_watch <9c>
jiffies <1009ac040>
next_to_watch.status <0>
MAC Status <80083>
PHY Status <796d>
PHY 1000BASE-T Status <7800>
PHY Extended Status <3000>
PCI Status <10>
Oct 05 21:29:10 desktop-pve systemd[1]: pve-ha-crm.service: Deactivated successfully.
Oct 05 21:29:10 desktop-pve systemd[1]: Stopped pve-ha-crm.service - PVE Cluster HA Resource Manager Daemon.

Ain’t that but a bitch…. and it turns out this is a known problem, now only was the first AI source noting this, here’s a PVE thread started in 2019 still on going, this guy exact same signs and symptoms from April this year, this Garret guy who for some reason used AI to created a service script that is way over kill when you can just disable the offloading feature via a simple edit to the interfaces file…just add

post-up ethtool -K {interface} tso off

under the specific interface you want to disable. verify with:

ethtool -k nic0 | grep tcp-segmentation-offload
tcp-segmentation-offload: off

This time its fixed…. 11PM at night, I swear… fucking tech man…

Posted on

Least-Privilege ZFS over iSCSI for Proxmox VE via TrueNAS

How I Configured Least-Privilege ZFS over iSCSI for Proxmox VE via TrueNAS

The built-in Proxmox VE “ZFS over iSCSI” storage provider relies on direct root SSH access to TrueNAS to dynamically spin up block devices. However, this legacy method is broken on modern TrueNAS SCALE versions (like Dragonfish and Electric Eel) because TrueNAS no longer uses the target engines (like Comstar/LIO) expected by PVE.
To solve this securely without enabling risky root SSH access, I installed the “official TrueNAS Proxmox VE Storage Plugin” aka plugin managed by “TheGreatWazoo”, worked around network anomalies, and configured a custom Role-Based Access Control (RBAC) policy for a true least-privilege service account.

Step 1: Setup TrueNAS

Virtualizing TrueNAS: CPU NUMA and Memory Logistics
    • Bypassing the CPU NUMA Trap: Since my TrueNAS virtual machine is small enough to fit comfortably inside a single physical CPU socket and its local memory pool, I left virtual NUMA (vNUMA) disabled in my hypervisor. Enabling it on a small VM would force the guest OS to waste processing cycles managing artificial boundaries. It would also restrict hypervisor scheduling, pinning my storage VM to specific sockets even if other sockets are completely idle. Leaving it disabled allows the hypervisor to handle optimization automatically via UMA (Uniform Memory Access) mode.
    • Fixing the RAM Allocation: TrueNAS was initially starved at a default 2048 MiB (2 GB), which is an instant recipe for an Out-Of-Memory (OOM) kernel crash under iSCSI loads. I scaled the allocation up to 8192 MiB (8 GB) (with a ceiling of 12288 MiB / 12 GB if needed). This provides enough overhead for the base OS middleware and the 4TB ZFS metadata.
    • Disabling Memory Ballooning: Because ZFS calculates its cache (ARC) based on a fixed expectation of available memory, I disabled the ballooning device and set fixed memory. If the host tried to dynamically reclaim RAM mid-test, the TrueNAS kernel would instantly panic and drop my storage network offline.

After successfully installing TrueNAS, I presses 1 at the console to configure the interface.

I noticed I could only toggle DHCPv4 and DHCPv6 on or off, but the actual IP address input field seemed completely missing.
Here is how I finally managed to configure my static IP:
    • I navigated down to the Aliases field. In TrueNAS SCALE, static IPs are assigned as aliases rather than a traditional static IP entry field.
    • I selected the empty list. I used my arrow keys to highlight aliases: <empty list> and pressed Enter to open the edit sub-menu.
    • I entered my IP with CIDR notation. I typed my desired address and subnet mask combined (for example, 192.168.1.100/24) instead of typing the netmask separately.
    • I saved the changes. I scrolled down to the <Save> button at the bottom of the screen and pressed Enter to apply the network settings.

Only then I could go to configure network settings and define the gateway and DNS servers. lil weird but whatever, easy enough.

Then just like before HBA to passthrough the hard drives to the VM

Just like before, created 2 isolated networks (one mgmt plane (HTTPS/SSH) and one data plane (iSCSI). *Note* I’m using VLANs (on the hypervisor for the VM guest) to simulate what you normally would do physically. So the configuration on TrueNAS looks the exact same as physical (two dedicated NICs for the two planes) AND… MPIO will not be covered in the scope of this blog. I’ll have to cover that in the future when I get some hosts with multiple NICs.

Step 2: TrueNAS Service Account & API Configuration

To avoid assigning global administrative rights, I built a dedicated service user and a custom privilege profile to authorize specific storage metadata calls.
1. Account Creation Constraints
I navigated to Credentials > Users to create my service account container:
    • Username: pve-iscsi
    • TrueNAS Access: Checked the box and selected Sharing Administrator rather than a generic read-only or full admin profile.

      • Create the account, Give it a Name, Check off TrueNAS Access, select read-only admin.
    • Shell/SSH Access: Kept unchecked. The plugin communicates purely via HTTPS REST API calls, so command-line execution is unnecessary.

🛠️ Caveat & Tweak: This creates a less restrictive account, it also creates a group with the same name as the account. We will be creating a custom “Privilege” Role with only the required “Roles” permissions, then bind that role to this group to complete the least privilege configuration.
User:
Group:
As you can see the group created here shows each permission already defined on this account (snippet here is after the needful was done), but you need to create the actual Role by clicking on the Privileges button next to add. Create a “Privilege” and name it after this setup.
Under “Roles” aka Permissions I granted this “Privilege”, what I call a Role,
1. Sharing (iSCSI Track)
    • Sharing iSCSI Extent Read
    • Sharing iSCSI Extent Write
    • Sharing iSCSI Initiator Read
    • Sharing iSCSI Initiator Write
    • Sharing iSCSI Portal Read
    • Sharing iSCSI Portal Write
    • Sharing iSCSI Target Read
    • Sharing iSCSI Target Write
    • Sharing iSCSI Target Extent Read
    • Sharing iSCSI Target Extent Write

2. Storage & Snapshots (ZFS Track)
    • Dataset Read
    • Dataset Write
    • Dataset Delete (Crucial so Proxmox can delete VM disks)
    • Snapshot Read
    • Snapshot Write
    • Snapshot Delete (Crucial so Proxmox can clear old snapshots/backups)
    • Pool Read (Allows the API to safely query the storage layout tree)

3. System Base Discovery
    • System General Read (Allows the API to check system info and uptime metrics)

4. The Breakthrough Secret (Filesystem Track)
    • Filesystem Read
    • Filesystem Write

(Note: Keeping Allow All Sudo Commands and Sudo without Password unchecked on the user account ensures this remains a strict least-privilege setup.)
As you can see in the snip once you bind this custom role the GUI only supports the built in “Privileges”, so if you edit this account it’ll show the TrueNAS Access checked off with the dropdown empty:

Step 3. Generating the API Token as Admin

Instead of logging out and accessing the UI under the new service account context, I provisioned the token from my master administrator account:
    1. Navigated to Credentials > Users and clicked on my new pve-iscsi user row.
    2. Inside the profile card’s Access widget, I clicked Add API Key.
    3. Named the token (e.g., PVE-iSCSI-Token) and copied the generated string immediately.

Step 4. Preparing the TrueNAS iSCSI Network Targets

Before connecting the hypervisors, I gathered configuration parameters and eased network access limits under Apps/Services > iSCSI.
    1. Bound iSCSI: Ensure iSCSI Service is bound to iSCSI NIC only.
    2. The Portal ID: Verified my active iSCSI listening group ID under the Portals tab (typically ID 1).
    3. The Target IQN: Copied my system base name string under Target Global Configuration (iqn.2005-10.org.freenas.ctl).
    4. The Initiator Workaround: Under the Initiators tab, I enabled Allow All Initiators and saved it as Initiator ID 1.

💡 Network Tweak: In a trusted home lab or private storage network, allowing all initiators simplifies initial setup. For strict isolation, this can be unchecked later by pasting the unique IQN strings of each individual Proxmox node directly into the allowed targets box.

Step 5. Driver Installation on Proxmox VE Nodes

Because storage modules run at the system level rather than the cluster level, I executed these steps directly on every node in my cluster via SSH/Shell.
# 0.  Firewall:
I had to configure a firewall rule to allow access to the DEVs githubsource.
URL: thegrandwazoo.github.io/
Application: Github-Pages
# 1. Download and register the official repository GPG key:
curl -fsSL https://thegrandwazoo.github.io/freenas-proxmox/public.gpg.key | gpg --dearmor | tee /etc/apt/keyrings/truenas-proxmox.gpg > /dev/null
# 2. Inject the driver repository metadata block using the DEB822 standard format:
cat << 'SOURCES' | tee /etc/apt/sources.list.d/truenas-proxmox.sources
Types: deb
URIs: https://thegrandwazoo.github.io/freenas-proxmox
Suites: v3
Components: main
Signed-By: /etc/apt/keyrings/truenas-proxmox.gpg
SOURCES
# 3. Synchronize package databases and install the kernel driver plugin:
apt update
apt install truenas-proxmox -y
Installing this package successfully registers the truenas-iscsi framework and automatically recycles local management services (pvedaemon, pveproxy) to render the integration options in the UI.

Step 6. Mounting Storage and Navigating GUI Flaws

I refreshed my Proxmox browser interface, navigated to Datacenter > Storage > Add, and selected the newly exposed TrueNAS iSCSI storage provider.
The Connection Profile Layout
    • ID: TrueNAS-ZFS (or any friendly cluster name)
    • Nodes: Set to All
    • TrueNAS Host: 172.16.21.2 (Management network interface)
    • Portal IP: 69.69.69.1 (The isolated network interface bound to the iSCSI portal)
    • Pool: LexarPool (Case-sensitive master storage pool name)
    • API Token: [Pasted the custom service token generated in Phase 1]

Hitting Add automatically builds the connection link, turning my cluster resource status icon green.

Core Architecture Takeaway: The Proxmox Offline Migration “Flaw”

During testing, I stumbled on a major counterintuitive design choice in how Proxmox handles moving virtual machine storage: Proxmox allows you to move local storage blocks to another node effortlessly if the VM is powered ON, but throws a hard block if the VM is powered OFF.

Why this happens under the hood:
    • Online Migration (VM ON): Proxmox delegates operations to the QEMU/KVM virtualization engine. QEMU handles the network stream natively, actively duplicating running disk blocks to an entirely different storage name on the destination node on the fly.
    • Offline Migration (VM OFF): Proxmox bypasses QEMU and drops back to its legacy internal cluster scripts. This simple script reads the text configuration of the VM, checks the source storage name (e.g., msiNM620), and checks if an identical storage name exists on the destination host. If the destination node does not have a local storage engine with that exact name, the script panics and blocks the migration through the UI.

How TrueNAS fixes this behavior:
Because my new TrueNAS integration is defined as shared storage named identically (TrueNAS-ZFS) across all nodes in my datacenter, it completely satisfies Proxmox’s basic offline script constraints. When a VM’s disks live on the TrueNAS storage block, the offline migration wizard unblocks instantly, allowing lightning-fast configuration transfers between cluster nodes without moving any underlying data blocks.

BONUS

I had weird errors when moving from shared storage back to local (this was when I had the setup using full local administrator “Privileges”, it didnt’ cause the task to fail and AI said this about.

When copying virtual machine disk blocks off of the TrueNAS iSCSI pool back onto local storage, you may notice repeating logs stating:
qemu-img: iSCSI GET_LBA_STATUS failed... SENSE KEY:ILLEGAL_REQUEST(5)
Why it triggers:
During a file transfer copy block, qemu-img sequentially pings the TrueNAS target engine with GET_LBA_STATUS calls to query which specific disk block ranges are blank or sparse so it can bypass copying empty space. Because the TrueNAS storage framework handles these optimizations at a different filesystem descriptor tier, it rejects the application block format structure, generating the standard warning lines.
The Impact:
This error is entirely harmless. The script immediately falls back to basic raw block-by-block copying and finishes streaming the virtual machine disk data over the network safely with zero block corruption.
What happens on hosts that have no installed the plug in? I dunno haven’t tested, I’m assuming they won’t have access to the storage.
What happens if the iSCSI server has a hard fail? Dunno, I’ll find out soon in my lab as I need to move the server. Any new findings I’ll update this blog post.
Hope this helps someone, feel free to leave a comment.
Posted on

PVE: A Storage Deep Dive – LVMThin vs ZFS vs EXT4

Testing different Storage Architectures against a cheap consumer SSD

When I first spun up my virtualization host, I knew that storage configuration would make or break my performance. But nothing prepared me for just how wildly my benchmarks would swing based purely on software definitions. Using the exact same underlying physical SSD, I ran a series of tests to uncover how different virtual controllers, file systems, and storage architectures handle real-world disk operations.
Here is the raw data from my benchmarking journey, followed by a breakdown of what I learned, and a definitive pros and cons guide for every setup.

📊 The Cumulative Benchmarks: My Raw Results

To keep the comparisons fair, I tracked sequential and random operations across four distinct iterations of my storage stack.
1. The Baseline: LVMthin with a Virtual SATA Controller (64 MiB Test)
My journey started here, and the results were a painful wake-up call to the limits of legacy hardware emulation.
[Read]  SEQ Q8T1:  429.221 MB/s  |  SEQ Q1T1:  318.640 MB/s  |  RND Q32T1:  13.760 MB/s  |  RND Q1T1:   5.507 MB/s
[Write] SEQ Q8T1:   44.661 MB/s  |  SEQ Q1T1:   16.980 MB/s  |  RND Q32T1:   2.382 MB/s  |  RND Q1T1:   1.655 MB/s
2. The Protocol Flip: LVMthin upgraded to VirtIO SCSI Single (64 MiB Test)
By changing a single dropdown menu in my VM settings from SATA to VirtIO SCSI, I unbottlenecked the communication protocol.
[Read]  SEQ Q8T1:  493.329 MB/s  |  SEQ Q1T1:  383.567 MB/s  |  RND Q32T1:  88.362 MB/s  |  RND Q1T1:  12.480 MB/s
[Write] SEQ Q8T1:   90.589 MB/s  |  SEQ Q1T1:   21.391 MB/s  |  RND Q32T1:   2.403 MB/s  |  RND Q1T1:   1.879 MB/s
3. The RAM Illusion: ZFS with VirtIO SCSI Single (Cached 64 MiB Test)
Next, I swapped my backend volume manager to ZFS. With a tiny test file size, I accidentally benchmarked the sheer speed of my host’s RAM via the ZFS ARC cache.
[Read]  SEQ Q8T1: 4401.052 MB/s  |  SEQ Q1T1: 1921.489 MB/s  |  RND Q32T1:  84.354 MB/s  |  RND Q1T1:  19.714 MB/s
[Write] SEQ Q8T1:  205.138 MB/s  |  SEQ Q1T1: 1791.981 MB/s  |  RND Q32T1:  68.788 MB/s  |  RND Q1T1:  17.412 MB/s

4. The True Hardware Limit: ZFS with VirtIO SCSI Single (Raw 8 GiB Test)
By cranking the test file size up to 8 GiB, I forced the data straight through the RAM buffers and onto the raw NAND flash chips. This represents my real physical drive performance under ZFS.
[Read]  SEQ Q8T1:  739.249 MB/s  |  SEQ Q1T1:  715.466 MB/s  |  RND Q32T1:  77.281 MB/s  |  RND Q1T1:  17.737 MB/s
[Write] SEQ Q8T1:  382.726 MB/s  |  SEQ Q1T1:  219.378 MB/s  |  RND Q32T1:  49.969 MB/s  |  RND Q1T1:  15.344 MB/s
5. The Image File Contender: qcow2 on an EXT4 Host File System (64 MiB Test)
Finally, I tested the classic approach—placing a qcow2 virtual disk file on top of a standard Linux EXT4 partition.
[Read]  SEQ Q8T1:  189.151 MB/s  |  SEQ Q1T1:  154.374 MB/s  |  RND Q32T1:  41.495 MB/s  |  RND Q1T1:  10.420 MB/s
[Write] SEQ Q8T1:  124.732 MB/s  |  SEQ Q1T1:  115.363 MB/s  |  RND Q32T1:   2.419 MB/s  |  RND Q1T1:   1.726 MB/s

🔍 Structural Deep Dive: Pros, Cons, and My Key Architectural Takeaways

Every storage backend has a specific target audience. Based on my benchmarking data and the structural architecture of these systems, here is how they stack up.

Setup A: LVMthin (Block Storage)

LVMthin acts as a pure logical volume manager natively inside the Linux kernel. It gives the VM direct block-level access to the storage layer without an intervening file system.
    • The VirtIO SCSI Upgrade Difference: My data showed a 542% explosion in random reads (from 13.76 MB/s to 88.36 MB/s) just by moving away from the virtual SATA controller. SATA forced my VM into a shallow, serialized hardware queue. Upgrading to VirtIO SCSI unlocked deep parallel queue depths (Q=32), passing commands straight down to the kernel block layer efficiently.

Pros
    • Ultra-Low Overhead: It consumes almost zero host RAM and negligible CPU power.
    • Zero Write Amplification: Blocks are written directly to the SSD, maximizing the physical lifespan (TBW) of consumer drives.

Cons
    • No Cache Cushioning: Because it lacks an aggressive RAM cache layer, your performance is entirely capped by the physical drive hardware. My random writes hit a miserable 2.4 MB/s wall in both tests because the raw SSD couldn’t keep up with heavy random write thrashing.
    • Feature Light: It lacks advanced data integrity features like inline file self-healing.

Setup B: ZFS (Advanced Copy-on-Write File System)

ZFS merges the file system and volume manager into a single entity. It uses a Copy-on-Write (CoW) design, meaning existing data is never overwritten; modifications are written to entirely new blocks, and metadata addresses are updated afterward.
    • The RAM vs. Real Disk Difference: In my 64 KiB test, ZFS hid my operations completely inside the Adaptive Replacement Cache (ARC) and transaction memory, showing an insane 4.4 GB/s read rate. Once I used an 8 GiB file to smash past the RAM barrier, I saw my true physical drive speeds (~740 MB/s Read / ~380 MB/s Write).
    • The Random Write Fix: Remarkably, ZFS managed to boost my raw un-cached random writes up to 49.97 MB/s (compared to LVMthin’s 2.4 MB/s). This is because ZFS groups messy random writes together in memory and flushes them to the physical disk as a clean, orderly sequential stream.

Pros
    • Unmatched Burst Speed: Turns unused host RAM into an ultra-fast read/write cache buffer.
    • Enterprise Features: Offers built-in data checksums to prevent silent data corruption (“bit-rot”), easy live replication, and zero-latency snapshots.

Cons
    • Resource Hungry: ZFS expects a steep memory tax (often quoted as 1GB of RAM per 1TB of storage) to keep its ARC cache happy.
    • The SSD Lifespan Killer: The Copy-on-Write mechanism causes heavy write amplification. On consumer-grade SSDs without power loss protection, ZFS constantly flushes transaction metadata, which can burn through the drive’s endurance metrics in a fraction of its normal lifespan.

Setup C: qcow2 on EXT4 (File-on-File Storage)

This architecture represents a double-layer virtualization penalty. The hypervisor creates a structured image file (.qcow2), which handles its own internal allocations, snapshots, and clusters, and then places that file on top of a standard host Linux file system like EXT4.
    • The Double-Abstraction Penalty: My benchmarks reveal that this setup struggled heavily with throughput. Sequential reads completely tanked out to just 189 MB/s, and sequential writes crawled at 124 MB/s. Because every single block request from the VM has to be translated twice—first by the qcow2 file allocation table and then by the host’s EXT4 file tracker—latency skyrocketed to 44–66ms, severely choking sequential bandwidth.

Pros
    • Extreme Portability: The entire VM drive is just a single file on the host. You can easily copy, move, back up, or email a .qcow2 file across systems.
    • Thin-Provisioning Simplicity: The file grows dynamically on disk automatically as the VM consumes space, making storage management incredibly intuitive.

Cons
    • Severe Throughput Bottlenecks: Easily the slowest sequential performer of the group due to file system translation overhead.
    • Terrible Random Writes: Like LVMthin, it lacked the intelligent memory-grouping mechanics of ZFS, collapsing down to a sluggish 2.4 MB/s on parallel random 4KiB writes.

⚠️ Important Architectural Caveats & Exceptions

Storage performance does not exist in a vacuum. Before choosing a design based on my charts, you must consider your underlying host hardware infrastructure.
1. Hardware RAID Controllers vs. ZFS (The Golden Rule)
If your server utilizes a physical Hardware RAID controller (e.g., a Dell PERC or HPE Smart Array), do not use ZFS.
    • Why: ZFS requires total, direct control over the physical disk topology to manage cache flushing and data integrity.
    • The Conflict: If you place ZFS on top of a hardware RAID logical volume, the hardware controller will hide the physical disk health from ZFS. Even worse, the controller’s onboard cache will conflict with the ZFS ARC, severely increasing the risk of cataclysmic data corruption during an unexpected power loss.
    • The Winner: For Hardware RAID arrays, LVMthin or EXT4 are the proper structural choices.

2. Battery-Backed Write Cache (BBWC) Controllers
If your host server features an enterprise RAID card paired with a physical Battery Backup Unit (BBU) or supercapacitor flash cache, the playing field changes entirely.
    • Why: A BBWC allows the host to immediately report a write operation as “complete” the millisecond it hits the controller’s physical onboard RAM cache, safely knowing that if the power cuts out, the battery will preserve the data until reboot.
    • The Impact: If I had a BBWC under my LVMthin or qcow2 tests, that terrible 2.4 MB/s random write bottleneck would have completely vanished, skyrocketing up into hundreds of megabytes per second. It essentially provides ZFS-like memory caching speeds without the software complexity or RAM overhead.

🏁 Final Verdict: Which One Should You Build?

    • Choose ZFS if you have plenty of host RAM to spare, you want elite enterprise data safety features, you rely on rapid multi-node backup replication, and your storage drives are connected to an HBA (Host Bus Adapter) in IT/Passthrough mode.
    • Choose LVMthin if you are using consumer-grade hardware or mini-PCs, you want to maximize the physical lifespan of your SSDs, you need every single drop of host RAM allocated to running your actual VMs, or you are deploying on top of a Hardware RAID card.
    • Choose qcow2 on EXT4 if absolute peak performance takes a back seat to sheer simplicity, storage portability, and easy file-level drive movement between isolated hypervisors.

I’m still trying to figureout the root cause of Veeam restore being unreal poor on a LVMThin target. Stay tuned!
Posted on

Migrating Veeam as a VMware VM to a Proxmox VM

Running Veeam as a Windows VM on Proxmox VE

With Broadcom completely upending the virtualization landscape, like many admins, I’ve been heavily evaluating Proxmox VE (PVE) as a viable enterprise alternative. While Veeam now natively supports PVE, I wanted to see what happens when you run the Veeam Backup & Replication (VBR) server as a Windows VM hosted directly on a PVE node, rather than on dedicated hardware.
Here is the technical reality of this architecture, the hypervisor mechanics, and the exact step-by-step process I used to execute a zero-duplication migration from ESXi into a tight local LVM-Thin storage layout.

1. The Core Architecture: How VBR Operates Inside PVE

When I deployed VBR into a Windows VM on Proxmox, the fundamental architecture shifted away from what I was used to in the VMware world:
  • The VBR VM is just the brain: The Windows VM hosting Veeam acts strictly as the management plane, controlling schedules, maintaining the PostgreSQL database, and exposing the console.
    – Works the same way as a VMware VM, it just can’t act as it’s own VMware hotadd node when on PVE VM
  • The PVE Worker handles the heavy lifting: Because PVE lacks a monolithic management layer like vCenter, Veeam requires me to add each individual node in the cluster to the console. Veeam then provisions a lightweight, Linux-based Worker VM onto each host to act as a stateless data mover.
  • The Control vs. Data Paths: When a job triggers, my Windows VBR VM talks directly to the Proxmox API to initiate a hypervisor-level snapshot. Once the snapshot is ready, VBR instructs the local Worker VM to mount that snapshot, compress the data blocks, and stream them directly to my backup repository.

                  ┌─── [ Veeam VBR Server (Windows) ] ───┐
                  │                                      │
                  ▼ (1. Orders Snapshot via API)         ▼ (3. Orders Data Processing)
       [ Proxmox VE Host ]                       [ Veeam PVE Worker VM ]
                  │                                      │
                  ▼ (2. Triggers Quiescing)              │
       [ Target VM + QEMU Agent ]                        │
                  │                                      │
                  └─────── (4. Reads Snapshot Data) ─────┘

2. Technical Friction Points & Lessons Learned

While the setup works, moving from ESXi to PVE exposed several architectural limitations and maturity gaps that I had to account for.
The Infuriating VLAN Blind Spot in the Worker Wizard
This is one of the clearest signs that Veeam’s Proxmox integration is still green. When deploying the Linux-based Worker VMs across my cluster nodes, the Veeam Deployment Wizard lacks any field to define a VLAN ID. It lets me select the network bridge (e.g., vmbr0), but that’s it.

  • The Hack: Because my management network is VLAN-tagged, the newly deployed Worker VMs immediately drop offline upon creation because they can’t get an IP address. I have to manually jump into the Proxmox GUI, open the hardware settings of the newly provisioned Worker VM, inject the VLAN tag into its virtual NIC, and reboot it.
  • The Catch: Every time Veeam pushes an update that redeploys or patches these Worker VMs, this manual configuration gets wiped out. It requires immediate administrative intervention in the PVE GUI to fix the network interfaces after every lifecycle update.

3. The Migration Strategy: Zero-Duplication Streaming to LVM-Thin

Moving my Veeam server presented a unique chicken-and-egg problem. To make matters worse, my old ESXi lab layout was messy: my Veeam backup repository wasn’t an independent iSCSI target—it was a raw, bloated 1.2 TiB VMDK sitting directly on local mechanical spindle datastores.
To avoid the pain of building a fresh Windows OS, configuring IPs, and rebuilding my environment from scratch, I chose to use the Self-Backup and Restore method. I backed up just the C: drive of the Veeam VM, restored it to Proxmox, and brought the compute layer online.
However, migrating that massive data disk into a tight, local LVM-Thin pool presented a storage Catch-22: LVM-Thin does not use files like VMDK or QCOW2. It is a block-level storage architecture. I couldn’t just copy the file over first because my PVE host had no local path large enough to store a temporary 1.2 TiB file.
To bypass this hurdle and ensure zero file duplication, I streamed the conversion directly over the network from the ESXi host using the command line. Which involves adding the ESXi host to PVE, then using PVEs CLI.

Phase 1: Pre-Migration Trim (Dropping the Bloat)

Inside the Windows Veeam VM, the OS reported only 600GB of actual data in use, meaning I had over 200GB of “ghost bloat” on the 808GB allocated ESXi VMDK. Before shutting down the VM, I ran a quick optimization check to ensure Windows explicitly issued zero-discard commands to the hypervisor:
powershell
Optimize-Volume -DriveLetter D -Defrag -Verbose
 Optimize-Volume -DriveLetter D -ReTrim -Verbose
(After trimming, my actual active file space dropped to exactly 519 GB).

Phase 2: Live Network Streaming via qm importdisk

When you add an ESXi storage provider to Proxmox, PVE transparently maps the remote files to a hidden local runtime directory on the PVE host. This allows you to target the file directly with native Proxmox CLI utilities without copying anything locally.
Add ESXi Storage:

  • Log in to the Proxmox Web UI.
  • Go to Datacenter > Storage > Add and select ESXi.
  • Enter a unique ID, your ESXi host Server IP/hostname, and your ESXi administrator Username and Password.
  • Check Skip Certificate Verification if your ESXi node uses a self-signed certificate.
  • Click Add.
I created a dummy placeholder virtual disk shell on my newly restored Proxmox Veeam VM (let’s assume VM ID 108), hopped into the Proxmox CLI, and initiated a fileless stream:
qm importdisk 108 /run/pve/import/esxi/desktopesxi/mnt/ha-datacenter/Desktop-2TB/Veeam/Veeam.vmdk msiNM620

Beating the Scary LVM-Thin Warnings
During the transfer, Proxmox threw a terrifying set of warnings:
WARNING: Sum of all thin volume sizes (<1.53 TiB) exceeds the size of thin pool nvme-nm620/nvme-nm620 and the size of whole volume group (<953.87 GiB).
Because the 1.2 TiB virtual boundary of the VMDK was larger than my physical 953 GiB NVMe drive, LVM-Thin immediately flagged that I was overprovisioning.
However, because we trimmed the drive beforehand, the qm importdisk zero-detection engine kicked in. As the empty space traveled over the LAN, the engine detected the logical zeros and completely dropped them instead of writing them to disk.
The progress bar crept through the full 1.2 TiB virtual structure, but it stopped consuming physical blocks when it hit empty space. It finished flawlessly at 100%, consuming an exact physical footprint of 518.88 GB on my NVMe pool—matching my source data perfectly without a single byte of temporary file bloat.

4. The Finish Line

To wrap up the migration, I executed the final hookups:
  1. Attach the Block Storage: Inside the PVE GUI for VM 108, I went to Hardware, double-clicked the newly created Unused Disk, set it to SCSI, and checked the Discard box (ensuring that future Veeam deletions will actively reclaim space on my LVM-Thin pool).
  2. Mount the ISO & Fix the NIC: Because the restored Windows VM lacked KVM network drivers, it booted up offline. I used scp to copy the virtio-win.iso from my other cluster node over to this host’s local directory (/var/lib/vz/template/iso/), mounted it to the VM’s virtual CD drive, opened Device Manager, and updated the Ethernet Controller driver.
  3. Rescan & Run: I brought the data disk Online in Windows Disk Management (which instantly recognized the original NTFS/ReFS structure perfectly), opened the Veeam console, and ran a Repository Rescan.

Within minutes, Veeam mapped my existing backup chains, and my entire infrastructure was fully functional on Proxmox VE with zero data loss and zero configuration rebuilding.
This was nice but I didn’t move the backup data to a dedicated block level storage, did I… No… will I… mhmmm maybe eventually, but I’m surprised how well the conversation went, another step to a VMware -> PVE migration. Also, the backup jobs moved from hotadd to NBD cause the Veeam server is not longer on a ESXi host. And you can’t map a backup set when re-adding a backup job sourcing a PVE host.

BONUS

When I tried running a Full VM Restore from my VMware backups directly onto an LVMthin pool on my high-speed SSD. The restore immediately cratered to a miserable 3 MB/s, pinning my host at a massive iowait percentage. Same problem in my previous blog post when I migrated my email server.
When I cancelled the job in Veeam, the console said “Failed,” but the helper VM refused to die, and the host remained completely paralyzed. Even running a basic Linux sync command via SSH completely hung and stole my terminal shell.
Here is the juicy reality of what happened under the hood:
  • The Block-Level Choke: Veeam streams raw, unallocated data blocks during standard restores. On an un-provisioned LVMthin target, this forces the Proxmox host to perform real-time metadata lookups and extend block structures on every single write loop. This created a massive write-amplification loop that totally choked the SSD controller queue.

  • The Unkillable “D-State”: Hard-killing the VM mid-stream locked the dm-thin kernel driver into an uninterruptible sleep state (Linux D-state). Because the storage layer was frozen waiting on hardware, the kernel completely ignored standard ACPI shutdown signals.

How I broken it loose without pulling the physical power cord:
With the storage layer totally unresponsive and sync frozen, standard reboot commands were completely useless. I had to open a secondary SSH shell and talk directly to the Linux kernel using aggressive, low-level emergency triggers.
First, I enabled the kernel’s magic SysRq interface:
echo 1 > /proc/sys/kernel/sysrq
Then, I sent the emergency hardware reboot signal (b):
bash
echo b > /proc/sysrq-trigger

This acted as a software-level reset button, forcing the motherboard to instantly reboot without trying to unmount or flush the corrupted, frozen storage queues. On startup, the LVMthin subsystem automatically repaired its metadata boundaries and the host came back up healthy.
I found restoring to the same SSD configured as a DIR Ext4 storage had to problem and restored the VM in 5 minutes. So, not sure how to properly handle a Veeam restore to a LVM-thin without having this problem.
Posted on

Proxmox Cluster Findings

Proxmox Cluster Findings

When I set up my new MSI PVE host, I decided to use the “Golden Blueprint” for storage: installing the base Proxmox OS on a cheap SATA SSD to shield it from heavy log writes, leaving my lightning-fast NVMe drive completely clean and unpartitioned as a dedicated LVM-Thin block storage pool for my VMs and containers.
But things got complicated when I tried to link my new host to my old host. Here is what I learned while wrestling with Proxmox cluster logic, VMware design differences, and hardcoded network traps.

1. The Populated Host Conundrum

My initial plan was straightforward: create a fresh cluster on my pristine new host, and then have my old host (which currently runs all my active workloads) join it. I quickly hit a hard wall. Unlike VMware vCenter—which effortlessly imports populated ESXi hosts and reindexes VM identifiers on the fly—Proxmox operates on a decentralized, shared-filesystem architecture (pmxcfs).
Because Proxmox uses hardcoded VMIDs (like 100, 101) to define physical storage paths and configuration maps, a host cannot join an existing cluster if it has any virtual machines on it. Joining a cluster completely overwrites the local /etc/pve directory, which would instantly orphan any existing virtual disks.
The Fix: I flipped the workflow. I created the cluster directly on my old host first. Because it initialized the cluster, its existing VMs remained perfectly safe.

2. Leaving a Cluster Requires the CLI

Before I figured out the correct order of operations, I had already initialized a test cluster on my new node. When I went to undo it, I discovered there is no “Delete Cluster” button in the Proxmox Web GUI. Because tearing down a cluster can cause severe data corruption if done incorrectly, Proxmox forces you into the shell.
To completely reset my new node back to standalone mode without reinstalling the entire OS, I had to open the terminal and force-clear the configuration database using these steps:
# Stop cluster synchronization services
systemctl stop pve-cluster
systemctl stop corosync

# Force the configuration filesystem into a temporary local mode
pmxcfs -l

# Permanently erase the old cluster registries
rm -f /etc/pve/corosync.conf
rm -rf /etc/corosync/*

# Kill the background lock process and restart standalone services
killall pmxcfs
systemctl start pve-cluster

3. The Hardcoded IP/VLAN Trap

While running pmxcfs -l, I noticed the terminal spat out an old IP address (192.168.0.68) that I had previously removed when migrating the host to a new VLAN-tagged management network.
I realized that Proxmox doesn’t query active network interfaces to resolve its own name; it relies entirely on a static registry. The old IP address was still hardcoded inside my /etc/hosts file. I used nano /etc/hosts to update the line to my new VLAN IP (172.16.21.20) so the host could resolve itself properly. (Blog post covering IP change updated)

4. Breaking the Handshake Hang

When I finally went to join my clean new node (172.16.21.20) to the old node’s cluster (172.16.21.60), the Web GUI installer completely hung on the message: “Request addition of this node.”
However, in classic homelab fashion, a simple web browser refresh cleared the stalled API cache, the handshake successfully completed on its own, and both nodes cleanly populated into a single sidebar.

5. Local Disks Show Up on the Wrong Nodes

Right after my two nodes successfully clustered, I noticed something deeply alarming in the sidebar UI: my old host’s massive local storage drive (mass-storage) was suddenly showing up underneath my shiny new msi-pve node.
I knew for a fact it wasn’t shared storage—there were no NFS, SMB, or iSCSI links connecting them. The drive was physically plugged into the old hardware. So why was my new host pretending it owned it?
The Cause: Global Cluster Definitions
This is one of Proxmox’s quirky design traits. Proxmox stores every single storage path inside a single, cluster-wide configuration file (/etc/pve/storage.cfg). The moment my new node joined the cluster, it downloaded this file and blindly copied the layout.
By default, Proxmox assumes any storage listed in that file is accessible by all nodes unless you explicitly state otherwise. It draws the icon under every host in the sidebar, creating a dangerous phantom placeholder. If I had tried to spin up a VM on my new node and targeted that ghost storage, the deployment would have crashed instantly with activation errors.
Luckily the fix, correcting this is incredibly simple and doesn’t require the command line:
    1. I clicked on Datacenter at the very top of the sidebar.
    2. I went to Storage, highlighted the phantom mass-storage pool, and clicked Edit.
    3. I found the Nodes dropdown—which was completely blank (Proxmox-speak for “Allow All Nodes”).
    4. I changed it to explicitly select only my old host (g9-pve) and saved it.
The second I clicked OK, the phantom icon vanished from underneath my new node, keeping my environment clean and preventing any catastrophic accidental deployments.

6. Migration Failure due to VMs bound virtual NIC

Coming from VMware, hitting a hard wall during a migration because of a network name mismatch feels entirely unnecessary. In VMware—even on standard vSwitches without distributed virtual switching—the migration wizard natively handles network remapping. If a target host doesn’t have a matching network, the wizard stops and lets you choose a new path on the fly.
This highlights a fundamental architectural difference in how these two hypervisors handle networking:
  • VMware uses Port Groups (VMPGs): This creates a layer of abstraction. The VM connects to a named Port Group, and that group handles the VLAN tagging down at the vSwitch level.
  • Proxmox uses Direct Bridging: By default, Proxmox expects you to point the VM’s virtual NIC directly at a specific host bridge (like vmbr1) and explicitly type the VLAN tag directly into the VM’s network device settings.
Because Proxmox binds the VM directly to a specific host bridge string rather than an abstract port group, its migration wizard is completely rigid. If the destination host doesn’t have an identically named bridge, the migration fails with an error instead of letting you remap it during the transfer. To make migrations seamless, you have no choice but to ensure your Linux bridge names match exactly across every single node in your cluster.

7. Migrating an LXC

With a container healthy, I attempted to clone it over to the new host’s NVMe drive. The Proxmox UI threw a sudden error:

Full clone of a running container is only possible from a snapshot (500)
Because containers share live kernel space, they cannot be live-cloned without a frozen snapshot boundary. I attempted to take a snapshot, only to hit a secondary brick wall:

The current guest configuration does not support taking new snapshots
This error points directly to the underlying physical storage. While advanced file systems like ZFS and LVM-Thin support snapshots out of the box, traditional file-based directory storages and thick LVM volume groups do not. I couldn’t take a snapshot, which meant I couldn’t live-clone. A cold migration was my only path forward, or so I thought…

Overriding the Migration Wizard with Manual Backup Streams

I shut the container down and clicked Migrate, but Proxmox aborted yet again:

ERROR: migration aborted: storage 'Mass-Storage' is not available on node 'msi-pve'
When dealing with local storage migration, the standard Proxmox wizard stubbornly expects the exact same storage pool name to exist on the target host. Because my old host’s 4TB drive was named Mass-Storage, and my new host only possessed an unmapped LVM-Thin layout, the handshake failed.
To bypass the rigid migration UI entirely, I resorted to a manual Secure Copy (SCP) backup stream. First, I shut down the source container on g9-pve and took a standard uncompressed cold backup file to my local directory. Then, because LVM block devices don’t show up via standard df -h folders, I targeted Proxmox’s universal root backup directory (/var/lib/vz/dump/) and pushed the data over the network via SCP:
bash
scp /mnt/pve/mass-storage/dump/vzdump-lxc-105-2026_09_21-19_57_35.tar.zst root@172.16.21.20:/var/lib/vz/dump/

The Final Handshake

Once the transfer finished, the backup file appeared beautifully under the new host’s local storage tab in the Web UI. I hit Restore, pointed the destination target directly into the brand-new 1TB NVMe LVM-Thin pool, and let it extract.
To guarantee no IP collisions occurred on my network while validating the data, I went into the original source container on g9-pve and checked the “Disconnected” flag on its network interface. With the original container safely blinded, I fired up the newly restored container on msi-pve. I adjusted its hardware bridge settings from vmbr1 to the new host’s active vmbr0 bridge, and the pings immediately started flowing.

The service came up perfectly healthy, allowing me to finally go back and permanently shut down the original source container on the old hardware.

8. Proxmox “Cluster Join Aborted – No Quorum” Errors

When adding a new node to an existing Proxmox VE cluster, you might encounter this frustrating terminal error:
An error occurred on the cluster node: cluster not ready - no quorum? TASK ERROR: Cluster join aborted!

Here is exactly why this happens and how to bypass it safely.

🔍 The Root Cause
Proxmox requires a strict majority (>50%) of voting nodes to be online and communicating via Corosync to allow administrative changes.
    • If a node in your existing cluster is offline, powered down, or network-isolated, the remaining online nodes lose quorum.
    • To prevent accidental data corruption, Proxmox drops into a protective, read-only state and blocks any new nodes from joining.

🛠️ The Fix: Bypassing Quorum
If you cannot easily boot the offline node back up, you can manually force the surviving node to lower its expected vote count.
Step 1: Force Expected Votes
SSH into your active cluster node and run the following command to temporarily lower the expected vote threshold to 1:
pvecm expected 1
This instantly restores cluster write-privileges to the remaining online node.
Step 2: Join the New Node
Return to your new node (via CLI or the Web GUI) and re-run the cluster join process. The join operation will now succeed.
Step 3: Boot the Offline Node
Once the new node has safely integrated into the cluster, power your original offline node back on. It will automatically check in with the rest of the network, detect the new node, and sync the updated cluster configuration seamlessly.
⚠️ Important Safety Checklist
Before running pvecm expected 1, always verify your storage layout:
  • Shared Storage (Ceph, NFS, iSCSI): Ensure the offline node is completely powered off. If it boots up isolated, it could cause a “split-brain” scenario and corrupt shared virtual disks.
  • Local Storage Only: If your cluster strictly uses local disks, the risk is near zero. The offline node cannot physically touch or corrupt data on the other hosts.

If you do Bring back your old host and didn’t remove it from the cluster, it may not be happy with you and remain in its own island cluster and not play nice with your old hosts…

9. Proxmox Cluster Split-Brain and Corosync Version Mismatch

When a Proxmox VE node is taken offline and changes are made to the remaining cluster (such as adding a new host), bringing that offline node back online can trigger a silent cluster failure. The node will refuse to rejoin, and the Proxmox Web UI will show the hosts as completely disconnected from one another.

Here is a breakdown of why this happens and how to safely force the node back into the cluster.

The Problem: The “Chicken and Egg” Loop
Proxmox relies on two core components to maintain a cluster:
    1. Corosync: The underlying network messaging engine that handles cluster quorum and node handshakes.
    2. pmxcfs (Proxmox Cluster File System): The distributed filesystem that mirrors configuration files (like /etc/pve/corosync.conf) across all nodes.

This creates a dependency loop: pmxcfs needs Corosync to sync files across the network, but Corosync needs to read pmxcfs to get its startup configuration.
If a node is offline when the active cluster undergoes an update, the cluster increments its configuration version (e.g., from config_version: 2 to 3). When the offline node boots back up, a security mechanism triggers:
    1. Corosync reads its local file and sees Version 2.
    2. It reaches across the network and hears Version 3 from the active cluster.
    3. To prevent a “split-brain” scenario (where two isolated parts of a cluster might blindly overwrite each other’s storage configs), Corosync panics and immediately exits with the error:
      [CMAP] Received config version (3) is different than my config version (2)! Exiting

Because Corosync crashes, the network pipe never initializes. Because the pipe never initializes, the node can never automatically download the updated file from the healthy cluster.

The Fix: Trick Corosync to Open the Pipe
To resolve this, you must force the local node’s configuration filesystem into read/write mode, manually spoof the configuration version number to match the cluster, and restart the services.
Step 1: Kill Zombie Processes & Force Local Mode
Because the cluster is offline, Proxmox locks /etc/pve into a read-only state. Clear out any stuck systemd loops and force it to run locally:
# Kill any hung cluster processes holding filesystem locks
killall -9 pmxcfs corosync pve-cluster

# Stop the systemd tracking services
systemctl stop pve-cluster corosync

# Force the cluster filesystem to run in independent local mode
pmxcfs -l
Step 2: Manually Update the Config Version
With the filesystem unlocked, edit the configuration file to match the version expected by the active cluster:
nano /etc/pve/corosync.conf
    • locate the config_version: line in the file.
    • Change the number to match the cluster’s current version (e.g., change 2 to 3).
    • Save and exit (Ctrl+O, Enter, Ctrl+X).

Step 3: Restart Services
Now that the version numbers match on paper, kill your manual filesystem override and let systemd start the cluster services normally:
# Clear the manual local mode instance
killall -9 pmxcfs

# Start the cluster services sequentially
systemctl start pve-cluster
systemctl start corosync
The Result: The True Config Wins
By manually matching the version numbers, Corosync is tricked into thinking the nodes are synchronized, allowing it to safely open the network tunnel.
The moment the connection stabilizes, Proxmox’s internal generation rules take over. It recognizes that the active cluster holds the true, updated configuration file. The cluster immediately pushes the complete configuration down to the returning node, gracefully overwriting your temporary edit and cleanly reintegrating the host.
You can verify the fix directly from the CLI:
pvecm status

If the above didn’t work cause your host was offline when you brought this one back online, and that host already has the same version number, but get connection issues in the GUI check out the next section.

10. Wiping Stale State and Bypassing SSL Mismatches

When a Proxmox VE cluster shifts structurally while a specific node is offline (such as adding new nodes or changing quorum dynamics), that offline node can fall too far out of sync. Even if configuration version numbers match, the node may completely fail to authenticate with the live cluster due to outdated cryptographic tokens or security states.
Here is how to safely wipe a node’s stale cluster data, preserve existing virtual machines, and force-join it back into production using an SSH fallback.
The Problem: Authentication Locks & The SSL 500 Error
When a node falls severely behind the cluster’s state, trying to rejoin it often leads to a dead end. To fix this, you must first reset the node back into a standalone host by purging its old configuration database files:
# Stop the stuck cluster loops
systemctl stop pve-cluster corosync
killall -9 pmxcfs corosync

# Force the configuration filesystem to open up locally
pmxcfs -l

# Purge the outdated corosync database files entirely
rm /etc/pve/corosync.conf
rm -rf /etc/corosync/*
rm -rf /var/lib/corosync/*

# Restart the local filesystem daemon as a clean standalone service
killall -9 pmxcfs
systemctl start pve-cluster
Once the node is wiped and returned to a standalone state, running a standard pvecm add [CLUSTER_IP] often triggers two scary security blocks:
  1. The Guest Warning: * this host already contains virtual guests.
  2. The SSL Failure: 500 Can't connect to [IP]:8006 (hostname verification failed).

Understanding the SSL 500 Error
By default, Proxmox attempts to execute cluster joins using its secure HTTPS API on port 8006. The target node serves an SSL certificate that matches its actual hostname (e.g., g9-pve), but the joining node is hitting it via a raw IP address. Because the IP address does not match the name listed on the security certificate, the API strictly rejects the connection and throws the 500 error.
The Solution: Forcing and Bypassing via SSH
To safely resolve both issues, use the following advanced join command:
pvecm add [IP of neigbouring host] --force --use_ssh
Why This Works:
  • The --force Flag: This tells Proxmox to intentionally ignore the virtual guest warning. Your VM’s raw virtual disks sitting on local storage are 100% safe and are never modified or deleted. The flag simply allows Proxmox to safely merge the node’s local configuration directory into the cluster’s unified database.
  • The --use_ssh Flag: This is the magic bullet for the certificate error. It instructs Proxmox to completely bypass the strict HTTPS API verification loop. Instead, it creates a secure tunnel over standard SSH to pull the fresh cluster schema, synchronize cryptographic keys, and launch Corosync cleanly.

The Result
The joining host will pull the clean, master configuration down, automatically register itself into the cluster map, and instantly turn green across your unified Proxmox dashboard.
Your existing VMs will reappear right where they belong under the node’s localized tree in the Web UI, fully integrated and ready to manage.
Posted on

Proxmox “Unable to Install Initramfs” Error

How I Fixed the Proxmox “Unable to Install Initramfs” Error on an MSI Z97 Motherboard

I recently tried installing the latest Proxmox VE 9.2.1 on my trusty MSI Z97 PC Mate motherboard, and it turned into an absolute nightmare. First, the installer flat out crashed with an unable to install initramfs error.
To fix it, I had to dig through the absolute dumbest BIOS layout in existence. On this motherboard, Secure Boot is completely hidden. I had to go to Settings ➔ Advanced ➔ Windows OS Configuration, turn on the Windows 8/8.1 Feature, and only then did the secret Secure Boot submenu pop up so I could finally disable it.
I wiped the disk and tried again, but immediately hit a new roadblock: installation of package pve-i18n... failed. It turns “out the modern Proxmox kernel sends power management and queuing commands that older Z97 SATA controllers just can’t keep up with, causing the target drive to drop offline mid-installation.” Source less claim, tiss a theory AI suggested.
The ultimate fix? I bypassed the hardware’s native command queuing entirely. At the Proxmox boot menu, I pressed e to edit the installer parameters, found the linux boot line, and appended these specific kernel safe-flags to the very end of it:
intel_iommu=on iommu=pt libata.force=noncq pci=nomsi
I hit Ctrl + X to boot, and the installation finally sailed right through without a single hiccup. If you are struggling with a modern Proxmox install on older Intel hardware, force these flags at boot—it will save your sanity!
Posted on

How to Transfer Files to an Offline Windows 11 VM on Proxmox

How I Transferred Files to an Offline Windows 11 VM on Proxmox (Without Network or USBs)

I recently ran into a frustrating loop trying to get application files (like CrystalDiskMark) onto a completely isolated, offline Windows 11 VM running on my Proxmox VE (PVE) server. Because the VM had no network connection, standard network shares (SMB) or Remote Desktop drag-and-drop were out of the question. I also didn’t have any physical USB sticks lying around to use hardware passthrough.
I initially tried to natively generate an ISO using PowerShell scripts on my Windows 11 management PC, but I quickly discovered that modern PowerShell lacks a native, out-of-the-box one-liner to build ISOs without throwing corrupt pipeline errors or requiring Microsoft Store utilities (which were blocked by my firewall).
To bypass all of this, I found a clever workaround using a native Virtual Hard Disk (VHDX) file and a simple file extension trick to fool the Proxmox GUI. Here is exactly how I did it:

Step 1: I Created a VHDX in PowerShell

Instead of fighting with broken ISO streams, I opened PowerShell as an Administrator on my management PC and ran a quick script to build a virtual disk, format it, and pack it with my files.
powershell
# I defined my folder source and output path
$SourceFolder = "D:\Apps\CrystalDiskMark"
$VHDXPath     = "C:\temp\Transfer.vhdx"

# I created, mounted, and formatted a fresh 1GB virtual drive
$VHDX = New-VHD -Path $VHDXPath -SizeBytes 1GB -Dynamic | Mount-VHD -Passthru | Initialize-Disk -PartitionStyle GPT -Passthru | New-Partition -AssignDriveLetter -UseMaximumSize
Format-Volume -Partition $VHDX -FileSystem NTFS -Confirm:$false

# I grabbed the temporary drive letter and copied my files over
$DriveLetter = $VHDX.DriveLetter
Copy-Item -Path "$SourceFolder\*" -Destination "${DriveLetter}:\" -Recurse

# I safely unmounted the virtual disk from my host machine
Dismount-VHD -Path $VHDXPath

Step 2: I tricked the PVE UI

Proxmox’s web upload utility is hardcoded to only accept certain file formats inside the ISO template container, so it threw an “extension error” when I tried to upload my raw .vhdx file.
To trick the system, I simply renamed the file extension on my Windows PC from Transfer.vhdx to Transfer.iso. Changing the label doesn’t hurt the data inside, and it completely bypassed the Proxmox GUI gate. I hit upload, and it went right through.

Step 3: I Imported the Disk via Proxmox Shell

Once the file finished uploading to my local storage, I jumped into the main Proxmox Node Shell and ran the built-in storage manager tool (qm) to natively import the disk into my target VM layout. Proxmox’s backend engine automatically identified the actual internal disk layout regardless of my fake .iso file extension:
bash
qm importdisk 100 /var/lib/vz/template/iso/Transfer.iso local-lvm

(Note: I replaced 100 with my specific VM ID and local-lvm with my target storage environment).

Step 4: I Attached the Unused Hardware and Brought it Online

After running the shell command, I went back to the Proxmox Web UI, clicked on my Windows 11 VM, and opened the Hardware tab. The imported disk was sitting there at the bottom, safely detached as an “Unused Disk 0”.
I double-clicked the unused disk line and selected SATA as the bus type for maximum compatibility, then clicked Add.
Finally, I hopped into my offline Windows 11 VM console, opened Disk Management, right-clicked the new 1GB disk block, and toggled it to Online. It mounted immediately in File Explorer with all of my folders intact!
Hope this helps someone.
Posted on

Backing up an LXC with Veeam

Backing up an LXC with Veeam

If you use Proxmox VE, you already know how amazing and lightweight Linux Containers (LXCs) are. So, when Veeam announced native support for Proxmox VE, I was incredibly excited. But my excitement turned to pure frustration when I realized a major catch: Veeam’s native Proxmox plug-in completely ignores LXC containers. It only supports QEMU/KVM virtual machines.
Veeam justifies this by saying enterprise workflows rely on VMs and that they require block-level Changed Block Tracking (CBT) to prevent massive network overhead. But for my lightweight setup, I just wanted a simple way to back up my containers without the corporate bloat.
Instead of walking away, I decided to test out a theory and found a bulletproof workaround using Proxmox’s native tools alongside Veeam. Here is how I set it up, step-by-step:

Step 1: I leverage Proxmox’s Native vzdump

First, I use Proxmox’s built-in CLI or Web GUI to generate standard backups. By navigating to Datacenter > Backup in PVE, I schedule a job that takes a live snapshot of my LXC and compresses it into a single .tar.zst file. Because these files pile up quickly, I set the PVE retention policy to Keep Last: 1. This ensures Proxmox doesn’t do shit here automatically it just fails a backup job if one already exists, kind of dumb it should just create one and purge the old one. cause technically me being able to delete it goes against the rentention policy, where as creating a new one and purging the old one does not.

Step 2: I hook Veeam into the Host Operating System

Since Proxmox stores these dumps locally on the host’s Debian filesystem (usually under /var/lib/vz/dump/), I need a way for Veeam to reach them. I went into the Veeam Backup & Replication console and added my Proxmox host as a Managed Linux Server.
A quick warning if you try this: The Veeam wizard tries to force-install a massive list of unnecessary corporate storage packages (like Dell Data Domain and NetApp drivers) onto your lean Proxmox host. I aggressively unchecked all optional components, keeping only the absolute essentials: the Installer Service and the Veeam Data Mover.

Step 3: I Use a Veeam File Copy Job for Offsite Retention

With Veeam now able to safely browse my Proxmox filesystem, I built a File Copy Job. I pointed the source to my Proxmox dump folder, used a wildcard filter (like *.tar.zst), and set my standard Veeam repository as the destination. Now, Veeam automatically pulls the full compressed backup file off my host every night. Even though Proxmox deletes its local copy the next day, Veeam keeps my historical recovery points safe in my long-term repository. From here we can complete the 3-2-1-0 Rule for backups.

Step 4: Seamless Disaster Recovery to a Second Host

The best part about this architecture is how incredibly simple the restore process is. Proxmox backup files are completely self-contained; they don’t rely on a central database.
If my primary Proxmox host dies, I can add a secondary Proxmox host to Veeam as a managed server. I then use Veeam to copy the .tar.zst backup files directly into the second host’s local dump folder. The exact second the file transfer completes, the backup instantly populates in the second Proxmox GUI under the storage menu. From there, I just hit “Restore,” assign a unique Container ID if needed, and my LXC is back online.
It might not be the “official” automated method Veeam envisioned, but it bypasses their limitations perfectly and works flawlessly for my environment.
It also doesn’t provide any dedup as the base file is literally replaced in place.
Hope this helps someone.
Posted on

Deploying Pi-hole v6 Offline on Proxmox VE with a Palo Alto Firewall

Deploying Pi-hole v6 Offline on Proxmox VE with a Palo Alto Firewall

An architectural summary and step-by-step rebuilding guide for the containerized offline deployment.

🏗️ Technical Architecture Overview

    • Hypervisor: Proxmox VE (PVE)
    • Parent Compute Layer: Privileged Linux Container (LXC) running a minimal Debian 12 base template.
    • Network Blueprint: Bound to vmbr1, isolated natively via VLAN 7 (172.16.7.2/24), routing through gateway 172.16.7.1.
    • Runtime Engine: Decoupled, standalone static binary installation of Docker Engine v27.3.1.
    • Application Tenant: Pi-hole v6 running in native Host Networking Mode (--net=host).
    • Upstream Security: Border perimeter managed by a Palo Alto Networks (PAN) Layer 7 firewall with strict application checking and URL filtering.

🛑 Phase 1: Perimeter Firewall Provisioning (Palo Alto)

Because the Pi-hole requires internet access to initialize its database but must remain highly secured, a tailored Layer 7 inspection ruleset was applied.
    1. Custom URL Category Object: Created an object named Pihole-Gravity-Sources populated with explicit domains to pull source databases:
        • *.github.com
        • *.githubusercontent.com
        • *.pi-hole.net
        • install.pi-hole.net

    2. Security Policy Mapping:
        • Source: VLAN 7 Zone (172.16.7.2)
        • Destination: WAN/External Zone (any)
        • Applications Permit List: ssl, web-browsing, dns, github-base (Crucial: required to pass deep inspection chunking packets during data stream ingestion).
        • Service Profile: Standard ports 80/http and 443/https.
        • URL Category Constraint: Enforced to match Pihole-Gravity-Sources.

💾 Phase 2: Staging Offline Files (Management PC)

To populate an offline PVE node, core files were staged using an online workstation browser:
    1. PVE Debian 12 Template:
      Index of /images/system/
    2. Docker Standalone Core:
      Index of linux/static/stable/x86_64/
    3. Pi-hole v6 Docker Image: Pulled down via an archive generator engine as a clean deployment container tarball (pihole-pihole-latest-linux-amd64.image.tar).

🛠️ Phase 3: Parent LXC Deployment & Pre-Staging

All instructions were passed natively via the Proxmox Host Shell CLI to build the environment and seed assets before booting the container.
    1. Create the Parent Host: 
      pct create 105 /mnt/pve/mass-storage/template/cache/debian-12-standard_12.7-1_amd64.tar.zst \
  -cores 1 \
  -memory 1024 \
  -swap 512 \
  -ostype debian \
  -storage mass-storage \
  -rootfs mass-storage:8 \
  -unprivileged 0 \
  -features nesting=1 \
  -onboot 1 \
  -net0 name=eth0,bridge=vmbr1,tag=7,ip=172.16.7.2/24,gw=172.16.7.1 \
  -nameserver 1.1.1.1

    2. Mount the Raw Virtual Disk for Staging: 
      pct mount 105

    3. Decompress and Inject Standalone Docker Binaries into System Paths: 
      tar -xf /mnt/pve/mass-storage/template/cache/docker-27.3.1.tgz -C /var/lib/lxc/105/rootfs/usr/bin/ --strip-components=1

    4. Decompress and Inject Pi-hole Image Template: 
      tar -xf /mnt/pve/mass-storage/template/cache/pihole-pihole-latest-linux-amd64.image.tar.gz -C /var/lib/lxc/105/rootfs/root/

    5. Close and Fire Up Compute Layer: 
      pct unmount 105
pct start 105

🐳 Phase 4: Initializing Engine Layers (Inside Container Shell)

Navigating straight inside the container environment using pct enter 105.
    1. Iptables isn’t natively available in the Debian Image so…
    2. Mitigate Missing Network Engine Elements (iptables Bypass JSON Config):
      Because the thin container lacked the legacy iptables stack, we commanded Docker to use a hypervisor-safe block driver storage matrix and turn off firewall manipulation:       
      mkdir -p /etc/docker
cat << 'EOF' > /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=cgroupfs"],
  "storage-driver": "vfs",
  "iptables": false
 }
EOF

    3. Start Core Engine Systems: 
      dockerd > /var/log/dockerd.log 2>&1 &

    4. Ingest Web Registry Layouts: 
      docker load -i /root/pihole-pihole-latest-linux-amd64.image.tar

🎯 Phase 5: Executing the Pi-hole v6 Application Tenant

Deploying with variables matching the newly restructured Pi-hole v6 engine specifications.
    1. The Live Run Sequence: 
      docker run -d \
  --name=pihole \
  --net=host \
  -e TZ="America/Chicago" \
  -e FTLCONF_webserver_api_password="WebLoginPasswordHere" \
  -e FTLCONF_dns_listeningMode="ALL" \
  -e FTLCONF_dns_upstreams="1.1.1.1;9.9.9.9" \
  -v pihole_config:/etc/pihole \
  -v dnsmasq_config:/etc/dnsmasq.d \
  --restart=unless-stopped \
  pihole/pihole:latest

        • --net=host: Bypassed broken virtual bridge layers, mapping directly to VLAN 7 network cards.
        • FTLCONF_webserver_api_password: Resolved the deprecated WEBPASSWORD v5 variable problem.
        • FTLCONF_dns_listeningMode="ALL": Opened cross-subnet resolution rules, allowing external VLAN query targets.
        • FTLCONF_dns_upstreams: Seaded upstream parameters to prevent silent drops.

🔒 Phase 6: Core System Persistence (Automation on Boot)

Because raw binary files do not auto-load rules or daemons natively when an LXC starts, a custom boot initialization script was injected to force persistence across power outages.
cat << 'EOF' > /etc/init.d/docker
#!/bin/sh
### BEGIN INIT INFO
# Provides:          docker
# Required-Start:    $network $remote_fs
# Required-Stop:     $network $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start Docker Daemon
### END INIT INFO

case "$1" in
  start)
    echo "Starting Docker Daemon..."
    # Carve runtime networks back into memory paths
    nft add table inet filter 2>/dev/null
    nft add chain inet filter input { type filter hook input priority 0 \; policy accept \; } 2>/dev/null
    nft add rule inet filter input udp dport 53 accept 2>/dev/null
    nft add rule inet filter input tcp dport 53 accept 2>/dev/null
    # Run core daemon engine
    /usr/bin/dockerd > /var/log/dockerd.log 2>&1 &
    ;;
  stop)
    echo "Stopping Docker Daemon..."
    killall dockerd
    ;;
  restart)
    $0 stop
    sleep 2
    $0 start
    ;;
  *)
    echo "Usage: $0 {start|stop|restart}"
    exit 1
    ;;
esac
exit 0
EOF

chmod +x /etc/init.d/docker
update-rc.d docker defaults
Holy Ballssacks bro, why does LXc containers have to be such a bitch.. all it should be is define the imagine you want to use, and the properties you want to define just like the docker deployment, i shouldn’t have to setup a LXC to run docker to run a proper container… fucking LXC.
Direct Comparison of Approaches
Feature 🛠️ The Video’s Approach (Native Script) 🟦 Our Approach (Docker inside LXC)
Architecture LXC ➡️ Pi-hole Script LXC ➡️ Docker ➡️ Pi-hole Container
How it works The script installs software packages (lighttpd, php, pihole-FTL) directly into the Debian template file system, just like a traditional VM or physical server. The Debian template is treated as a clean, blank slate. Docker manages the dependencies, keeping the Pi-hole completely sealed off in its own application layer.
Internet Needs Requires constant, wide-open internet during installation to pull dozens of Linux package dependencies via apt. Can be done 100% offline. We only needed a tiny firewall hole for the single pre-packaged image file.
Upgrades You must run pihole -up inside the console, which modifies the system packages and can break if dependencies drift. To upgrade, you simply delete the container and spin up a new image version. Your underlying configuration directories stay untouched.
Flexibility If you want to run another tool (like a dashboard or home automation), its packages mix with Pi-hole’s web server settings, often causing conflicts. You can spin up 5 more isolated Docker containers inside this same LXC host without them ever interfering with Pi-hole.
Why did the video creator do it that way?
The video creator used the native script because it is the easiest path for a standard home lab with wide-open internet access. You just paste one command (curl -sSL https://pi-hole.net | bash), click “Next” on a few menus, and it handles everything automatically. They treat the LXC exactly like a lightweight VM because they don’t have strict network constraints.
Why were we forced to use our approach?
Your specific environment made the video’s method nearly impossible for two massive reasons:
    1. Your PVE Host is Offline: The native Pi-hole installer script expects to talk to official Debian package repositories to download web servers and compilation tools. Because your host couldn’t reach the open web, that script would have crashed immediately on missing dependencies.
    2. Your Strict Palo Alto Firewall: Even if we got the script running, a standard Linux installation queries dozens of different repository mirrors across random IP addresses and domain spaces. Your Palo Alto would have blocked 90% of them. By using Docker, we only had to tell the firewall to trust one single destination (github.com) to get the container fully populated.
Our approach required a lot more upfront engineering, but you ended up with a far more resilient setup. Your core Debian system remains pristine, and your Pi-hole is perfectly contained.
But wait you must be asking, how do we manage it. Well, the web interface will be available for most administrative work.
Why it doesn’t just go to the dashboard login… unno… anyway.  “Nooo, like the container…” well you should have access to the PVE host, enter the LXC with “pct enter 105”, replacing 105 with the ID of your LXC container.  “nooo, the docker container”
ohh…
root@docker-host:~# docker exec -it pihole /bin/bash
failed to create runc console socket: stat /run/user/0: no such file or directory: unknown
root@docker-host:~# mkdir -p /run/user/0
root@docker-host:~# docker exec -it pihole /bin/bash
docker-host:/#
look we’re in… but why doesn’t it have its own hostname?
“The reason you are seeing docker-host inside the container is because your container is running on the host network mode (–net=host).In Docker, when a container uses the host network stack, it bypasses Docker’s standard network isolation. It shares the host’s network namespace, IP addresses, network interfaces, and hostname directly.”
What about updates? … Well, adjusted security rule to allow app: apt-get, and destination security.debian.org and deb.debian.org, run apt update and apt upgrade, good to go. Just make a backup first. Pi-hole updates? whent he latest container release is available, simple bring down the instance, pull the latest image, and bring the container back up.
I wish containers were a bit different on PVE, but I also understand their complexities over advanced deployments. but it would be nice if they had simple FOSS containers for simple stuff like Pi-Hole, let me click add containerized app in the PVE host menu, pick the app (in this case Pi-Hole) specify the options (IP[vlan], DNS, Hostname) click deploy and it just runs the container on the PVE host itself and be done with it, updates then would just be stop the container, pull latest image, bring the container up.. none of this having to manage an LXC with its own apps and libraries and have to run apt update on it… ehhh whatever, I’ll just move on at this point lol…
Man, but I can’t back it up using Veeam, you can only backup VMs. So, unless I want to manage to backup solutions Veeam + PVE Backup Server, I’d have to convert to a VM instead of an LXC anyway. The main advantage was RAM savings..
vs
well **** I might just end up throwing this in the trash and replacing it with a VM running docker instead… **** me, well.. guess you live and you learn. but it’s still a super tight setup if someone wanted to do PiHole on LXC, while being able to do a regular docker style pull instead of a raw dawg install,
Hope this helps someone.
*Update* I had an issue where pihole wasn’t starting on boot, so used crontab to resolve it.
  • Run crontab -e .
  • Arrow down to the very bottom line.
  • Paste the fallback rule:
    text
    @reboot dockerd > /var/log/dockerd.log 2>&1 &
Legal Information Proudly powered by WordPress