Posted on by Aemilianus Kehler

The Poor Man’s VAAI for ZFS-over-iSCSI

The Poor Man’s VAAI: Orchestrating an Offline ZFS-over-iSCSI Storage Migration Manually

If you run a homelab or an enterprise environment utilizing Proxmox VE (PVE) backed by a TrueNAS storage array via a community ZFS-over-iSCSI plugin (like the popular TheGrandWazoo implementation), you have likely run into the limits of the API abstraction layer.
Try to perform a cross-pool storage migration (moving a virtual hard drive from one ZFS pool to another on the same TrueNAS machine), and you are likely to get slapped with this cryptic failure:
TASK ERROR: storage migration failed: TrueNAS API POST /iscsi/extent: 422 Unprocessable Entity — iscsi_extent_create.name: Extent name must be unique

Here is a deep technical breakdown of why this happens, how the hypervisor configuration misleads you, and how to execute a manual, bare-metal block migration—essentially building a poor man’s offline VAAI (vSphere Storage API for Array Integration) for ZFS over iSCSI.

The Architectural Root Cause

To understand the failure, you have to look at the intersection of three components: TrueNAS’s global iSCSI architecture, the Proxmox plugin layer, and the PVE VM configuration file.

1. TrueNAS Manage iSCSI Globally

TrueNAS maintains a centralized, system-wide configuration database for its iSCSI daemon (ctld or SCST). When an initiator requests a target, TrueNAS maps that connection globally. It enforces strict namespace uniqueness across the entire machine for Extents, Targets, and Associated Targets. TrueNAS does not care if you have two isolated ZFS pools; you cannot register the same extent name twice on the same machine.

2. The Proxmox Plugin Collision

When you define two separate TrueNAS-backed iSCSI datastores in Proxmox, they usually point to the same TrueNAS Host Portal IP.
The storage plugin hardcodes the iSCSI extent naming convention using the VM ID and slot number: vm-ID-disk-X.
When you initiate a storage migration to the other datastore via the Proxmox GUI, the plugin tries to provision the new storage before tearing down the old one. It fires off a POST request to the TrueNAS API to create an extent named vm-108-disk-1 on the destination pool. Because that exact name already exists in TrueNAS’s global registry (pointing to the source pool), TrueNAS immediately rejects it with a 422 Unprocessable Entity.

3. The PVE .conf File Illusion

In an iSCSI architecture, the storage configuration string in Proxmox is just a pointer. It tells Proxmox to talk to the storage daemon associated with that label and map a specific disk ID. Because both Proxmox datastore definitions map to the same target IP portal, Proxmox connects to the exact same network destination. TrueNAS sees the incoming request for vm-108-disk-1 and points the hypervisor right back to the original active iSCSI session on the old pool.

Poor Man’s Offline VAAI: The Manual Fix

In an enterprise environment with a SAN that natively supports VAAI block offloading, a storage migration issues a single command telling the array to clone blocks and re-map the LUN at the hardware layer—consuming zero hypervisor CPU or network bandwidth.
Since the Proxmox plugin lacks the logic to orchestrate this natively across multiple TrueNAS pools, we can act as the orchestration engine ourselves using raw ZFS and iSCSI manipulation.

Step 1: Detach the Disk in Proxmox

To safely move a block storage device without corrupting data, the file systems must be unmounted and the iSCSI lock freed.
  1. Shut down the target VM completely.
  2. In Proxmox, navigate to the VM ➔ Hardware ➔ Select the target drive (e.g., scsi0) ➔ Click Detach.
  3. The drive will drop down to the bottom of the list as an Unused Disk.

Step 2: Backend ZFS Block Migration (TrueNAS CLI)

Instead of forcing data across your network adapter via the hypervisor, leverage native ZFS snapshot streaming directly inside the TrueNAS kernel.
  1. SSH into TrueNAS or open the Web UI Shell.
  2. Escalate privileges to root using sudo.
  3. Take an atomic, read-only snapshot of the source Zvol: 
    sudo zfs snapshot BigBurtha/vm-108-disk-1@move

  4. Stream the raw data blocks directly into the destination pool (LexarPool) using a ZFS send/receive pipe. The -vR flags enable a progress bar and maintain replication properties: 
    sudo zfs send -vR BigBurtha/vm-108-disk-1@move | sudo zfs recv LexarPool/vm-108-disk-1

Step 3: Atomic LUN Mapping Flip (TrueNAS GUI)

Now that the data blocks reside on the new physical disks, you must rewrite the iSCSI pointer inside the TrueNAS database so the global network target routes to the new destination pool.
  1. In the TrueNAS Web UI, navigate to Shares ➔ Block Shares (iSCSI) ➔ Extents.
  2. Find the extent matching your disk (vm-108-disk-1) and click Edit.
  3. Locate the Source / Zvol Path dropdown menu. It will currently point to your old pool path: /dev/zvol/BigBurtha/vm-108-disk-1.
  4. Flip this dropdown to point to the new pool path: /dev/zvol/LexarPool/vm-108-disk-1.
  5. Save the configuration.

Step 4: Re-link and Boot

Go back to the Proxmox Web UI. Select the Unused Disk at the bottom of the VM hardware configuration panel, click Edit/Add, and reassign it back to its original slot (e.g., scsi0).
When you boot the VM, Proxmox will query the iSCSI target portal exactly as it did before. TrueNAS receives the request, references its updated global registry database, and serves the blocks directly out of LexarPool. Run a disk I/O benchmark utility inside the guest OS, and you will see the hardware statistics instantly register activity on the new storage array.

Step 5: Post-Migration Cleanup

Once the VM is verified stable and running on the new pool, the iSCSI service will release its execution locks on the old Zvol path, allowing you to cleanly reclaim your storage capacity.
From the TrueNAS Shell, execute the final destruction commands:
# Delete the migration snapshot from the original pool
sudo zfs destroy BigBurtha/vm-108-disk-1@move

# Wipe the old Zvol block device entirely
sudo zfs destroy BigBurtha/vm-108-disk-1

# Optional: Clean up the temporary snapshot on the new pool
sudo zfs destroy LexarPool/vm-108-disk-1@move

You might be wondering if utilizing zfs snapshot did I need to power down and detach the disk as step 1?
Yes, you are 100% correct. You absolutely could have done the ZFS snapshot and copy task completely live while the VM was running, reducing your actual downtime to just a few seconds.
Taking a ZFS snapshot is an atomic operation at the block layer; it takes less than a second to freeze the blocks, regardless of whether the VM is actively writing data or not.
By restructuring the order of operations, you can turn this into a near-live storage migration with minimal disruption.

The Live Migration Workflow (Zero-Downtime Copy)

Here is how you would optimize the steps for a blog post to achieve an almost seamless migration:

1. The Live ZFS Send/Recv (Zero Downtime)

While the VM is fully booted and handling active I/O, you run the initial snapshot and block transfer. ZFS will copy all the existing data blocks over to LexarPool.
sudo zfs snapshot BigBurtha/vm-108-disk-1@live-migrate
sudo zfs send -vR BigBurtha/vm-108-disk-1@live-migrate | sudo zfs recv LexarPool/vm-108-disk-1

(The VM stays online the entire time this data streams across your pools).

2. The Catch-Up Snapshot (Optional but Recommended)

If the block copy took a long time (e.g., a massive multi-terabyte drive), the VM will have written new data to BigBurtha while the copy was running. To minimize downtime even further, you take a second incremental snapshot to sync just the changes:
sudo zfs snapshot BigBurtha/vm-108-disk-1@sync
sudo zfs send -vR -i BigBurtha/vm-108-disk-1@live-migrate BigBurtha/vm-108-disk-1@sync | sudo zfs recv LexarPool/vm-108-disk-1

3. The “Micro-Downtime” Cutover

This is the only part where you have to stop the VM, because you cannot re-route an active iSCSI session while TrueNAS holds an open kernel lock on the running file system.
  1. Shut down the VM in Proxmox.
  2. Run a final incremental ZFS sync to catch the last few blocks written during the shutdown sequence: 
    sudo zfs snapshot BigBurtha/vm-108-disk-1@final
sudo zfs send -vR -i BigBurtha/vm-108-disk-1@sync BigBurtha/vm-108-disk-1@final | sudo zfs recv LexarPool/vm-108-disk-1

  3. In the TrueNAS Web UI (Shares ➔ Block Shares ➔ Extents), edit the vm-108-disk-1 extent and flip the dropdown path from BigBurtha to LexarPool.
  4. Update your Proxmox .conf file label if you want the storage names to look tidy (though as we discovered, this part is just cosmetic for the iSCSI portal routing).
  5. Boot the VM back up.

Why this is even closer to real VAAI

By doing the heavy lifting (the bulk data transfer) while the VM is fully operational, your total downtime drops from hours (waiting for network copies or local block transfers to finish while the VM is offline) to literally 30 seconds—just long enough to stop the VM, flip the TrueNAS iSCSI mapping dropdown, and hit start again.
This approach makes the “Poor Man’s VAAI” title fit even better, because it mirrors exactly how enterprise storage arrays handle asynchronous storage migration cutovers!

Summary

When software automation plugins fail to scale across architectural boundaries, understanding the underlying storage routing allows you to manipulate data directly at the block layer. By combining raw ZFS snapshot pipes with target remapping, you bypass API namespace limits completely—achieving zero-network-overhead backend replication.

Leave a Reply

Your email address will not be published. Required fields are marked *

Legal Information Proudly powered by WordPress