{"id":1894,"date":"2026-10-09T00:12:05","date_gmt":"2026-10-09T05:12:05","guid":{"rendered":"https:\/\/zewwy.ca\/?p=1894"},"modified":"2026-10-09T00:12:05","modified_gmt":"2026-10-09T05:12:05","slug":"the-poor-mans-vaai-for-zfs-over-iscsi","status":"publish","type":"post","link":"https:\/\/zewwy.ca\/index.php\/2026\/10\/09\/the-poor-mans-vaai-for-zfs-over-iscsi\/","title":{"rendered":"The Poor Man\u2019s VAAI for ZFS-over-iSCSI"},"content":{"rendered":"
<\/div>\n

<\/span>The Poor Man\u2019s VAAI: Orchestrating an Offline ZFS-over-iSCSI Storage Migration Manually<\/span><\/h1>\n
If you run a homelab or an enterprise environment utilizing Proxmox VE (PVE)<\/span> backed by a TrueNAS<\/span> storage array via a community ZFS-over-iSCSI plugin (like the popular TheGrandWazoo<\/em> implementation), you have likely run into the limits of the API abstraction layer.<\/div>\n
<\/div>\n
Try to perform a cross-pool storage migration (moving a virtual hard drive from one ZFS pool to another on the same TrueNAS machine), and you are likely to get slapped with this cryptic failure:<\/div>\n
\n
TASK ERROR: storage migration failed: TrueNAS API POST \/iscsi\/extent: 422 Unprocessable Entity \u2014 iscsi_extent_create.name: Extent name must be unique\r\n<\/code><\/pre>\n
<\/div>\n
Here is a deep technical breakdown of why this happens, how the hypervisor configuration misleads you, and how to execute a manual, bare-metal block migration\u2014essentially building a poor man\u2019s offline VAAI (vSphere Storage API for Array Integration)<\/span> for ZFS over iSCSI.<\/div>\n
<\/span>The Architectural Root Cause<\/span><\/h1>\n
To understand the failure, you have to look at the intersection of three components: TrueNAS’s global iSCSI architecture, the Proxmox plugin layer, and the PVE VM configuration file.<\/div>\n
<\/span>1. TrueNAS Manage iSCSI Globally<\/span><\/h2>\n
TrueNAS maintains a centralized, system-wide configuration database for its iSCSI daemon (ctld<\/code> or SCST<\/code>). When an initiator requests a target, TrueNAS maps that connection globally. It enforces strict namespace uniqueness<\/span> across the entire machine for Extents, Targets, and Associated Targets. TrueNAS does not care if you have two isolated ZFS pools; you cannot register the same extent name twice on the same machine.<\/div>\n
<\/span>2. The Proxmox Plugin Collision<\/span><\/h2>\n
When you define two separate TrueNAS-backed iSCSI datastores in Proxmox, they usually point to the same TrueNAS Host Portal IP<\/span>.
The storage plugin hardcodes the iSCSI extent naming convention using the VM ID and slot number: vm-ID-disk-X<\/code>.<\/div>\n
When you initiate a storage migration to the other<\/em> datastore via the Proxmox GUI, the plugin tries to provision the new storage before tearing down the old one. It fires off a POST<\/code> request to the TrueNAS API to create an extent named vm-108-disk-1<\/code> on the destination pool. Because that exact name already exists in TrueNAS\u2019s global registry (pointing to the source pool), TrueNAS immediately rejects it with a 422 Unprocessable Entity<\/span>.<\/div>\n
<\/span>3. The PVE .conf<\/code> File Illusion<\/span><\/h2>\n
In an iSCSI architecture, the storage configuration string in Proxmox is just a pointer<\/span>. It tells Proxmox to talk to the storage daemon associated with that label and map a specific disk ID. Because both Proxmox datastore definitions map to the same target IP portal, Proxmox connects to the exact same network destination. TrueNAS sees the incoming request for vm-108-disk-1<\/code> and points the hypervisor right back to the original active iSCSI session on the old pool.<\/div>\n
<\/span>Poor Man’s Offline VAAI: The Manual Fix<\/span><\/h1>\n
In an enterprise environment with a SAN that natively supports VAAI block offloading<\/span>, a storage migration issues a single command telling the array to clone blocks and re-map the LUN at the hardware layer\u2014consuming zero hypervisor CPU or network bandwidth.<\/div>\n
<\/div>\n
Since the Proxmox plugin lacks the logic to orchestrate this natively across multiple TrueNAS pools, we can act as the orchestration engine ourselves using raw ZFS and iSCSI manipulation.<\/div>\n
<\/span>Step 1: Detach the Disk in Proxmox<\/span><\/h2>\n
To safely move a block storage device without corrupting data, the file systems must be unmounted and the iSCSI lock freed.<\/div>\n
\n
<\/div>\n
    \n
  1. Shut down the target VM completely.<\/li>\n
  2. In Proxmox, navigate to the VM \u2794 Hardware<\/span> \u2794 Select the target drive (e.g., scsi0<\/code>) \u2794 Click Detach<\/span>.<\/li>\n
  3. The drive will drop down to the bottom of the list as an Unused Disk<\/em>.<\/li>\n<\/ol>\n
    <\/div>\n
    <\/span>Step 2: Backend ZFS Block Migration (TrueNAS CLI)<\/span><\/h2>\n
    Instead of forcing data across your network adapter via the hypervisor, leverage native ZFS snapshot streaming directly inside the TrueNAS kernel.<\/div>\n
    \n
    <\/div>\n
      \n
    1. SSH into TrueNAS or open the Web UI Shell.<\/li>\n
    2. Escalate privileges to root using sudo<\/code>.<\/li>\n
    3. Take an atomic, read-only snapshot of the source Zvol:\n
      \n
      sudo zfs snapshot BigBurtha\/vm-108-disk-1@move\r\n<\/code><\/pre>\n
      <\/div>\n
      <\/li>\n
    4. Stream the raw data blocks directly into the destination pool (LexarPool<\/code>) using a ZFS send\/receive pipe. The -vR<\/code> flags enable a progress bar and maintain replication properties:\n
      \n
      sudo zfs send -vR BigBurtha\/vm-108-disk-1@move | sudo zfs recv LexarPool\/vm-108-disk-1\r\n<\/code><\/pre>\n
      <\/div>\n
      <\/li>\n<\/ol>\n
      <\/div>\n
      <\/span>Step 3: Atomic LUN Mapping Flip (TrueNAS GUI)<\/span><\/h2>\n
      Now that the data blocks reside on the new physical disks, you must rewrite the iSCSI pointer inside the TrueNAS database so the global network target routes to the new destination pool.<\/div>\n
      \n
      <\/div>\n
        \n
      1. In the TrueNAS Web UI, navigate to Shares<\/span> \u2794 Block Shares (iSCSI)<\/span> \u2794 Extents<\/span>.<\/li>\n
      2. Find the extent matching your disk (vm-108-disk-1<\/code>) and click Edit<\/span>.<\/li>\n
      3. Locate the Source \/ Zvol Path<\/span> dropdown menu. It will currently point to your old pool path: \/dev\/zvol\/BigBurtha\/vm-108-disk-1<\/code>.<\/li>\n
      4. Flip this dropdown to point to the new pool path: \/dev\/zvol\/LexarPool\/vm-108-disk-1<\/code><\/span>.<\/li>\n
      5. Save the configuration.<\/li>\n<\/ol>\n
        <\/div>\n
        <\/span>Step 4: Re-link and Boot<\/span><\/h2>\n
        Go back to the Proxmox Web UI. Select the Unused Disk<\/em> at the bottom of the VM hardware configuration panel, click Edit\/Add<\/span>, and reassign it back to its original slot (e.g., scsi0<\/code>).<\/div>\n
        When you boot the VM, Proxmox will query the iSCSI target portal exactly as it did before. TrueNAS receives the request, references its updated global registry database, and serves the blocks directly out of LexarPool<\/code>. Run a disk I\/O benchmark utility inside the guest OS, and you will see the hardware statistics instantly register activity on the new storage array.<\/div>\n
        <\/span>Step 5: Post-Migration Cleanup<\/span><\/h2>\n
        Once the VM is verified stable and running on the new pool, the iSCSI service will release its execution locks on the old Zvol path, allowing you to cleanly reclaim your storage capacity.<\/div>\n
        From the TrueNAS Shell, execute the final destruction commands:<\/div>\n
        \n
        # Delete the migration snapshot from the original pool\r\nsudo zfs destroy BigBurtha\/vm-108-disk-1@move\r\n\r\n# Wipe the old Zvol block device entirely\r\nsudo zfs destroy BigBurtha\/vm-108-disk-1\r\n\r\n# Optional: Clean up the temporary snapshot on the new pool\r\nsudo zfs destroy LexarPool\/vm-108-disk-1@move\r\n<\/code><\/pre>\n
        <\/div>\n
        You might be wondering if utilizing zfs snapshot did I need to power down and detach the disk as step 1?<\/div>\n
        <\/div>\n
        \n
        Yes, you are 100% correct.<\/span><\/mark> You absolutely could have done the ZFS snapshot and copy task completely live<\/span> while the VM was running, reducing your actual downtime to just a few seconds.<\/div>\n
        Taking a ZFS snapshot is an atomic operation<\/span> at the block layer; it takes less than a second to freeze the blocks, regardless of whether the VM is actively writing data or not.<\/div>\n
        By restructuring the order of operations, you can turn this into a near-live storage migration<\/span> with minimal disruption.<\/div>\n
        <\/span>The Live Migration Workflow (Zero-Downtime Copy)<\/span><\/h2>\n
        Here is how you would optimize the steps for a blog post to achieve an almost seamless migration:<\/div>\n
        <\/span>1. The Live ZFS Send\/Recv (Zero Downtime)<\/span><\/h2>\n
        While the VM is fully booted and handling active I\/O, you run the initial snapshot and block transfer. ZFS will copy all the existing data blocks over to LexarPool<\/code>.<\/div>\n
        \n
        sudo zfs snapshot BigBurtha\/vm-108-disk-1@live-migrate\r\nsudo zfs send -vR BigBurtha\/vm-108-disk-1@live-migrate | sudo zfs recv LexarPool\/vm-108-disk-1\r\n<\/code><\/pre>\n
        <\/div>\n
        (The VM stays online the entire time this data streams across your pools).<\/em><\/div>\n
        <\/span>2. The Catch-Up Snapshot (Optional but Recommended)<\/span><\/h2>\n
        If the block copy took a long time (e.g., a massive multi-terabyte drive), the VM will have written new data to BigBurtha<\/code> while the copy was running. To minimize downtime even further, you take a second incremental snapshot to sync just the changes:<\/div>\n
        \n
        sudo zfs snapshot BigBurtha\/vm-108-disk-1@sync\r\nsudo zfs send -vR -i BigBurtha\/vm-108-disk-1@live-migrate BigBurtha\/vm-108-disk-1@sync | sudo zfs recv LexarPool\/vm-108-disk-1\r\n<\/code><\/pre>\n
        <\/div>\n
        <\/span>3. The “Micro-Downtime” Cutover<\/span><\/h2>\n
        This is the only part where you have to stop the VM, because you cannot re-route an active iSCSI session while TrueNAS holds an open kernel lock on the running file system.<\/div>\n
        \n
        <\/div>\n
          \n
        1. Shut down the VM<\/span> in Proxmox.<\/li>\n
        2. Run a final incremental ZFS sync to catch the last few blocks written during the shutdown sequence:\n
          \n
          sudo zfs snapshot BigBurtha\/vm-108-disk-1@final\r\nsudo zfs send -vR -i BigBurtha\/vm-108-disk-1@sync BigBurtha\/vm-108-disk-1@final | sudo zfs recv LexarPool\/vm-108-disk-1\r\n<\/code><\/pre>\n
          <\/div>\n
          <\/li>\n
        3. In the TrueNAS Web UI (Shares \u2794 Block Shares \u2794 Extents<\/span>), edit the vm-108-disk-1<\/code> extent and flip the dropdown path from BigBurtha<\/code> to LexarPool<\/code>.<\/li>\n
        4. Update your Proxmox .conf<\/code> file label if you want the storage names to look tidy (though as we discovered, this part is just cosmetic for the iSCSI portal routing).<\/li>\n
        5. Boot the VM back up.<\/span><\/li>\n<\/ol>\n
          <\/div>\n
          <\/span>Why this is even closer to real VAAI<\/span><\/h2>\n
          By doing the heavy lifting (the bulk data transfer) while the VM is fully operational, your total downtime drops from hours<\/span> (waiting for network copies or local block transfers to finish while the VM is offline) to literally 30 seconds<\/span>\u2014just long enough to stop the VM, flip the TrueNAS iSCSI mapping dropdown, and hit start again.<\/div>\n
          This approach makes the “Poor Man’s VAAI” title fit even better, because it mirrors exactly how enterprise storage arrays handle asynchronous storage migration cutovers!<\/div>\n
          \n<\/div>\n
          \n
          <\/span>Summary<\/span><\/h1>\n
          When software automation plugins fail to scale across architectural boundaries, understanding the underlying storage routing allows you to manipulate data directly at the block layer. By combining raw ZFS snapshot pipes with target remapping, you bypass API namespace limits completely\u2014achieving zero-network-overhead backend replication.<\/div>\n<\/div>\n
          \n<\/div>\n
          <\/div>\n
          <\/div>\n<\/div>\n
          <\/div>\n
          \n<\/div>\n
          <\/div>\n
          <\/div>\n","protected":false},"excerpt":{"rendered":"
          The Poor Man\u2019s VAAI: Orchestrating an Offline ZFS-over-iSCSI Storage Migration Manually If you run a homelab or an enterprise environment utilizing Proxmox VE (PVE) backed by a TrueNAS storage array via a community ZFS-over-iSCSI plugin (like the popular TheGrandWazoo implementation), you have likely run into the limits of the API abstraction layer. Try to perform … <\/p>\n
          Continue reading “The Poor Man\u2019s VAAI for ZFS-over-iSCSI”<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,6,8,7],"tags":[369,461,507,504],"class_list":["post-1894","post","type-post","status-publish","format-standard","hentry","category-hypervisors","category-networking","category-server-administration","category-storage","tag-iscsi","tag-pve","tag-vaai","tag-zfs"],"_links":{"self":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts\/1894","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/comments?post=1894"}],"version-history":[{"count":1,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts\/1894\/revisions"}],"predecessor-version":[{"id":1895,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts\/1894\/revisions\/1895"}],"wp:attachment":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/media?parent=1894"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/categories?post=1894"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/tags?post=1894"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}