How I Configured Least-Privilege ZFS over iSCSI for Proxmox VE via TrueNAS

The built-in Proxmox VE “ZFS over iSCSI” storage provider relies on direct root SSH access to TrueNAS to dynamically spin up block devices. However, this legacy method is broken on modern TrueNAS SCALE versions (like Dragonfish and Electric Eel) because TrueNAS no longer uses the target engines (like Comstar/LIO) expected by PVE.

To solve this securely without enabling risky root SSH access, I installed the “official TrueNAS Proxmox VE Storage Plugin” aka plugin managed by “TheGreatWazoo”, worked around network anomalies, and configured a custom Role-Based Access Control (RBAC) policy for a true least-privilege service account.

Step 1: Setup TrueNAS

Virtualizing TrueNAS: CPU NUMA and Memory Logistics

- Bypassing the CPU NUMA Trap: Since my TrueNAS virtual machine is small enough to fit comfortably inside a single physical CPU socket and its local memory pool, I left virtual NUMA (vNUMA) disabled in my hypervisor. Enabling it on a small VM would force the guest OS to waste processing cycles managing artificial boundaries. It would also restrict hypervisor scheduling, pinning my storage VM to specific sockets even if other sockets are completely idle. Leaving it disabled allows the hypervisor to handle optimization automatically via UMA (Uniform Memory Access) mode.
- Fixing the RAM Allocation: TrueNAS was initially starved at a default 2048 MiB (2 GB), which is an instant recipe for an Out-Of-Memory (OOM) kernel crash under iSCSI loads. I scaled the allocation up to 8192 MiB (8 GB) (with a ceiling of 12288 MiB / 12 GB if needed). This provides enough overhead for the base OS middleware and the 4TB ZFS metadata.
- Disabling Memory Ballooning: Because ZFS calculates its cache (ARC) based on a fixed expectation of available memory, I disabled the ballooning device and set fixed memory. If the host tried to dynamically reclaim RAM mid-test, the TrueNAS kernel would instantly panic and drop my storage network offline.

After successfully installing TrueNAS, I presses 1 at the console to configure the interface.

I noticed I could only toggle DHCPv4 and DHCPv6 on or off, but the actual IP address input field seemed completely missing.

Here is how I finally managed to configure my static IP:

- I navigated down to the Aliases field. In TrueNAS SCALE, static IPs are assigned as aliases rather than a traditional static IP entry field.
- I selected the empty list. I used my arrow keys to highlight aliases: <empty list> and pressed Enter to open the edit sub-menu.
- I entered my IP with CIDR notation. I typed my desired address and subnet mask combined (for example, 192.168.1.100/24) instead of typing the netmask separately.
- I saved the changes. I scrolled down to the <Save> button at the bottom of the screen and pressed Enter to apply the network settings.

Only then I could go to configure network settings and define the gateway and DNS servers. lil weird but whatever, easy enough.

Then just like before HBA to passthrough the hard drives to the VM

Just like before, created 2 isolated networks (one mgmt plane (HTTPS/SSH) and one data plane (iSCSI). *Note* I’m using VLANs (on the hypervisor for the VM guest) to simulate what you normally would do physically. So the configuration on TrueNAS looks the exact same as physical (two dedicated NICs for the two planes) AND… MPIO will not be covered in the scope of this blog. I’ll have to cover that in the future when I get some hosts with multiple NICs.

Step 2: TrueNAS Service Account & API Configuration

To avoid assigning global administrative rights, I built a dedicated service user and a custom privilege profile to authorize specific storage metadata calls.

1. Account Creation Constraints

I navigated to Credentials > Users to create my service account container:

- Username: pve-iscsi
- TrueNAS Access: Checked the box and selected Sharing Administrator rather than a generic read-only or full admin profile.
  - Create the account, Give it a Name, Check off TrueNAS Access, select read-only admin.
- Shell/SSH Access: Kept unchecked. The plugin communicates purely via HTTPS REST API calls, so command-line execution is unnecessary.

🛠️ Caveat & Tweak: This creates a less restrictive account, it also creates a group with the same name as the account. We will be creating a custom “Privilege” Role with only the required “Roles” permissions, then bind that role to this group to complete the least privilege configuration.

User:
Group:

As you can see the group created here shows each permission already defined on this account (snippet here is after the needful was done), but you need to create the actual Role by clicking on the Privileges button next to add. Create a “Privilege” and name it after this setup.

Under “Roles” aka Permissions I granted this “Privilege”, what I call a Role,

1. Sharing (iSCSI Track)

- Sharing iSCSI Extent Read
- Sharing iSCSI Extent Write
- Sharing iSCSI Initiator Read
- Sharing iSCSI Initiator Write
- Sharing iSCSI Portal Read
- Sharing iSCSI Portal Write
- Sharing iSCSI Target Read
- Sharing iSCSI Target Write
- Sharing iSCSI Target Extent Read
- Sharing iSCSI Target Extent Write

2. Storage & Snapshots (ZFS Track)

- Dataset Read
- Dataset Write
- Dataset Delete (Crucial so Proxmox can delete VM disks)
- Snapshot Read
- Snapshot Write
- Snapshot Delete (Crucial so Proxmox can clear old snapshots/backups)
- Pool Read (Allows the API to safely query the storage layout tree)

3. System Base Discovery

- System General Read (Allows the API to check system info and uptime metrics)

4. The Breakthrough Secret (Filesystem Track)

- Filesystem Read
- Filesystem Write

(Note: Keeping Allow All Sudo Commands and Sudo without Password unchecked on the user account ensures this remains a strict least-privilege setup.)

As you can see in the snip once you bind this custom role the GUI only supports the built in “Privileges”, so if you edit this account it’ll show the TrueNAS Access checked off with the dropdown empty:

Step 3. Generating the API Token as Admin

Instead of logging out and accessing the UI under the new service account context, I provisioned the token from my master administrator account:

1. Navigated to Credentials > Users and clicked on my new pve-iscsi user row.
2. Inside the profile card’s Access widget, I clicked Add API Key.
3. Named the token (e.g., PVE-iSCSI-Token) and copied the generated string immediately.

Step 4. Preparing the TrueNAS iSCSI Network Targets

Before connecting the hypervisors, I gathered configuration parameters and eased network access limits under Apps/Services > iSCSI.

1. Bound iSCSI: Ensure iSCSI Service is bound to iSCSI NIC only.
2. The Portal ID: Verified my active iSCSI listening group ID under the Portals tab (typically ID 1).
3. The Target IQN: Copied my system base name string under Target Global Configuration (iqn.2005-10.org.freenas.ctl).
4. The Initiator Workaround: Under the Initiators tab, I enabled Allow All Initiators and saved it as Initiator ID 1.

💡 Network Tweak: In a trusted home lab or private storage network, allowing all initiators simplifies initial setup. For strict isolation, this can be unchecked later by pasting the unique IQN strings of each individual Proxmox node directly into the allowed targets box.

Step 5. Driver Installation on Proxmox VE Nodes

Because storage modules run at the system level rather than the cluster level, I executed these steps directly on every node in my cluster via SSH/Shell.

# 0. Firewall:

I had to configure a firewall rule to allow access to the DEVs githubsource.

URL: thegrandwazoo.github.io/

Application: Github-Pages

# 1. Download and register the official repository GPG key:

curl -fsSL https://thegrandwazoo.github.io/freenas-proxmox/public.gpg.key | gpg --dearmor | tee /etc/apt/keyrings/truenas-proxmox.gpg > /dev/null

# 2. Inject the driver repository metadata block using the DEB822 standard format:

cat << 'SOURCES' | tee /etc/apt/sources.list.d/truenas-proxmox.sources
Types: deb URIs: https://thegrandwazoo.github.io/freenas-proxmox
Suites: v3
Components: main
Signed-By: /etc/apt/keyrings/truenas-proxmox.gpg
SOURCES

# 3. Synchronize package databases and install the kernel driver plugin:

apt update apt install truenas-proxmox -y

Installing this package successfully registers the truenas-iscsi framework and automatically recycles local management services (pvedaemon, pveproxy) to render the integration options in the UI.

Step 6. Mounting Storage and Navigating GUI Flaws

I refreshed my Proxmox browser interface, navigated to Datacenter > Storage > Add, and selected the newly exposed TrueNAS iSCSI storage provider.

The Connection Profile Layout

- ID: TrueNAS-ZFS (or any friendly cluster name)
- Nodes: Set to All
- TrueNAS Host: 172.16.21.2 (Management network interface)
- Portal IP: 69.69.69.1 (The isolated network interface bound to the iSCSI portal)
- Pool: LexarPool (Case-sensitive master storage pool name)
- API Token: [Pasted the custom service token generated in Phase 1]

Hitting Add automatically builds the connection link, turning my cluster resource status icon green.

Core Architecture Takeaway: The Proxmox Offline Migration “Flaw”

During testing, I stumbled on a major counterintuitive design choice in how Proxmox handles moving virtual machine storage: Proxmox allows you to move local storage blocks to another node effortlessly if the VM is powered ON, but throws a hard block if the VM is powered OFF.

Why this happens under the hood:

- Online Migration (VM ON): Proxmox delegates operations to the QEMU/KVM virtualization engine. QEMU handles the network stream natively, actively duplicating running disk blocks to an entirely different storage name on the destination node on the fly.
- Offline Migration (VM OFF): Proxmox bypasses QEMU and drops back to its legacy internal cluster scripts. This simple script reads the text configuration of the VM, checks the source storage name (e.g., msiNM620), and checks if an identical storage name exists on the destination host. If the destination node does not have a local storage engine with that exact name, the script panics and blocks the migration through the UI.

How TrueNAS fixes this behavior:

Because my new TrueNAS integration is defined as shared storage named identically (TrueNAS-ZFS) across all nodes in my datacenter, it completely satisfies Proxmox’s basic offline script constraints. When a VM’s disks live on the TrueNAS storage block, the offline migration wizard unblocks instantly, allowing lightning-fast configuration transfers between cluster nodes without moving any underlying data blocks.

I had weird errors when moving from shared storage back to local (this was when I had the setup using full local administrator “Privileges”, it didnt’ cause the task to fail and AI said this about.

When copying virtual machine disk blocks off of the TrueNAS iSCSI pool back onto local storage, you may notice repeating logs stating:
qemu-img: iSCSI GET_LBA_STATUS failed... SENSE KEY:ILLEGAL_REQUEST(5)

Why it triggers:

During a file transfer copy block, qemu-img sequentially pings the TrueNAS target engine with GET_LBA_STATUS calls to query which specific disk block ranges are blank or sparse so it can bypass copying empty space. Because the TrueNAS storage framework handles these optimizations at a different filesystem descriptor tier, it rejects the application block format structure, generating the standard warning lines.

The Impact:

This error is entirely harmless. The script immediately falls back to basic raw block-by-block copying and finishes streaming the virtual machine disk data over the network safely with zero block corruption.

What happens on hosts that have no installed the plug in? I dunno haven’t tested, I’m assuming they won’t have access to the storage.

What happens if the iSCSI server has a hard fail? Dunno, I’ll find out soon in my lab as I need to move the server. Any new findings I’ll update this blog post.

Hope this helps someone, feel free to leave a comment.

Least-Privilege ZFS over iSCSI for Proxmox VE via TrueNAS