Using FreeNAS as a VM to Test Storage Speeds

OK sooo this is gonna seem kinda commical… I

1. set up a PVE host. Most of which ended up being about storage, I didn’t even blog the actual installation process, I just got right into the storage after a completed installation lol.
2. wrote a blog attempting to discuss managing a PVE host. A little better cover some basic host management stuff, but again end up talking about VM storage options and performance.. lol
3. wrote a blog post above recovering a VM from ESXi to PVE using Veeam. Which was also poor, most a just a video reference to a Veeam tech who shows the technical steps, then just me wondering why I got such unreal poor performance from my setup.

I don’t suggest you read any of them cause they are some of the worst blogs I have ever written. They are, however, not entirely useless as they provide some bases to the tests I continue to complete.

So now I had another thought, and can you guess what it was around… yeah… storage… anyway, so if LVMthin is not the best choice for Random IO, and even using the Linux kernel as cache causes issues, what if I just throw the SSDs to a FreeNAS VM. Will it perform better?

🛠️ Phase 1: Deconstructing My Original Proxmox LVM Storage

1. 1. Remove GUI mapping: I commanded Proxmox to stop monitoring the LVM-Thin dashboard pool:
      bash

      pvesm remove Striped-SSDs
      

   2. Deactivate and scrub the swap space: I disabled the active swap space running on the SSDs:
      bash

      swapoff -v /dev/pve-fast/fast-swap
      

   3. Remove swap from the boot configuration: I edited the filesystem table:
      bash

      nano /etc/fstab
      

      I located the active line /dev/pve-fast/fast-swap none swap sw 0 0 and deleted it (or added a # at the beginning) to prevent my system from hanging or crashing on its next boot.

   4. Destroy the Logical Volumes: I permanently purged the inner allocation containers:
      bash

      lvremove /dev/pve-fast/fast-data -y
      lvremove /dev/pve-fast/fast-swap -y
      
   5. Destroy the Volume Group: I deleted the master pool layout itself:
      bash

      vgremove pve-fast
      

   6. Wipe LVM Labels from Raw Disks: I forced LVM to entirely release its ownership tags over the physical hardware:
      bash

      pvremove /dev/sda /dev/sdb /dev/sdc
      
   7. Verify Everything is Clean: I ran the validation checks to ensure only my main OS boot drives remained:
      bash

      pvs
      vgs
      lvs
      

💾 Phase 2: Passing Individual Raw Disks to TrueNAS

1. 1. Locate My Persistent Disk IDs: I knew using changing identifiers like /dev/sdb would break my VM mapping if I added or removed hardware down the line. I ran this command to pull the unique hardware serial strings:
      bash

      ls -l /dev/disk/by-id/
      

      I copied down my three distinct Kingston identifiers:

      • • ata-KINGSTON_SA400S37240G_50026B7785138139
        • ata-KINGSTON_SA400S37240G_50026B77851380CD
        • ata-KINGSTON_SA400S37240G_50026B7785137D21

   2. Map the Disks into the VM: Using the Proxmox host shell, I manually bound the raw block devices to my TrueNAS VM (VM ID 101) sequentially on the SCSI controller, appending critical flags to disable host-level caching and keep Proxmox backup routines from touching my storage array:
      bash

      qm set 101 -scsi1 /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B7785138139,cache=none,backup=0
      qm set 101 -scsi2 /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B77851380CD,cache=none,backup=0
      qm set 101 -scsi3 /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B7785137D21,cache=none,backup=0
      
   3. Force Hardware Serial Numbers: When I first booted the VM, TrueNAS’s ZFS middleware threw validation errors. QEMU passes virtual disks as generic entities, meaning TrueNAS saw three separate paths sharing a blank or overlapping virtual serial identifier. I shut down the TrueNAS VM and edited the backend configuration file on my Proxmox host:
      bash

      nano /etc/pve/qemu-server/101.conf
      

      I navigated to the scsi1, scsi2, and scsi3 entry lines and appended ,serial= followed by their real hardware suffixes directly to the end of the config text:

      text

      scsi1: /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B7785138139,backup=0,cache=none,size=234431064K,serial=50026B7785138139
      scsi2: /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B77851380CD,backup=0,cache=none,size=234431064K,serial=50026B77851380CD
      scsi3: /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B7785137D21,backup=0,cache=none,size=234431064K,serial=50026B7785137D21
      

      After saving and booting TrueNAS up, the duplicate ID issue disappeared completely.

🌐 Phase 3: Building an Isolated Layer 2 Virtual Network

1. 1. Create the Private Bridge in Proxmox: In the Proxmox Web GUI, I went to Node -> System -> Network -> Create -> Linux Bridge.
      • • Name: vmbr1
        • IPv4/CIDR: 10.10.10.1/24
        • Gateway / Bridge Ports: Left completely blank.
        • This isolated the bridge entirely inside host memory, enabling packets to move at CPU speed without hitting a physical switch. I clicked Apply Configuration to spin it up live.

   2. Add a Second NIC to TrueNAS: In VM 101 -> Hardware -> Add -> Network Device.
      • • Bridge: Selected vmbr1
        • Firewall Checkbox: Unchecked. This was a critical step. By turning off the Proxmox software firewall for this card, I bypassed heavy packet inspection overhead, saving CPU cycles and ensuring lower latency for my storage loop.

   3. Configure the Storage IP in TrueNAS: I logged into my TrueNAS SCALE web dashboard, opened Network -> Interfaces, and edited the newly populated unconfigured adapter (e.g., vtnet1). I unchecked DHCP and manually assigned a flat Layer 2 static IP configuration:
      • • IP Address: 10.10.10.2
        • CIDR: 24
        • Gateway: Left completely blank to eliminate any potential multihoming asymmetric routing loops.

   4. Verify the Pipeline: I went into the Proxmox terminal and ran a quick check to make sure the internal memory link was intact:
      bash

      ping -c 3 10.10.10.2
      

      🏗️ Phase 4: Carving out Zvol and Provisioning the iSCSI SAN

   5. Create My Zvol Container: In TrueNAS SCALE, I went to the Datasets tab in the left panel. I selected my master pool (FastPool), clicked Add Zvol, and input the following configuration parameters:
      • • Zvol Name: pve-zvol
        • Size: 450 GiB
        • Sparse Volume: Unchecked (Thick Provisioned). I did this to intentionally carve out and lock down this exact slice of my 639 GiB total raw pool upfront. This prevents Proxmox from accidentally over-allocating storage down the line and protects my ZFS array from hitting 100% capacity and freezing.
        • Compression: LZ4 (Lightning fast, low CPU overhead, reduces physical write amplification by compressing data blocks before they hit flash).
        • ZFS Deduplication: OFF. (I verified this was disabled because dedupe consumes roughly 5GB of system RAM per 1TB of data tracked, which would quickly starve and crash my 16GB TrueNAS VM).

   6. Execute the iSCSI Wizard: I navigated to Shares -> Block (iSCSI) -> Wizard:
      • • Target Page: Named it pve-target and set Target Intent to Modern OS.
        • Extent Page: Set Extent Type to Device and selected my new volume FastPool/pve-zvol (450G). Under sharing platform, I selected Modern OS to guarantee proper 512-byte block alignment mapping.
        • Protocol Options Page: Switched the Portal dropdown to Create New, and selected my static storage IP address link 10.10.10.2. I hit save and ensured the global iSCSI service was flipped to Running and configured to Start Automatically.

   7. Map the Target in Proxmox: In the Proxmox Web GUI, I went to Datacenter -> Storage -> Add -> iSCSI:
      • • ID: TrueNAS-iSCSI
        • Portal: 10.10.10.2
        • Target: I clicked the drop-down box, and Proxmox instantly queried the virtual switch, auto-populating my exact target IQN string: iqn.2005-10.org.freenas.ctl:pve-target.
        • Use LUNs Directly: Unchecked. By leaving this unchecked, I prevented Proxmox from locking the raw connection down to one exclusive VM.

   8. Layer LVM for Dynamic Multi-VM Support: To allow Proxmox to carve up that 450 GiB network block into multiple separate virtual machine hard drives, I added a management layer. Still under Datacenter -> Storage, I clicked Add -> LVM:
      • • ID: iscsi-storage (I discovered this field requires alphanumeric characters/text and cannot be a plain integer like 69).
        • Base Storage: Selected TrueNAS-iSCSI
        • Base Volume: Selects the auto-discovered 450 GiB LUN block.
        • Volume Group: Named it tg-pool.
        • Content: Selected both Disk Image and Container.
        • Shared: Checked.
        • Wipe removed volumes / Allow snapshots as volume-chains: Left both Unchecked to save unnecessary write wear on my consumer SSDs and to bypass broken thin-provisioned snapshot metadata lookups over standard iSCSI blocks.

📊 Performance Testing, Caveats, & Troubleshooting Lessons

• • Sequential Reads (~6,193 MB/s): I hit incredibly high, near-PCIe numbers. This proved that TrueNAS’s ZFS ARC (Adaptive Replacement Cache) was working flawlessly—intercepting my test read requests and streaming them directly out of my VM RAM across the high-speed virtual memory switch.
  • The Initial Write Bottleneck (120 MB/s Seq / 1.07 MB/s Random 4K Q1T1): My write metrics initially hit a performance wall. Because Proxmox handles iSCSI network targets with strict write-safety guarantees, it flags every transaction as a Synchronous Write. My consumer-grade Kingston A400 SSDs do not have a physical onboard RAM battery protection module (PLP – Power Loss Protection). As a result, every time a sync request came down the pipe, the drive controllers were forced to freeze operations and perform a hard cache flush down to physical flash cells, tanking my speed.
  • The Performance Fix (283 MB/s Seq / 4.00 MB/s Random 4K Q1T1): To fix this, I adjusted my parameters in the TrueNAS dashboard by going to Datasets, selecting my pve-zvol, and modifying its advanced properties to change Sync from “Standard” to “Disabled”. This forced TrueNAS to treat incoming IO as Asynchronous, allowing my write commands to buffer safely in TrueNAS RAM first. My sequential speeds more than doubled, and my random 4K write speeds instantly surged by 400%.

⚠️ My Lab Warnings & Core Caveats To Remember

1. 1. The “Virtualization Tax” on RND4K Q1T1: Even with ZFS Sync disabled, my single-threaded, single-queue random writes max out at 4 MB/s (whereas a standalone, bare-metal Windows installation on this same single SSD can easily hit ~25 MB/s). I now understand that this is standard for virtual storage. Forcing a single 4K file over a deep chain of abstraction layers (Windows File System → VirtIO Driver → Proxmox LVM → iSCSI Network → TrueNAS Kernel → ZFS Allocation → Physical Storage Controller) introduces microscopic amounts of computational latency. Because a Q1T1 test forbids parallel actions, the system must wait for a full round-trip confirmation before sending the next block. My parallel performance is healthy, however, as shown by my high RND4K Q32T1 queue numbers.
   2. Why My Old Striped LVM-Thin Setup Died: This project helped me diagnose why my original configuration suffered from terrible 4K performance. Layering an LVM-Thin allocation pool on top of an LVM striped storage block caused severe sector misalignment and block-write amplification. A tiny 4K operating system write would get split across physical drive block boundaries, forcing the host controller to continuously run slow read-modify-write loops across multiple SSDs simultaneously just to update a single 4K data sector.
   3. The Data Integrity Tradeoff: Setting ZFS Sync to Disabled is perfect for my home test lab to achieve fast performance, but it carries a risk. Because TrueNAS is caching incoming writes inside its RAM buffer before they actually finish sinking onto the physical SSD flash chips, a sudden home power outage or a hard freeze of the physical Proxmox host will cause data loss for whatever was floating in memory. This can easily lead to a corrupted VM operating system filesystem.
   4. No Native GUI Snapshot Functionality: Because standard LVM sits on top of raw network blocks, Proxmox’s blue “Take Snapshot” button is greyed out/unsupported for these VMs. If I want to schedule automated backup states or snapshots for my testing, I must manage them directly through TrueNAS’s native ZFS snapshot tasks dashboard at the Zvol level.