Category: Networking

All posts related to Networking.

Posted on

The Poor Man’s VAAI for ZFS-over-iSCSI

The Poor Man’s VAAI: Orchestrating an Offline ZFS-over-iSCSI Storage Migration Manually

If you run a homelab or an enterprise environment utilizing Proxmox VE (PVE) backed by a TrueNAS storage array via a community ZFS-over-iSCSI plugin (like the popular TheGrandWazoo implementation), you have likely run into the limits of the API abstraction layer.
Try to perform a cross-pool storage migration (moving a virtual hard drive from one ZFS pool to another on the same TrueNAS machine), and you are likely to get slapped with this cryptic failure:
TASK ERROR: storage migration failed: TrueNAS API POST /iscsi/extent: 422 Unprocessable Entity — iscsi_extent_create.name: Extent name must be unique

Here is a deep technical breakdown of why this happens, how the hypervisor configuration misleads you, and how to execute a manual, bare-metal block migration—essentially building a poor man’s offline VAAI (vSphere Storage API for Array Integration) for ZFS over iSCSI.

The Architectural Root Cause

To understand the failure, you have to look at the intersection of three components: TrueNAS’s global iSCSI architecture, the Proxmox plugin layer, and the PVE VM configuration file.

1. TrueNAS Manage iSCSI Globally

TrueNAS maintains a centralized, system-wide configuration database for its iSCSI daemon (ctld or SCST). When an initiator requests a target, TrueNAS maps that connection globally. It enforces strict namespace uniqueness across the entire machine for Extents, Targets, and Associated Targets. TrueNAS does not care if you have two isolated ZFS pools; you cannot register the same extent name twice on the same machine.

2. The Proxmox Plugin Collision

When you define two separate TrueNAS-backed iSCSI datastores in Proxmox, they usually point to the same TrueNAS Host Portal IP.
The storage plugin hardcodes the iSCSI extent naming convention using the VM ID and slot number: vm-ID-disk-X.
When you initiate a storage migration to the other datastore via the Proxmox GUI, the plugin tries to provision the new storage before tearing down the old one. It fires off a POST request to the TrueNAS API to create an extent named vm-108-disk-1 on the destination pool. Because that exact name already exists in TrueNAS’s global registry (pointing to the source pool), TrueNAS immediately rejects it with a 422 Unprocessable Entity.

3. The PVE .conf File Illusion

In an iSCSI architecture, the storage configuration string in Proxmox is just a pointer. It tells Proxmox to talk to the storage daemon associated with that label and map a specific disk ID. Because both Proxmox datastore definitions map to the same target IP portal, Proxmox connects to the exact same network destination. TrueNAS sees the incoming request for vm-108-disk-1 and points the hypervisor right back to the original active iSCSI session on the old pool.

Poor Man’s Offline VAAI: The Manual Fix

In an enterprise environment with a SAN that natively supports VAAI block offloading, a storage migration issues a single command telling the array to clone blocks and re-map the LUN at the hardware layer—consuming zero hypervisor CPU or network bandwidth.
Since the Proxmox plugin lacks the logic to orchestrate this natively across multiple TrueNAS pools, we can act as the orchestration engine ourselves using raw ZFS and iSCSI manipulation.

Step 1: Detach the Disk in Proxmox

To safely move a block storage device without corrupting data, the file systems must be unmounted and the iSCSI lock freed.
  1. Shut down the target VM completely.
  2. In Proxmox, navigate to the VM ➔ Hardware ➔ Select the target drive (e.g., scsi0) ➔ Click Detach.
  3. The drive will drop down to the bottom of the list as an Unused Disk.

Step 2: Backend ZFS Block Migration (TrueNAS CLI)

Instead of forcing data across your network adapter via the hypervisor, leverage native ZFS snapshot streaming directly inside the TrueNAS kernel.
  1. SSH into TrueNAS or open the Web UI Shell.
  2. Escalate privileges to root using sudo.
  3. Take an atomic, read-only snapshot of the source Zvol: 
    sudo zfs snapshot BigBurtha/vm-108-disk-1@move

  4. Stream the raw data blocks directly into the destination pool (LexarPool) using a ZFS send/receive pipe. The -vR flags enable a progress bar and maintain replication properties: 
    sudo zfs send -vR BigBurtha/vm-108-disk-1@move | sudo zfs recv LexarPool/vm-108-disk-1

Step 3: Atomic LUN Mapping Flip (TrueNAS GUI)

Now that the data blocks reside on the new physical disks, you must rewrite the iSCSI pointer inside the TrueNAS database so the global network target routes to the new destination pool.
  1. In the TrueNAS Web UI, navigate to Shares ➔ Block Shares (iSCSI) ➔ Extents.
  2. Find the extent matching your disk (vm-108-disk-1) and click Edit.
  3. Locate the Source / Zvol Path dropdown menu. It will currently point to your old pool path: /dev/zvol/BigBurtha/vm-108-disk-1.
  4. Flip this dropdown to point to the new pool path: /dev/zvol/LexarPool/vm-108-disk-1.
  5. Save the configuration.

Step 4: Re-link and Boot

Go back to the Proxmox Web UI. Select the Unused Disk at the bottom of the VM hardware configuration panel, click Edit/Add, and reassign it back to its original slot (e.g., scsi0).
When you boot the VM, Proxmox will query the iSCSI target portal exactly as it did before. TrueNAS receives the request, references its updated global registry database, and serves the blocks directly out of LexarPool. Run a disk I/O benchmark utility inside the guest OS, and you will see the hardware statistics instantly register activity on the new storage array.

Step 5: Post-Migration Cleanup

Once the VM is verified stable and running on the new pool, the iSCSI service will release its execution locks on the old Zvol path, allowing you to cleanly reclaim your storage capacity.
From the TrueNAS Shell, execute the final destruction commands:
# Delete the migration snapshot from the original pool
sudo zfs destroy BigBurtha/vm-108-disk-1@move

# Wipe the old Zvol block device entirely
sudo zfs destroy BigBurtha/vm-108-disk-1

# Optional: Clean up the temporary snapshot on the new pool
sudo zfs destroy LexarPool/vm-108-disk-1@move

You might be wondering if utilizing zfs snapshot did I need to power down and detach the disk as step 1?
Yes, you are 100% correct. You absolutely could have done the ZFS snapshot and copy task completely live while the VM was running, reducing your actual downtime to just a few seconds.
Taking a ZFS snapshot is an atomic operation at the block layer; it takes less than a second to freeze the blocks, regardless of whether the VM is actively writing data or not.
By restructuring the order of operations, you can turn this into a near-live storage migration with minimal disruption.

The Live Migration Workflow (Zero-Downtime Copy)

Here is how you would optimize the steps for a blog post to achieve an almost seamless migration:

1. The Live ZFS Send/Recv (Zero Downtime)

While the VM is fully booted and handling active I/O, you run the initial snapshot and block transfer. ZFS will copy all the existing data blocks over to LexarPool.
sudo zfs snapshot BigBurtha/vm-108-disk-1@live-migrate
sudo zfs send -vR BigBurtha/vm-108-disk-1@live-migrate | sudo zfs recv LexarPool/vm-108-disk-1

(The VM stays online the entire time this data streams across your pools).

2. The Catch-Up Snapshot (Optional but Recommended)

If the block copy took a long time (e.g., a massive multi-terabyte drive), the VM will have written new data to BigBurtha while the copy was running. To minimize downtime even further, you take a second incremental snapshot to sync just the changes:
sudo zfs snapshot BigBurtha/vm-108-disk-1@sync
sudo zfs send -vR -i BigBurtha/vm-108-disk-1@live-migrate BigBurtha/vm-108-disk-1@sync | sudo zfs recv LexarPool/vm-108-disk-1

3. The “Micro-Downtime” Cutover

This is the only part where you have to stop the VM, because you cannot re-route an active iSCSI session while TrueNAS holds an open kernel lock on the running file system.
  1. Shut down the VM in Proxmox.
  2. Run a final incremental ZFS sync to catch the last few blocks written during the shutdown sequence: 
    sudo zfs snapshot BigBurtha/vm-108-disk-1@final
sudo zfs send -vR -i BigBurtha/vm-108-disk-1@sync BigBurtha/vm-108-disk-1@final | sudo zfs recv LexarPool/vm-108-disk-1

  3. In the TrueNAS Web UI (Shares ➔ Block Shares ➔ Extents), edit the vm-108-disk-1 extent and flip the dropdown path from BigBurtha to LexarPool.
  4. Update your Proxmox .conf file label if you want the storage names to look tidy (though as we discovered, this part is just cosmetic for the iSCSI portal routing).
  5. Boot the VM back up.

Why this is even closer to real VAAI

By doing the heavy lifting (the bulk data transfer) while the VM is fully operational, your total downtime drops from hours (waiting for network copies or local block transfers to finish while the VM is offline) to literally 30 seconds—just long enough to stop the VM, flip the TrueNAS iSCSI mapping dropdown, and hit start again.
This approach makes the “Poor Man’s VAAI” title fit even better, because it mirrors exactly how enterprise storage arrays handle asynchronous storage migration cutovers!

Summary

When software automation plugins fail to scale across architectural boundaries, understanding the underlying storage routing allows you to manipulate data directly at the block layer. By combining raw ZFS snapshot pipes with target remapping, you bypass API namespace limits completely—achieving zero-network-overhead backend replication.
Posted on

Virtualized TrueNAS Crashing

So, remember when I first build a test Frankenstein TrueNAS box to edge out some performance on otherwise shady consumer SSDs?  and then I ran with that Idea to test ZFS over iSCSI using TrueNAS?

Well since I was using a poormans setup, something wasn’t happy (PVE Virtio network stack? TrueNAS stack? I unno but I was using both VLAN segregation… Virtual NIC setup via Virtio, not sure but the host would become unresponsive to any network requests (pings or otherwise) but the host and VM would be otherwise fine (if I went to the console, I could issue commands without problem, but if I restarted the networking service only the host PVE would become responsive, TrueNAS would stay dead I had to reboot the whole host. AI assumes its the saturation, I have my doubts as I throttled the VLANs which didn’t help. But we’ll try anyway, I’ll move the iSCSI NICs to a PCI network card I apparently forgot I already had in this host. via hardware passthrough.. ok Let’s see how this goes…

I didn’t realize this one active VM is still on the ZFS over iSCSI datastore… move it and… host crash, so doing some testing interesting finds:

  1. Just as I stated above, restoring the network stack on the PVE host brings it back up fine, but
  2. Restarting the nework stack or even restarting the TrueNAS VM does not bring it back
  3. I have to reboot the PVE host as a whole
  4. The GUI task viewer, showing the task will stall on the transfer but the hosts stay perfectly stable and running fine, the transfer resumes where it left off after the storage PVE host and TrueNAS VM boot back up (takes roughly 20 seconds for PVE to boot, and 1 minute 30 seconds for TrueNAS) 2 min.
  5. Even when testing had a 6 minute outage window and the transfer resumed just fine when the backend storage came back. Is there a timeout out that checks actual transfer and if its stalled? I don’t know.
  6. The ZFS storage status in the left hand nav doesn’t reflect unknown status till roughly about 5 minutes of the storage becoming unavailable. But showing unknown back to good is almost instant.
The Solution: Hardware Isolation via PCIe Passthrough
The definitive fix is to physically separate the traffic by cutting Proxmox completely out of the storage loop. My motherboard has a spare PCIe slot, added a dedicated network card (a cheap, enterprise-stable multi-port Intel Gigabit adapter) and passed it directly to the storage VM.
Find the dedicated network card’s IOMMU group and add it as a Raw PCI Device to the TrueNAS VM hardware tab in Proxmox. Ensure All Functions and PCI-Express are checked.
This completely bypasses the Proxmox network stack, the virtual Linux bridge (vmbr0), and the VirtIO translation layers. TrueNAS now talks directly to bare-metal Intel hardware using native, highly optimized FreeBSD drivers.
After doing this I haven’t had the hosting PVE server or TrueNAS VM stop responding to pings, or services. It’s a lil less Frankensteiny now.
… or so I thought that separating the iSCSI traffic resolved it, everything far more stable, my usual instant test (Booting Windows VM against the SSD backed ZFS over iSCSI target) was working just fine… So, while I thought this resolved it, I jumped back to my AI chat window to state that this fix didn’t in fact work. I took a snip of my intuitional moment…
Which it then gave some examples of what it could be, I was kinda shocked when one of it’s hunches, was backed by hard evidence in the logs after the fact.. it stated and I quote..
“Clear out the e1000e Driver Bugs on your Management NIC

Your onboard management NIC (00:19.0) uses the Intel Linux e1000e driver, which is notorious for dropping off the network during heavy CPU or PCIe bus loads due to power management and interrupt coalescing.”

then told me to check the log….

journalctl -b -1 -n 200
and low and behold…
Oct 05 21:29:09 desktop-pve kernel: e1000e 0000:00:19.0 nic0: Detected Hardware Unit Hang:
TDH <9c>
TDT <c4>
next_to_use <c4>
next_to_clean <9a>
buffer_info[next_to_clean]:
time_stamp <100904184>
next_to_watch <9c>
jiffies <1009ac040>
next_to_watch.status <0>
MAC Status <80083>
PHY Status <796d>
PHY 1000BASE-T Status <7800>
PHY Extended Status <3000>
PCI Status <10>
Oct 05 21:29:10 desktop-pve systemd[1]: pve-ha-crm.service: Deactivated successfully.
Oct 05 21:29:10 desktop-pve systemd[1]: Stopped pve-ha-crm.service - PVE Cluster HA Resource Manager Daemon.

Ain’t that but a bitch…. and it turns out this is a known problem, now only was the first AI source noting this, here’s a PVE thread started in 2019 still on going, this guy exact same signs and symptoms from April this year, this Garret guy who for some reason used AI to created a service script that is way over kill when you can just disable the offloading feature via a simple edit to the interfaces file…just add

post-up ethtool -K {interface} tso off

under the specific interface you want to disable. verify with:

ethtool -k nic0 | grep tcp-segmentation-offload
tcp-segmentation-offload: off

This time its fixed…. 11PM at night, I swear… fucking tech man…

Posted on

Least-Privilege ZFS over iSCSI for Proxmox VE via TrueNAS

How I Configured Least-Privilege ZFS over iSCSI for Proxmox VE via TrueNAS

The built-in Proxmox VE “ZFS over iSCSI” storage provider relies on direct root SSH access to TrueNAS to dynamically spin up block devices. However, this legacy method is broken on modern TrueNAS SCALE versions (like Dragonfish and Electric Eel) because TrueNAS no longer uses the target engines (like Comstar/LIO) expected by PVE.
To solve this securely without enabling risky root SSH access, I installed the “official TrueNAS Proxmox VE Storage Plugin” aka plugin managed by “TheGreatWazoo”, worked around network anomalies, and configured a custom Role-Based Access Control (RBAC) policy for a true least-privilege service account.

Step 1: Setup TrueNAS

Virtualizing TrueNAS: CPU NUMA and Memory Logistics
    • Bypassing the CPU NUMA Trap: Since my TrueNAS virtual machine is small enough to fit comfortably inside a single physical CPU socket and its local memory pool, I left virtual NUMA (vNUMA) disabled in my hypervisor. Enabling it on a small VM would force the guest OS to waste processing cycles managing artificial boundaries. It would also restrict hypervisor scheduling, pinning my storage VM to specific sockets even if other sockets are completely idle. Leaving it disabled allows the hypervisor to handle optimization automatically via UMA (Uniform Memory Access) mode.
    • Fixing the RAM Allocation: TrueNAS was initially starved at a default 2048 MiB (2 GB), which is an instant recipe for an Out-Of-Memory (OOM) kernel crash under iSCSI loads. I scaled the allocation up to 8192 MiB (8 GB) (with a ceiling of 12288 MiB / 12 GB if needed). This provides enough overhead for the base OS middleware and the 4TB ZFS metadata.
    • Disabling Memory Ballooning: Because ZFS calculates its cache (ARC) based on a fixed expectation of available memory, I disabled the ballooning device and set fixed memory. If the host tried to dynamically reclaim RAM mid-test, the TrueNAS kernel would instantly panic and drop my storage network offline.

After successfully installing TrueNAS, I presses 1 at the console to configure the interface.

I noticed I could only toggle DHCPv4 and DHCPv6 on or off, but the actual IP address input field seemed completely missing.
Here is how I finally managed to configure my static IP:
    • I navigated down to the Aliases field. In TrueNAS SCALE, static IPs are assigned as aliases rather than a traditional static IP entry field.
    • I selected the empty list. I used my arrow keys to highlight aliases: <empty list> and pressed Enter to open the edit sub-menu.
    • I entered my IP with CIDR notation. I typed my desired address and subnet mask combined (for example, 192.168.1.100/24) instead of typing the netmask separately.
    • I saved the changes. I scrolled down to the <Save> button at the bottom of the screen and pressed Enter to apply the network settings.

Only then I could go to configure network settings and define the gateway and DNS servers. lil weird but whatever, easy enough.

Then just like before HBA to passthrough the hard drives to the VM

Just like before, created 2 isolated networks (one mgmt plane (HTTPS/SSH) and one data plane (iSCSI). *Note* I’m using VLANs (on the hypervisor for the VM guest) to simulate what you normally would do physically. So the configuration on TrueNAS looks the exact same as physical (two dedicated NICs for the two planes) AND… MPIO will not be covered in the scope of this blog. I’ll have to cover that in the future when I get some hosts with multiple NICs.

Step 2: TrueNAS Service Account & API Configuration

To avoid assigning global administrative rights, I built a dedicated service user and a custom privilege profile to authorize specific storage metadata calls.
1. Account Creation Constraints
I navigated to Credentials > Users to create my service account container:
    • Username: pve-iscsi
    • TrueNAS Access: Checked the box and selected Sharing Administrator rather than a generic read-only or full admin profile.

      • Create the account, Give it a Name, Check off TrueNAS Access, select read-only admin.
    • Shell/SSH Access: Kept unchecked. The plugin communicates purely via HTTPS REST API calls, so command-line execution is unnecessary.

🛠️ Caveat & Tweak: This creates a less restrictive account, it also creates a group with the same name as the account. We will be creating a custom “Privilege” Role with only the required “Roles” permissions, then bind that role to this group to complete the least privilege configuration.
User:
Group:
As you can see the group created here shows each permission already defined on this account (snippet here is after the needful was done), but you need to create the actual Role by clicking on the Privileges button next to add. Create a “Privilege” and name it after this setup.
Under “Roles” aka Permissions I granted this “Privilege”, what I call a Role,
1. Sharing (iSCSI Track)
    • Sharing iSCSI Extent Read
    • Sharing iSCSI Extent Write
    • Sharing iSCSI Initiator Read
    • Sharing iSCSI Initiator Write
    • Sharing iSCSI Portal Read
    • Sharing iSCSI Portal Write
    • Sharing iSCSI Target Read
    • Sharing iSCSI Target Write
    • Sharing iSCSI Target Extent Read
    • Sharing iSCSI Target Extent Write

2. Storage & Snapshots (ZFS Track)
    • Dataset Read
    • Dataset Write
    • Dataset Delete (Crucial so Proxmox can delete VM disks)
    • Snapshot Read
    • Snapshot Write
    • Snapshot Delete (Crucial so Proxmox can clear old snapshots/backups)
    • Pool Read (Allows the API to safely query the storage layout tree)

3. System Base Discovery
    • System General Read (Allows the API to check system info and uptime metrics)

4. The Breakthrough Secret (Filesystem Track)
    • Filesystem Read
    • Filesystem Write

(Note: Keeping Allow All Sudo Commands and Sudo without Password unchecked on the user account ensures this remains a strict least-privilege setup.)
As you can see in the snip once you bind this custom role the GUI only supports the built in “Privileges”, so if you edit this account it’ll show the TrueNAS Access checked off with the dropdown empty:

Step 3. Generating the API Token as Admin

Instead of logging out and accessing the UI under the new service account context, I provisioned the token from my master administrator account:
    1. Navigated to Credentials > Users and clicked on my new pve-iscsi user row.
    2. Inside the profile card’s Access widget, I clicked Add API Key.
    3. Named the token (e.g., PVE-iSCSI-Token) and copied the generated string immediately.

Step 4. Preparing the TrueNAS iSCSI Network Targets

Before connecting the hypervisors, I gathered configuration parameters and eased network access limits under Apps/Services > iSCSI.
    1. Bound iSCSI: Ensure iSCSI Service is bound to iSCSI NIC only.
    2. The Portal ID: Verified my active iSCSI listening group ID under the Portals tab (typically ID 1).
    3. The Target IQN: Copied my system base name string under Target Global Configuration (iqn.2005-10.org.freenas.ctl).
    4. The Initiator Workaround: Under the Initiators tab, I enabled Allow All Initiators and saved it as Initiator ID 1.

💡 Network Tweak: In a trusted home lab or private storage network, allowing all initiators simplifies initial setup. For strict isolation, this can be unchecked later by pasting the unique IQN strings of each individual Proxmox node directly into the allowed targets box.

Step 5. Driver Installation on Proxmox VE Nodes

Because storage modules run at the system level rather than the cluster level, I executed these steps directly on every node in my cluster via SSH/Shell.
# 0.  Firewall:
I had to configure a firewall rule to allow access to the DEVs githubsource.
URL: thegrandwazoo.github.io/
Application: Github-Pages
# 1. Download and register the official repository GPG key:
curl -fsSL https://thegrandwazoo.github.io/freenas-proxmox/public.gpg.key | gpg --dearmor | tee /etc/apt/keyrings/truenas-proxmox.gpg > /dev/null
# 2. Inject the driver repository metadata block using the DEB822 standard format:
cat << 'SOURCES' | tee /etc/apt/sources.list.d/truenas-proxmox.sources
Types: deb
URIs: https://thegrandwazoo.github.io/freenas-proxmox
Suites: v3
Components: main
Signed-By: /etc/apt/keyrings/truenas-proxmox.gpg
SOURCES
# 3. Synchronize package databases and install the kernel driver plugin:
apt update
apt install truenas-proxmox -y
Installing this package successfully registers the truenas-iscsi framework and automatically recycles local management services (pvedaemon, pveproxy) to render the integration options in the UI.

Step 6. Mounting Storage and Navigating GUI Flaws

I refreshed my Proxmox browser interface, navigated to Datacenter > Storage > Add, and selected the newly exposed TrueNAS iSCSI storage provider.
The Connection Profile Layout
    • ID: TrueNAS-ZFS (or any friendly cluster name)
    • Nodes: Set to All
    • TrueNAS Host: 172.16.21.2 (Management network interface)
    • Portal IP: 69.69.69.1 (The isolated network interface bound to the iSCSI portal)
    • Pool: LexarPool (Case-sensitive master storage pool name)
    • API Token: [Pasted the custom service token generated in Phase 1]

Hitting Add automatically builds the connection link, turning my cluster resource status icon green.

Core Architecture Takeaway: The Proxmox Offline Migration “Flaw”

During testing, I stumbled on a major counterintuitive design choice in how Proxmox handles moving virtual machine storage: Proxmox allows you to move local storage blocks to another node effortlessly if the VM is powered ON, but throws a hard block if the VM is powered OFF.

Why this happens under the hood:
    • Online Migration (VM ON): Proxmox delegates operations to the QEMU/KVM virtualization engine. QEMU handles the network stream natively, actively duplicating running disk blocks to an entirely different storage name on the destination node on the fly.
    • Offline Migration (VM OFF): Proxmox bypasses QEMU and drops back to its legacy internal cluster scripts. This simple script reads the text configuration of the VM, checks the source storage name (e.g., msiNM620), and checks if an identical storage name exists on the destination host. If the destination node does not have a local storage engine with that exact name, the script panics and blocks the migration through the UI.

How TrueNAS fixes this behavior:
Because my new TrueNAS integration is defined as shared storage named identically (TrueNAS-ZFS) across all nodes in my datacenter, it completely satisfies Proxmox’s basic offline script constraints. When a VM’s disks live on the TrueNAS storage block, the offline migration wizard unblocks instantly, allowing lightning-fast configuration transfers between cluster nodes without moving any underlying data blocks.

BONUS

I had weird errors when moving from shared storage back to local (this was when I had the setup using full local administrator “Privileges”, it didnt’ cause the task to fail and AI said this about.

When copying virtual machine disk blocks off of the TrueNAS iSCSI pool back onto local storage, you may notice repeating logs stating:
qemu-img: iSCSI GET_LBA_STATUS failed... SENSE KEY:ILLEGAL_REQUEST(5)
Why it triggers:
During a file transfer copy block, qemu-img sequentially pings the TrueNAS target engine with GET_LBA_STATUS calls to query which specific disk block ranges are blank or sparse so it can bypass copying empty space. Because the TrueNAS storage framework handles these optimizations at a different filesystem descriptor tier, it rejects the application block format structure, generating the standard warning lines.
The Impact:
This error is entirely harmless. The script immediately falls back to basic raw block-by-block copying and finishes streaming the virtual machine disk data over the network safely with zero block corruption.
What happens on hosts that have no installed the plug in? I dunno haven’t tested, I’m assuming they won’t have access to the storage.
What happens if the iSCSI server has a hard fail? Dunno, I’ll find out soon in my lab as I need to move the server. Any new findings I’ll update this blog post.
Hope this helps someone, feel free to leave a comment.
Posted on

Deploying Pi-hole v6 Offline on Proxmox VE with a Palo Alto Firewall

Deploying Pi-hole v6 Offline on Proxmox VE with a Palo Alto Firewall

An architectural summary and step-by-step rebuilding guide for the containerized offline deployment.

🏗️ Technical Architecture Overview

    • Hypervisor: Proxmox VE (PVE)
    • Parent Compute Layer: Privileged Linux Container (LXC) running a minimal Debian 12 base template.
    • Network Blueprint: Bound to vmbr1, isolated natively via VLAN 7 (172.16.7.2/24), routing through gateway 172.16.7.1.
    • Runtime Engine: Decoupled, standalone static binary installation of Docker Engine v27.3.1.
    • Application Tenant: Pi-hole v6 running in native Host Networking Mode (--net=host).
    • Upstream Security: Border perimeter managed by a Palo Alto Networks (PAN) Layer 7 firewall with strict application checking and URL filtering.

🛑 Phase 1: Perimeter Firewall Provisioning (Palo Alto)

Because the Pi-hole requires internet access to initialize its database but must remain highly secured, a tailored Layer 7 inspection ruleset was applied.
    1. Custom URL Category Object: Created an object named Pihole-Gravity-Sources populated with explicit domains to pull source databases:
        • *.github.com
        • *.githubusercontent.com
        • *.pi-hole.net
        • install.pi-hole.net

    2. Security Policy Mapping:
        • Source: VLAN 7 Zone (172.16.7.2)
        • Destination: WAN/External Zone (any)
        • Applications Permit List: ssl, web-browsing, dns, github-base (Crucial: required to pass deep inspection chunking packets during data stream ingestion).
        • Service Profile: Standard ports 80/http and 443/https.
        • URL Category Constraint: Enforced to match Pihole-Gravity-Sources.

💾 Phase 2: Staging Offline Files (Management PC)

To populate an offline PVE node, core files were staged using an online workstation browser:
    1. PVE Debian 12 Template:
      Index of /images/system/
    2. Docker Standalone Core:
      Index of linux/static/stable/x86_64/
    3. Pi-hole v6 Docker Image: Pulled down via an archive generator engine as a clean deployment container tarball (pihole-pihole-latest-linux-amd64.image.tar).

🛠️ Phase 3: Parent LXC Deployment & Pre-Staging

All instructions were passed natively via the Proxmox Host Shell CLI to build the environment and seed assets before booting the container.
    1. Create the Parent Host: 
      pct create 105 /mnt/pve/mass-storage/template/cache/debian-12-standard_12.7-1_amd64.tar.zst \
  -cores 1 \
  -memory 1024 \
  -swap 512 \
  -ostype debian \
  -storage mass-storage \
  -rootfs mass-storage:8 \
  -unprivileged 0 \
  -features nesting=1 \
  -onboot 1 \
  -net0 name=eth0,bridge=vmbr1,tag=7,ip=172.16.7.2/24,gw=172.16.7.1 \
  -nameserver 1.1.1.1

    2. Mount the Raw Virtual Disk for Staging: 
      pct mount 105

    3. Decompress and Inject Standalone Docker Binaries into System Paths: 
      tar -xf /mnt/pve/mass-storage/template/cache/docker-27.3.1.tgz -C /var/lib/lxc/105/rootfs/usr/bin/ --strip-components=1

    4. Decompress and Inject Pi-hole Image Template: 
      tar -xf /mnt/pve/mass-storage/template/cache/pihole-pihole-latest-linux-amd64.image.tar.gz -C /var/lib/lxc/105/rootfs/root/

    5. Close and Fire Up Compute Layer: 
      pct unmount 105
pct start 105

🐳 Phase 4: Initializing Engine Layers (Inside Container Shell)

Navigating straight inside the container environment using pct enter 105.
    1. Iptables isn’t natively available in the Debian Image so…
    2. Mitigate Missing Network Engine Elements (iptables Bypass JSON Config):
      Because the thin container lacked the legacy iptables stack, we commanded Docker to use a hypervisor-safe block driver storage matrix and turn off firewall manipulation:       
      mkdir -p /etc/docker
cat << 'EOF' > /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=cgroupfs"],
  "storage-driver": "vfs",
  "iptables": false
 }
EOF

    3. Start Core Engine Systems: 
      dockerd > /var/log/dockerd.log 2>&1 &

    4. Ingest Web Registry Layouts: 
      docker load -i /root/pihole-pihole-latest-linux-amd64.image.tar

🎯 Phase 5: Executing the Pi-hole v6 Application Tenant

Deploying with variables matching the newly restructured Pi-hole v6 engine specifications.
    1. The Live Run Sequence: 
      docker run -d \
  --name=pihole \
  --net=host \
  -e TZ="America/Chicago" \
  -e FTLCONF_webserver_api_password="WebLoginPasswordHere" \
  -e FTLCONF_dns_listeningMode="ALL" \
  -e FTLCONF_dns_upstreams="1.1.1.1;9.9.9.9" \
  -v pihole_config:/etc/pihole \
  -v dnsmasq_config:/etc/dnsmasq.d \
  --restart=unless-stopped \
  pihole/pihole:latest

        • --net=host: Bypassed broken virtual bridge layers, mapping directly to VLAN 7 network cards.
        • FTLCONF_webserver_api_password: Resolved the deprecated WEBPASSWORD v5 variable problem.
        • FTLCONF_dns_listeningMode="ALL": Opened cross-subnet resolution rules, allowing external VLAN query targets.
        • FTLCONF_dns_upstreams: Seaded upstream parameters to prevent silent drops.

🔒 Phase 6: Core System Persistence (Automation on Boot)

Because raw binary files do not auto-load rules or daemons natively when an LXC starts, a custom boot initialization script was injected to force persistence across power outages.
cat << 'EOF' > /etc/init.d/docker
#!/bin/sh
### BEGIN INIT INFO
# Provides:          docker
# Required-Start:    $network $remote_fs
# Required-Stop:     $network $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start Docker Daemon
### END INIT INFO

case "$1" in
  start)
    echo "Starting Docker Daemon..."
    # Carve runtime networks back into memory paths
    nft add table inet filter 2>/dev/null
    nft add chain inet filter input { type filter hook input priority 0 \; policy accept \; } 2>/dev/null
    nft add rule inet filter input udp dport 53 accept 2>/dev/null
    nft add rule inet filter input tcp dport 53 accept 2>/dev/null
    # Run core daemon engine
    /usr/bin/dockerd > /var/log/dockerd.log 2>&1 &
    ;;
  stop)
    echo "Stopping Docker Daemon..."
    killall dockerd
    ;;
  restart)
    $0 stop
    sleep 2
    $0 start
    ;;
  *)
    echo "Usage: $0 {start|stop|restart}"
    exit 1
    ;;
esac
exit 0
EOF

chmod +x /etc/init.d/docker
update-rc.d docker defaults
Holy Ballssacks bro, why does LXc containers have to be such a bitch.. all it should be is define the imagine you want to use, and the properties you want to define just like the docker deployment, i shouldn’t have to setup a LXC to run docker to run a proper container… fucking LXC.
Direct Comparison of Approaches
Feature 🛠️ The Video’s Approach (Native Script) 🟦 Our Approach (Docker inside LXC)
Architecture LXC ➡️ Pi-hole Script LXC ➡️ Docker ➡️ Pi-hole Container
How it works The script installs software packages (lighttpd, php, pihole-FTL) directly into the Debian template file system, just like a traditional VM or physical server. The Debian template is treated as a clean, blank slate. Docker manages the dependencies, keeping the Pi-hole completely sealed off in its own application layer.
Internet Needs Requires constant, wide-open internet during installation to pull dozens of Linux package dependencies via apt. Can be done 100% offline. We only needed a tiny firewall hole for the single pre-packaged image file.
Upgrades You must run pihole -up inside the console, which modifies the system packages and can break if dependencies drift. To upgrade, you simply delete the container and spin up a new image version. Your underlying configuration directories stay untouched.
Flexibility If you want to run another tool (like a dashboard or home automation), its packages mix with Pi-hole’s web server settings, often causing conflicts. You can spin up 5 more isolated Docker containers inside this same LXC host without them ever interfering with Pi-hole.
Why did the video creator do it that way?
The video creator used the native script because it is the easiest path for a standard home lab with wide-open internet access. You just paste one command (curl -sSL https://pi-hole.net | bash), click “Next” on a few menus, and it handles everything automatically. They treat the LXC exactly like a lightweight VM because they don’t have strict network constraints.
Why were we forced to use our approach?
Your specific environment made the video’s method nearly impossible for two massive reasons:
    1. Your PVE Host is Offline: The native Pi-hole installer script expects to talk to official Debian package repositories to download web servers and compilation tools. Because your host couldn’t reach the open web, that script would have crashed immediately on missing dependencies.
    2. Your Strict Palo Alto Firewall: Even if we got the script running, a standard Linux installation queries dozens of different repository mirrors across random IP addresses and domain spaces. Your Palo Alto would have blocked 90% of them. By using Docker, we only had to tell the firewall to trust one single destination (github.com) to get the container fully populated.
Our approach required a lot more upfront engineering, but you ended up with a far more resilient setup. Your core Debian system remains pristine, and your Pi-hole is perfectly contained.
But wait you must be asking, how do we manage it. Well, the web interface will be available for most administrative work.
Why it doesn’t just go to the dashboard login… unno… anyway.  “Nooo, like the container…” well you should have access to the PVE host, enter the LXC with “pct enter 105”, replacing 105 with the ID of your LXC container.  “nooo, the docker container”
ohh…
root@docker-host:~# docker exec -it pihole /bin/bash
failed to create runc console socket: stat /run/user/0: no such file or directory: unknown
root@docker-host:~# mkdir -p /run/user/0
root@docker-host:~# docker exec -it pihole /bin/bash
docker-host:/#
look we’re in… but why doesn’t it have its own hostname?
“The reason you are seeing docker-host inside the container is because your container is running on the host network mode (–net=host).In Docker, when a container uses the host network stack, it bypasses Docker’s standard network isolation. It shares the host’s network namespace, IP addresses, network interfaces, and hostname directly.”
What about updates? … Well, adjusted security rule to allow app: apt-get, and destination security.debian.org and deb.debian.org, run apt update and apt upgrade, good to go. Just make a backup first. Pi-hole updates? whent he latest container release is available, simple bring down the instance, pull the latest image, and bring the container back up.
I wish containers were a bit different on PVE, but I also understand their complexities over advanced deployments. but it would be nice if they had simple FOSS containers for simple stuff like Pi-Hole, let me click add containerized app in the PVE host menu, pick the app (in this case Pi-Hole) specify the options (IP[vlan], DNS, Hostname) click deploy and it just runs the container on the PVE host itself and be done with it, updates then would just be stop the container, pull latest image, bring the container up.. none of this having to manage an LXC with its own apps and libraries and have to run apt update on it… ehhh whatever, I’ll just move on at this point lol…
Man, but I can’t back it up using Veeam, you can only backup VMs. So, unless I want to manage to backup solutions Veeam + PVE Backup Server, I’d have to convert to a VM instead of an LXC anyway. The main advantage was RAM savings..
vs
well **** I might just end up throwing this in the trash and replacing it with a VM running docker instead… **** me, well.. guess you live and you learn. but it’s still a super tight setup if someone wanted to do PiHole on LXC, while being able to do a regular docker style pull instead of a raw dawg install,
Hope this helps someone.
*Update* I had an issue where pihole wasn’t starting on boot, so used crontab to resolve it.
  • Run crontab -e .
  • Arrow down to the very bottom line.
  • Paste the fallback rule:
    text
    @reboot dockerd > /var/log/dockerd.log 2>&1 &
Posted on

Using FreeNAS as a VM to Test Storage Speeds

Using FreeNAS as a VM to Test Storage Speeds

OK sooo this is gonna seem kinda commical… I

  1. set up a PVE host. Most of which ended up being about storage, I didn’t even blog the actual installation process, I just got right into the storage after a completed installation lol.
  2. wrote a blog attempting to discuss managing a PVE host. A little better cover some basic host management stuff, but again end up talking about VM storage options and performance.. lol
  3. wrote a blog post above recovering a VM from ESXi to PVE using Veeam. Which was also poor, most a just a video reference to a Veeam tech who shows the technical steps, then just me wondering why I got such unreal poor performance from my setup.

I don’t suggest you read any of them cause they are some of the worst blogs I have ever written. They are, however, not entirely useless as they provide some bases to the tests I continue to complete.

So now I had another thought, and can you guess what it was around… yeah… storage… anyway, so if LVMthin is not the best choice for Random IO, and even using the Linux kernel as cache causes issues, what if I just throw the SSDs to a FreeNAS VM. Will it perform better?

My Storage Engineering Journey: Rebuilding Proxmox Local SSD Storage into an Isolated iSCSI San

🛠️ Phase 1: Deconstructing My Original Proxmox LVM Storage

Originally, my Proxmox host had three 240GB Kingston SSDs (sda, sdb, sdc) grouped into a striped LVM volume group called pve-fast. This group hosted an LVM-Thin data pool (fast-data) and an active striped system swap space (fast-swap).
To completely free these drives up for raw passthrough without throwing GUI errors, I opened my Proxmox host SSH shell and dismantled the entire architecture in reverse order:
    1. Remove GUI mapping: I commanded Proxmox to stop monitoring the LVM-Thin dashboard pool:
      bash
      pvesm remove Striped-SSDs
    2. Deactivate and scrub the swap space: I disabled the active swap space running on the SSDs:
      bash
      swapoff -v /dev/pve-fast/fast-swap

    3. Remove swap from the boot configuration: I edited the filesystem table:
      bash
      nano /etc/fstab

      I located the active line /dev/pve-fast/fast-swap none swap sw 0 0 and deleted it (or added a # at the beginning) to prevent my system from hanging or crashing on its next boot.

    4. Destroy the Logical Volumes: I permanently purged the inner allocation containers:
      bash
      lvremove /dev/pve-fast/fast-data -y
lvremove /dev/pve-fast/fast-swap -y
    5. Destroy the Volume Group: I deleted the master pool layout itself:
      bash
      vgremove pve-fast

    6. Wipe LVM Labels from Raw Disks: I forced LVM to entirely release its ownership tags over the physical hardware:
      bash
      pvremove /dev/sda /dev/sdb /dev/sdc
    7. Verify Everything is Clean: I ran the validation checks to ensure only my main OS boot drives remained:
      bash
      pvs
vgs
lvs

💾 Phase 2: Passing Individual Raw Disks to TrueNAS

I spun up a brand-new TrueNAS SCALE virtual machine assigned with a flat, stable 16GB of RAM. Because memory ballooning breaks ZFS caching calculations, I disabled ballooning by keeping the Minimum Memory and Maximum Memory values identical inside the Proxmox UI.
    1. Locate My Persistent Disk IDs: I knew using changing identifiers like /dev/sdb would break my VM mapping if I added or removed hardware down the line. I ran this command to pull the unique hardware serial strings:
      bash
      ls -l /dev/disk/by-id/

      I copied down my three distinct Kingston identifiers:

        • ata-KINGSTON_SA400S37240G_50026B7785138139
        • ata-KINGSTON_SA400S37240G_50026B77851380CD
        • ata-KINGSTON_SA400S37240G_50026B7785137D21

    2. Map the Disks into the VM: Using the Proxmox host shell, I manually bound the raw block devices to my TrueNAS VM (VM ID 101) sequentially on the SCSI controller, appending critical flags to disable host-level caching and keep Proxmox backup routines from touching my storage array:
      bash
      qm set 101 -scsi1 /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B7785138139,cache=none,backup=0
qm set 101 -scsi2 /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B77851380CD,cache=none,backup=0
qm set 101 -scsi3 /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B7785137D21,cache=none,backup=0
    3. Force Hardware Serial Numbers: When I first booted the VM, TrueNAS’s ZFS middleware threw validation errors. QEMU passes virtual disks as generic entities, meaning TrueNAS saw three separate paths sharing a blank or overlapping virtual serial identifier. I shut down the TrueNAS VM and edited the backend configuration file on my Proxmox host:
      bash
      nano /etc/pve/qemu-server/101.conf

      I navigated to the scsi1, scsi2, and scsi3 entry lines and appended ,serial= followed by their real hardware suffixes directly to the end of the config text:

      text
      scsi1: /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B7785138139,backup=0,cache=none,size=234431064K,serial=50026B7785138139
scsi2: /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B77851380CD,backup=0,cache=none,size=234431064K,serial=50026B77851380CD
scsi3: /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B7785137D21,backup=0,cache=none,size=234431064K,serial=50026B7785137D21

      After saving and booting TrueNAS up, the duplicate ID issue disappeared completely.

🌐 Phase 3: Building an Isolated Layer 2 Virtual Network

To keep my heavy, high-throughput iSCSI storage data entirely off my flat subnet and isolated from my Management VLAN 21, I engineered an in-memory virtual switch pipeline.
    1. Create the Private Bridge in Proxmox: In the Proxmox Web GUI, I went to Node -> System -> Network -> Create -> Linux Bridge.
        • Name: vmbr1
        • IPv4/CIDR: 10.10.10.1/24
        • Gateway / Bridge Ports: Left completely blank.
        • This isolated the bridge entirely inside host memory, enabling packets to move at CPU speed without hitting a physical switch. I clicked Apply Configuration to spin it up live.

    2. Add a Second NIC to TrueNAS: In VM 101 -> Hardware -> Add -> Network Device.
        • Bridge: Selected vmbr1
        • Firewall Checkbox: Unchecked. This was a critical step. By turning off the Proxmox software firewall for this card, I bypassed heavy packet inspection overhead, saving CPU cycles and ensuring lower latency for my storage loop.

    3. Configure the Storage IP in TrueNAS: I logged into my TrueNAS SCALE web dashboard, opened Network -> Interfaces, and edited the newly populated unconfigured adapter (e.g., vtnet1). I unchecked DHCP and manually assigned a flat Layer 2 static IP configuration:
        • IP Address: 10.10.10.2
        • CIDR: 24
        • Gateway: Left completely blank to eliminate any potential multihoming asymmetric routing loops.

    4. Verify the Pipeline: I went into the Proxmox terminal and ran a quick check to make sure the internal memory link was intact:
      bash
      ping -c 3 10.10.10.2

      🏗️ Phase 4: Carving out Zvol and Provisioning the iSCSI SAN

    5. Create My Zvol Container: In TrueNAS SCALE, I went to the Datasets tab in the left panel. I selected my master pool (FastPool), clicked Add Zvol, and input the following configuration parameters:
        • Zvol Name: pve-zvol
        • Size: 450 GiB
        • Sparse Volume: Unchecked (Thick Provisioned). I did this to intentionally carve out and lock down this exact slice of my 639 GiB total raw pool upfront. This prevents Proxmox from accidentally over-allocating storage down the line and protects my ZFS array from hitting 100% capacity and freezing.
        • Compression: LZ4 (Lightning fast, low CPU overhead, reduces physical write amplification by compressing data blocks before they hit flash).
        • ZFS Deduplication: OFF. (I verified this was disabled because dedupe consumes roughly 5GB of system RAM per 1TB of data tracked, which would quickly starve and crash my 16GB TrueNAS VM).

    6. Execute the iSCSI Wizard: I navigated to Shares -> Block (iSCSI) -> Wizard:
        • Target Page: Named it pve-target and set Target Intent to Modern OS.
        • Extent Page: Set Extent Type to Device and selected my new volume FastPool/pve-zvol (450G). Under sharing platform, I selected Modern OS to guarantee proper 512-byte block alignment mapping.
        • Protocol Options Page: Switched the Portal dropdown to Create New, and selected my static storage IP address link 10.10.10.2. I hit save and ensured the global iSCSI service was flipped to Running and configured to Start Automatically.

    7. Map the Target in Proxmox: In the Proxmox Web GUI, I went to Datacenter -> Storage -> Add -> iSCSI:
        • ID: TrueNAS-iSCSI
        • Portal: 10.10.10.2
        • Target: I clicked the drop-down box, and Proxmox instantly queried the virtual switch, auto-populating my exact target IQN string: iqn.2005-10.org.freenas.ctl:pve-target.
        • Use LUNs Directly: Unchecked. By leaving this unchecked, I prevented Proxmox from locking the raw connection down to one exclusive VM.

    8. Layer LVM for Dynamic Multi-VM Support: To allow Proxmox to carve up that 450 GiB network block into multiple separate virtual machine hard drives, I added a management layer. Still under Datacenter -> Storage, I clicked Add -> LVM:
        • ID: iscsi-storage (I discovered this field requires alphanumeric characters/text and cannot be a plain integer like 69).
        • Base Storage: Selected TrueNAS-iSCSI
        • Base Volume: Selects the auto-discovered 450 GiB LUN block.
        • Volume Group: Named it tg-pool.
        • Content: Selected both Disk Image and Container.
        • Shared: Checked.
        • Wipe removed volumes / Allow snapshots as volume-chains: Left both Unchecked to save unnecessary write wear on my consumer SSDs and to bypass broken thin-provisioned snapshot metadata lookups over standard iSCSI blocks.

📊 Performance Testing, Caveats, & Troubleshooting Lessons

I deployed a Windows guest VM directly onto my new iscsi-storage pool and ran benchmarks using CrystalDiskMark. The numbers revealed exactly how complex, multi-layered storage virtualization behaves under the hood.
My Benchmark Results & The ZFS Sync Breakthrough
    • Sequential Reads (~6,193 MB/s): I hit incredibly high, near-PCIe numbers. This proved that TrueNAS’s ZFS ARC (Adaptive Replacement Cache) was working flawlessly—intercepting my test read requests and streaming them directly out of my VM RAM across the high-speed virtual memory switch.
    • The Initial Write Bottleneck (120 MB/s Seq / 1.07 MB/s Random 4K Q1T1): My write metrics initially hit a performance wall. Because Proxmox handles iSCSI network targets with strict write-safety guarantees, it flags every transaction as a Synchronous Write. My consumer-grade Kingston A400 SSDs do not have a physical onboard RAM battery protection module (PLP – Power Loss Protection). As a result, every time a sync request came down the pipe, the drive controllers were forced to freeze operations and perform a hard cache flush down to physical flash cells, tanking my speed.
    • The Performance Fix (283 MB/s Seq / 4.00 MB/s Random 4K Q1T1): To fix this, I adjusted my parameters in the TrueNAS dashboard by going to Datasets, selecting my pve-zvol, and modifying its advanced properties to change Sync from “Standard” to “Disabled”. This forced TrueNAS to treat incoming IO as Asynchronous, allowing my write commands to buffer safely in TrueNAS RAM first. My sequential speeds more than doubled, and my random 4K write speeds instantly surged by 400%.

⚠️ My Lab Warnings & Core Caveats To Remember

    1. The “Virtualization Tax” on RND4K Q1T1: Even with ZFS Sync disabled, my single-threaded, single-queue random writes max out at 4 MB/s (whereas a standalone, bare-metal Windows installation on this same single SSD can easily hit ~25 MB/s). I now understand that this is standard for virtual storage. Forcing a single 4K file over a deep chain of abstraction layers (Windows File System → VirtIO Driver → Proxmox LVM → iSCSI Network → TrueNAS Kernel → ZFS Allocation → Physical Storage Controller) introduces microscopic amounts of computational latency. Because a Q1T1 test forbids parallel actions, the system must wait for a full round-trip confirmation before sending the next block. My parallel performance is healthy, however, as shown by my high RND4K Q32T1 queue numbers.
    2. Why My Old Striped LVM-Thin Setup Died: This project helped me diagnose why my original configuration suffered from terrible 4K performance. Layering an LVM-Thin allocation pool on top of an LVM striped storage block caused severe sector misalignment and block-write amplification. A tiny 4K operating system write would get split across physical drive block boundaries, forcing the host controller to continuously run slow read-modify-write loops across multiple SSDs simultaneously just to update a single 4K data sector.
    3. The Data Integrity Tradeoff: Setting ZFS Sync to Disabled is perfect for my home test lab to achieve fast performance, but it carries a risk. Because TrueNAS is caching incoming writes inside its RAM buffer before they actually finish sinking onto the physical SSD flash chips, a sudden home power outage or a hard freeze of the physical Proxmox host will cause data loss for whatever was floating in memory. This can easily lead to a corrupted VM operating system filesystem.
    4. No Native GUI Snapshot Functionality: Because standard LVM sits on top of raw network blocks, Proxmox’s blue “Take Snapshot” button is greyed out/unsupported for these VMs. If I want to schedule automated backup states or snapshots for my testing, I must manage them directly through TrueNAS’s native ZFS snapshot tasks dashboard at the Zvol level.

Summary, was it faster? Well in terms of I/O performance, technically yes although at the cost of a lot of implementation steps, and at the cost of Server Memory, and CPU threads. Would I recommend this, even for a home lab… meh I mean for learning its cool, but the performance while the SEQ read is kind of insane, it doesn’t provide much practical use.
Here you can see the amount of memory the FreeNAS has to do its ZFS magic, and how much CPU it takes on a high SEQ operation:
no matter what the RAN4K Q1T1 always seem to perform poorly in my tests:
if You have the Memory to spare, and have a decent CPU server with a poor storage controller, this isn’t really that bad of an option, you can also tie it into other PVE bridges/networks and serve other storage needs.
Would I recommend this, over all probably not, not honestly this is more robost then the LVMthin on the striped LVM group of the same SSDs, and disabling the write protection, that caused the entire storage stack to come to a halt and made my one server become unresponsive. “So, I tried this, and with a 64M target saw speeds up to 5x to 10x better results. So, I figured really test it and pick 8GB target file. And the other VM I just migrated onto this host lost it pings. Apparently… Optimizing Proxmox storage using a VirtIO SCSI Single controller paired with io_uring,IO Thread, and Write Back caching can yield a massive 5x to 10x performance boost in small bursts. However, executing a massive storage stress test (like an 8GB CrystalDiskMark run) on budget, DRAM-less hardware (such as Kingston A400 SSDs) can cause a severe cascading system freeze.”
So, this not only performed better, it also did cause a storage kernel panic on the same PVE host. I still tore it down cause it was too much overhead. Still neat to see it work though.
Posted on

Using Veeam to Migrate from ESXi to Proxmox

Step 1) Have Veeam with Backups from an ESXi Host.

Check.

Step 2) Have a PVE Host.

Check.

Step 3) Add PVE Host to Veeam.

Check. I had a whole bunch of images saved on Img but I lost all the links so the above is a YouTube video that I basically followed to get er done.

The only thing of note here that was annoying is I wanted the worker VM’s network to be in a certain VLAN and the wizard in Veeam didn’t have an option to set it, so I had to enter the network config, and when the wizard was at the testing stage, connect to the PVE host and apply the VLAN tag on the network of the worker VM.

This problem can either be resolved using VNets, SDN for PVE, but the real solution (having a simple text field and applying it into an API call) is “on the roadmap” for Veeam after 2 years knowing about this limitation, that the fix is so simple, it’s mind boggling it not in the initial offering… 

Another thing I found weird with my particular setup (step 2), is that for the snapshot storage I could only pick my EXT4 storage and not the LVMthin.

Step 4) Restore VM to PVE

Even with the worker VM on the host, Veeam wouldn’t give me recovery speed or estimate to recovery, I used the glances command on the PVE host and noticed it was indicating CPU-IOWAIT was the bottle neck,  and seeing the logical disk and each SSD in glances showing only roughly 3 MB/s. I believe this might be due to how the worker VM was configured for its storage settings and how it coded. Took 6 hours but it did work once I did these steps after Veeam said success.

It worked but it wouldn’t boot even with the SCSI controller set to VMware SCSI. I had to detach the HDD, and reattach using SATA and then under VM options pick it for the boot order.

Network wasn’t working had to apply VLAN manually, then install virtio drivers to see the NIC, then manually re-IP and it said IP on old phantom NIC, so remove from that? yes, and network back up.

*Note you should really uninstall VMware tools…. cause for some reason the UN-installer does a hardware check to see if it is a VMware VM, I remember this from when I did a V2P a while back, what does Broadcom have to say about it? “Fuck you, if you didn’t remove the application before converting… fuck you. Uninstaller won’t work, sit there like a tattoo, fuck you bruuuh.”

Quoted from this KB

Issue/Introduction

  • A Windows virtual machine was migrated from vSphere to a non-vSphere environment without uninstalling VMware Tools.
  • Microsoft installer fails to uninstall the VMware tools.
  • No errors are identified during the uninstallation process.

Cause

After migrating the VM to a different platform, the VMware Tools uninstallation process fails because the virtual machine is not running within a vSphere environment.

Resolution

  • This is an expected behavior. <- AKA: We coded this deliberately
  • VMware Tools should be uninstalled prior to migrating the virtual machines out of the vSphere environment. <- AKA: You should of been a perfect admin.
  • Once the migration is completed, the virtual machine is no longer under the support of VMware by Broadcom. <- AKA: Fuck you!

Maybe I’ll cover a blog doing that, but I forgot in my test example. So, make sure you have the latest backup of the VM after removing VMware tools.

Issue #1

Storage Speeds

If you check out, Managing a Proxmox Host – Zewwy’s Info Tech Talks, you can see on the Test VM my Crystal Disk mark speeds and it performed poor on the RAN-t1q1 R/W but the others were fine. I asked AI, it mentioned that LVMThin and the lack of the battery write cache, and the fact it’s DRAM less SSDs creates the issue.

1. The LVM-Thin “Allocation on Commit” Penalty
LVM-Thin allocates space dynamically on demand. When a brand new Windows VM runs CrystalDiskMark, it writes to sectors that have never been written to before.
Every single time a tiny 4KB write occurs, the host operating system has to pause, check the hidden LVM metadata tracker, carve out a raw block from the pool, update the metadata index, and then commit the write. Doing this chunk-by-chunk at Queue Depth 1 (one file at a time) destroys random I/O performance.
2. The Interleaved Stripe Stripe-Size Mismatch
When you created the volume, you explicitly declared a 64k stripe size (-I 64k) across 3 disks (-i 3). This means LVM expects data chunks to be written in 192KB sweeps (64KB x 3) to evenly split the load.
  • CrystalDiskMark is attempting to write a tiny 4KB packet.
  • 4KB is a fraction of a single 64KB stripe.
  • Because it doesn’t span all three drives, the kernel doesn’t gain parallel execution speed. Instead, the storage driver must execute a Read-Modify-Write (RMW) cycle, adding physical disk latency overhead to a minute transaction.
3. Missing VirtIO SCSI Drivers & Write Caching
If you are testing this inside a fresh Windows VM without optimizations, Windows defaults to a generic IDE/SATA controller emulation, which forces synchronous, non-cached writes. Coupled with your HPE B140i controller lacking a battery-backed physical cache, Windows is forcing every single 4KB block to wait until the flash chips on the Kingston A400 SSDs send an physical acknowledgment back up the chain.
For point 3, I didn’t tell AI about my test and second blog post where I actually did cover the difference in the different virtual controllers. However, as we saw we didn’t get much better performance in the RANIO results, OK double in the reads but nothing in the writes. It suggested to change the cache to write back

So I tried this, and with a 64M target saw speeds up to 5x to 10x better results. So I figured really test it and pick 8GB target file. And the other VM I just migrated onto this host lost it pings. Apparently…

Optimizing Proxmox storage using a VirtIO SCSI Single controller paired with io_uring,IO Thread, and Write Back caching can yield a massive 5x to 10x performance boost in small bursts. However, executing a massive storage stress test (like an 8GB CrystalDiskMark run) on budget, DRAM-less hardware (such as Kingston A400 SSDs) can cause a severe cascading system freeze.
Here is exactly what happens behind the scenes when a heavy synthetic workload breaks a virtualized storage layer:
  • The Host RAM Trap (Linux Dirty Throttling): When a VM uses Write back caching, the Proxmox host intercepts writes and absorbs them instantly into its own memory pool. However, once the cache volume hits Linux’s internal threshold (dirty_ratio), the kernel hits an emergency brake. It forcefully halts all concurrent disk I/O requests across the entire storage layer to flush the data down to the physical disks.
  • The DRAM-less Wall: Consumer-grade SSDs lack dedicated onboard DRAM to map where files live. Under a massive, continuous random 4K write assault, their internal Flash Translation Layer (FTL) becomes heavily bottlenecked. Once their small, temporary SLC burst cache fills up, write speeds plunge down to a crawl (1–2 MB/s), causing I/O latency to spike into full seconds.
  • The LVM Storage Deadlock: With the physical drives running at a snail’s pace, the Linux kernel thread managing the thin-LVM volume pool drops into an Uninterruptible Sleep (D state). While the main Proxmox Web GUI stays responsive, any management process trying to hook directly into the VM’s active hardware layer—such as the vncproxy console stream—locks up instantly. The target guest VM drops completely off the network because its virtual hard drive stops responding.
The Fix & Takeaway: To safely benchmark real-world storage limits without triggering a host-level queue lockup, bypass the host RAM cache entirely by setting the VM disk cache mode to Default (No Cache) or Write through.
More testing and learning to commence. interesting finds.
Posted on

Managing a Proxmox Host

In my last post we went over installing a Proxymox host and we did a fair bit of managing already… ok mostly just storage but we had to manage the host after the initial install of the base OS. This should be pretty obvious, it’s a web interface, which is stated right on the Console output after you install. All the commands in the previous blog could have all been done from the direct system console, but also via remote SSH.

So, first act is to change the update repo, by removing the enterprise ones and adding the no-sub repo for updates. This alone won’t resolve the nagging pop up when you log in about having no subscription. to get rid of that:

Remove the annoying subscription pop-up

  1. Open the Shell terminal from your Proxmox web UI or connect via SSH as root.
  2. Navigate to the widget toolkit directory: 
    cd /usr/share/javascript/proxmox-widget-toolkit/

    Make a backup copy of proxmoxlib.js:

    cp proxmoxlib.js proxmoxlib.js.bak
  3. Open the file in a text editor like nano: 
    nano proxmoxlib.js
  4. Search for the text active (press Ctrl + W in nano).
  5. Locate the conditional check that looks for an active status, which typically contains !== 'active' or !res logic. Change the inequality exclamation mark ! to make it an equality check == 'active' (removing the ! so it evaluates positively instead of triggering the warning when inactive). Alternatively, comment out or bypass the function call according to your specific Proxmox minor version.
  6. Save the file (Ctrl + O, then Enter) and exit (Ctrl + X).
  7. Restart the Proxmox proxy service to apply the change: 
    systemctl restart pveproxy.service
  8. Perform a hard refresh or clear your browser cache (Ctrl + F5)

System Resources

I basically just click on the host summary tab. or a VMs summary tab.

Or install “glances” on the terminal shell.

Networking

I know, I know, you’re probably screaming about authentication and user management, groups, permissions. probably yelling “RBAC!!” I’m gonna stick to using root for now and concentrate on infrastructure stuff for now.

When configuring Proxmox in a multi-subnet or VLAN environment, you quickly run into the limitations of the Linux kernel’s “Weak Host Model,” which handles routing very differently than enterprise firewalls like Palo Alto Networks (PAN-OS). Unlike zone-based firewalls that use policy-based forwarding to automatically reply out of the same interface a packet arrived on, Linux relies strictly on destination-based routing tables. This becomes a major trap if you assign identical or overlapping subnets to multiple network interfaces; even if you physically unplug a network cable, the Linux kernel holds onto that dead route at the top of its table. This causes traffic to be shoved down a disconnected interface, resulting in frustrating “No Route to Host” errors and dropped connections, even when your other live interface is perfectly configured.

Furthermore, setting up multi-homed access to the Proxmox Web UI introduces asymmetric routing challenges, as Linux only permits a single global default gateway by default. If traffic arrives on a secondary VLAN interface, Proxmox will mistakenly attempt to send the reply back out the primary management gateway, causing firewalls to log “aged out” or “incomplete” states due to the routing mismatch. To resolve this, administrators must either use the CLI to inject custom policy-based routing rules (ip rule and separate routing tables) into the network configuration file or cleanly isolate their subnets by stripping duplicate IP layers off disconnected bridges. Additionally, when testing these secondary access points, remember that the Proxmox web service strictly binds to its local hosts configuration and requires explicit HTTPS formatting over port 8006 (https://<IP>:8006) to successfully initialize a session.

That’s a long winded way to say that when I was trying to keep the flat home network (untagged) ip address on the Proxmox server, while also giving it a virtual interface attached to another VLAN tagged subnet. It wouldn’t connect (or it wouldn’t load) the web interface from my home untagged network, even though it was routed, and tagged properly on all network devices along the network path.

So from what I can tell:

a “Linux Bridge” is like a vmware vSwitch. You define the physical connection the host has to these bridges.

a “Linux VLAN” is like a VMK. This is where you define another IP address the host can use. You select which device by defining VLAN raw device, which seem you can pic the physical NIC or the bridge, I don’t know the implication if you pic the nic when its already configured for a bridge though… I’m still learning here.

When you edit a VMs NIC settings you pick a bridge, and you can define what vlan the traffic will be at the VMs NIC settings, this is like the VMPG on ESXi.

Did I break updates from this… yup looks like it.. DNS works.. but can’t reach out anywhere or yeah.. locked down subnet, that was easily fixed.

Edited a VM NIC settings, change bridge, added VLAN Tag. Disconnect, power on VM, apply static IP, change to connect, yup.. works just fine.

To move the MGMT IP of the PVE host from untagged, to tagged follow these steps. Here’s a chart to help with making a decision on how to configure your network stacks accorindgly:

Goal Right Approach How it works
Give TrueNAS multiple IPs on the same network IP Aliases Keep a single physical network cable connected. Go to network settings and add multiple IP addresses (aliases) to that single interface.
Increase Bandwidth / Redundancy Link Aggregation (LACP / Bond) Combine both physical connections into a single logical Bond interface (bond0). Your switch must support LACP (802.3ad) to gain multi-user bandwidth. (If you want to have an IP address work on any interface, then create the bond (as failover type) and assign IP, if you need it tagged leave IP undefined and create a VLAN interface and attach to this bond.)
Isolate Virtual Machines / Apps Network Bridge Create a br0 bridge interface, strip the IP from your physical NIC, and assign it to the bridge instead. Your apps, VMs, and TrueNAS host will share it cleanly.
SMB Multichannel / High-Speed Storage Distinct Subnets Change the IP scheme on the second interface to a completely separate subnet (e.g., 192.168.2.X). Ensure your client machine also has a NIC configured on that separate subnet.

Step 1) Bridge Needs to be VLAN aware

In my case using the base bridge vmbr0.

PVE host (left hand side) -> System -> Network -> vmbr0 -> edit -> Check off VLAN Aware.

Step 2) Create a VLAN

Under name give it vlan#, where # corresponds to the VLAN tag you need applied.

Set the IP address and the new Gateway (if it complains about gateway already set on the bridge network, make sure you remove the gateway from it, and it’s IP address else you’ll fall into the problem I described at the beginning of this networking section.)

Step 3) Apply the Config

Hit Apply at the top and watch the ping flip over…

Step 4) Change IP under /etc/hosts

nano /etc/hosts

Find your old IP and update, otherwise things like creating cluster info will bind to old IP in the cluster info.

Interacting with VMs

Virt-Viewer and Virt-Manager

PVE has the web console built right in, so you can just manage the VM directly that way. I like being able to have an app window for the connection much like VMRC for VMware. Which PVE has, called Virt-viewer, get it here: Virtual Machine Manager. I installed the Winx64 binaries.

One trick I like to do is connect a USB stick to my main mgmt machine, then copy files to it, then to a VM if I need to get files on to said VM if the VM is an offline only machine.

Since I deployed this VM from a generalized image I had created I needed to pick a storage controller that I knew would be natively available to the image I was using so I stuck with the LSI, I installed virt-viewer so SPICE as my GPU, and again a native supported NIC, so the E1000.

As you can also see, they are all generic drivers, but.. working:

So as you see, not terrible, but also not crazy, I know those 3 SSDs can perform better then these results since I did an I/O test on them via the host backend, so I’m assuming I have so loss in the virtual bus controller (the LSI 53C) or the standard Windows drivers. So, the first thing I want to test is installed the guest tools, will they change how devices show in the device manger, and will there be any performance increases?

Spice Guest Tools

So downloaded them on the guest VM from “www.spice-space.org/download.html”

not sure what was up with the serial driver, but I just accepted it:

Well…

Windows Main Device? No USB Trick for you!

  1. Even after all that, the video drivers showed up without basic drivers, and I can move in and out of the VM in the virt-viewer with having to press CTRL+ALT+R. That’s Good.
  2.  The Storage device in device manager still shows generic SATA ACHI so I don’t believe I’ll get any better I/O results.
  3. Attempting to add a SPICE USB port to the VM hardware worked but…

after shutting down the VM and power it back on, the device list wasn’t greyed out and showed one free channel. but picking any of my devices…

ok… this might be cause my mgmt machine is Windows?

I want to like PVE, but there are a lot of little niche things that are pissing me off about it. Then when you want to use SPICE with virt-viewer, it downloads a spice.vv file that you have to open, which auto deletes when the VM is shutdown or close (haven’t tested this). just feels like weird UX. anyway…

Storage Controller vs Virtual Hard Drives

I changed the SCSI controller from LSI 53C to VirtIO SCSI Single. But when I booted the VM back up I still saw the same generic SATA ACHI Controller. I felt like there was some ignorance on my part so I asked AI for any insights, it informed me to add a drive cause the type on the actual virtual disk could still be bound to the old type. So I temp added a disk (just for testing) and changed the connection from SATA to SCSI and checked the dev mgmt and ran a test and the performance was a fair bit better…

compared to

Performance Increase Overview

Benchmark Test Metric Type Performance Change Percentage Increase
Seq1M-Q8T1 Read 433 MB/s → 651 MB/s +50.3%
Write 56 MB/s → 95 MB/s +69.6%
Seq1M-Q1T1 Read 400 MB/s → 511 MB/s +27.8%
Write 51 MB/s → 54 MB/s +5.9%
Ran4K-Q32T1 Read 14 MB/s → 136 MB/s +871.4%
Write 7 MB/s → 7 MB/s 0.0% (No Change)
Ran4K-Q1T1 Read 6 MB/s → 13 MB/s +116.7%
Write 1.5 MB/s → 2 MB/s +33.3%

Key Takeaways
  • Massive Random Read Improvement: The biggest leap is in Ran4K-Q32T1 Read, sky-rocketing by 871.4%. This means heavy multi-threaded background random tasks will feel exponentially faster.
  • Solid Sequential Gains: Large file transfers (Seq1M) see a great bump, with reads improving by roughly 28% to 50%, and multi-queued writes jumping by nearly 70%.
  • Lagging Write Speeds: Random deep-queue writes (Ran4K-Q32T1) didn’t improve at all, and sequential single-thread writes (Seq1M-Q1T1) only crawled up by 5.9%.

That’s a bit improvment, I need to get the base OS HDD on this new type to gain the performance increase. Do to that:

Swap the Real Drive to SCSI

  1. In the Proxmox Hardware tab, select the 1 GB dummy disk you just made and click Detach. Then select the detached unused disk and click Remove.
  2. Select your main Windows boot disk (currently sitting on sata0 or ide0) and click Detach. It will immediately drop down to the bottom of the hardware list as an Unused Disk 0.
  3. Double-click that Unused Disk 0.  (I don’t know why double click seemed the only option I couldn’t see any action items at the top)
  4. In the pop-up window, change the Bus/Device dropdown to SCSI (it will likely assign scsi0). Click Add.

Fix the Boot Order

  1. Go to the VM’s Options tab in Proxmox.
  2. Double-click Boot Order.
  3. Check the box for your newly reattached scsi0 drive and drag/button it to the very top of the list so it is the primary boot device. Click OK.

Yeah for some reason it wasn’t checked off, so reattaching a vHDD has this implication something I didn’t instinctively had to do, in the snip above I unchecked net boot and checked off the scsi0.

Start your VM. before I ran the test I wanted to make sure the baseline VM was fine for it since now it was the Windows main OS drive that was running on the new virtual SCSI bus. however sure enough windows updates were alerady hitting the disk and the CPU.. I noticed it in task a manager, which was also showing me…
like what?! 84% active time constant, with a contant 800+ ms repsonse time and a measly 1.7MB/s … is windows doing insane I/o and bottle necking the I/O bus? was the theory all BS, or would this have happened on the settings I had before…? So many questions, so little answers… but the results are not good the Windows updates process is low CPU and high wait time on disk it seems the disk is slowing things down….
well system is back to idle windows updates completed.. lets see what diskmark has to say… shows the same results as “D:\” so we should have got the I/O performance increase, yet.. I remain skeptical….

Summary

So, we touched a bit on some basic management of a Proxmox host, like checking system resources, networking, storage, and managing VMs. Each of these are not covered in depth by any means, but just the simple fundamentals to getting a VM up and running and basic management of them.
These fundamentals are needed for the next stage, migrating VMs from ESXi to ProxMox. I know, I know, you’re saying I already did a basic pilot of that in the past here: Migrate ESXi VM to Proxmox – Zewwy’s Info Tech Talks but that was a bare metal, bare FS and using a linux VM with a convertion tool to just convert the  base HDD and it’s associated FS intact the version required by the hypervisor. It also took a lot of space, bandwidth, I didn’t explain what each step was really doing in detail. Anyway, long story short in the next blog post I’m gonna see how we can use Veeam to do a migration instead of a linux machine.
Posted on

Strong(er) authentication required

Strong(er) authentication required

Time for another annoying story… So, I wanted to configure my personal VPN at home using Global Protect… So, I went back to view my old blog posts on how to do this to polish up on the process again. And low and behold on following Step one, authentication I already hit a new road block. IT is such a fun time *sarcasm*, so when I went to enumerate the groups in the group mapping section of the PAN I was hit with the good ol’ error “Strong(er) authentication required” as you can see right here:

Looking this up online I found a Reddit post linking to a PAN KB. Which states this happens when you have LDAP hardening enabled, at least for older Windows Server (2008 referenced), when I wrote my old blog post I was running 2016, and I had updated it to 2022. So, asking AI about it, (by copying and pasting the line from the KB) if this hardening was enabled by default at first it was like “No” then after a couple back n forth was like yeah but “cause of CBT (LDAP Channel Binding)”…

Classic pedantic AI… So… what are my options?

Option 1) Disable CBT LDAP Channel Binding

The “not recommended option”

Registry Path

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters

Value Name

LdapEnforceChannelBinding

Value Type

REG_DWORD

Possible Values

  • 0 — Disable enforcement (CBT not required; effectively disables CBT requirement)
  • 1 — Enable enforcement for supported clients only
  • 2 — Always enforce CBT (strict)

To disable CBT enforcement, set:

LdapEnforceChannelBinding = 0
Registry Path
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters
Value Type
(REG_DWORD)
Value Name
LDAPServerIntegrity
0 = None 1 = Negotiate 2 = Require Signing
Then, reboot… and…
Problem solved, *dusts hands*. Now in my case with a single DC, and a home lab this def could be good enough… but in most cases, you’ll probably have to implement then next option.

Option 2) LDAPS

This is the more annoying but secure option.

1. Plan the certificate setup

You need each domain controller that will serve LDAPS to have a certificate with:

  • Key usage: Digital Signature, Key Encipherment
  • Enhanced Key Usage: Server Authentication (OID 1.3.6.1.5.5.7.3.1)
  • Subject / SAN: Includes the DC’s FQDN (e.g. dc01.contoso.com)

You can use:

  • Internal AD CS (most common)
  • Or a public CA if clients are external and not domain‑joined.

2. Install Certificate Authority (if you don’t already have one)

Setup Offline Root CA (Part 1) – Zewwy’s Info Tech Talks

Remove Existing Enterprise Root CA (Part 2) – Zewwy’s Info Tech Talks

Setup Subordinate CA (Part 3) – Zewwy’s Info Tech Talks

Or just install a primary Enterprise CA if you don’t want to do it the secure way.

3. Create a certificate template for domain controllers (optional but recommended)

On the CA:

  1. Open Certification Authority → right‑click Certificate Templates → Manage.
  2. Duplicate “Kerberos Authentication” (recommended) or “Computer” template.
  3. On the new template:
    • General: Give it a name like “Domain Controller LDAPS”.
    • Subject Name: Set to “Build from this Active Directory information” with DNS name checked.
    • Extensions: Confirm Server Authentication is present in EKU.
      I removed Smart card, and Client Auth.
  4. Security tab: Allow Domain Controllers group Enroll (and Autoenroll if you want auto‑deployment).
  5. Close, then in Certification Authority, right‑click Certificate Templates → New → Certificate Template to Issue, and select your new template.

4. Enroll the certificate on the domain controller

On each DC:

  1. Open mmc.exe → File → Add/Remove Snap-in → add Certificates for Computer account.
  2. Navigate to Personal → Certificates.
  3. Right‑click Personal → All Tasks → Request New Certificate.
  4. Choose your “Domain Controller LDAPS” (or equivalent) template → Enroll.
  5. Confirm the new cert appears under Personal → Certificates, with:
    • Private key present
    • Intended purposes includes Server Authentication
    • Subject/SAN includes the DC’s FQDN.

*Bonus* – I got hung up here for a while with no templates showing in the CA snapin on the DC, and it turns out it was cause the OFFLINE root CA cert somehow was on in the trust root store. I’m have no idea how that happened, but yeah… shrug….

5. Verify LDAPS is active on port 636

On the DC:

  1. Restart the Active Directory Domain Services service or reboot the DC (simpler).
  2. Use ldp.exe (built‑in tool):
    • Run ldp.exe.
    • Connection → Connect…
    • Server: DC FQDN, Port: 636, check SSL → OK.
    • If the certificate is correct and trusted, the connection should succeed.

6. Import the Offline Root and SubCA Certs into PAN Firewall

Import the certificates as Base64. Then edit the LDAP profile for port636 and check off SSL. You’ll need to create a dedicated rule to allow SSL on a nonstandard port by either having service set to any on the rule or creating a custom application and port for SSL on port 636. Then testing again…

Hope this helps someone.

Bonus verifying Plain LDAP bind on a DC:

Get-WinEvent -FilterHashtable @{LogName='Directory Service'; ID=2889} | Select Message | FL}

Note this may require configuring additional logging to be found:

Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics\" -Name "16 LDAP Interface Events" -Value 2
Posted on

Wireless Hyper-V Host

Back Story

Now a while back I wrote a blog post about creating a wireless ESXi hypervisor. A lot of lessons learnt, so why would I attempt this again? *ASMR* Cause you have an idea…. Sigh these usually end up bad… but here we go!!

Where did this idea come from, if I already knew all the limitations around Wireless? Cause I asked the same questions as last time, knowing I get the same answers:

Off-topic: Is there a wifi trunk port? : r/firewalla

“Not possible unfortunately. You can’t do VLAN tagging on WiFi except by separating the SSIDs.”
However this time, the OP came back acknowledging the limitation, then planted that seed, like I’m being manipulated like in the move inception.
“Thanks for the post. The radio bridge mode is interesting. There is another article here (https://forum.openwrt.org/t/trunking-over-wireless/27517) about achieving it using tunnels.”
Then I debated with AI, which first was using technical differences, to denote I can’t do the same thing, (WDS vs STA) for connecting. The thread stated using a WiFi extender via WDS, where as I have a Hypervisor connected to an ap via STA. Done deal, we still can’t do this.. *idea in head*… but what if we spun up two nodes one on a hypervisor physically connected and another on the wireless hypervisor? We did the same trick with our Wireless ESXi host, but instead of layer3 routing traffic, we tunnel the layer2… making our whole broadcast domain work, and VLANs (at the cost of MTU cause of encapsulation)… I showed AI a basic ASCII network design of this and stated it in theory should work… so here I go… ready to immensely suffer though something that I could simply plug a hardwired cable into and be done with it…

Step 1) Hyper-V Base

Since I have no clue what I’m doing, I’m gonna start with a base.. a Hyper-V Server (on Server 2025), running on a laptop. We configured a second one on an old PC mainboard, which will be physically plugged into the network. (Making it the easiest setup ever). The only point of this one is to have another node for the tunnels endpoints, as discussed above.

Step 2) OpenWRT

Why OpenWRT instead of OPNsense… I used it before, I’m familiar with it… well mostly for one main reason (ok 2)…

1. OpenWRT expects:

  • 100–500 MHz CPUs
  • 64–256 MB RAM

OPNsense expects:

  • 2–4 core x86 CPUs
  • 4–8 GB RAM

2. Two VERY important traits for this dumb idea.. and why not learn a new UI… and commands… why not.. anyway… first we have to source the installer.

Took me a bit but I believe I found what I’m looking for here: Index of /releases/25.12.0-rc1/targets/x86/64/

At least at the time of this writing, I’m assuming I can just DD the download img file to the base HDD of my VM… let’s find out… OK I asked AI for help here, I’ll admit it… so it turns I COULD have done that and it technically would have worked. However you can apparently just convert the image using qemu-img.

qemu-img convert -f raw -O vhdx openwrt.img openwrt.vhdx

Now, you may notice this is not a native Windows command (probably not native in most Linux Distro either) but we options;

1. Install QEMU for Windows (the simplest way)

2. Use the “qemu-img‑win64” standalone builds

3. Use WSL (Windows Subsystem for Linux)

If you have WSL installed:

sudo apt install qemu-utils
qemu-img convert ...
user@DESKTOP:/mnt/c/temp$ qemu-img convert -f raw -O vhdx openwrt-25.12.0-rc1-x86-64-generic-ext4-combined-efi.img openwrt.vhdx
user@DESKTOP:/mnt/c/temp$

Wow something worked for once…

Create VM… First did Gen 2, gave a random error “start_image() returned 0x8000000000000000009)” riiiiight the whatever the fuck that means error.. after chattin to AI some more… turns out even though I downloaded the EFI based image of OpenWRT… Hyper-v won’t boot it (even with secure boot disbaled), created Gen1 VM, and it booted just fine… dude whatever with this stuff:

OK, I did a quick test with 2 ubuntu VMs on each host and they were able to ping each other (Hyper-v wired [Ubi1] {172.16.51.1}) <– Ping –>  (Hyper-v wireless [Ubi2] {172.16.51.2}) and they were able to ping each other, so this should be the bases of the two nodes communication… but well try different IPs… man the way all these OS’s configure their IP address are ridiculous.. on Ubuntu I had to use network Manger, and files under netplan that were YAML based (gross)… and what about OpenWRT?!?!

Look at all those crazy uci commands… any whooooo… moving on, time to make a second OpenWRTon my other Hyper-v host…

OK it’s done….

Alright primary plumbing is in place… now we need to build our tunnels… then, 2nd NICs on both VMs tied to internal switches on the Hyper-V hosts for the different VLANs.

*UPDATE | FYI* – uci commands appear to just save things in memory then write them to specific files (E.G uci commit network -> /etc/config/network), so often times if you need to make quick changes it can be easier to edit the config files manually then simply restart the service (but do this only if you know exactly what you’re doing, otherwise stick to the commands provided by the supporting vendor.)

Step 3) Tunnels

Now, I had to change the IP addresses above to that of my local LAN subnet which has internet (*cough NAT*) cause apparently AI forgot to tell me that I need to install the GRE package on the OpenWRT clients…

*Note* if you see gre0, but do not see a gre1 interface, kmod-gre was installed but gre base app was not. It took me a long time to figure that problem out.

apk add kmod-gre
apk add gre

Now setup the tunnel interfaces…

On OpenWrt1:

uci set network.gret=interface
uci set network.gret.proto='gre'
uci set network.gret.peeraddr='10.1.1.2'
uci set network.gret.tunlink='wan'
uci set network.gret.mtu='1476'

uci set network.gres=interface
uci set network.gres='static'
uci set network.gres.device='@gret'
uci set network.gres.ipaddr='10.10.10.1'
uci set network.gres.netmask='255.255.255.252'

uci commit network
/etc/init.d/network restart

On OpenWrt2:

uci set network.gret=interface
uci set network.gret.proto='gre'
uci set network.gret.peeraddr='10.1.1.1'
uci set network.gret.tunlink='wan'
uci set network.gret.mtu='1476'

uci set network.gres=interface
uci set network.gres='static'
uci set network.gres.device='@gret'
uci set network.gres.ipaddr='10.10.10.2'
uci set network.gres.netmask='255.255.255.252'

uci commit network
/etc/init.d/network restart

*SIDE RANT*  Can you scroll the OpenWRT TTY on a Hyper-v console window? Nooooo Why the **** would that be possible….

“I get why this hits a nerve. When you’re just trying to test something simple and you run into a wall because of some arbitrary platform limitation, it feels like the whole stack is fighting you instead of helping. And Hyper‑V is full of these little “gotchas” that make no sense unless you’ve already spent hours tripping over them.

You’re not wrong for being annoyed. Anyone would be.

Hyper‑V is great for Windows workloads, but the moment you try to run anything outside its comfort zone — Linux consoles, unsigned EFI loaders, non‑Microsoft boot paths — it becomes this rigid, unforgiving box. And it’s frustrating because none of these limitations are technical necessities. They’re just design choices that make life harder for no good reason.

You’re not overreacting. This kind of friction wears people down.”

Now disable the firewall cause even creating a proto 47 rule for some reason won’t make it work, if you’re connecting these nodes to the internet I wouldn’t recommend this one bit, these are “offline” nodes, in that they have no gateway defined so they can’t comm with devices outside their flat network *yes I should have dropped it to a /30 instead of /24, what ya gonna do* Any whoooo…

/etc/init.d/firewall stop
/etc/init.d/firewall disable

that took me way longer then you’d believe to get up to this point, learning is hard. So now that we have ping across of nodes inside the tunnel, we should be good for the next step. (Note this is not need [L3 tunnel], this is just to ensure a tunnel can properlly be established and used).

Not sure whats with the first lost pings, it was working just before and it came back.. maybe I have a keepalive problem.. anyway I’ll just ignore that for now.

PHASE 1 — Create the GRETAP tunnel (L2)

OpenWrt1

uci set network.gt01='interface'
uci set network.gt01.proto='gretap'
uci set network.gt01.ipaddr='10.1.1.1'
uci set network.gt01.peeraddr='10.1.1.2'
uci set network.gt01.delegate='0'
uci set network.gt01.mtu='1558'
uci commit network
/etc/init.d/network restart

OpenWrt2

uci set network.gt01='interface'
uci set network.gt01.proto='gretap'
uci set network.gt01.ipaddr='10.1.1.2'
uci set network.gt01.peeraddr='10.1.1.1'
uci set network.gt01.delegate='0'
uci set network.gt01.mtu='1558'
uci commit network
/etc/init.d/network restart

This will create an interface named something like:

gre4t-gt01
The exact name varies slightly by build, but it will start with gre4t-.

Nothing is bridged yet. Nothing breaks.

I told my router a joke. It didn’t get it — must’ve been a layer 8 issue.

So, on the wired Hyper-V host OpenWRT has 2 NICs (one for its main untagged traffic, and one for each VLAN traffic, tagged all connected to the external switch). This is easily possible cause a wired link can easily support VLAN tags.

On the wiresless Hyper-V host the set up is slight different, The OpenWRT config looks the same, but instead of a second NIC on the external switch tagged, it’s instead connected to an internal switch.

But as you can see, the OpenWRT configs appear exactly the sme (outside of different IPs), by keeping the tagging outside the VM it allows us to keep the configs int he VMs the same, making the setup a bit easier, IMHO).

Final notes here on these config:

  • WAN = The primary NIC of the OpenWRT device (This is commonly known as “router on a stick”), it won’t be doing any actual routing).
  • gret = The virtual interface for the L3 Tunnel (this is technically not needed but was used for troubleshooting and connectivity testing).
  • gres = A static IP assigned on to gret (this is technically not needed but was used for troubleshooting and connectivity testing).
  • gtl2 = The virtual interface for the L2 Tunnel
  • v12t = The virtual sub-interface for the VLAN 12 on gtl2
  • br12 = The bridge that connects the internal Switch (eth1) to the sub-interface gre4t-gtl2.12 (on the wiressless host), or connects the tagged traffic to the sub-interface (on the wired host)

*NOTE* You have to expand the network adapter in Hyper-V, and check off allow MAC spoofing, else the tunnel won’t work. You’ve been warned.

Veeam

So, I went to backup a VM from this host, but I couldn’t create the Job it would always fail with “the specified network name is no longer available”. Checking online pointed to the usual KB… KB1230: Win32 error: The network path was not found. Code 53

But everything seemed to be in place, but I went to verify all the things anyway. (Remember I recently connected these guys, and had it working so I went through all the pre-reqs)…  Turns out the SMB share simply wasn’t loading from the Veeam server… but why? I could access it from other networked devices just fine, so the service is up and working, permission are fine, the networks fine, what gives? So I asked AI for help and it actually helped me find the reason (the key was providing all the relevant information when asking for help). Here’s it’s response based on my setup and issue and symtops:

“You’re thinking about this in a very sane way, so let’s be precise about where our assumptions can quietly be wrong.

You’re absolutely right about this:

  • TCP handshake works (Test-NetConnection 445 = True).
  • SMB works from your desktop over a routed path.
  • SMB from the VM on the same subnet to the same host hangs.

So:

  • It is not a basic 445 reachability issue.
  • It is not that “SMB is globally broken.”
  • It is path‑specific behavior between that VM and that host IP.

The key point: “If MTU was the problem, it would break everything” is only true if all paths behave the same. They don’t.

Why MTU/fragmentation can break one path and not another?

MTU problems are per path, not “per service” or “per machine”.

Your desktop and your VM are not taking the same path, even though they end up on the same host:

  • Desktop → Router/Firewall → GRE → Host
    • Many firewalls/routers do MSS clamping (e.g., force TCP MSS down to 1360/1400).
    • That keeps TCP segments small enough to survive GRE/Wi‑Fi without fragment issues.
  • VM → Inner switch → GRE → Host
    • Often no MSS clamping here.
    • PMTUD may be broken or ICMP “fragmentation needed” never makes it back.
    • Result: TCP handshake (small packets) succeeds, SMB negotiation (big packets) silently drops.

So you can absolutely have:

  • Desktop: works fine, because the firewall “fixes” the path.
  • VM on same “subnet”: hangs, because it hits the raw GRE/Wi‑Fi path with no MSS clamping.

That’s why I keep hammering on MTU and MSS: it’s the one class of bugs that very naturally behaves like this.

But let’s not stay theoretical — let’s go to things that will prove or disprove it for your exact case.

  1. Directly test MTU from the VM to the host IP that SMB hangs on

From the VM:

ping <HOST-IP> -f -l 1472
ping <HOST-IP> -f -l 1400
ping <HOST-IP> -f -l 1300

If 1472 fails but 1300/1400 succeeds, that’s a big hint the path cannot sustain full 1500‑sized packets once GRE/Wi‑Fi overhead is in play.

Then, as a brutal, simple test, temporarily drop the VM NIC MTU:

netsh interface ipv4 show interfaces    # get Id for the VM’s NIC
netsh interface ipv4 set subinterface <ID> mtu=1400 store=persistent

Bounce the NIC or reboot the VM, then try \\<HOST-IP>\c$ again.

If it suddenly stops hanging, you’ve just proved it’s a path/MTU issue that your desktop is being “protected” from by MSS clamping on the router.”

Sure enough, the MTU on Veeam’s NIC was 1500, dropping it to 1400, and testing the SMB path it worked perfectly… Learn something new everyday.

Summary

This is a huge PITA, but it IS technically possible. It took me serveral days to figure all this out, that for something that would otherwise simply be tagging ethernet frames on a physical hard wired connection… all because “You can’t tag Ethernet frames over Wi‑Fi because 802.11 wireless doesn’t carry 802.1Q VLAN tags the way wired Ethernet does. Wi‑Fi frames have a completely different header format, and access points strip off the wireless framing and rebuild Ethernet frames on the wired side. Since VLAN tags live inside Ethernet framing, they never survive that translation step.”

AKA the engineers that designed the farmwork figured no one would ever have a need for this, so fuck designing for it.

I hope this blog post helps someone out. It took me several days to figure all this out and I learnt a lot along the way, even if it’s not practical.

Posted on

Hyper-V Networking

Install Hyper-V, and let’s learn it’s Networking

Hyper-V Networking

On VMware, when you install the OS (ESXi) on the hardware in the DCUI you simply set the VLAN tag right on the network interface you configure, and bind it to a physical NIC in the menu area, simple.

We leave the network connected to a non tagged, internet allowed connection. Now we can configure our networks.

1. Create one external switch

  • Bind it to your physical NIC (configured as a trunk on the upstream switch).
  • Check “Allow management OS to share this network adapter” once — this creates the first host vNIC.

2. Add more host vNICs via PowerShell:

# Add another vNIC for the host bound to the same external switch
Add-VMNetworkAdapter -ManagementOS -Name "HostVLAN20" -SwitchName "ExternalSwitch"
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "HostVLAN20" -Access -VlanID 6

3. Verify in Windows

  • You’ll now see multiple vEthernet adapters in the host OS (e.g., “vEthernet (HostVLAN20)”, “vEthernet (HostVLAN30)”).
  • Assign IP addresses or enable DHCP as needed.

*Note* this is for doing it on a single network server (for my lab use) usually you’d have dedicated NICs for other networks designated on those servers… anyway.. where was I… right now that we got additional network on the server we can work on a shared storage, you know how I like my iSCSI.

So, when I setup Hyper-V, and connect a VM to the “default” virtual switch, the Hyper-V server acts as a DHCP provider and NATs the connections from that vSwitch to whatever other network the host is connected to?

Great question — this trips up a lot of people when they first use Hyper‑V! Let’s break it down clearly:

🖧 Hyper‑V Default Switch Behavior

  • Introduced in Windows 10 (1709): The “Default Switch” is automatically created by Hyper‑V and cannot be deleted or modified.
  • NAT (Network Address Translation): Yes — the host machine provides NAT services. This means your VM’s traffic is translated and routed through the host’s network connection, regardless of whether that’s Wi‑Fi, Ethernet, or even VPN.
  • DHCP Services: The host also runs a lightweight DHCP service for the Default Switch. This is what hands out IP addresses to your VMs automatically (usually in the 172.16.x.x range).
  • Connectivity: VMs connected to the Default Switch can reach the internet and other devices on the host’s network, but inbound connections from the LAN to the VM are limited. It’s designed for convenience, not for complex networking setups.

⚖️ Comparison with Other Switch Types

Switch Type Provides DHCP? Provides NAT? Use Case
Default Switch ✅ Yes (host DHCP) ✅ Yes (host NAT) Quick setup, internet access without config
External Switch ❌ No (relies on external network DHCP) ❌ No VM directly on physical network, full LAN visibility
Internal Switch ❌ No ❌ No Communication between host and VMs only
Private Switch ❌ No ❌ No Communication between VMs only

✅ So to answer directly:

Yes — when you connect a VM to the Default Switch, Hyper‑V’s host acts as a DHCP provider and NATs the VM’s traffic to whatever network the host is connected to. It’s essentially a convenience bridge so you don’t have to manually configure networking.

“That’s what I did configured a external virtual switch so the hosts NIC would be VLAN tagged in a certain network, I was figuring the default switch would act more like a … well switch at layer2 of the hosts Nic, so VMs on it would get DHCP from the main network, as if it was connected to a switch in the base network, not layer3 and then 4 NAT like c’mon…”

Exactly — you’ve nailed the distinction. The Default Switch in Hyper‑V is deliberately not a true Layer‑2 bridge to your host NIC. It’s a convenience construct that Microsoft built to “just work” for internet access without requiring you to mess with VLANs, external switches, or DHCP servers on your LAN.

Here’s the breakdown:

🔌 Why the Default Switch behaves like NAT

  • It’s implemented using Windows Internet Connection Sharing (ICS) under the hood.
  • ICS is a Layer‑3 service: it provides a private subnet (usually 172.16.0.0/12), runs a DHCP service, and NATs traffic out through whatever adapter the host is using.
  • That’s why VMs on the Default Switch don’t see your LAN’s DHCP server — they only see the host’s mini DHCP/NAT service.
  • It’s intentionally isolated so you don’t accidentally expose VMs directly to your corporate or home LAN without realizing it.

⚖️ Why it’s not Layer‑2 bridging

  • A true Layer‑2 bridge would forward broadcasts (like DHCP Discover) from the VM to your LAN. That’s what an External Virtual Switch does.
  • The Default Switch instead terminates those broadcasts at the host and answers them itself. That’s why you only ever get the host‑provided DHCP.
  • VLAN tagging, trunking, or other advanced networking features aren’t supported on the Default Switch — it’s locked down.

✅ When to use each

  • Default Switch (NAT + DHCP): Quick internet access, testing, dev boxes, laptops where you don’t want to reconfigure NICs.
  • External Switch (Layer‑2 bridge): When you want VMs to behave like physical machines on your LAN, get DHCP from the real network, or use VLAN tagging.
  • Internal/Private Switches: For host‑VM communication or VM‑VM only networks.

So your expectation — that the Default Switch would act like a transparent Layer‑2 bridge — makes sense, but Microsoft intentionally designed it as a Layer‑3 NAT gateway instead. If you want “real switch” behavior, the External Switch is the only way to go.

“So, whatever my host machine is allowed to do, the guest VMs in the default switch are also allowed to do? like access the internet, and the firewall that gave the host the ability to access the internet would never see the private NATed IP address the Hyper-V host is dishing out to those VMs?”

You’ve got it — that’s exactly how the Default Switch works in Hyper‑V. Let me spell it out clearly:

🌐 How traffic flows with the Default Switch

  • VMs get private IPs: Each VM connected to the Default Switch is assigned an IP from the host’s internal DHCP service (usually 172.16.x.x).
  • Host does NAT: The Hyper‑V host translates those private IPs into its own IP before sending traffic out to the physical network.
  • Firewall visibility: From the perspective of your upstream firewall or router, all traffic from those VMs looks like it’s coming from the host’s IP. The firewall never sees the VM’s private NATed addresses.
  • Permissions: Whatever the host is allowed to do (e.g., access the internet, reach certain networks), the VMs can also do — because their traffic is piggybacking on the host’s connection.

⚖️ Important implications

  • Outbound access: VMs can reach the internet or LAN resources as long as the host can. They inherit the host’s connectivity.
  • Inbound access: External devices cannot directly initiate connections to those VMs, because NAT blocks unsolicited inbound traffic unless you set up port forwarding (which the Default Switch doesn’t expose for configuration).
  • Firewall rules: The upstream firewall only enforces rules against the host’s IP. If you need per‑VM visibility or control, you’ll need an External Switch so each VM gets its own LAN IP.

✅ So yes: the Default Switch makes your VMs “ride along” with the host’s network permissions. They’re hidden behind NAT, which is why your firewall only ever sees the host’s traffic.

Additional Information

If you want to apply a VLAN tag to a physical NIC that’s not a Hyper-V host on Windows:

Since I didn’t see anything in the usual Network  IPv4 configuration window in Windows. I decided to see how useful this AI stuff is.. so I asked it…

“how do I VLAN tag the IP on windows 11 interface?”

Quick Answer: On Windows 11, VLAN tagging isn’t enabled by default. You need a NIC that supports IEEE 802.1Q VLAN tagging, then configure the VLAN ID either through the NIC’s Device Manager properties (Advanced tab → VLAN ID) or by using Intel PROSet/ANS tools or PowerShell (Set-NetAdapter -VlanID) if your driver supports it.

🖥️ Steps to Configure VLAN Tagging on Windows 11

1. Check NIC Compatibility

  • VLAN tagging requires a network card (NIC) that supports IEEE 802.1Q.
  • Not all NICs expose VLAN options in Windows. Intel and Broadcom NICs often support it, but you may need vendor-specific drivers.

2. Configure via Device Manager

  • Press Win + R, type devmgmt.msc, and hit Enter.
  • Expand Network adapters, right-click your NIC → Properties.
  • Go to the Advanced tab.
  • Look for VLAN ID or Priority & VLAN.
  • Enter the VLAN ID (e.g., 100) and click OK.

3. Configure via Intel PROSet/ANS (if supported)

  • Install Intel PROSet/Advanced Network Services (ANS).
  • In the NIC properties, Intel’s tab allows creating multiple VLAN interfaces with different IDs.
  • Each VLAN interface can then be assigned its own IP address.

4. Configure via PowerShell

  • Open PowerShell as Administrator.
  • Run: 
    Set-NetAdapter -Name "Ethernet" -VlanID 100
  • Replace "Ethernet" with your adapter name and 100 with your VLAN ID.
  • Note: This only works if your NIC driver supports VLAN tagging. Otherwise, you’ll get an error like “Requested operation not supported on adapter”.

5. Assign IP Address

  • Once VLAN tagging is enabled, go to Network & Internet settings → Change adapter options.
  • Right-click the VLAN-enabled adapter → Properties → Internet Protocol Version 4 (TCP/IPv4).
  • Assign the IP address, subnet mask, and gateway for that VLAN.

⚠️ Important Notes

  • Driver Support is Key: If your NIC driver doesn’t expose VLAN options, you cannot tag VLANs directly in Windows without Hyper-V or third-party tools.
  • Hyper-V Alternative: You can create VLAN-tagged virtual NICs using Hyper-V’s Virtual Switch Manager if your hardware doesn’t support native tagging.
  • Testing: Use ping or tracert to verify connectivity once VLAN tagging is applied.

✅ Bottom Line: To VLAN tag an IP on Windows 11, you must enable VLAN tagging on the NIC (via Device Manager, Intel PROSet/ANS, or PowerShell if supported), then assign the IP to that VLAN interface. If your NIC doesn’t support VLAN tagging, you’ll need Hyper-V or vendor-specific utilities.

My response “I’ve got an intle 82579LM Gig NIC, I see it says VLAN enabled in the advanced tab, but I don’t see an area to configure it, I’m sure the NIC supports it since I was using ESXi with vlan tagging just fine.”

That’s a great observation — the Intel 82579LM definitely supports VLAN tagging at the hardware level, but here’s the catch: Windows drivers for that NIC don’t expose VLAN configuration in the same way ESXi or Linux does.

🔎 Why You Don’t See VLAN Config in Windows

  • The Advanced tab in Device Manager shows “Priority & VLAN” or “VLAN Enabled,” but that option only toggles whether the NIC can process VLAN-tagged frames.
  • Unlike ESXi, Windows doesn’t let you directly assign VLAN IDs to a physical NIC unless the driver provides a VLAN interface creation utility (like Intel PROSet/ANS).
  • For the 82579LM, Intel stopped including VLAN configuration in their Windows drivers after Windows 8/10. On Windows 11, the driver exposes VLAN capability but no UI to configure VLAN IDs.

✅ Workarounds

Here are the practical options:

1. Intel PROSet/ANS (if available)

  • Older Intel drivers came with Advanced Network Services (ANS), which let you create VLAN sub-interfaces.
  • On Windows 11, Intel no longer bundles ANS with drivers for the 82579LM, so you won’t see the VLAN tab.

2. PowerShell Command

  • Windows has a Set-NetAdapter -VlanID cmdlet, but it only works if the NIC driver supports VLAN tagging through the OS.
  • On the 82579LM with current Windows 11 drivers, this usually fails with “Requested operation not supported”.

3. Hyper-V Virtual Switch

  • If you enable Hyper-V, you can create a virtual NIC and assign VLAN IDs there.
  • This is the most reliable way to do VLAN tagging on Windows when the physical NIC driver doesn’t expose VLAN config.

4. Third-Party NIC Utilities

  • Some OEMs (Dell, HP) ship custom drivers/utilities that re-enable VLAN tagging. If your system vendor has one, that’s worth checking.

⚠️ Bottom Line

Your Intel 82579LM can tag VLANs, but on Windows 11 the Intel driver doesn’t expose VLAN configuration. That’s why you don’t see a place to set the VLAN ID.

  • On ESXi/Linux → VLAN tagging works natively.
  • On Windows → You need Intel PROSet/ANS (no longer supported for this NIC), Hyper-V, or vendor-specific drivers.

Well ok I guess once we install Hyper-V hopefully it’ll be more like ESXi in a sense and we can tag.

*UPDATE* ExternalSwitch, WiFi NIC

I was trying to use a Hyper-V server with an External Switch, bound to a WiFi NIC. and was getting unexpected results based on the above information. but my results were NOT as I had expected, I was assuming Default Switch behavior. You may be asking, “Why would you expect that behavior if you’re creating an External Switch?”  Now, if you read my Wireless ESXi host blog, you probably are well aware of the VLAN limitations of WiFi, and can never truly be used as a trunk port (Limitation of the 802 standard, not of OS or software).

So how could a ExternalSwitch work, via WiFi if the layer 2 broadcast doesn’t work and can’t “speak” with the rest of the layer 2 stack? Yet I create a VM and it DOES get a DHCP lease address from my local subent?! What the heck is going on here…

So I had to ask AI, what was going on here, it says, yeah… that’s expected… here’s the deets… get ready.. it’s a long one….

BAH-LETED, there was way tooooo much BS from the AI response to justify keeping this data in here… long story short… Local subnet VMs work fine (it does ARP Masquerading), VLANs will never work per the usual BS I’ve talked about in my Wireless ESXi host blog.

Legal Information Proudly powered by WordPress