{"id":1888,"date":"2026-10-05T20:28:28","date_gmt":"2026-10-06T01:28:28","guid":{"rendered":"https:\/\/zewwy.ca\/?p=1888"},"modified":"2026-10-05T22:43:58","modified_gmt":"2026-10-06T03:43:58","slug":"virtualized-truenas-crashing","status":"publish","type":"post","link":"https:\/\/zewwy.ca\/index.php\/2026\/10\/05\/virtualized-truenas-crashing\/","title":{"rendered":"Virtualized TrueNAS Crashing"},"content":{"rendered":"

So, remember when I first build a test Frankenstein TrueNAS box to edge out some performance on otherwise shady consumer SSD<\/a>s?\u00a0 and then I ran with that Idea to test ZFS over iSCSI using TrueNAS<\/a>?<\/p>\n

Well since I was using a poormans setup, something wasn’t happy (PVE Virtio network stack? TrueNAS stack? I unno but I was using both VLAN segregation… Virtual NIC setup via Virtio, not sure but the host would become unresponsive to any network requests (pings or otherwise) but the host and VM would be otherwise fine (if I went to the console, I could issue commands without problem, but if I restarted the networking service only the host PVE would become responsive, TrueNAS would stay dead I had to reboot the whole host. AI assumes its the saturation, I have my doubts as I throttled the VLANs which didn’t help. But we’ll try anyway, I’ll move the iSCSI NICs to a PCI network card I apparently forgot I already had in this host. via hardware passthrough.. ok Let’s see how this goes…<\/p>\n

I didn’t realize this one active VM is still on the ZFS over iSCSI datastore… move it and… host crash, so doing some testing interesting finds:<\/p>\n

    \n
  1. Just as I stated above, restoring the network stack on the PVE host brings it back up fine, but<\/li>\n
  2. Restarting the nework stack or even restarting the TrueNAS VM does not bring it back<\/li>\n
  3. I have to reboot the PVE host as a whole<\/li>\n
  4. The GUI task viewer, showing the task will stall on the transfer but the hosts stay perfectly stable and running fine, the transfer resumes where it left off after the storage PVE host and TrueNAS VM boot back up (takes roughly 20 seconds for PVE to boot, and 1 minute 30 seconds for TrueNAS) 2 min.<\/li>\n
  5. Even when testing had a 6 minute outage window and the transfer resumed just fine when the backend storage came back. Is there a timeout out that checks actual transfer and if its stalled? I don’t know.<\/li>\n
  6. The ZFS storage status in the left hand nav doesn’t reflect unknown status till roughly about 5 minutes of the storage becoming unavailable. But showing unknown back to good is almost instant.<\/li>\n<\/ol>\n
    The Solution: Hardware Isolation via PCIe Passthrough<\/div>\n
    The definitive fix is to physically separate the traffic by cutting Proxmox completely out of the storage loop. My motherboard has a spare PCIe slot, added a dedicated network card (a cheap, enterprise-stable multi-port Intel Gigabit adapter) and passed it directly to the storage VM.<\/div>\n
    <\/div>\n
    \n
    Find the dedicated network card’s IOMMU group and add it as a Raw PCI Device<\/strong> to the TrueNAS VM hardware tab in Proxmox. Ensure All Functions<\/strong> and PCI-Express<\/strong> are checked.<\/div>\n
    <\/div>\n
    This completely bypasses the Proxmox network stack, the virtual Linux bridge (vmbr0<\/code>), and the VirtIO translation layers. TrueNAS now talks directly to bare-metal Intel hardware using native, highly optimized FreeBSD drivers.<\/div>\n<\/div>\n
    <\/div>\n
    After doing this I haven’t had the hosting PVE server or TrueNAS VM stop responding to pings, or services. It’s a lil less Frankensteiny now.<\/div>\n
    <\/div>\n
    … or so I thought that separating the iSCSI traffic resolved it, everything far more stable, my usual instant test (Booting Windows VM against the SSD backed ZFS over iSCSI target) was working just fine… So, while I thought this resolved it, I jumped back to my AI chat window to state that this fix didn’t in fact work. I took a snip of my intuitional moment…<\/div>\n
    <\/div>\n
    \"\"<\/a><\/div>\n
    <\/div>\n
    Which it then gave some examples of what it could be, I was kinda shocked when one of it’s hunches, was backed by hard evidence in the logs after the fact.. it stated and I quote..<\/div>\n
    <\/div>\n
    “Clear out the <\/span>e1000e<\/code> Driver Bugs on your Management NIC<\/span><\/p>\n
    \n

    Your onboard management NIC (00:19.0<\/code>) uses the Intel Linux e1000e<\/code> driver, which is notorious for dropping off the network during heavy CPU or PCIe bus loads due to power management and interrupt coalescing.<\/a>”<\/p>\n

    then told me to check the log….<\/p>\n<\/div>\n

    journalctl -b -1 -n 200<\/pre>\n<\/div>\n
    and low and behold…<\/div>\n
    Oct 05 21:29:09 desktop-pve kernel: e1000e 0000:00:19.0 nic0: Detected Hardware Unit Hang:\r\nTDH <9c>\r\nTDT <c4>\r\nnext_to_use <c4>\r\nnext_to_clean <9a>\r\nbuffer_info[next_to_clean]:\r\ntime_stamp <100904184>\r\nnext_to_watch <9c>\r\njiffies <1009ac040>\r\nnext_to_watch.status <0>\r\nMAC Status <80083>\r\nPHY Status <796d>\r\nPHY 1000BASE-T Status <7800>\r\nPHY Extended Status <3000>\r\nPCI Status <10>\r\nOct 05 21:29:10 desktop-pve systemd[1]: pve-ha-crm.service: Deactivated successfully.\r\nOct 05 21:29:10 desktop-pve systemd[1]: Stopped pve-ha-crm.service - PVE Cluster HA Resource Manager Daemon.<\/pre>\n
    Ain’t that but a bitch…. and it turns out this is a known problem, now only was the first AI source noting this, here’s a PVE thread started in 2019 still on going<\/a>, this guy exact same signs and symptoms from April this year,<\/a> this Garret guy who for some reason used AI to created a service script that is way over kill<\/a> when you can just disable the offloading feature via a simple edit to the interfaces file…<\/a>just add<\/p>\n
    post-up ethtool -K {interface} tso off<\/code><\/pre>\n
    under the specific interface you want to disable. verify with:<\/p>\n
    ethtool -k nic0 | grep tcp-segmentation-offload\r\ntcp-segmentation-offload: off<\/pre>\n
    This time its fixed…. 11PM at night, I swear… fucking tech man…<\/p>\n","protected":false},"excerpt":{"rendered":"
    So, remember when I first build a test Frankenstein TrueNAS box to edge out some performance on otherwise shady consumer SSDs?\u00a0 and then I ran with that Idea to test ZFS over iSCSI using TrueNAS? Well since I was using a poormans setup, something wasn’t happy (PVE Virtio network stack? TrueNAS stack? I unno but … <\/p>\n
    Continue reading “Virtualized TrueNAS Crashing”<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,7],"tags":[461,465],"class_list":["post-1888","post","type-post","status-publish","format-standard","hentry","category-networking","category-storage","tag-pve","tag-truenas"],"_links":{"self":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts\/1888","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/comments?post=1888"}],"version-history":[{"count":4,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts\/1888\/revisions"}],"predecessor-version":[{"id":1893,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts\/1888\/revisions\/1893"}],"wp:attachment":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/media?parent=1888"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/categories?post=1888"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/tags?post=1888"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}