Tag: PVE

Posted on

Managing a Proxmox Host

In my last post we went over installing a Proxymox host and we did a fair bit of managing already… ok mostly just storage but we had to manage the host after the initial install of the base OS. This should be pretty obvious, it’s a web interface, which is stated right on the Console output after you install. All the commands in the previous blog could have all been done from the direct system console, but also via remote SSH.

So, first act is to change the update repo, by removing the enterprise ones and adding the no-sub repo for updates. This alone won’t resolve the nagging pop up when you log in about having no subscription. to get rid of that:

Remove the annoying subscription pop-up

  1. Open the Shell terminal from your Proxmox web UI or connect via SSH as root.
  2. Navigate to the widget toolkit directory: 
    cd /usr/share/javascript/proxmox-widget-toolkit/

    Make a backup copy of proxmoxlib.js:

    cp proxmoxlib.js proxmoxlib.js.bak
  3. Open the file in a text editor like nano: 
    nano proxmoxlib.js
  4. Search for the text active (press Ctrl + W in nano).
  5. Locate the conditional check that looks for an active status, which typically contains !== 'active' or !res logic. Change the inequality exclamation mark ! to make it an equality check == 'active' (removing the ! so it evaluates positively instead of triggering the warning when inactive). Alternatively, comment out or bypass the function call according to your specific Proxmox minor version.
  6. Save the file (Ctrl + O, then Enter) and exit (Ctrl + X).
  7. Restart the Proxmox proxy service to apply the change: 
    systemctl restart pveproxy.service
  8. Perform a hard refresh or clear your browser cache (Ctrl + F5)

System Resources

I basically just click on the host summary tab. or a VMs summary tab.

Or install “glances” on the terminal shell.

Networking

I know, I know, you’re probably screaming about authentication and user management, groups, permissions. probably yelling “RBAC!!” I’m gonna stick to using root for now and concentrate on infrastructure stuff for now.

When configuring Proxmox in a multi-subnet or VLAN environment, you quickly run into the limitations of the Linux kernel’s “Weak Host Model,” which handles routing very differently than enterprise firewalls like Palo Alto Networks (PAN-OS). Unlike zone-based firewalls that use policy-based forwarding to automatically reply out of the same interface a packet arrived on, Linux relies strictly on destination-based routing tables. This becomes a major trap if you assign identical or overlapping subnets to multiple network interfaces; even if you physically unplug a network cable, the Linux kernel holds onto that dead route at the top of its table. This causes traffic to be shoved down a disconnected interface, resulting in frustrating “No Route to Host” errors and dropped connections, even when your other live interface is perfectly configured.

Furthermore, setting up multi-homed access to the Proxmox Web UI introduces asymmetric routing challenges, as Linux only permits a single global default gateway by default. If traffic arrives on a secondary VLAN interface, Proxmox will mistakenly attempt to send the reply back out the primary management gateway, causing firewalls to log “aged out” or “incomplete” states due to the routing mismatch. To resolve this, administrators must either use the CLI to inject custom policy-based routing rules (ip rule and separate routing tables) into the network configuration file or cleanly isolate their subnets by stripping duplicate IP layers off disconnected bridges. Additionally, when testing these secondary access points, remember that the Proxmox web service strictly binds to its local hosts configuration and requires explicit HTTPS formatting over port 8006 (https://<IP>:8006) to successfully initialize a session.

That’s a long winded way to say that when I was trying to keep the flat home network (untagged) ip address on the Proxmox server, while also giving it a virtual interface attached to another VLAN tagged subnet. It wouldn’t connect (or it wouldn’t load) the web interface from my home untagged network, even though it was routed, and tagged properly on all network devices along the network path.

So from what I can tell:

a “Linux Bridge” is like a vmware vSwitch. You define the physical connection the host has to these bridges.

a “Linux VLAN” is like a VMK. This is where you define another IP address the host can use. You select which device by defining VLAN raw device, which seem you can pic the physical NIC or the bridge, I don’t know the implication if you pic the nic when its already configured for a bridge though… I’m still learning here.

When you edit a VMs NIC settings you pick a bridge, and you can define what vlan the traffic will be at the VMs NIC settings, this is like the VMPG on ESXi.

Did I break updates from this… yup looks like it.. DNS works.. but can’t reach out anywhere or yeah.. locked down subnet, that was easily fixed.

Edited a VM NIC settings, change bridge, added VLAN Tag. Disconnect, power on VM, apply static IP, change to connect, yup.. works just fine.

Interacting with VMs

Virt-Viewer and Virt-Manager

PVE has the web console built right in, so you can just manage the VM directly that way. I like being able to have an app window for the connection much like VMRC for VMware. Which PVE has, called Virt-viewer, get it here: Virtual Machine Manager. I installed the Winx64 binaries.

One trick I like to do is connect a USB stick to my main mgmt machine, then copy files to it, then to a VM if I need to get files on to said VM if the VM is an offline only machine.

Since I deployed this VM from a generalized image I had created I needed to pick a storage controller that I knew would be natively available to the image I was using so I stuck with the LSI, I installed virt-viewer so SPICE as my GPU, and again a native supported NIC, so the E1000:

As you can also see, they are all generic drivers, but.. working:

So as you see, not terrible, but also not crazy, I know those 3 SSDs can perform better then these results since I did an I/O test on them via the host backend, so I’m assuming I have so loss in the virtual bus controller (the LSI 53C) or the standard Windows drivers. So, the first thing I want to test is installed the guest tools, will they change how devices show in the device manger, and will there be any performance increases?

Spice Guest Tools

So downloaded them on the guest VM from “www.spice-space.org/download.html”

not sure what was up with the serial driver, but I just accepted it:

Well…

Windows Main Device? No USB Trick for you!

  1. Even after all that, the video drivers showed up without basic drivers, and I can move in and out of the VM in the virt-viewer with having to press CTRL+ALT+R. That’s Good.
  2.  The Storage device in device manager still shows generic SATA ACHI so I don’t believe I’ll get any better I/O results.
  3. Attempting to add a SPICE USB port to the VM hardware worked but…

after shutting down the VM and power it back on, the device list wasn’t greyed out and showed one free channel. but picking any of my devices…

ok… this might be cause my mgmt machine is Windows?

I want to like PVE, but there are a lot of little niche things that are pissing me off about it. Then when you want to use SPICE with virt-viewer, it downloads a spice.vv file that you have to open, which auto deletes when the VM is shutdown or close (haven’t tested this). just feels like weird UX. anyway…

Storage Controller vs Virtual Hard Drives

I changed the SCSI controller from LSI 53C to VirtIO SCSI Single. But when I booted the VM back up I still saw the same generic SATA ACHI Controller. I felt like there was some ignorance on my part so I asked AI for any insights, it informed me to add a drive cause the type on the actual virtual disk could still be bound to the old type. So I temp added a disk (just for testing) and changed the connection from SATA to SCSI and checked the dev mgmt and ran a test and the performance was a fair bit better…

compared to

Performance Increase Overview

Benchmark Test Metric Type Performance Change Percentage Increase
Seq1M-Q8T1 Read 433 MB/s → 651 MB/s +50.3%
Write 56 MB/s → 95 MB/s +69.6%
Seq1M-Q1T1 Read 400 MB/s → 511 MB/s +27.8%
Write 51 MB/s → 54 MB/s +5.9%
Ran4K-Q32T1 Read 14 MB/s → 136 MB/s +871.4%
Write 7 MB/s → 7 MB/s 0.0% (No Change)
Ran4K-Q1T1 Read 6 MB/s → 13 MB/s +116.7%
Write 1.5 MB/s → 2 MB/s +33.3%

Key Takeaways
  • Massive Random Read Improvement: The biggest leap is in Ran4K-Q32T1 Read, sky-rocketing by 871.4%. This means heavy multi-threaded background random tasks will feel exponentially faster.
  • Solid Sequential Gains: Large file transfers (Seq1M) see a great bump, with reads improving by roughly 28% to 50%, and multi-queued writes jumping by nearly 70%.
  • Lagging Write Speeds: Random deep-queue writes (Ran4K-Q32T1) didn’t improve at all, and sequential single-thread writes (Seq1M-Q1T1) only crawled up by 5.9%.

That’s a bit improvment, I need to get the base OS HDD on this new type to gain the performance increase. Do to that:

Swap the Real Drive to SCSI

  1. In the Proxmox Hardware tab, select the 1 GB dummy disk you just made and click Detach. Then select the detached unused disk and click Remove.
  2. Select your main Windows boot disk (currently sitting on sata0 or ide0) and click Detach. It will immediately drop down to the bottom of the hardware list as an Unused Disk 0.
  3. Double-click that Unused Disk 0.  (I don’t know why double click seemed the only option I couldn’t see any action items at the top)
  4. In the pop-up window, change the Bus/Device dropdown to SCSI (it will likely assign scsi0). Click Add.

Fix the Boot Order

  1. Go to the VM’s Options tab in Proxmox.
  2. Double-click Boot Order.
  3. Check the box for your newly reattached scsi0 drive and drag/button it to the very top of the list so it is the primary boot device. Click OK.

Yeah for some reason it wasn’t checked off, so reattaching a vHDD has this implication something I didn’t instinctively had to do, in the snip above I unchecked net boot and checked off the scsi0.

Start your VM. before I ran the test I wanted to make sure the baseline VM was fine for it since now it was the Windows main OS drive that was running on the new virtual SCSI bus. however sure enough windows updates were alerady hitting the disk and the CPU.. I noticed it in task a manager, which was also showing me…
like what?! 84% active time constant, with a contant 800+ ms repsonse time and a measly 1.7MB/s … is windows doing insane I/o and bottle necking the I/O bus? was the theory all BS, or would this have happened on the settings I had before…? So many questions, so little answers… but the results are not good the Windows updates process is low CPU and high wait time on disk it seems the disk is slowing things down….
well system is back to idle windows updates completed.. lets see what diskmark has to say… shows the same results as “D:\” so we should have got the I/O performance increase, yet.. I remain skeptical….

Summary

So, we touched a bit on some basic management of a Proxmox host, like checking system resources, networking, storage, and managing VMs. Each of these are not covered in depth by any means, but just the simple fundamentals to getting a VM up and running and basic management of them.
These fundamentals are needed for the next stage, migrating VMs from ESXi to ProxMox. I know, I know, you’re saying I already did a basic pilot of that in the past here: Migrate ESXi VM to Proxmox – Zewwy’s Info Tech Talks but that was a bare metal, bare FS and using a linux VM with a convertion tool to just convert the  base HDD and it’s associated FS intact the version required by the hypervisor. It also took a lot of space, bandwidth, I didn’t explain what each step was really doing in detail. Anyway, long story short in the next blog post I’m gonna see how we can use Veeam to do a migration instead of a linux machine.
Posted on

Setting up a Proxmox host

Install Proxmox

Hardware

Step 1) Pick Hardware. Important is CPU support. Mostly ARM or x86_64.
– My host a HPE DL160 G9.

Software

Step 2) Install using appropriate installer image.
– In my case x86_64 version 9.2
– I installed to an internal 32GB sd card.

Storage (Physical)

Step 3) VM Storage.
I’ve discussed this in the past specially when it comes to shared storage options. There you can see a picture of all the options available to Proxmox, and if it supports snapshots or if its shared. For ease sake of this post we’re going to stick to local storage.  With the minted information from that chart alone ZFS would seem the winner, however….

Choosing the right storage architecture for a virtualization host requires a careful balance between resource allocation, hardware capabilities, and performance goals. For this build—featuring 50 GB of memory, an HPE B140i controller running in SATA AHCI pass-through mode, and a mix of SSDs and a mechanical drive—maximizing raw performance and preserving system RAM for virtual machines is our primary objective. By selecting LVM-Thin instead of ZFS, we bypass the heavy computational and memory overhead of a Copy-on-Write filesystem, ensuring that nearly all 50 GB of RAM remains dedicated strictly to our workloads. The design stripes multiple solid-state drives into a single, high-performance LVM-Thin volume group to multiply IOPS and throughput for VM boot disks. Meanwhile, the standalone 4TB spindle drive is formatted as a standard, zero-RAM-footprint Linux directory to act as an isolated target for possible Proxmox backups and static file shares. This hybrid, LVM-centric approach eliminates storage controller bottlenecks, maximizes the lifespan and speed of our SSDs, and relies on a robust backup strategy rather than restrictive hardware or software redundancy.

After messing around about an hour, I found out the reason I wasn’t seeing the drives was due to a controller configuration (it was already set to Sata AHCI support mode), which I double verified by seeing the disks and running dd commands against them to get sequential performance numbers. The reason, was cause apparently in this mode drives are not hot swapable.

Could attempt a manual rescan via the shell backend, I guess but as noted there. “If your SATA controller supports hot swap, it should “just work(tm).”

<rant> Stupid ass HP, always causing me to waste my life away cause of their stupid ass storage controller and firmware/driver choices.. ughhh </rant>

Turn off Swap

SIDE QUEST! Congratulations you just entered a side quest on your way to configuring your storage for your PVE hypervisor. SWAP!

An critical optimization step for any Proxmox host booting from flash media is managing the system’s swap space. By default, the Debian-based Proxmox installer creates a virtual memory swap partition directly on the boot drive. When running Proxmox from an internal SD card, leaving swap enabled is a hardware hazard; Linux will continuously shift idle processes onto the card, exhausting its low write-endurance and risking boot environment corruption. Because this host boasts a healthy 50 GB of physical RAM, we immediately disabled and removed the default swap volume to shield the SD card from unnecessary wear. However, completely lacking a swap space can lead to kernel instability under unexpected memory spikes. Our strategy resolves this by re-establishing a dedicated swap space directly on our new solid-state storage tier. Crucially, this swap will not be placed inside the dynamic LVM-Thin pool—which can cause file system deadlocks and severe latency—but will instead be carved out as a fixed, pre-allocated ‘Thick’ LVM volume. This hybrid approach ensures the SD card remains read-heavy and protected, while giving the host an ultra-fast, safe SSD safety net without sacrificing valuable system memory to ZFS.

1. Turn off active swap immediately

swapoff -v /dev/mapper/pve-swap

2. Stop it from turning back on when you reboot

Open your filesystem table:
nano /etc/fstab
Look for the line that mentions pve-swap. It will look similar to this:
/dev/pve/swap none swap sw 0 0
Add a # at the very beginning of that line to comment it out and disable it permanently:
# /dev/pve/swap none swap sw 0 0

3. Delete the volume entirely (Optional but recommended)

To ensure the OS never touches it again, remove the logical volume entirely:
bash
lvremove /dev/pve/swap
Now with swap off we can finally build our LVM groups and move the swap to the SSDs.

Back to Storage (Logical)

Step 1: Create the Physical Volumes (PV)

First, we tell LVM that these three specific SSDs are ready to be used as raw storage building blocks.
pvcreate /dev/sda /dev/sdb /dev/sdc
Expected output: Physical volume "/dev/sda" successfully created. x3

Step 2: Combine them into a Volume Group (VG)

Now, we pool those three independent drives into one large, unified storage pool. We will name this group pve-fast.
vgcreate pve-fast /dev/sda /dev/sdb /dev/sdc
Expected output: Volume group "pve-fast" successfully created.

Step 3: Verify the Master Pool

To confirm everything was combined properly and to check your exact total available space, run:
bash
vgs pve-fast
You should see pve-fast listed with 3 physical volumes (#PV) and a total size that roughly equals the combined capacity of your three SSDs.
root@g9-pve:~# pvcreate /dev/sda /dev/sdb /dev/sdc
Physical volume "/dev/sda" successfully created.
Physical volume "/dev/sdb" successfully created.
Physical volume "/dev/sdc" successfully created.
root@g9-pve:~# vgcreate pve-fast /dev/sda /dev/sdb /dev/sdc
Volume group "pve-fast" successfully created
root@g9-pve:~# vgs pve-fast
VG #PV #LV #SN Attr VSize VFree
pve-fast 3 0 0 wz--n- <670.70g <670.70g

Creating the Volume Group (VG) only defines the boundaries of your master pool. It tells LVM: “You are allowed to use the storage blocks inside sda, sdb, and sdc.” It does not decide how data is laid out yet.

The master pool itself is neutral. The choice between Linear or Striped happens entirely in the next step when we create the Logical Volumes (LVs) inside that pool. Why it’s like this, I dunno, I’m just here to figure out how it works.

Back to Swap

We will allocate 4 GB of space for this safety net. We will use the -i 3 flag to guarantee that any memory swapped to disk is interleaved across all three SSD controllers simultaneously for maximum throughput.
Run these four commands sequentially in your Proxmox CLI:

1. Create the Striped Logical Volume

We will carve out a new volume named fast-swap from your pve-fast volume group.
lvcreate -L 4G -i 3 -I 64k -n fast-swap pve-fast

    • -L 4G: Allocates exactly 4 Gigabytes of space.
    • -i 3: Forces the volume to stripe data across exactly 3 physical disks (RAID0 behavior).
    • -I 64k: Sets the optimal block stripe size for performance.

2. Format the Volume for Swap

Now we tell the operating system to format this new striped block device specifically as Linux swap space.
mkswap /dev/pve-fast/fast-swap

3. Activate the New Swap Space

Turn on the newly created SSD swap space right now so the system can begin utilizing it.
swapon /dev/pve-fast/fast-swap

4. Make it Permanent Across Reboots

We need to register this new location in your system’s filesystem table so it mounts automatically every time the server turns on. Run this command to append the new rule to your configuration file:
echo '/dev/pve-fast/fast-swap none swap sw 0 0' >> /etc/fstab

Verify the Configuration

To verify that your swap is active, running at top speed, and no longer touching your 32 GB SD card, run:
swapon --show
You should see /dev/mapper/pve--fast-fast--swap listed as the sole active device, confirming your SD card is safe.
root@g9-pve:~# lvcreate -L 4G -i 3 -I 64k -n fast-swap pve-fast
Rounding size 4.00 GiB (1024 extents) up to stripe boundary size <4.01 GiB (1026 extents).
Logical volume "fast-swap" created.
root@g9-pve:~# mkswap /dev/pve-fast/fast-swap
Setting up swapspace version 1, size = 4 GiB (4303351808 bytes)
no label, UUID=a6ec7580-dc9c-40de-8405-b9f8c66a41fd
root@g9-pve:~# swapon /dev/pve-fast/fast-swap
root@g9-pve:~# echo '/dev/pve-fast/fast-swap none swap sw 0 0' >> /etc/fstab
root@g9-pve:~# swapon --show
NAME TYPE SIZE USED PRIO
/dev/dm-0 partition 4G 0B -1
root@g9-pve:~# fdisk -l | grep mapper
Disk /dev/mapper/pve-root: 12.8 GiB, 13740539904 bytes, 26836992 sectors
Disk /dev/mapper/pve--fast-fast--swap: 4.01 GiB, 4303355904 bytes, 8404992 sectors
K we’re finally done with swaps. I asked AI why I wasn’t seeing any partitions on the SSDs after allocating them to the LVM pool. it stated:

“When we ran the pvcreate command earlier, LVM took over the raw, unpartitioned block devices (sda, sdb, sdc) directly, rather than writing a traditional partition table like MBR or GPT.

Why LVM Prefers No Partition Layout
Tools like fdisk and gparted look for standard partition tables to tell them where a drive starts and ends. LVM bypasses this entirely:
    1. Direct Disk Ownership: LVM wipes out the standard partition header area and writes its own metadata directly to the very first sectors of the raw disk.
    2. Simplified Layout: Because we are striping these drives together, creating individual partitions (like sda1, sdb1) is an unnecessary extra layer of management. By giving LVM the raw disks, it can seamlessly manage the entire drive boundary.

How to Properly View Your LVM Layout
Because fdisk doesn’t natively parse LVM metadata structures, it will report that the disks look empty. To see the true reality of your storage, you must use LVM-specific inspection commands instead:
    • To see the raw disks LVM owns: pvdisplay or pvs
    • To see your combined master pool: vgdisplay or vgs
    • To see the striped swap volume we just carved out: lvdisplay or lvs"

Take this with a grain of sand I do not know if its true but seems that way.

root@g9-pve:~# fdisk -l
Disk /dev/sda: 223.57 GiB, 240057409536 bytes, 468862128 sectors
Disk model: KINGSTON SA400S3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sdb: 223.57 GiB, 240057409536 bytes, 468862128 sectors
Disk model: KINGSTON SA400S3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sdc: 223.57 GiB, 240057409536 bytes, 468862128 sectors
Disk model: KINGSTON SA400S3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@g9-pve:~# pvs
PV VG Fmt Attr PSize PFree
/dev/sda pve-fast lvm2 a-- <223.57g 222.23g
/dev/sdb pve-fast lvm2 a-- <223.57g 222.23g
/dev/sdc pve-fast lvm2 a-- <223.57g 222.23g
/dev/sde3 pve lvm2 a-- <29.22g 3.63g
root@g9-pve:~# vgs
VG #PV #LV #SN Attr VSize VFree
pve 1 2 0 wz--n- <29.22g 3.63g
pve-fast 3 1 0 wz--n- <670.70g 666.69g
root@g9-pve:~# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-a-tz-- <10.79g 0.00 1.58
root pve -wi-ao---- <12.80g
fast-swap pve-fast -wi-ao---- <4.01g

Back to Storage

So now I just need another logical volume for the VM high speed OS vHDDs.
When I tried to assign 100% of the remaining space to the VM data volume, I hit a common LVM roadblock: the thin-pool conversion failed due to insufficient free space (0 extents). This happens because an LVM-Thin pool needs a tiny bit of unallocated space left over to build its hidden metadata index for tracking snapshots. To fix this, I deleted the raw volume and recreated it using 99%FREE of the remaining pool instead. This small tweak left plenty of breathing room for the tracking database while keeping the 3-disk stripe perfectly aligned.

1: Recreate it with 99% of the pool space

By allocating 99%FREE instead of 100%FREE, we guarantee there is plenty of room left over for the metadata engines while still satisfying the stripe alignment requirements.
lvcreate -l 99%FREE -i 3 -I 64k -n fast-data pve-fast

2: Convert it to a Thin Pool

lvconvert --type thin-pool pve-fast/fast-data
Once it says successfully converted, run the final step to link it to your Proxmox dashboard:
pvesm add lvmthin Striped-SSDs --vgname pve-fast --thinpool fast-data

The conversion went through smoothly, and the high-speed storage tier is now online in the Proxmox GUI under the name Striped-SSDs

Quick Sequential I/O test:

root@g9-pve:~# swapoff /dev/pve-fast/fast-swap
root@g9-pve:~# dd if=/dev/zero of=/dev/pve-fast/fast-swap bs=1M count=2000 status=progress conv=fdatasync
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 3.59334 s, 584 MB/s
root@g9-pve:~# swapon /dev/pve-fast/fast-swap
swapon: /dev/mapper/pve--fast-fast--swap: read swap header failed
root@g9-pve:~# mkswap /dev/pve-fast/fast-swap
Setting up swapspace version 1, size = 4 GiB (4303351808 bytes)
no label, UUID=aa6557d6-626b-4680-9741-2b63a1f55a13
root@g9-pve:~# swapon /dev/pve-fast/fast-swap

More Storage

Yes even more storage, while we used LVM to stripe across our 3 SSDs. We are going to use Ext4 on the 4TB Drive to host ISOs, or large disk virtual drives on the VMs.

1. Create a Standard Partition Table

We will write a clean, modern GPT partition table to the raw drive and create a single partition that takes up 100% of the 4TB space.
parted -s /dev/sdd mklabel gpt mkpart primary ext4 0% 100%

2. Format the Partition as Ext4

Now, we format that fresh partition (/dev/sdd1) with the standard Linux Ext4 filesystem. This handles sequential data streams beautifully on mechanical platters.
mkfs.ext4 /dev/sdd1

3. Create a Mount Point and Mount the Drive

We will create a permanent folder on your host OS and mount the physical drive into it.
mkdir -p /mnt/pve/mass-storage
mount /dev/sdd1 /mnt/pve/mass-storage

4. Make the Mount Permanent Across Reboots

To make sure Debian hooks this drive back up every time the server boots, we add its unique identification to your filesystem table (fstab). Run this command to fetch the drive’s unique ID and automatically write the mount rule:
echo "/dev/sdd1 /mnt/pve/mass-storage ext4 defaults,noatime,nofail 0 2" >> /etc/fstab
(Note: noatime eliminates unnecessary write cycles to track when files are read, and nofail ensures your Proxmox host still boots perfectly even if the 4TB drive is unplugged).

4. Register the storage for the Proxmox GUI

Finally, run this command to tell Proxmox that this folder is ready to accept backup files, ISOs, and VM disks:
pvesm add dir Mass-Storage --path /mnt/pve/mass-storage --content backup,iso,images

Verify Your Entire Server Storage Layout

Now that everything is fully configured, your storage is split perfectly into two distinct, high-efficiency worlds. If you run:
df -h /mnt/pve/mass-storage
root@g9-pve:~# parted -s /dev/sdd mklabel gpt mkpart primary ext4 0% 100%
root@g9-pve:~# mkfs.ext4 /dev/sdd1
mke2fs 1.47.2 (1-Jan-2025)
Creating filesystem with 976754176 4k blocks and 244195328 inodes
Filesystem UUID: 84d88803-a246-4067-8cec-3a93ba188169
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

root@g9-pve:~# mkdir -p /mnt/pve/mass-storage
root@g9-pve:~# mount /dev/sdd1 /mnt/pve/mass-storage
mount: (hint) your fstab has been modified, but systemd still uses
the old version; use 'systemctl daemon-reload' to reload.
root@g9-pve:~# systemctl daemon-reload
root@g9-pve:~# echo "/dev/sdd1 /mnt/pve/mass-storage ext4 defaults,noatime,nofail 0 2" >> /etc/fstab
root@g9-pve:~# dd if=/dev/zero of=/mnt/pve/mass-storage/zerofile bs=1M status=progress conv=fdatasync
78979792896 bytes (79 GB, 74 GiB) copied, 293 s, 270 MB/s
root@g9-pve:~# dd if=/dev/zero of=/mnt/pve/mass-storage/zerofile bs=1M status=progress oflag=direct
2867855360 bytes (2.9 GB, 2.7 GiB) copied, 33 s, 86.9 MB/s

Summary: Proxmox VE Storage Architecture: Maximizing Performance and RAM with a Non-Cached Controller

When designing local storage for a hypervisor host, the default answer is often to throw everything into a ZFS pool. However, storage architecture should never be a one-size-fits-all solution. For this Proxmox VE build—featuring 50 GB of physical RAM, an HPE B140i controller running in SATA AHCI pass-through mode, a trio of SSDs, and a single 4TB mechanical spindle drive—we chose a hybrid, LVM-centric approach designed specifically to prioritize raw performance and maximize available system memory for virtual workloads.

Phase 1: Protecting the Boot Media and Reclaiming Swap

The server boots Proxmox from an internal 32 GB SD card. By default, the Debian-based installer creates an active swap volume directly on the boot drive. Because SD cards utilize low-endurance flash memory, leaving an active swap partition on this media is a hardware hazard that would quickly wear out the card. Furthermore, with 50 GB of physical RAM available, host-level swapping should be incredibly rare.
We immediately disabled and purged the default pve-swap volume from the SD card. To preserve a host safety net without introducing latency or deadlocks, we moved the swap partition to our incoming solid-state pool. Crucially, this swap space was created as a fixed, pre-allocated “Thick” LVM volume rather than being nested inside a thin data pool, ensuring kernel stability under unexpected memory spikes.

Phase 2: The Performance Tier (3-Disk LVM-Thin Stripe)

To give our VM operating systems maximum IOPS and unthrottled throughput, we grouped our three zeroed SSDs (sda, sdb, sdc) into a single LVM Volume Group named pve-fast.
Because data safety is managed via a strict backup strategy rather than local fault tolerance, we chose to strip data evenly across all three disks using LVM’s interleaved striping parameter (-i 3). This acts as a high-efficiency software RAID0 array directly inside the Linux kernel. During configuration, we encountered a classic LVM hurdle: allocating 100% of the remaining pool to a raw data container left 0 extents behind for the metadata tracker, causing the LVM-Thin conversion to fail. Re-provisioning the container at 99%FREE provided the required breathing room for the tracking database while maintaining perfect alignment across the three controllers.
This performance tier consumes virtually 0 MB of host RAM, leaving almost all 50 GB available for our VMs. A sequential write test using dd directly against the raw striped blocks clocked in at a blistering 584 MB/s, successfully compounding the bandwidth of our independent controllers.

Phase 3: The Mass Storage Tier (Zero-RAM Spindle Directory)

For our 4TB mechanical drive (sdd), we chose to completely bypass LVM and ZFS, formatting it directly as a standard Ext4 Directory partition mapped straight to the Proxmox dashboard.
Using ZFS here would have starved our host by demanding a massive chunk of RAM for its ARC cache, while its Copy-on-Write architecture would have severely choked write performance on a controller lacking a battery-backed hardware cache. LVM-Thin was also discarded for this drive; Proxmox backup files require a standard filesystem folder, and thin block-level provisioning creates massive physical fragmentation on spinning platters over time.
By using a standard Ext4 directory, we can provision large virtual hard disks (vHDDs) for backup servers like Veeam using the QCOW2 file format. QCOW2 handles thin-provisioning intelligently at the virtual file level, preventing the hypervisor from scattering blocks chaotically across the physical disk.

Phase 4: Benchmarking and the Reality of Caching

We ran two distinct write benchmarks against our newly formatted 4TB Ext4 storage directory to observe how the operating system handles a mechanical drive:
  1. The Buffered Test (conv=fdatasync): After an initial RAM-buffered burst, the sequential write stream stabilized at an impressive 270 MB/s. This is significantly faster than the drive’s raw hardware capability. The boost is entirely driven by Ext4 optimizations like Delayed Allocation (delalloc) and sequential extents, which neatly arrange incoming data blocks on the fast, outer edge of the empty platter.
  2. The Direct I/O Test (oflag=direct): To expose the raw physical limits of the drive, we bypassed the Linux kernel’s RAM page cache completely. Stripped of its file-system optimizations, the performance leveled off at 100 MB/s, exposing the exact mechanical floor of the spindle and proving how vital the filesystem’s caching layer is for normal operation.

The Power-Safety Tradeoff

The 270 MB/s buffered speed comes with an engineering tradeoff: write safety. Because our AHCI pass-through controller lacks a physical battery-backed write cache, any data floating in the host’s volatile RAM cache during a sudden power outage will be lost.
While this risk would be unacceptable for a live production database, it is perfectly suited for this specific architecture. The 4TB tier is dedicated strictly to static ISOs and compressed Veeam backup repositories; a power failure simply invalidates a running backup job, which can easily be restarted once the system boots back up. To completely mitigate this, the host will be plugged into an Uninterruptible Power Supply (UPS) integrated with automated shutdown software, ensuring all memory caches are safely flushed to the physical disks before the server powers down.
This finished architecture leaves us with a highly optimized, dual-tier environment: a blazing fast 584 MB/s SSD stripe for active VMs, a highly efficient 270 MB/s sequential mass storage folder for backups, and a completely unburdened 50 GB pool of RAM dedicated entirely to running workloads.
This is the bare basics of setting up a Proxmox server. Things we haven’t covered yet are networking, updating, clustering, shared storage, managing VMs, etc. These will be covered in the upcoming blog posts. This one is just the fundamental requirement to all those other topics. This is just the foundation. Hope this helps someone.
BONUS MATERIAL!!!!
If you read this far amazing, you may wonder how to figure out how to know how much actual disk space a vm’s disk is using when configured on a LVM thin. well the GUI won’t tell you. You can run “lvs” and do math… why a native command gives you this data in a percentage instead of actual size? Beats me.. but here paste this into the shell to create a better new command “lvu” which I call “logical Volume Usage”
cat << 'EOF' >> ~/.bashrc
alias lvu="lvs -o lv_name,lv_size,data_percent --noheadings --units g | awk '{
name = \$1;
alloc = \$2;
pct = \$3;
gsub(/[A-Za-z]/, \"\", alloc);
gsub(/%/, \"\", pct);
if (pct == \"\" || pct == 0) {
used = alloc;
} else {
used = (alloc * pct / 100);
}
printf \"%-16s | Allocated: %6.2f G | Used Space: %6.2f G\n\", name, alloc, used
}'"
EOF
source ~/.bashrc

Now just type “lvu”

root@g9-pve:~# lvu
data                         | Allocated: 10.79 G       | Used Space: 10.79 G
root                          | Allocated: 12.80 G       | Used Space: 12.80 G
fast-data               | Allocated: 660.02 G   | Used Space: 8.12 G
fast-swap             | Allocated: 4.01 G          | Used Space: 4.01 G
vm-100-disk-0 | Allocated: 0.00 G          | Used Space: 0.00 G
vm-100-disk-1  | Allocated: 32.00 G       | Used Space: 8.12 G

Why this isn’t a native command, also beats me.

Posted on

PVE Hosts Won’t Boot, Missing Drive

This is pretty dumb… every other hypervisor I’ve ever played with, if the boot drive is fine… the OS boots… period….

Yet the other day I tried to boot my PVE host and it just won’t boot it would get stuck stating that a datastore (nothing that’s a dependency for the OS to actually boot) was causing the OS not to boot….

I found this PVE thread that was more recent with a comment that worked for the OP.

“if you created the partition via the gui, we create a systemd-mount unit under /etc/systemd/system

(e.g. mnt-datastore-foo.mount)

you can disable that unit with ‘systemctl disable <unit>’

or delete the file

we’ll improve the docs in the near future and have planned to make the gui disk management a bit easier in regards to removing/reusing”

This was posted in 2021, yet when I checked that path (booting into Recovery mode) it didn’t contain any file with a ending of .mount… So not sure what this is about. I did however find this thread which was exactly my problem… and funny enough the OP literally posted their own answer (which is the answer here as well) and no other comments made on the post, which was created in  2018…

[SOLVED] – Reboot Proxmox Host now will not boot | Proxmox Support Forum

“So I had upgraded some packages and the proxmox host recommended rebooting the system. After rebooting the system hangs at the screen showing [DEPEND] in yellow for 3 lines:
Dependency failed for /mnt/Media
Dependency failed for local file system
Dependency failed for File system check on /dev/Data-Storage/Media

I tried running control-D to continue but it does not continue.

I’m guessing I need to clean up the entries how can i do that? I’m assuming I just need to boot into emergency and edit /etc/fstab and remove the entries?

OK yes removing those from /etc/fstab fixed it and now it boots.”

This is exactly what I did as well… I saw the offending entry which was a BTRFS storage I had configured in the past and that storage unit had been shutdown. (I thought I blogged this, but I only blogged about using LVM over iscsi.. Configuring shared LVM over iSCSI on Proxmox – Zewwy’s Info Tech Talks)

Anyway, removing the entry from fstab and rebooting.. bam PVE host came right up.

Constructive criticism to PVE, while yes any knowledgeable Linux sysadmin will figure out how to fix this, as I just did here. However, how about NOT having the boot process fail simply cause a configured storage is not available… like all other hypervisors… BOOT the host and show the storage as failed in the management UI to clean it up that way…. Just.. food for thought….

Posted on

Configuring shared LVM over iSCSI on Proxmox

So, I’ve been recently playing with Proxmox for virtualization. It’s pretty nice, but in my cluster (which consisted of two old laptops) whenever I would migrate VM’s or Containers it would have to migrate the storage over the network as well. Since they are just old laptops everything connects together with 1 gbps to switches with the same rated ports.

I’m used to iSCSI so I checked the Proxmox storage guidance to see what I could use.

I was interested in ZFS over iSCSI. However, I temporarily gave up on this cause for some reason… you have to allow root access to the FreeNAS box over SSH, on the same network that the iSCSI is for….

“First of all we need to setup SSH keys to the freenas box, the SSH connection needs to be on the same subnet as the iSCSI Portal, so if you are like me and have a separate VLAN and subnet for iSCSI the SSH connection needs to be established to the iSCSI Portal IP and not to the LAN/Management IP on the FreeNAS box.
The SSH connection is only used to list the ZFS pools”

Also mentioned in this guide.

This was further verified when I attempted to setup ZFS on an iSCSI disk, I go this error message:

Since I didn’t want to configure my NAS to have root access over SSH, on the iSCSI network. I was still curious then what the point of iSCSI was for PVE if you can’t use a drive shared… Reviewing the chart above, and this comment “i guess the best way to do it, is to create a iscsi storage via the gui and then an lvm storage also via the gui (if you want to use lvm to manage the disks) or directly use the luns (they have to be managed on the storage server side)”

I ended up using LVM on the disk “3: It is possible to use LVM on top of an iSCSI or FC-based storage. That way you get a shared LVM storage”

However, using this model you can’t use snapshots. 🙁
You can use LVM-Thin but that’s not shared.

Step 1) Setup Storage Server

In my case I’m using a FreeNAS server, with spare drive ports, so for this test I took a 2TB drive (3.5″), plugged it in and wiped it from the web UI.

After this I configured a new extent as a raw device share.

Created the associated targets and portals. Once this was done (since I had dynamic discovery on my ESXi hosts) they discovered the disk. I left them be, but probably best to have separate networks…. but I’ll admit… I was lazy.

Step 2) Configure PVE hosts

In my case I had to add the iSCSI network (VLAN tagged) on to my hosts. This is easy enough Host -> System -> Network -> Create Linux VLAN

OK, so where in ESXi you simply add an iSCSI adapter, in PVE you have to install it first? Sure ok lets do that… Turns out it was already installed.
after reading that and seeing what my ESXi did, I managed to edit my /etc/pve/storage.cfg and added

iscsi: freenas
portal 172.16.69.2
target iqn.2005-10.org.freenass.ctl:proxhdd
content none

To my surprise… it showed as a storage unit on both my PVE hosts. :O

mhmm doing a df -h, I don’t see anything… but doing a fdisk -l sure enough I see the drive.. so cool 🙂
So now that I got both hosts to see the same disk, I guess it simply comes down to creating a file system on the raw disk.
Or not… when I try to create a ZFS using the WebUI it just says no disk are available.

Step 3) Setup LVM

However, adding an LVM works:

After setting up LVM the data source should show up on all nodes in the cluster that have access to the disk. One on of my nodes it wasn’t showing as accessible until I rebooted the node that had no problems accessing it. ¯\_(ツ)_/¯

So, there’s no option to pick storage when migrating a VM, you have to go into the VM’s hardware settings and “move the disk”.

When I went to do my first live VM migration, I got an error:

I soon realized this was just my mistake by not having selected “delete source” since when “moving the disk” it actually converted the disk from qcow2 to raw and didn’t delete the old qcow2 file. So I simply deleted it. then tried again…

and it worked! Now the only problem is no snapshots. I attempted to create an LVM-Thin on top the LVM, and it did create it, but as noted in the chart both my hosts could not access it at the same time, so not shared.

Guess I’ll have to see how Ceph works. That’ll be a post for another day. Cheers.

*Update* I’ll have to implement a filter on FreeNAS cause Proxmox I guess won’t implement a fix that was given to them for free.

https://forum.proxmox.com/threads/iscsi-reconnecting-every-10-seconds-to-freenas-solution.21205/#post-163412

https://bugzilla.proxmox.com/show_bug.cgi?id=957

*UPDATE May 2025*

Ohhh Looks like they may have finally got off their butts and implemented a fix…

Fedrich “As Victor notes, their patch is applied and available in libpve-storage-perl >= 8.3.4, which is part of Proxmox VE 8.4. Thanks for your contribution!

One thing I want to point out is that, even with this patch, the Proxmox VE node will still perform a connection check (via TCP ping) when there is no active session (yet) on some occasions, e.g., when first logging in or after boot. However, with this patch it will not do TCP pings against a portal if there is an active session to the portal, and this should get rid of the large majority of (recurring) TCP pings against portals.”

I have not personally had a chance to test or verify this however.

Legal Information Proudly powered by WordPress