Migrating/Restoring Veeam

In one of my pervious posts I discussed upgrading Veeam, today I want to discuss migrating it entirely. Or recovering it, as this process here is essentially the same.

Disclaimer what you do in your own environment is on you, everything in this blog is for educational purposes only. This also doesn’t cover encryption management all data is moved in-place (E.G disconnecting, and reconnecting an HDD from one machine to another), with the data at rest being unencrypted.

Step 1) Sign in to Veeam portal

I didn’t have a paid product license, so my download section was full of free trial links. Since I’m using CE (community edition) from here: Free Backup Software For Windows, VMware, & More – Veeam

Step 2) Download the ISO

it’s a doosy at 13 GBs

Step 3) Read the update notes for any expected issues/outcomes.

For all the FAQs go here: Veaam Upgrade FAQs

For basic System Requirements and release notes see here: Veeam Backup & Replication 12.3 Release Notes

The main thing will be the change of the server SQL service, moving from MS SQL Express, to PostgresDB, Though it’s not directly mentioned from what I can see other than the step 8 in the Upgrade path: Upgrading to Veeam Backup & Replication 12.3 – User Guide for VMware vSphere

Step 4) Attach the ISO

Attach it to the server being upgraded or installed on.

in my case this time, I’m simply cloning my freshly semi hardened Windows11 image, giving it a whopping 8GB of RAM, and 64Gig HDD for the OS and Veeam App to live on. While that’s being prepared lets take a config backup of our veeam server to make our lives easier.

Step 5) Backup Config.

I’d hope you’d have this configured before your Veeam server failed.

Veeam B&R -> File -> Backup Config, in our case save it to backup data drive as that will be moved and mounted first thing, we can then use that to load the config and should be good to go.

Now it shows up under Drive:\VeeamConfigBackup\Hostname\Hostname_Datestamp.bco

Step 6) Install Veeam on New Server

Depending on your Uptime requirements, you can either spin up the new server with a temp different IP, get the Veeam app and services installed, then move your discs and change IP’s. Since I don’t care in my lab, I’ll fully shutdown my existing server to free up the IP and system resources. then boot up my new server, attach the downloaded ISO in step 1, and install Veeam.

Hostname, networking, and other prerequisites are not discussed in details here.

I like how it knows, click install…

Install B&R

How long we wait is based on the Matrix. Looking at the VM resource usage, and my machines based on the setup, looks like it’s reading from the ISO to load installation files. and writing it somewhere to disk, my setup only yielded me about 40 MB’s and took roughly 8 minutes.

Agree to the EULA.

License upgrade: (I’ll try not selecting this since CE, nope wizard wouldn’t let me for CE, shucks hahah)

Service account, Local System (recommended). I left this default, next.

This is why I like Veeam, made by sysadmins for sysadmins.

Install, and now we wait… once complete

Step 7) Attach disk with backup data

How you do this is up to you, I got the needful done.

Step 8) Open Veeam B&R Console, and import config backup.

In Veeam B&R Console, click what should be file -> Config Backup, then click restore button.

Now, I picked restore since I shutdown my OG server to move the data as a whole, so I picked restore:

The config deets check em over, I don’t know what the minimum gap between version is allowed, but in this case 12.3.1 source, to target 12.3.2

Target Data is localhost, pay attention to the login name, if you ever change the local admin account or whatever account installs Veeam, this could be an issue to your SQL Veeam config.

yes…

Restore…

Yes…

Wait for services to all stop…

success… until it’s not…

This for some reason failed…

I clicked start and it seemed to start everything up just fine…

But no matter what when I tried to rescan any repos in the console it would complain that not all components were upgraded. Everything AI was telling me was off and felt wrong.. I found this one thread with the statement “It seems that not all Windows 10 installations are facing this problem. We’ll try to figure out of certain builds are involved in this. On the other hand, a fresh v12 install in Win10 works without any problems.” Well This is a fresh install, it happened after the backup import, when I did the last upgrade back in March, it was ain in place upgrade from 12.1 to 12.3, and I didn’t have this problem.

After enough fooling around I found my answer here, which was to run the provided script. finding the component listed with 0.0 as noted in the thread. Strange.

Then finally the part of the wizard completed:

October 11, 2025November 17, 2025

Docker on Core Linux

Docker Running on Core Linux

Is it possible? … Yes? However I didn’t write down some of the pre-requisites for the Core Server (whoops, maybe one day I’ll redo it from scratch). But if you do manage to get the base binaries installed this post should be helpful for all the caveats I faced along the way…

In my previous post I mentioned that Docker wouldn’t run unless it was 64bit machine, so I created a Core Linux 64 bit image and showed how to get the base OS up and running… but what about Docker itself.

Now I got this “working” but I didn’t exactly write down all my steps (it took a long time to figure out). From looking at the VM’s history looks like I simply used the tc account to download and extract the base Docker binaries:

now this doesn’t tell me the relative path I was on when some of the relative paths are called, but I do know it was the tc account so some safe assumptions can be made.

Reviewing my AI chat and notes I took, and getting it running again after a reboot, it seem after the “install” (copy base files to path shown above image, line 51) I also added “var/lib/docker” and “etc/docker” to the filetool.lst file, so they stay persisted after reboot. Strangely only /var/lib/docker is populated, but I can’t see how that’s the case from the history review. I was pretty positive the script itself failed to execute… I really should start from scratch else this post will be a bit useless…. butt…. F*** it….

The next issues seems to be tired to cgroups and certificates…

Fixing Cgroups Error

sudo mount -t tmpfs cgroup_root /sys/fs/cgroup/
sudo mkdir /sys/fs/cgroup/devices
sudo mount -t cgroup -o devices none /sys/fs/cgroup/devices

That should be it… but we need this to be persisted and auto run at boot time so we don’t have to do this every time…

sudo vi /opt/dockerd.sh
i
mount -t tmpfs cgroup_root /sys/fs/cgroup/
mkdir /sys/fs/cgroup/devices
mount -t cgroup -o devices none /sys/fs/cgroup/devices
ESC
:wq
sudo vi /opt/bootlocal.sh
*append with*
/opt/dockerd.sh
:wq
sudo chmod +x /opt/dockerd.sh
filetool.sh -b

The next issue seems that docker would load, but when pulling a container to load it would just seem to fail complaining about certificates.

Fixing Certificate Error

I found the point in my notes rambling with AI when I figured it out…

“NO F***KIN WAY!!!!!!! https://stackoverflow.com/questions/75696690/how-to-resolve-tls-failed-to-verify-certificate-x509-certificate-signed-by-un I read this thread and read the answer by Andrei Nicolae… which said just durr copy ca certs to /etc/ssl/certs I was like, I bet docker is hard coded to look there, which is why it was first suggested but all other apps on tiny core linux know to use /usr/local/etc/ssl/certs, so yeah docker never was using the expectects paths liek I suspected from the begining cause we manualy installed it for a OS not supported. so with this I did sudo mkdir -p /etc/ssl/certs sudo cp /usr/local/etc/ssl/certs/* /etc/ssl/certs sudo pkill dockerd sudo dockerd & sudo docker pull hello-world and guess what it finally freaking worked”

But I realized instead of copying them I could just make a symlink

sudo mkdir /etc/ssl/
ln -s /usr/local/etc/ssl/certs/ /etc/ssl/

I simply placed these lines in /opt/dockerd.sh file I created earlier, rebooted and verified that /etc/ssl/certs was populated with certs and it was.

And finally…

Running Dockerd

sudo DOCKER_RAMDISK=true dockerd &

Pulling Image

sudo docker pull hello-world

Running Image

sudo docker run --rm hello-world

Yay we actually ran a container from Core Linux.. Mind Blown… I swear I had it all running at only 90MB of RAM, but checking now show 116MB Bah…

To get Docker to run at boot my final /opt/dockerd.sh looked like this:

*Update* outisde of hello world I wasn’t able to spin up any other real containers due to other technical limitations, this was a huge waste of time.

October 11, 2025October 11, 2025

Installing CorePure64

Back Story

So in my previous post I shared how to setup a very small footprint Linux server called Linux Core: Installing Core Linux – Zewwy’s Info Tech Talks

but…… I tried getting docker running on it was hit with an error “Line 1: ELF: File not found”.

AI, after giving all the required command to do a “manual install”, stated, “duuuuurrrrrrrrrrrrrr docker don’t give 32 bit binaries”, to which I was replied huh… I guess I installed 32 bit Core Linux… do they have 64bit versions?

It gave me some dumb link to some dumb third party source.. the answer is yes.. here: Index of /16.x/x86_64/release/

So here we go again….

Installing CorePure64

Step 1) Download Install image CorePure64-16.0.iso

Step 2) get x64 bit hardware, or create a VM that supports 64 bit. I have 64 bit hypervisors, so I will create a VM as I did in my first post.

This time 2 CPU, 1 GB RAM, 60GB HDD, thin, VMparavirtual scsi controller, EFI enabled with secure boot, let’s see if this works out…. No boot… Flip boot settings to BIOS mode… ISO boots.. ah man FFS its x64 based but still relies on BIOS for booting… that sucks… of well moving on….

Booting and Installing Core Linux

Attach ISO boot. Core Linux boots automatically from ISO:

For some reason the source doesn’t tell you what to do next. type tc-install and the console says doesn’t know what you are talking about:

AI Chat was kind enough to help me out here, and told me I had to run:

tce-load -wi tc-install

Which required an internet connection:

However even after this, attempting to run gave the same error.. mhmm, using the find command I find it, but it needs to be run as root, so:

sudo su
/tmp/tcloop/tc-install/usr/local/bin/tc-install.sh

C for install from CDrom:

Lets keep things frugal around here:

1 for the whole disk:

y we want a bootloader (It’s extlinux btw located [/mnt/sda1/boot/extlinux/extlinux.conf}):

Press enter again to bypass “Install Extensions from..”

3 for ext4:

Like the install source guide says add boot options for HDD (opt=sda1 home=sda1 tce=sda1)

last chance… (Dooo it!) y:

Congrats… you installed TC-Linux:

Once rebooted the partition and disk free will look different, before reboot, running from memory:

after reboot:

Cool, the install process was 100% the same as the 32bit process…

but running uname -m we see we are now 64 bit instead of 32 bit.

Changing TC Password

Step 1) Edit /opt/.filetool.lst (use vi as root)
– add etc/passwd and etc/shadow

Step 2) run:

filetool.sh -b

Step 3) run

passwd tc

Step 4) run

filetool.sh -b

Now reboot, you may not notice that it applied due to the auto login, however, if you type exit to get back to the actual login banner, type in tc and you will be prompted for the password you just set. Now we can move on to the next step which is to disable the auto login.

Disable Auto-Login

Step 1) Run

sudo su
echo 'echo "booting" > /etc/sysconfig/noautologin' >> /opt/bootsync.sh

Step 2) Run

filetool.sh -b
reboot

K on to the next fun task… static IP…

Static IP Address

For some reason AI said I had to create a script that runs the manual step… not sure if this is the proper way… I looked all over the Wiki: wiki:start – Tiny Core Linux Wiki I can’t find nothing.. I know this works so we’ll just do it this way:

Step 1) Run:

echo "ifconfig eth0 192.168.0.69 netmask 255.255.255.0 up" > /opt/eth0.sh
echo "route add default gw 192.168.0.1" >> /opt/eth0.sh
echo 'echo "nameserver 192.168.0.7" > /etc/resolv.conf' >> /opt/eth0.sh
chmod +x /opt/eth0.sh
echo "/opt/eth0.sh" >> /opt/bootlocal.sh
filetool.sh -b

Step 2) reboot to apply and verify.

It didn’t work, but is I add “sleep 2” just before /opt/eth0.sh in the bootsync.sh file, then it works, not the greatest but I’ll take it for now.

Updates?

Tiny Core Linux, updating is a bit different from traditional distros. Here’s how it works:

🔄 Updating Tiny Core Linux

Tiny Core doesn’t have a single tc-update command for the whole system, but it does have tools for updating extensions and managing packages:

🧰 Extension Update

Use the tce-update command to update installed extensions:

bash

tce-update

This checks for newer versions of your installed .tcz extensions and updates them.
📦 Other Useful Commands

tce-load: Loads extensions from your /tce directory.
tce-ab: Opens the AppBrowser (if you have GUI).
tce-audit: Audits installed extensions.
tce-remove: Removes extensions.

🧱 Core System Update

To update the core system itself (like the kernel and initrd), you’ll need to manually download the latest bzImage and core.gz (or tinycore.gz) from the Tiny Core Mirrors then replace them in your boot partition.

Steps:

Download the latest files.
Mount your boot partition.
Replace bzImage and core.gz.
Update your bootloader config (e.g., GRUB or syslinux).
Reboot.

June 4, 2025

VMware Changes Update URLs

If you run a home lab, or manage systems for companies you may have noticed updates not working in VAMI… something like…. Ohhh I dunno.. this:

Check the URL and try again.

Unable to patch the vCenter via VAMI as it fails to download the updates from Broadcom public repositories

Cause

Public facing repository URLs and authentication mechanisms are changing. Download URLs are no longer common but unique for each customer therefore will require to be re-configured.

Well… wow thank you Broadcom for being so… amazing.

If you want to be overly confused about the whole thing you can this this KB: Authenticated Download Configuration Update Script

As the original link I shared above all you have to do is login to the Broadcom support portal, and get a token, and edit the URL…. but….

Notes:

- The custom URL is not preserved post migration upgrade, FBBR restore and VCHA failover
- If there is a proxy device configured between vCenter and the internet, ensure it is configured to allow communications to the new URL
- Further patches automatically update this URL. For example, if 8.0.3.00400 is patched to 8.0.3.00500, the default URL will change to end in 8.0.3.00500.

Looks like this was enforced just a couple days ago … Sooooo, happy patching? ¯\_(ツ)_/¯

June 3, 2025

Permission to perform this operation was denied. NoPermission.message.format

For anyone who may use my site as a source of informational references, I do apologies, for the following:

My Site Cert expiring. ACME is great, I’m just a bit upset they refuse to announce their HTTP auth sources so I can’t create a security rule for it. Right now it would be restricted to App Type. While not bad.. not good enough, so I manually have to allow the traffic for the cert to be renewed.
No… I have no interest in allowing ACME access to my DNS for DNS auth.
Site was down for 24 hours. If anyone noticed at all, yes my site was down for over 24 hours. This was due to a power outage that lasted over 12 hours after a storm hit. No UPS could have saved me from this. Though one is in the works even after project “STFU” has completed.
No, I have no interest in clouding my site.

I have a couple blog post ideas roaming around, I’m just having a hard time finding the motivation.

Anyway, if you get “Permission to perform this operation was denied. NoPermission.message.format” while attempting to move a ESXi host into a vCenter cluster. Chances are you may have a orphaned vCLS VM.

If so, log into VAMI and restart the ESX Agent Manager (EAM) service.

After restarting that service everything should hunky dory…

Cheers.

March 19, 2025

Update Veeam 12.3

Grab Update file from Veeam.

Step 1) Sign in to Veeam portal

Step 2) Download the ISO, it’s a doosy at 13 GBs

Step 3) Read the update notes for any expected issues/outcomes.

For all the FAQs go here: Veaam Upgrade FAQs

For basic System Requirements and release notes see here: Veeam Backup & Replication 12.3 Release Notes

Step 4) Attach the ISO to the server being upgraded or installed on

My case a 12.1 based server.

My case it’s a VM, so I just attach it via VMRC.

Step 5) Run the Installer

Make sure you stop any “continuous” jobs, and close the B&R Console.

Double Click Setup.exe on the mounted ISO’s main directory.

If you haven’t guessed it, click Upgrade. Yes, nice to see coding done where it just does a check and knows it’s a Veeam server, so the only option is to Upgrade.

In my case I again only have one option to choose from.

Agree to the EULA.

Upgrade the server, here’s you have a checkbox to update remote components automatically (such as Veeam proxies). In my lab the setup is very simply so I have none. I just click next.

License upgrade: (I’ll try not selecting this since CE, nope wizard wouldn’t let me for CE, shucks hahah)

Service account, Local System (recommended). I left this default, next.

Here’s the OG MS SQL instance:

… yes?

For the Veeam Hunter service… ignore (Shrug)

free space… needs more than 40 Gigs… holy molly….

43.1 GB required, 41 GB Available. Unreal, guess I’ll extend the drive, great part of running VMs. 🙂

Finally! Let’s Gooooo! and sure enough first step.. here comes the new SQL instance.. this is probably why it requires over 40 gigs to do the install, to migrate the SQL instance from MS SQL to Postgres…. Wonder if space will be reclaimed by removal of the MS SQL Express instance….

Roughly half hour later…

Mhmmm checking the services I see the orginal MS SQL instance is still there running. I see a postgres service.. not running… uhhhh mhmmm…

All Veeam services are running, open the Veeam B&R console, connect, and yup it opens. The upgrade component wizard automatically opened, and it updated the only item.. itself.

*UPDATE* Patch for latest CVE of 9.9. If you have a domain joined Veeam server.

KB4724: CVE-2025-23120

*thumbs up* It’s another 8 gig btw…

March 16, 2025March 24, 2025

Installing Core Linux

Installing TC-Linux (Core Only)

Sources

Source: wiki:install_hd – Tiny Core Linux Wiki

On, ESXi VM: wiki:vmware_installation – Tiny Core Linux Wiki

FAQs: http://www.tinycorelinux.net/faq.html

Setting up VM

VM Type: Other Linux 32bit kernel 4.x
CPU: 1
Mem: 256 MB
HDD: 20 Gig
Network: DHCP + Internet Access

Change boot to BIOS (instead of EFUI)

Booting and Installing Core Linux

Attach ISO boot. Core Linux boots automatically from ISO:

For some reason the source doesn’t tell you what to do next. type tc-install and the console says doesn’t know what you are talking about:

AI Chat was kind enough to help me out here, and told me I had to run:

tce-load -wi tc-install

Which required an internet connection:

However even after this, attempting to run gave the same error.. mhmm, using the find command I find it, but it needs to be run as root, so:

sudo su
/tmp/tcloop/tc-install/usr/local/bin/tc-install.sh

C for install from CDrom:

Lets keep things frugal around here:

1 for the whole disk:

y we want a bootloader (It’s extlinux btw located [/mnt/sda1/boot/extlinux/extlinux.conf}):

Press enter again to bypass “Install Extensions from..”

3 for ext4:

Like the install source guide says add boot options for HDD (opt=sda1 home=sda1 tce=sda1)

last chance… (Dooo it!) y:

Congrats… you installed TC-Linux:

Once rebooted the partition and disk free will look different, before reboot, running from memory:

after reboot:

Installing OpenSSH?

tce-load -wi openssh

This is where things got a little weird. Installing an app… Not as root TC-Linux says…

This is when things got a bit annoying n weird, even though the guide says using -wi installs it in the on boot section, I found it wasn’t loading on boot, well at first I noticed it didn’t start at all after install, as I couldn’t SSH in, this was cause of a missing config file…

Even if I got it running it still wouldn’t run at boot and that apparently was cause the file disappeared after reboot. This is apparently cause the system mostly run entirely in RAM. If you didn’t notice even after install the root filesystem was still only roughly 200 MB in size (enough to fit into the RAM we configured for this VM).

Notice the no password on the tc account? Set it, reboot. doesn’t stick…

Notice the auto login on tty1? Attempt to disable.. doesn’t stick…

Configuring Core Linux

Long story short apparently you have to define what paths are to be considered persistent via a file:

/opt/.filetool.lst

These files are saved to mydata.gz via the command:

filetool.sh -b

So here’s what we have to do:

Configure the system to ensure settings we configure stay persistent across reboots.
Change the tc account password.
Disable auto login on TTY1.
Configure Static IP address.
Install and run on boot OpenSSH.

Changing TC Password

Step 1) Edit /opt/.filetool.lst (use vi as root)
– add etc/passwd and etc/shadow

Step 2) run:

filetool.sh -b

Step 3) run

passwd tc

Step 4) run

filetool.sh -b

Disable Auto-Login

Step 1) Run

sudo su
echo 'echo "booting" > /etc/sysconfig/noautologin' >> /opt/bootsync.sh

Step 2) Run

filetool.sh -b
reboot

K on to the next fun task… static IP…

Static IP Address

Step 1) Run:

echo "ifconfig eth0 192.168.0.69 netmask 255.255.255.0 up" > /opt/eth0.sh
echo "route add default gw 192.168.0.1" >> /opt/eth0.sh
echo 'echo "nameserver 192.168.0.7" > /etc/resolv.conf' >> /opt/eth0.sh
chmod +x /opt/eth0.sh
echo "/opt/eth0.sh" >> /opt/bootlocal.sh
filetool.sh -b

Step 2) reboot to apply and verify.

What about SSH?!

Oh right.. we got it installed but we never got it running did we?!

Step 1) Run:

cp /usr/local/etc/ssh/sshd_config.orig /usr/local/etc/ssh/sshd_config
vi /usr/local/etc/ssh/sshd_config

Edit and uncomment:
Port: 22
Address: 0.0.0.0
PasswordAuthedAllowed:true

Step 2) Run:

echo "usr/local/etc/ssh/" >> /opt/.filetool.lst
echo "/usr/local/etc/init.d/openssh start" >> /opt/bootlocal.sh
filetool.sh -b
reboot

congrats you got openSSH working on TC-Linux.

Hostname

Most systems you run the hostname command… ooooeee not so easy not TC-Linux.

Option 1 (Clean)

Edit the first line of /opt/bootsync.sh which sets the hostname.

Then just run filetool.sh -b, done.

Option 2 (Dirty)

To ensure the hostname persists across reboots, you need to modify the /etc/sysconfig/hostname file:

Edit the hostname configuration file:
```
sudo vi /etc/sysconfig/hostname
```
Add or modify the line to include your desired hostname:
```
your_new_hostname
```
Save and close the file.

Add /etc/sysconfig/hostname to the persistence list:

echo "etc/sysconfig/hostname" >> /opt/.filetool.lst
echo "hostname $(cat /etc/sysconfig/hostname)" >> /opt/bootlocal.sh

Save the configuration:
```
filetool.sh -b
```

reboot

That’s it for now, next blog post we’ll get to installing other goodies!

Managing Apps

Installing Apps

As you can see it’s most running:

tce-load -wi

for all the details see their page on this, or run -h.

Source of app (x86): repo.tinycorelinux.net/15.x/x86/tcz/

For the most it’s install app. Edit files as needed, saved edited files to /opt/.filetool.lst. Then run backup command, test service, edit /opt/bootlocal.sh with commands needed to get app/service running. again run filetool.sh and bobs your uncle.

Deleting Apps

To remove a package on Tiny Core Linux that was installed using tce-load, here’s what you can do:

For Extensions in the onboot.lst File:
- First, remove the package name from the /etc/sysconfig/tcedir/onboot.lst file to prevent it from being loaded at boot. You can edit the file with:
  bash
  sudo nano /etc/sysconfig/tcedir/onboot.lst
- Delete the entry corresponding to the package you wish to remove, then save and exit.
Delete the Extension File:
- Navigate to the directory where the extensions are stored:
  bash
  cd /etc/sysconfig/tcedir/optional
- Remove the .tcz file associated with the package:
  bash
  sudo rm package-name.tcz
Clean Up Dependency Files (Optional):
- To clean up leftover dependency files related to the removed package, you can check and delete them from the same directory (/etc/sysconfig/tcedir/optional).

November 27, 2024

Veeam VM Restore failed: Cannot apply encryption policy. You must set the default key provider.

So in my Lab vCenter went completely POOOOOF. So, I installed it fresh.

After vCenter was installed, I updated my Veeam configuration to ensure my backup chains wouldn’t break which still works great by the way.

One VM was missing from my vSphere. So I went to restore it when all of a sudden:

I remembered by post about configuring a Native Key Provider cause it was required as such to have a vTPM. So I thought, is this a “PC Load Letter” problem, and it’s actually just complaining that I didn’t configure a NKP for it to “apply encryption policy”?

Follow the same old steps to configure a NKP.

Log in to the vSphere Client:
- Open the vSphere Client and log in with your credentials.
Navigate to Key Providers:
- Select the vCenter Server instance.
- Click on the Configure tab.
- Under Security, click on Key Providers.
Add a Native Key Provider:
- Click on Add.
- Select Add Native Key Provider.
- Enter a name for the Native Key Provider.
- If you want to use hosts with TPM 2.0, select the option Use key provider only with TPM protected ESXi hosts.
Complete the Setup:
- Click Add Key Provider.
- Wait for the process to complete. It might take a few minutes for the key provider to be available on all hosts.
Backup the Native Key Provider:
- After adding the Native Key Provider, you must back it up.
- Click on the Native Key Provider you just created.
- Click Backup.
- Save the backup file and password in a secure location.

Once I did all that…

No way that actually worked. But will it boot? Well it def “booted” but it asked for the BitLocker key (which makes sense since we created a new TPM and it doesn’t have the old keys). I checked my AD and sadly enough for some reason it didn’t have any BitLocker keys saved for this AD object/VM.

Guess this one is a loss and the importance of saving your encryption keys.

September 29, 2024

Careful Cloning ESXi Hosts

I’ll keep this post short. I was doing some ESXi host deployments in my home lab, and I noticed that when I would install on a 120GB SSD, the install would go smoothly, but I wasn’t able to use any of the storage as a Datastore. However, if I took a fresh install copy of ESXi from installing onto an 8GB USB Stick and DD’d it to the 120GB SSD I got several advantages from this:

When done via a USB3 Pipe of Linux live holding a copy of my base image to deploy I could get speeds in excess of 100 MB/s, and with only 8GB of data to transfer, the “install” would complete in a mere 90 seconds.
The IP address and root password are preconfigured to what I already now, and I can simply change the IP address from the DCUI and call it a day.

Using this method I could have a host up in less than 5 minutes (2 min to boot linux live, 90 seconds to install the base ESXi OS image, and 2 more to boot ESXi). This was of course on machine without ECC and all the server hardware firmware jazz… in those cases install times are always longer. anyway…

This was an amazing option, until I noticed that when I connect one machine in I just deployed and changed the IP address, and (since I’m super anal about networking during this type of project/operations) I noticed my ping from one machine (a completely different IP address) started to drop when the new device came up… and after a while the ping responses would come back but drop from the new host, and vice versa, flip and flop it goes. I’m used to this usually if there’s an IP conflict and two devices have the same IP address. In this case they were different IP addresses… after enough symptom gathering and logical deduction of because I had to assume that the MAC address just be the same and this is the same problem in reverse (different IP’s but same MAC) and as such experiencing the same symptoms.

To validate this I simply deployed my image to a new machine, then I went on the hunt to figure out how to see the MAC address, since I couldn’t plug in the NIC and get to the web based MGMT interface I had to figure out how to do that via the console CLI directly… mhmm after enough googling on my phone I found this spiceworks thread with my answer:

vim-cmd hostsvc/net/info | grep “mac =”

I then checked this against the ESXi host that I saw the flipping flopping with, and sure enough they matched… After doing a fresh install I noticed that the first 3 sections match the physical MAC, but in my DD deployed ones they retain the MAC of the system from which it was installed and those when I ran the command above, I could tell which ones were deployed via my method. This was further mentioned in this reddit thread by a commenter who goes by the name of sryan2K1:

“The physical NIC MACs are never used. vmk ports, along with VMs themselves will all use VMWare’s OUI as the first half of the address on the wire.”

OK, now maybe I can still salvage my deployment method by simply deleting and recreating the VMK after deployment, but I’d guess it best be done via the DCUI or direct console… I found one KB by VMware/Broadcom but it gave a 404 but Luckly there was a wayback machine link for it here.

Which states the following:

“During Initial Installation and DCUI, ESXi management interface (default vmk0) is created during installation.

The MAC address assigned will be the primary active physical NIC (pnic) associated.

If the associated vmnic is modified with the management interface vmkernel will once again assign MAC address of the associated physical NIC.

To create a VMkernel port and attach it to a portgroup on a Standard vSwitch, run these commands:

esxcli network ip interface add --interface-name=vmkX --portgroup-name=portgroup
esxcli network ip interface ipv4 set --interface-name=vmkX --ipv4=ipaddress --netmask=netmask --type=static"

Alternatively, you can also use esxcli to create the management interface vmkernel on the VDS.

Creation of the management interface with the ‘esxcli network’ will generate a VMware Universally Unique address instead of the pnic MAC address.

It is recommended to use the esxcli network IP interface method to create the management interface and not use DCUI.

Workarounds: None

Additional Information:
Using DCUI to remove vmnic binding from management vmkernel or any modification will apply change at vSwitch level. Management interface is associated with propagating the change to any port groups within the vSwtich level.

Impact/Risks: None.”

I”m assuming it means if you use the DCUI to reconfigure the MGMT interface settings the MAC will automatically be reconfigured to match what I found during initial clean install and mentioned in the reddit thread of using the first 3 sections to derive the MAC of the VMK.

But what if you don’t have any additional interfaces to use to make the section change in the DCUI to have that actually happen? cause what I’ve noticed changing the IP address and disabling IPv6 and rebooting did not change the VMK’s MAC address. Oh there’s in option in the DCUI “Reset Network Settings” within there there’s several options, I simply picked reset to factory defaults. Said success, checked the MAC via the first command stated above and bam the VMK nic changed to what it should be! Sweet my deployment method is still viable.

Hope this helps someone.

September 25, 2024

The virtual machine must be encrypted

Sooo I lost a VM in my fray of re-organizing my server farm. Like a lost pup I figured I just rely on my good old Veeam backup sets. Recover VM, alright here we goo….

What.. what does that mean…. Oh wait is this cause of when I blogged about adding vTPMs to VMs?

Re-checked the linked video from VMware… 2 min in … “Failure to save your key backup will result in unrecoverable data loss”…. mhmmm, OK I thought all I did was add a TPM device to my VM and enabled secure boot, that’s the deal here?

Somewhere I read that the VM config files get encrypted, but I don’t think that’s the case here either. Even checking the Pre-reqs from VMware I can’t see anything nothing this:

Prerequisites

Ensure that your vSphere environment is configured with a key provider. See the following for more information:
Configuring vSphere Trust Authority
Configuring and Managing a Standard Key Provider
Configuring and Managing vSphere Native Key Provider
Ensure that host encryption mode is enabled. See Enable Host Encryption Mode Explicitly.
The guest OS you use can be Windows Server 2008 and later, Windows 7 and later, or Linux.
The ESXi hosts running in your environment must be ESXi 6.7 or later (Windows guest OS), or 7.0 Update 2 (Linux guest OS).
The virtual machine must use EFI firmware.
Verify that you have the required privileges:
Cryptographic operations.Clone
Cryptographic operations.Encrypt
Cryptographic operations.Encrypt new
Cryptographic operations.Migrate
Cryptographic operations.Register VM

What I think is happening here is my NKP that IS a Prerequisite went poof (the vCenter server that was used to create it is shutdown and not being used), and another temp vCenter is being used.

My first thought was maybe I could just add a new NKP and go as I figured the TPM physical module that’s installed simply needs this, and I think it’s this hardware that’s faulting the boot.

I didn’t want to muck the with original I just recovered so I tried to clone it, but the clone failed too complaining about encryption before adding a TPM, further validating my assumption. What I don’t understand it how the VM was allowed to be created from backup in the first place if I can’t even clone it…?

Any since I know recovery is possible (since I just did it), I guess maybe I can just remove it? Or I could also create a new VM and use vmkfstools to clone the hdd… let’s try that first…

Go to boot VM, well got past that error but the Machine was bitlocked, I was hoping it wasn’t going to be.. go to AD server, open ADUC… no bitlocker tab… ughhhh…

ADUC Missing BitLocker Recovery Tab in 1809 – Microsoft Community

Right but where is that in on a server, oh in server manager it moved…

Yay there’s the bitlocker tab and… it’s empty.. man give me a fucking break… so now I have a bunch of backups that are useless cause I lost the bitlocker key… shiiiiiiit

Well I don’t have anything to follow up on here but a lesson learnt to backup your bitlocker key (I don’t know why it wasn’t save to the AD computer object).