Posted on

ESXi 6.x Datastore Not Mounted

Quick post here, I had to recover from a flooded basement. Sorry for the day outage. I had to put my disc in another server and load FreeNAS, and import my ZFS volumes, recreate the iSCSI targets, and then I added them to my ESXi hosts, and rescanning the HBAs shows the disks…

but the datastores were not visible…

so I googled and found this VMware thread with some helpful commands to try. (I do kind of agree with the OP, that its annoying they removed the front end UI for import that could handle this)

esxcli storage vmfs snapshot list

esxcfg-volume -M UUID

Ehh it worked!

Hope this helps someone. If this doesn’t work you might have some other underling issue?

Posted on

Veeam As Built Report

Source: HomeLab – Veeam VBR Documentation with AsBuiltReport – A Technology Blog (zenprsolutions.net)

Prereqs:

Ughhh…

Google!! … Let’s try this…

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Nice it worked…

You can use the commands like in the source guide, I however simply downloaded the source files and extracted to Veeam server c:\temp.

Don’t forget to use recursive… lol

As you might be able to tell, my Veeam instance is non domain joined. Using the local Admin account to run everything.

Let’s GO!

Little missing field validation, but not important for what will just be text field in a report.

and failed cause bad credentials, right I forgot to put in my password when making the snippet, lets build the credentials again properlly and run through again…

Alright some file checking here, nice….

and success short lived… Enterprise Edition… like what??

Well I tried… I’ll update this blog once there’s an update from the Dev.

Posted on

Using StarTech USB3HDMI in OBS

Startech USB3HDMI

This is a quick post. I’ve lately been streaming and I picked up this sick capture device the “StarTech USB3HDMI” I got this after EposVox, check out his twitch channel here, did a review of the product and posted it on youtube here.

This video really helped me out in making this purchase and configuring it to work with OBS. I haven’t tested this card with other capturing based software, and technically OBS is a streaming based software, but can be used to record the canvas of multiple input sources, so in reality way more powerful then basic video capture alone.

Anyway here’s the pictures of it working on OBS, as a device capture input and the settings to make it work for:

Composite AKA the Red/White/Yellow RCA plugs

If you don’t see all these options try scrolling, the scroll bar is easy to miss. It’s important to pick the audio device here for the Red/White audio input to be picked up. There’s also addition settings when you click “Configure Video” which oddly enough includes some audio options…?

As you can see Audio input set to embedded. With video selected as 6/Composite. That’s it for that one. Sometimes I find you have to restart OBS or reseat the capture device for things to play nice.

This is the capture I do the most and I found I could actually play games from this source directly without a low latency splitter. For near zero latency game play though you might want to get a near zero latency splitter.

VGA

Time to test VGA, to start this test I’m actually going to take my computer and power it on a standard VGA monitor, after the output is good, I will then switch it over to the capture card.

Oddly it didn’t show up at first, after I restarted OBS the source showed fine.

Now for this setup you’d almost assume that the device has a 3.5mm headphone jack or maybe RCA plugs for Audio in… but the only one is the ones available for composite, now you possible could go from a 3.5mm audio out to RCA in on the composite input of the capture card and use the same audio settings as the above composite, however I personally have not tested that.

For now I use the Line-In on my mainboard on the computer running OBS to capture to audio in this setup. If you’re capture computer doesn’t have a dedicated Line-IN or Aux in, then attempt the trick I mentioned in the previous paragraph.

DVI

The VGA is actually physciall done via a  DVI-I to VGA adapter that comes with the kit, and the fact it has a DVI-I capture input means it can grab analog VGA and digital signal on the same interface, and probably why this card costed as much as it did. So if you need to capture DVI, simply unplug the DVI-I to VGA adapter and plug the DVI cable directly into the capture card and pick 2/DVI from the device settings/properties as shown in the previous snippets from the RCA and VGA input options discussed above.

I do not have snips of this as (at the time of this writing) I do not have a DVI based device to capture its source.

HDMI/DP

Since Display port and HDMI are (for the most part) interchangeable (won’t discuss the technical details about features of each, and which version) but for the most part, it’s like USB, backwards compatible.

That being said it has a limitation of only 1080p, so don’t expect 4k capturing here. Also while the box did not define it, I’m also assuming 30 FPS or so, and not from my testing but from EposVox apparently decent latency (a couple hundred ms?) nothing worse than what a bad TV would give.

Posted on

Email Stuck in Exchange Transport in 2022

Happy New Year!

If you are an exchange admin you may want to check out the notice from Microsoft. But you probably already have considering it started in the beginning of the new year: Email Stuck in Exchange On-premises Transport Queues – Microsoft Tech Community

So you probably already implemented this fix.

We have now created a solution to address the problem of messages stuck in transport queues on Exchange Server 2016 and Exchange Server 2019 because of a latent date issue in a signature file used by the malware scanning engine within Exchange Server. Customer action is required to implement this solution. When the issue occurs, you’ll see errors in the Application event log on the Exchange Server, specifically event 5300 and 1106 (FIPFS), as illustrated below:

Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 1:03:42 AM 
Event ID: 5300 
Level: Error 
Computer: server1.contoso.com
Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.
Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 11:47:16 AM 
Event ID: 1106 
Level: Error 
Computer: server1.contoso.com 
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.

Using the Automated Solution

  • Download the script here: https://aka.ms/ResetScanEngineVersion
  • Before running the script, change the execution policy for PowerShell scripts by running Set-ExecutionPolicy -ExecutionPolicy RemoteSigned.
  • Run the script on each Exchange mailbox server that downloads antimalware updates in your organization (use elevated Exchange Management Shell).

Edge Transport servers are unaffected by this issue. You can run this script on multiple servers in parallel. After the script has completed, you will see the following output:

[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Reset-ScanEngineVersion.ps1
EXCH1 Stopping services...
EXCH1 Removing Microsoft engine folder...
EXCH1 Emptying metadata folder...
EXCH1 Starting services...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start...
EXCH1 Starting engine update...
Running as EXCH1-DOM\Administrator.
--------
Connecting to EXCH1.CONTOSO.com.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
--------
[PS] Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
--------
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>Get-EngineUpdateInformation

Engine                : Microsoft
LastChecked           : 01/01/2022 08:58:22 PM -08:00
LastUpdated           : 01/01/2022 08:58:31 PM -08:00
EngineVersion         : 1.1.18800.4
SignatureVersion      : 1.355.1227.0
SignatureDateTime     : 01/01/2022 03:29:06 AM -08:00
UpdateVersion         : 2112330001 (note: higher version number starting with 211233... is also OK)
UpdateStatus          : UpdateAttemptSuccessful

Using the Manual Solution

In lieu of using the script, customers can also manually perform steps to resolve the issue and restore service. To manually resolve this issue, you must perform the following steps on each Exchange mailbox server in your organization that downloads antimalware updates. Edge Transport servers are unaffected by this issue.

Verify the impacted version is installed
Run Get-EngineUpdateInformation and check the UpdateVersion information. If it starts with “22…” then proceed. If the installed version starts with “21…” you do not need to take action.

Remove existing engine and metadata
1. Stop the Microsoft Filtering Management service.  When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

Update to latest engine
1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.

Verify engine update info
1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001 (or higher)

After updating the engine, we also recommend that you verify that mail flow is working and that FIPFS error events are not present in the Application event log.

If you want to know why this happened here’s a answer from the comments:

John_C_Kirk – “This wasn’t due to a change on 31st Dec. The problem is caused by an integer overflow error: the anti-malware component is converting the date/time into “YYMMDDHHMM” format and storing it as a signed 32-bit number (max value 2147483648). So, in Dec 2021, the number would start with “2112…” (below the threshold). In Jan 2022, the number would start with “2201…” (above the threshold).”

Two Thumbs up on implementation.

Posted on

ESXi /tmp is Full

I’ll keep this post short and to the point. Gott errors in the alerts.

I was like huh, interesting… go to validate it on the host by logging in via SSH then typing the command:

vdf -h

At the bottom you can see /tmp space usage:

I then found out about this cool command from this thread:

find /tmp/ -exec ls -larth '{}' \;

This will list all the files and their sizes to gander at, when I noticed a really large file:

I decided to look up this file and found this lovely VMware KB:

The Workaround:

echo > /tmp/ams-bbUsg.txt

The solution:

To fix the issue, upgrade to VMware AMS to version 11.4.5 (included in the HPE Offline Bundle for ESXi version 3.4.5), available at the following URLs:

HPE Offline Bundle for ESXi 6.7 Version 3.4.5

https://www.hpe.com/global/swpublishing/MTX-a38161c3e8674777a8c664e05a

HPE Offline Bundle for ESXi 6.5 Version 3.4.5

https://www.hpe.com/global/swpublishing/MTX-7d214544a7e5457e9bb48e49af

HPE Offline Bundle for ESXi 6.0 Version 3.4.5

https://www.hpe.com/global/swpublishing/MTX-98c6268c29b3435e8d285bcfcc

Procedure

  1. Power off any virtual machines that are running on the host and place the host into maintenance mode.
  2. Transfer the offline bundle onto the ESXi host local path, or extract it onto an online depot.
  3. Install the bundle on the ESXi host.
    1. Install remotely from client, with offline bundle contents on a online depot: 
      esxcli -s <server> -u root -p mypassword software vib install -d <depotURL/bundle-index.xml>
    2. Install remotely from client, with offline bundle on ESXi host: 
      esxcli -s <server> -u root -p mypassword software vib install -d <ESXi local path><bundle.zip>
    3. Install from ESXi host, with offline bundle on ESXi host: 
      esxcli software vib install -d <ESXi local path><bundle.zip>
  4. After the bundle is installed, reboot the ESXi host for the updates to take effect.
  5. (Optional) Verify that the vibs on the bundle are installed on your ESXi host. 
    esxcli software vib list
  6. (Optional) Remove individual vibs. <vib name> can be identified by listing the vibs as shown in #5. 
    esxcli software vib remove -n <vib name>

    Summary

    Use the commands shown to trace the source of the usage, your case may not be as easy. Once found hopefully find a solution. In my case I got super lucky and other people already found the problem and solution.

Posted on

Veeam – More Than One Replica Candidate Found

Story Time!

The Problem!

So real quick one here. I edited a Replication job and changed it source form production to a backup dataset within the Veeam Replication Job settings. I went to run the replication job and was presented with an error I have no seen before…

I had an idea of what happened (I believe the original ESXi host might have been rebuilt) I’m not 100% sure, but just speculating. I was pretty sure the change I made on the job was not the source of the problem.

Since I wasn’t concerned about the target VM being re-created entirely I decided to go to Veeam’s Replica’s, and right clicked the target VM, and picked Delete from Disk… to my amazement the same error was presented…

Alright… kind of sucks, but here’s how I resolved it.

The Solution

Sadly I had to right click the Target VM under Veeams Replicas, and instead picked “Remove from Configuration”. What’s really annoying about this is it will remove the source VM from the replication job itself as well.

Why? Unno Veeams coding choices...

So after successfully removing the target VM from Veeam’s configuration, I manually deleted the target VM on the host ESXi host. Then I had to reconfigure the replication job and point it to the source VM again. Again if your interested in why that’s the case see the link above.

After that the job ran successfully. Hope this helps someone.

Posted on

Exchange Certificates and SMTP

Exchange and the Certificates

Quick Post here… If you need to change Certificates on a SMTP receiver using TLS.. how do you do it?

You might be inclined to search and find this MS Doc source: Assign certificates to Exchange Server services | Microsoft Docs

What you might notice is how strange the UI is designed, you simple find the certificate, and in it’s settings check off to use SMTP.

Then in the connectors options, you simply check off TLS.

Any sensible person, might soon wonder… if you have multiple certificates, and they can all enable the check box for SMTP, and you can have multiple connectors with the checkbox enabled for TLS…. then… which cert is being used?

If you have any familiarity with IIS you know that you have multiple sites, then you go enable HTTPS per site, you define which cert to use (usually implying the use of SNI).

When I googled this I found someone who was having a similar question when they were receiving a unexpected cert when testing their SMTP connections.

I was also curious how you even check those, and couldn’t find anything native to Windows, just either python, or openSSL binaries required.

Anyway, from the first post seems my question was answered, in short “Magic”…

“The Exchange transport will pick the certificate that “fits” the best, based on the if its a third party certificate, the expiration date and if a subject name on the certificate matches what is set for the FQDN on the connector used.” -AndyDavid

Well that’s nice…. and a bit further down the thread someone mentions you can do it manually, when they source non other than the Exchange Guru himself; Paul Cunninham.

So that’s nice to know.

The Default Self Signed Certificate

You may have noticed a fair amount of chatter in that first thread about the default certificate. You may have even noticed some stern warnings:

“You can’t unless you remove the cert. Do not remove the built-in cert however. ” “Yikes. Ok, as I mentioned, do not delete that certificate.”-AndyDavid

Well the self signed cert looks like is due to expire soon, and I was kind of curious, how do you create a new self signed certificate?

So I followed along, and annoyingly you need an SMB shared path accessible to the Exchange server to accomplish this task. (I get it; for clustered deployments)

Anyway doing this and using the UI to assign the certificates to all the required services. Deleted the old Self Signed Cert, wait a bit, close the ECP, reopen it and….

I managed to find this ms thread with the same issue.

The first main answer was to “wait n hour or more”, yeah I don’t think that’s going to fix it…

KarlIT700 – ”

Our cert is an externally signed cert that is due to expire next year so we wanted to keep using it and not have to generate a new self sign one.

We worked around this by just running the three PS commands below in Exchange PS

Set-AuthConfig -NewCertificateThumbprint <WE JUST USED OUR CURRENT CERT THUMPRINT HERE> -NewCertificateEffectiveDate (Get-Date)
Set-AuthConfig -PublishCertificate
Set-AuthConfig -ClearPreviousCertificate

 

Note: that we did have issues running the first command because our cert had been installed NOT allowing the export of the cert key. once we reinstalled the same cert back into the (local Computer) personal cert store but this time using the option to allow export of the cert key, the commands above worked fine.

We then just needed to restart ISS and everything was golden. :D”

Huh, sure enough this MS KB on the same issue..

The odd part is running the validation cmdlet:

(Get-AuthConfig).CurrentCertificateThumbprint | Get-ExchangeCertificate | Format-List

Did return the certificate I renewed UI the ECP webUI… even then I decided to follow the rest of the steps, just as Karl has mentioned using the thumbprint from the only self signed cert that was there.

Which sure enough worked and everything was working again with the new self signed cert.

Anyway, figured maybe this post might help someone.

Posted on

vCenter Appliance Failed File Based Backup

Story Time

*UPDATE* VMware has pulled this garbage mess of an update version of vSphere. Why?

1) They PSOD ESXi Hosts...

2) Broke more shit then they fixed...

3) Broke and silently removed protocols for File Based Backups (This post)

As much as the backup failed, I failed along with it,

Task. Backup the vCenter Server using VAMI to create a file based backup.

Now for a ESXi host, you can do this super easy (at least the config so install new and simply load the config)

For a deep and better understanding of backing up and restoring ESXi host’s please read this really amazing blog post by Michael Bose from NAKIVO.

Back up ESXi configuration:

vim-cmd hostsvc/firmware/backup_config

and You will get a simple URL to download the file right to your management machine/computer.

Does vCenter have something like this? (from my research…) No.

“You use the vCenter Server Interface to perform a file-based backup of the vCenter Server core configuration, inventory, and historical data of your choice. The backed-up data is streamed over FTP, FTPS, HTTP, HTTPS, SFTP, NFS, or SMB to a remote system. The backup is not stored on the vCenter Server.”

Which hasn’t been updated since 2019. Let’s make a couple things here clear:

  1. The HTTP and HTTPS mentioned above are not like the ESXi style mentioned above where it creates a nice backup file locally on the VCSA and presents you with a simple URL to navigate to, to download it. It expects the HTTP/HTTPS to be a file based server to accept file transfers to (like dropbox).
  2. Lots of these “supported” protocols have pretty bad bugs, or simply don’t even work at all. Which well see below.

Doing the Theory

So OK, l log into VAMI, Click the Backup tab on the left hand nav, try to add a open SMB path I have available to use cause, why not, make my life some what easy…

Looking this up I get: VAMI Backup with SMB reports error: “Path not exported by the remote filesystem” (86069) (vmware.com) dated Oct 28,2021. Nice, nice.

Alrighty then, I’ll just spin up a dedicated FTP service on my freeNas box I guess. I learnt a couple things about chroot and local users via FTP, but the short and sweet was I created a local account on the FreeNAS box, created a Dataset under than existing mounted logical volume, and granted that account access to the path. Then enabled local user login for the FTP server, and specified that path as the user’s home path, and enabled chroot on the FTP service, so when this user logs in all they can see is their home path, which to that user appears as root. This (I felt) was a fair bit of security on it, even though its a lab and not needed, just nice…. ANYWAY… Once I had an FTP server ready….

Now I went to Start a File based backup of the vcenter server:

First Error: Service Not Running

In my case I got an error that the PSC Health service was not running, this might just be cause my lack of decent hardware for good performance might have caused some services to not start up in a timely manner. Either way, Navigating to Services in VAMI and started the PSC Health service. Lucky for me there was no further errors on this part.

If you have service errors you will have to check them out and get the required services up and running, which is out the scope of this post.

Second Error: Number of Connections

The next error I got complained about the allowed number of connections to the target.

Which in my case there was an option on the FreeNAS FTP service configurations for this, I adjusted it to “0” or unlimited in hopes to resolve this problem:

restart the service, and try again…

Third Error: Unknown

This is starting to get annoying…

What kind of vague error is that?!

Guy in this thread states the path has to be empty? what?

I tried that, cleared some more space, and it seems to have sorta worked?

Clear the FTP users home path, and try again:

Fourth Problem: Stuck @ 95%

The Job appeared to run but I noticed a couple things:

1) Even though the backup config said the overall size would only be roughly 400MB, the job ran to around 1.8 Gigs.

2)  All I/O appeared to stop and all Resources returned to an idle state, while the job remained stuck processing at 95%.

OK… I found this thread, which suggested to restart the autodeploy service, tried that and it didn’t work, the job remained stuck @ 95%.

I also found this VMware KB,  however,

1) I have a tiny deployment so no chance my DB would be 300Gigs.

2) When I went to check the “buggy python script” the “workaround” seemed to already have been implemented. So the versions of vCenter I was on (7.0u3a) already had this “fix” in place

3) The symptoms still remain to be exactly the same and the python scripts remain in a “sleeping” state.

FFS already….

Try Anyway

Well I saw the files were created, so I decided to try the restore method on the VCSA deployment wizard anyway…

I forgot to take a snippet here, but it basically stated there was a missing metafile.json file. I can only assume that when the backup process was stuck at 95% it never created this required json file…

FUCK….

One Scheduled Run

I noticed that I suppose overnight a scheduled job tried to run and provided yet a different error message:

Well that’s still pretty vague, as far as I know there should be no connectivity issues since file were created all the way up to 1.8 gigs, so I don’t see how it’s network, or permissions related, or even available space in this case, since all files were cleared, up to the already possible and shown to be written 1.8 gigs, which have been deleted to empty the path every time.

Liek seriously, wtf gives here. The fact there’s an entirely new KB with an entire Table of list of shit that apparently is wrong with this file based backup honestly begs the question, Where the FUCK is the QA in software these days? This shit is just fucking ridiculous already…

Check the Logs

*This Log file only gets created the first time you click “configure” under the backup section of VAMI.

Here’s how to access the logs:

Using putty or similar, SSH in as root on the appliance.
Type Shell at the prompt.
Type cd /var/log/vmware/applmgmt.
Type more backup.log or tail backup.log.

[VCDB-WAL-Backup:PID-42812] [VCDB::_backup_wal_files:VCDB.py:797] INFO: VCDB backup WAL start not received yet.

Checking the entry I find this thread. Along with this Reddit Post. Which leads right back to the first shared thread, which states some bitching about the /etc/issues files… and I have a strange feeling, just like the stuck @ 95% issue, I’ll look at the file and it will probably be correct just like the guy who created the Reddit post.

Try Alternative Protocols

When I tried alternative protocols I came across more issues:

NFS – Had the same path issue SMB did “Path not exported by remote system”

SCP – Was apparently silently dropped, much like what this thread mentioned. The amount of silence on that thread speaks volumes to me.

TFTP was also dropped.

You are so Fucked

Soo I wonder if I try to “upgrade” aka downgrade using the UI installer of a supposed version that works (7.0u2b)…

Alright so let me get this straight… I upgraded, and now I can’t make a backup cause the upgraded version is completely broken it terms of its File Basked Backups.

I can’t Roll back the upgrade without having kept the old VCSA, which was removed in my case since all other services was working, vSphere itself.

I can’t “downgrade” and existing one, I can’t make a backup to restore my old ones. OK fine well how about a huge FUCK YOU VMWARE. while I try to come up with some sort of work around for this utter fucking mess.

Infected Mushroom – U R So F**ked [HQ & 1080p] – YouTube

Work around option #1

Build a brand new vCenter, add hosts, and reconfigure.

The main issue here is the fact if you rely on CBT, you will be fucked and all the VM-IDs will have changed, so you will have to:

1) Edit and adjust all back up jobs to point to the new VM, via it’s new VM-IM.

2) Let the delta files be all recalculated (which can be major I/O on storage units depending on many different factors (# of VM, Size of VMs, change of files on VMs, etc)

Not and option I want to explore just yet.

Work Around option #2

Back and restore the config database?

Let’s try.. first backup…

copy python scripts (hope they not all buggy and messed up too..)

Stop required services:

service-control --stop vmware-vpxd
service-control --stop vmware-content-library

change the script permissions

chmod +x backup_lin.py

Run it:

Make a copy of it via WinSCP.

run the restore script… and

well was worth a shot but that failed too….

Lets try PG dump for shits…

I’d really recommend to read this blog post by Florian Grehl on Virden.net for great information around using postgres on vCenter.

Connect to server via SSH (SSH enabled required on vCenter).

“To connect to the database, you have to enable SSH for the vCenter Server, login as root, and launch the bash shell. When first connecting to the appliance, you see the “Appliance Shell”. Just enter “shell” to enter the fully-featured bash shell.

The simplest way to connect to the databases is by using the “postgres” user, which has no password. It is convenient to also use the -d option to directly connect to the VCDB instance.”

# /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

Cool, this lets us know the postgres DB service is running. The most important take away from Florian’s post is:

“When connecting, make sure that you use the psql binaries located in /opt/vmware/vpostgres/current/bin/ and not just the psql command. The reason is that VMware uses a more recent version than it is provided by the OS. In vSphere 7.0 for example, the OS binaries are at version 10.5 while the Postgres server is running 11.6”

Kool, I could use pg_dumpall but I found it didn’t work (maybe that was wrong version of vcenter being mixed, not sure) either way lets try just the VCDB instance…

interesting, lol, as you see I got an error about version mismatch. I found this thread about it and with the info from Florians post, had an idea, tried it out, and it actually worked. Mind… BLOWN.

rm /usr/bin/

OK let’s take this file and place it on the newly deployed vcenter.

even though restore appeared to have worked the vCenter instance booted and showed to be like new install. Was worth a shot I guess, but did not work.

Work Around Option #3

I’m not sure this is even a fair option, as it only works if you have existing backup of alternative types. In my case I use Veeam and its saved my bacon I don’t know how many times.

Sure enough Veeam saved my bacon again. I ended up restoring a copy of my vCenter before the 7.0u3a, which happened to be on 7.0u2d.

I managed to add a SMB path without it erroring, and unreal, I ran a File Based Backup and it actually succeeded!!

Now I just simply run the deploy wizard, and pick restore to build a new vCenter server from this backup.

Ahhh VMware… dammit you got me again!

alright fine… grabs yet another copy of vCenter…

and this time…

are you fucking kidding me? Mhmmm interesting… VCSA 7.0 restore issue – VMware Technology Network VMTN

ok… good to know…

From this… to this….

then Deploy again…

It stated it failed, due to user auth. However I was able to login and verify it worked, but sadly it also instantly expired the license as well. I was hoping I could get another 60 days without creating a new center, reconfiguring and breaking my VM-IDs and CBT delta points for my backup software.

Even this link states what I’m trying to do is not possible… ugh the struggles are real!

In the end just started from scratch, Ugh,

Posted on

When to VMware Snapshot

OK real quick short post here. I figured I’d take a snapshot of my vCenter server (reason will be next blog post).  In this case I decided to snapshot the VM with memory saving, I figured it would be faster than bringing the VM back up from a shutdown state as that’s what a normal snapshot would do.

In most cases that would probably be a fair assumption, but boy was I wrong.

It Took a short time save the snapshot but almost 15 min or more to bring the VM back to full operational status with all memory back in tact… just check out these charts:

Here you can see it took maybe 5 min to save the memory state to disk, this would have been a time of 0 minutes since a normal snapshot doesn’t save memory to disk. Then you can see the slower longer recovery time it took to get the same memory from disk and put it back into memory.

Of course taking a solid guess that disk I/O is much slower than Memory I/O the bottle would have to be non other than the actual disk….

Yup there’s the same matching results of fast disk writes, and slow disk reads…

and there’s the disk being 100% bust on the read requests. I’m not sure why the read performance on this drive was as bad as it was, but I have a feeling a regular boot would have been faster… I’ll update this post if I do an actual test.

Meh, pretty much same amount of time… I think I need some super fast local storage… yet I’m so cheap I never do…. cheap bastard…

Legal Information Proudly powered by WordPress