CU update not Showing in WSUS

The Story

Today was a bit annoying…

I did my usual updates sync, and approve required updates, and in the past this has included CU’s without much fuss. However today I did my usual and ran check for updates on a member machine, returned clean (which i was expecting as per the results on WSUS console) and then I did the followup “Check for updates from Microsoft Updates” to my dismay the server stated an update was available a CU (KB4516061)… ughhh OK…

Checking WSUS

Decided to double check WSUS, to my dismay, re-syncing, and checking unapproved updates yielded no new updates. But I know there’s a new CU/// what gives?

Doing some reseach I find this is nothing really new and has been a problem for a while due to what could be multiple problems, including apparently packaging certain updates into other updates… how lovely.

As the main answer from that one says you can Import them… Ughhh fine…

*Expectation* Download MSU, click Import update, update gets imported to WSUS, and approve.

*Reality* Well reality is generally always worse then the expectations…

Importing Update into WSUS

So I downloaded the 1.5 Gig KB from MS Catalog and on my MMC snap-in click Import update… What do I get, a Windows Explorer popup asking me where the msu file I want to import is… NO… a link to the MS Catalog website…

Ughh… I already downloaded it what gives…

after a bit more research (honestly software should not be this non-intuitive, but that’s how old software was… non-intuitive…) turns out this “Import Updates” is not even designed for remote use (uhhh isn’t that the whole point of MMC Snap-ins?!?!?) Anyway, OK so people state you have to use it directly on the WSUS server….

FINE,  Log directly into the server and open the WSUS console, click “Import Updates” IE opens and page can’t be loaded. Strange checking the IE security settings the site attempting to be navigated should be trusted.

Even grabbing the direct catalog link and pasting it in this IE window only gave me the option to Download, not “add” and then “view basket”. It turns out the option to add only becomes available after an ActiveX install for something.

Originally I was not getting this, it wasn’t until I read this , I found a golden egg on Technet, and very carefully read the answer:

MS WSUS Product Team:

“Just to let you know, a statement from the WSUS Product Team has been published: WSUS Catalog import failures

“We are aware of the issue and presently working on a fix. In the meantime, the following workarounds can be used to unblock your deployment:

After clicking on the “Import Updates…” option in the WSUS console, an Internet Explorer window will open on the following URL: http://catalog.update.microsoft.com/… &Protocol=1.20
Before proceeding with importing the updates, change the “1.20” protocol value in the URL to the previous protocol value “1.8”. The URL should look like this when you’re done: http://catalog.update.microsoft.com/… &Protocol=1.8″

Uhhh ok… so it turns out on the initial pop-up where you get the Windows can’t display this page:

Change end number to 1.8

Yes, and Yes

Once this page loads, you can add the Active X control at the bottom:

Now you get the add, and view basket, and finally get the import option:

Well that was an annoying morning…

Update Computer Group Membership without Reboot

Source

Purge the computer account kerberos tickets

klist -lh 0 -li 0x3e7 purge

Force the gpo re-evaluation

gpupdate /force

Any previous attempt for access via newly added group membership should work; such as in this example I created a new Group, added this computer object into it, created a gMSA granting the group permission to use it, however the computer was not rebooted since added it into the group which was allowed access to install the gMSA.

PS C:\Windows\system32> New-ADGroup -Name "gMSANewGroup" -SamAccountName gMSANewGroup -GroupCategory Security -GroupScope Domain -DisplayName "gMSANewGroup" -Path "CN=Managed Service Accounts,DC=zewwy,DC=ca" -Description "Members of this group get Access to gMSATest2"
PS C:\Windows\system32> Add-ADGroupMember "gMSANewGroup" -Members "THISCOMP$"
PS C:\Windows\system32> New-ADServiceAccount -name gMSATest2 -DNSHostName gMSATest2.zewwy.ca -PrincipalsAllowedToRetrieveManagedPassword "gMSANewGroup"

Then Attempting to install the gMSA fails as the computer object hasn’t updated its group memberships locally, even though it has replicated throughout the domain, but following the command above to purge the computers tickets worked:

Hope this helps someone who needs to do granular group control but also don’t have the ability to reboot the host machine for service disruptions. 🙂

Quick Managed Service Account Audit

First get the list of gMSAs from AD:

$gMSAlist = Get-ADServiceAccount -filter {samAccountName -like "*"}

Second Determine the systems allowed to use them:

ForEach ($gMSA in $gMSAlist) {(Get-ADServiceAccount $gMSA -properties *).PrincipalsAllowedToRetrieveManagedPassword}

Yay, we know who can use these accounts… but ARE they currently using it. If this returns a Group, look to see the systems in this group, else just access the system in question.

Third, verify the account is in use by listing all the services on the system and the accounts used to run them:

Get-Service | Select -ExpandProperty Name | ForEach{(Get-WmiObject Win32_Service -Filter "Name='$_'") | Select Name, StartName}

The above command simply lists out all the services and the account they run under, it’s not optimal as it is slow, but it gets it all, and if you need a more readable version pipe it into Output-CSV, or apply a more granular filter on the result for the gMSAs in question.

That’s about it, if you don’t see the gMSA listed on any service on the target machine, it’s rather safe to assume that the gMSA is not in use and can be safely removed from AD.

Remove-ADServiceAccount gMSAToBeRemoved

Fixing Veeam (Veeam Service won’t Start)

Veeam Won’t Start

Yeap, the one thing you don’t want can happen at the worst time. For me I was testing a hypervisor upgrade scenario, and my host sure enough failed to come up successfully. Well…. shit.

While I was going crazy trying to bring my host back up (the stock ESXi images wasn’t good enough cause…. RealTek, yeah… this Mobo I picked was an overall bad choice, sad cause it’s ASUS… anyway…

I went to go restore some VMs from backup onto other hosts till I could recover my main host (find that custom ESXi install image) and to my dismay… Veeam console failed to connect…

Failed to connect to the Veeam Backup & Replication server:
No connection could be made because the target machine actively refused it :9392

ughhhh, what? this is a standalone server, not domain joined, no special services account or MSAs, or separate servers, like what gives?

Event viewer is literally useless… as nothing shows anywhere for any hints.

First Fix Attempt

OK so, the usual, google, and let’s see here

Like other symptoms not much help and a generic console error, so this fix was worth a shot, what I took away from it was how to do a manual DB backup (assuming this is all the settings and configurations if re-install required) and some registry keys used by Veeam and that this was not the problem (not the droids you are after). I thought maybe I had updated and not tested, as I do tend to do shutdown instead of reboot, with my limited resources and well windows is heavy on resources.

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\SqlServerName (This is the server name where SQL is running)
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\SqlInstanceName (This is the instance name needed for the connection, which is in the format Servername\InstanceName)
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\SqlDatabaseName (This is the database name in the Databases folder once you connect)

But sadly no good, as I guess my issue is not related to any lock files on the SQL DB… ok so what else is there…

Second Fix

So I started reading this one and at first I was thinking, yup same problem, and reading along, I like Foggy but them not sharing the answer was rather annoying… then after some others reported the solution and my jaw literally dropped (probably why they tell you call support, cause this is some dirty laundry…)

as Tommy stated

“It is very likely to caused by the changing of the host name, do refer to the following link, i managed to my Veeam service started again.”

What….

sure enough running the req query command and hostname showed I had indeed changed the hostname to something more suitable AFTER installation.

Why they’d rely on a reg key vs a simply enviroment variable is really beyond me, cause the problem with using a reg key for this is pretty clear here….

So let’s try to fix this, thanks to the second guys reply by spacecrab:

“I know this is an old post, but thank you for replying with this information. I installed Veeam Backup and Replication before changing the default generated hostname, and it was really throwing me through a loop. The fix noted at that url worked perfectly after I rebooted to reset the services. I’ll relay the content here in case that sources goes away.

In my case I had renamed the computer from a default WIN234dfasd type name to a ‘much’ better alternative. Veeam refers to the local computer name in a couple of registry entries and promptly stopped working – which we didn’t notice until later.

The keys are:

HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\SqlServerName
HKLM\SOFTWARE\Veeam\Veeam Backup Catalog\CatalogSharedFolderPath

Backup of the site’s Virtual Machines is now running again.”

alright let’s update some keys to be Veeam…. I just used reg edit to do this vs figuring out the exact query (although I probably should figure out a query in-case other keys but meh….

and after a reboot… Woah! all the Veeam services are running, sure enough I can connect to my standalone Veeam Server! Wooo thanks Spacecrab!

 

 

Remove “inaccessable” datastore from VCSA

In my previous post I mentioned restoring my ESXi after a bad upgrade. Today when I attempted to add it back into vCenter, it complained stating a Datastore with the same name exists. I was a bit stumped when I saw it showing up under the datastore area as inaccessible, when there should be nothing referencing it. Googling led me to this gem where MikeOD states:

“I figured it out.  I was double checking on VM’s on those datastores.  Under “related objects”, there were no VM’s or hosts, but there were two old templates that were still referenced by the original VCenter.  When I right clicked on the template and selected “remove from inventory”, the data stores disappeared.”

mhmmm, looking at the associated VM, I checked one of it’s settings and sure enough, an old ISO was mounted on it:

just as Mike said, as soon as I removed the association, by changing the VM to client device, the inaccessible datastore went away.

You can also check for templates, snapshots, etc.

Using VMware Update Manager (VUM)

VUM

Overview

In this post I’m going to try and upgrade one of my ESXi 5.5 host to 6.5 using VCSA’s now built in by default VUM. I followed this video on youtube for reference.

First thing I noticed but the video doesn’t mention is that (at least for 6.5) VUM tab is only available using the flash based client. If you use the HTML5 based web client the Update tab isn’t shown:

HTML 5 Missing update Tab:

Flash based client with Update Tab:

As soon as I see a video, I can tell which version is being used as its sooo different. (Flash sucks)

Import Image

About a minute into the source video he gets to where the major images are for major host upgrades. Since I had no existing images provided like the video I decided to try the import wizard, which poped up a useful Windows Selection Dialog box (as I was testing this form the latest version of Windows 10 – 1903, with the latest Chrome (Built in flash after enabling it)), on top of that I uploaded the the image from a UNC path (\\ip\some\folder\file.iso) after verifying access via windows explorer, I simply pasted the UNC path into the Windows Selection Dialog box address bar, and selected the ISO image. and it worked.

only once a image is uploaded, and selected does the creation of a baseline option make itself shown.

Baseline

Let the beat drop! *Bass beat drops* What the point of this is I can’t exactly tell yet, it seems to be a one to one mapping between a name and the ISO image being used?

paraphrasing the video guide “Now that we completed this useless step, we navigate to the cluster needing to be upgraded” In my case a single 5.5 test host, and by clicking on “Go to compliance view”

Attach Baseline

It would seem the Update Tab is available, at either the vSphere host level, Datacenter level, cluster level or host level depending on the scope you wish to deploy a “baseline”. Once within the scope you choose, I’m at host, click on “attach baseline” after ensuring you are still on the update tab.

much like the source after attaching the baseline the compliance level was shown as unknown, let’s follow along and “scan for updates”.

Now I’m assuming cause I am at the host level I don’t see the tabs with compliant and others cause there is only one host. and in this case it does change to “non-compliant” cause as the speaker states “The hosts listed as non-compliant do not match the version of ESXi associated with the attached baseline” AKA these hosts need to be upgraded.

Remediate

Click it to being the upgrade process for the host/cluster, which will being a wizard! Ohhh might wizard guide me to the light at the end of the tunnel!

while I clicked next, flash gave me an error prompt telling me my session had expired, and kicked me out back to the login page, even though I was still pretty actively working on it (snippets don’t take that long). Stupid flash, logged back in and back to the wizard:

agree to the EULA

Schedule it or do it now by not checking a scheduled time.

Pick your additional options and remediation options (I picked for my test to suspend my VMs as they are unable to be vmotions live due to no EVC based cluster of the hosts. they are all stand alone at the time of this writing. so lets try that.

After clciking finish I didn’t see anyting much happening at the vcenter tasks, so I logge dinto the host being upgraded and saw it was suspending the vms in question:

Now it has to copy all the memory from these VMs to disk so this could take a bit of time… then I’d assume I’ll be disconnected from the host once it reboots.

Monitoring my pings for these servers the pings have dropped starting the suspend stat (makes sense) but the host is still responding (makes sense).

I decided at this point to go get some food, I’m lazy and don’t cook, so by the time I had returned I was rather shocked to see the host had succefully been updated, showed compliant and my systems were right back to operational…

Summary

Besides the flash rubbish, this was overall a rather good experience. :O

I think I may upgrade more hosts this way in the future. I didn’t even have to step into my basement at all. That was great!

Until…

I was going to update my second host at home and was hit with this…

well wtf… then it hit me in the face… oh yeah…. I forgot about that, this is a nice real possible word example of third party, unsupported drivers. When I checked my own blog, and lucky the reference to the driver, and where I got it, it appears it still works for 6.x, so I can only guess I’ll have to remove the VIB, run the VUM update procedure, then manually re-install the third party driver… lets try this!

Remove VIB

Following this as a reference, I did the same thing:

esxcli software vib list

esxcli software vib remove --vibname DLink-528T

Ughhhh…

I remember that script/VIB, He was generally really cool guy an dI really loved his blog posts, but his VIB has been rather garbage…. as others have mentioned

Errors

As the picture shows a reboot is required now…. I let VUM do its thing with the standard ESXi 6.5u2 baseline I was using, after the server rebooted I got a problem:

“There was a problem with the Network Device specified on the command line. Error: No NIC found with MAC address.”

Discussion :

The NIC to be used as the management NIC has no drivers installed for it.

Ohhh crap, I forgot when I installed ESXi on this desktop I had to make a custom image, and this is a requirement for systems with custom builds, I removed the drivers for the one NIC but it was not for the ESXi mgmt, but the built in NIC on the mainboard is Realtek and… yeah… anyway, I’ll make a post on creating a custom image, but after a good while of failing to get what I need (as it was my hypervisor with my internet providing VM). I managed to find my initial build and re-install it manually and re-register the VMs and re-create the vSwitches and brought everything back up.

In this case I could have used my Veeam server, however none of my other hypervisors have multiple NICs and thus not an option to use them. My lab is def no redundant lab setup.

Reset ESXi trial license

Quoted directly by Aaron from:

“This guide will give you the steps needed to reset the license file so that you can apply the evaluation license back to your ESXi host.

WARNING: This is for education/informational testing/development purposes only, and should not be used on a production server.

To reset your expired ESX 4.x, ESXi 4.x, ESXi 5.x or ESXi 6.x 60 day evaluation license:

  1. Login to the HOST via SSH or Shell
  2. Remove /etc/vmware/license.cfg
  3. Copy /etc/vmware/.#license.cfg to /etc/vmware/license.cfg
  4. Restart the vpxa service

Or simply copy the code below and paste it into your SSH session.

rm -r /etc/vmware/license.cfg
cp /etc/vmware/.#license.cfg /etc/vmware/license.cfg
/etc/init.d/vpxa restart

Then open the “Licensed Features” option in the configuration tab of the ESXi host through the vSphere Client.

Click on “Edit” in the top right of the “Licensed Features” page

Once the “Assign License” window opens you will see two options. There will be a category for “Evaluation Mode” and Assigned License. Click on the “(No License Key)” option and then click “OK”. This will set the host back to “evaluation” mode and will give you access to all features for 60-days!”

Installing vCenter

Installing vCenter

Since vCenter will not be support on Windows moving forward, all discussion of vCenter will simply be referenced by its new known acronym; VCSA. vCenter based on linux.

I just signed up for VMUG advantage as such I get to play with vCenter at home, yay, else get the required ISO from VMware’s product portal using your own VMware login ID.

Although 6.7 is out, and well polished, 6.7 cannot manage ESXi 5.5 hosts, since I still have a few I’d like to use in my cluster, I’m going to be using VSCA 6.5 for this guide.

Also, I technically only have 5.5 based hosts at this moment (I love the phat (C#) client).

new version PhotonOS?

VCSA CPU and RAM Requirements

VCSA Storage Requirements

Open/Mount the ISO on your OS of choice. For me in Windows, simply mount the ISO and navigate into the vcsa-ui-installer\Win32\installer.exe

Run it!

Stage 1

*Drools* I’m not sure what to do… *Clicks Install*

Introduction; Next
EULA; Accept; Next
VCSA + PSC
Target Host + port + username + Password; next
VM Name + Root Password
Select Datastore (I enabled Thin Disk)
Give a system name (which you’ll want to point to the IP address you define, in the DNS servers used by the VCSA and any client systems needing management access)
IP Address
IP MASK
Gateway + DNS Server


Finish.

Now it states this will take a few minutes as it depends on, the hardware specs of the ESXi host it was deployed to, and maybe internet speed if these RPMs are not on the OVF template that was deployed. Also the VM has to boot.

Quick Break time!

Interesting default… until it finally completes…

Stage 2

NEXT!

NTP servers (0.ca.pool.ntp.org,1.ca.pool.ntp.org,2.ca.pool.ntp.org)

Next

New SSO domain, create a password for administrator@vsphere.local (I’ll create a SSO domain for zewwy.ca later to allow my local AD based accounts to have logon rights later on in this or another tutorial).

DEPLOY!

Mhmm, after 2 attempts I kept getting a pschealth service error. I googled it but the VMware KB was rather useless.

On the third try, I set the system name to IP address, as well as set the vCenter to simply use the hosts time, instead of NTP (even though I used the same NTP server the host was using… so shrug), also waited a little bit longer when starting stage 2, and on the third try it finally succeeded the installation.

Then I added the license key and assigned it to vCenter. which was provided to me when I checked out the “purchase” on VMUG advantage partner site.

Summary

Over all the process is very straight forward. In the next post I’ll cover adding hosts, assigning keys, connecting VCSA to an AD server for an alternative SSO domain. Stay tuned!

8Bitdo Unresponsive with charge light stuck on

If you haven’t heard of 8Bitdo check them out. They have an amazing line of wireless controllers. I personally love my SN30 Pro.

The other day I wanted to play some Cuphead and to my dismay the controller wasn’t responding. When I disconnected the USB cable (as I was charging it and using ti at the same time just recently prior) the charge light remained lit.

baffled I tried looking in the manual, and trying different button combinations and holds, after all this failed I googled which lead me to this reddit form

To my amazement the reply by “Tamoketh”

“The orange light stayed on?

Hold the Start button until the orange light turns off. Then turn on as normal.”

actually worked, I don’t know why this wasn’t written in the user manual, considering it has no direct power button.