ESXi VM disconnected after applying patch

Keep this short.

I had to update a ESXi host locally as it’s mgmt connection would drop with all VMs having to go down on it. As it’s a single host with virtualized network components.

On one of the VMs this was an opportunity to update it since it needed to be shutdown temperately anyway. I took a snapshot of the VM, updated it, validated updates were fine, removed the snapshot. Then proceeded to update the host:

vim-cmd hostsvc/Maintenance_mode_enter
esxcli software vib update -d "/path/to/file.zip"
vim-cmd hostsvc/Maintenance_mode_exit
reboot

Nothing special here. However once the host came back up and I was able to access it via vCenter, one of the VM’s was shown as “disconnected” I’ve seen this with ESXi hosts before, but not particularly with a VM.

Oddly enough there’s only one datastore on the host and all other VMs are fine, and checking the datastore, all files are where they should be.

I figured maybe remove the VM from inventory and just re-add it via the vmx file, however the option was greyed out.

It turned out there was apparently still a snapshot left on the VM (noticed via delta files existing within the VMs folder path).

Removing all the snapshots resolved the issue. Turns out the VM was also running, but didn’t show the green play icon, thus I wasn’t aware of it’s powered on state. Which also explains the greyed out context menu for removing from inventory.

Hope this helps someone.

Validating Windows Creds

I wanna make a really quick post here about this. Normally I generally right click a app on the taskbar, and then shift+right click the app icon, and in the context menu pick “run as a different user”. then I get a credentials box prompt asking me to enter the creds of the user and their password, and if successful open the app (generally cmd).

This time I was testing some old credentials used for a particular service, but I wasn’t sure of the password, I also wasn’t sure exactly where this account was all used, so was hesitant to just go and change the accounts password.

I did my usual trick as stated above and got the user was not allowed local logon for this machine, which was a good thing, some standard best practices for the account were implemented. This however still left me with the assumption the user/password was correct, but not 100% sure.

attempting the same thing with a random known bad password sure enough responded with wrong username/password. Giving me pretty confident results the username and password I entered were correct.

I found this serverfault post about the same question, and I attempted the simple “net use” trick. Sure enough they also do the run as trick I stated in the first paragraph.

net use \\%userdnsdomain% /user:%userdomain%\%username% *

on my main machine I got an error of multiple connections not allowed, I attempted the fix posted by themadmax

net use /delete \\unc\path

which didn’t work probably cause this path I was testing against was a mapped drive for my local logged in user. I followed up by running the commands from an alternative machine I knew had access to the share and DC’s.

Sure enough this worked, I am now confident the username and password are correct.

Hope this helps someone!

Auto Install Defender Updates, but Not All Updates

Issue

Fun times! Updates. Which are not separated in the defined GPOs available to Sys Admins.

Many sources of this issue:
[SOLVED] Server 2016 – Auto install definition updates but nothing else? – Windows Server – Spiceworks

Autoupdate Windows Defender (microsoft.com)

Windows Server 2016 auto install security updates – Microsoft Q&A

Issue: Defender Definition updates come ever day, no separate GPO to differentiate other Windows system updates from these. Other updates require manual install for service availability reasons.

*NOTE* This is how to do this while retaining the update option #3: Auto Download and notify for install. Incase you need to maintain guided (human controlled) updates, but not for the definition updates.

Solution:

Use either:

For 2008 R2 (Source)
A) C:\Program Files\Windows Defender\MpCmdRun.exe -Signature Update

For 2016 + 2019 (Source)
B) PowerShell Cmdlet: Update-MpSignature

Implementation:

Create a script, configure a GPO to deploy it to server as a scheduled task.
*This post to be update with better, step by step tasks. Just a place holder for now with references.

Step 1: Create a script

If you need help with this, you can use my script as a reference, or just use it, similar to this.

Step 2: Determine shared location

Save the script to a share available to domain system (I heard SYSVOL is accessible by all). If this is not acceptable you can follow this guys guide in which he creates a standard SMB share from an alternative server.

Step 3: Fall Down a Rabbit Hole

OK… this is where things got a bit tricky. There’s one slight issue if you want to run a task from a systems’ perspective when the source is on a SMB share that requires domain creds. In the guide I provided about the Op simply created a shortcut link to the network shared script, which will run under a users context.

In this guide, by SysOps, he mentions the use of SYSTEM and the escalated privilege’s it has, but later mentions that he’s sourcing the script local but you could use a network share, however, not mentioning the issue I just did here.

Of course I figured, ok what a good time learn using gMSA accounts to run the task. It should be able to read the script file, it should have the required rights. (expect this is super good to know Thanks Leon! – If you have “Run whether user is logged in or not” your gMSA must be member of the Log on as a batch job or the local Administrators group to be able to run.) Also don’t have to worry about managing a password for the account, it should be a win all around. Let’s do it.

Pick Your Poison

You can either A) copy the script to a local path on the server, and create a scheduled task to run the script, either as system, or any standard user.

or B) create a domain account, or gMSA, and place the script in a SMB location and use a GPO to create the scheduled task on all machines.

I choose B…. but….

This is a bit of a rabbit hole so feel free to avoid this tangent by skipping to part B.

Turns out there’s no governance around the ExecutionPolicy in windows.

Microsoft has changed how definition updates are seen in update history.

Note usually you should grant access to manage the machine password permission to a group, instead of machines directly, and if done so permission to the gMSA can be applied without reboot. (Though I’m sure the same might apply when applied directly to the system as well, but I have tested).

Now my mind started to wonder a bit, Is there a limitation to how many machines can have access to a gMSA? Even this more nitty gritty blog post on gMSAs doesn’t seem to state any limitations. This reddit post asking specifically around gMSA limitation.. nothing.

“unique_username065
3 years ago
You also need to give the gMSA permission to run scripts. There is a technet blog article that explains all the necessary steps to run scheduled tasks and scripts. I am on mobile, so I can’t look for it.

Just be very careful because everyone with access to the machine can potentially exploit gMSAs AFAIK.

Disclaimer: all I wrote is based in theories”

Well that TechNet blog would have been useful, I’ll keep sourcing my findings as we move along here.  So I’ll test it on a single machine, but I have multiple systems in an OU which ties GPOs, so how to push a GPO to just one machine in a OU?

Of course this has it’s own rabbit hole you have to considerSee here for all the details.

In short… ‘If adding “Authenticated Users” with just “Read” permissions is not an option in your environment, then you will need to add the “Domain Computers” group with “Read” Permissions. If you want to limit it beyond the Domain Computers group: Administrators can also create a new domain group and add the computer accounts to the group so you can limit the “Read Access” on a Group Policy Object (GPO).’

In my case the computer account it’s self should suffice, or as mentioned a group with computer accounts. This was the scope and the read permission will both be applied via the same group, and if needing to add more machine only need to add them to the appropriate groups not mess with GPO scopes or permissions. (AKA scalable design)

Then I had one final question pop in my head, “If you can define a GPO to copy a file from a shared network path to the local machine, how does it do that? If scheduled Tasks can only run via ‘SYSTEM’.”

My highly intelligent friend said something, and seems to be backed by this source as well.

“This can only be done during system startup – you’re copying to a system protected folder. During system startup you’ll need to grant the computer itself read access to the source directory share. There are two ways of doing this.

– Create a computer group and grant that group read access. You’ll then need to add every computer to it. You could use the built in Domain Computers group for this as well

or

– Put all the files you want copied into the GPO folder. This folder is read-only for computers as they start.”

Ohhh weird… but you can’t use the computer account to run scheduled tasks?

Apparently not well poop. So that explains all that….

I wanted to test my script as a scheduled task, and noticed a random change from the last time I test.

Old results: Click Check for updates, Definition Update was available, but had to click Install for them to be installed.,

New results (without deploying this script): Click check for update, definition update installs by itself after clicking check for update.

Oh well in that case, lets just up the amount of times it checks for updates.

Apply the GPO setting “Automatic Updates Detection Frequency”

Check the next morning….

As you can see, the detection frequency was applied, but I guess it’s not being adhered to. The last update is well beyond 2 hours.

Time to deploy the script.

K so to pull this off…

Step 3: Create a gMSA

  1. create a group for granting access to manage the MSA password
    New-ADGroup -Name "Update Defender Definitions" -SamAccountName UpdateDefenderDefinitions -GroupCategory Security -GroupScope DomainLocal -DisplayName "Update Defender Definitions" -Description "This group is granted ManagePassword rights on the gMSAtskUDDspt" -Path "CN=Managed Service Accounts,DC=zewwy,DC=ca"
  2. create the gmsa
    New-ADServiceAccount -Name gMSAtskUDDspt -DNSHostName gMSAtskUDDspt.zewwy.ca -PrincipalsAllowedToRetrieveManagedPassword UpdateDefenderDefinitions
  3. grant the group access to the GMSA, by adding computers into the group created in step 1.
    Add-ADGroupMember -Identity SvcAccPSOGroup -Members SQL01,SQL02

    Step 4: Create a GPO that creates a Scheduled Task to run the script

Right click in GPMC where you want your GPO to be linked, and select “create new GPO and link it here”

Remove Authenticated Users from the scope (if you need to test this one one machine, when multiple machines are in one OU, else skip this stuff). Then add the computer account in the scope area. (It appears the computer account is granted read rights on the GPO now.)

Edit the GPO -> Computer -> Preferences -> Scheduled Tasks (at least Windows 7)

It’s super important to know the differences between the actions types.

Action : Update Create

Name: InstallDefinitionUpdates

[Document remaining steps]

Caveats

Shit… I forgot, you have to add the gMSA to all computers that would need this applied too.. and you can’t automate that via GPO, like you can everything else.

Did a update force and saw the scheduled task, finally something, but…

I clearly forgot about Leon’s advise… and double checking that the option to run if the user is logged in or not.

gpupdate /force….. no change to task…. what… ok delete task…..

gpupdate /force… No new task… what?

Go to GPO, switch the option back to run when user is logged on…

gpupdate /force… new task is there… OK what gives?

Try to set the task to run if user is logged on or not manually by editing the task…. I get a cred box pop-up. As for most services using gMSAs, left the password field blank and clicked ok…

I love IT work….

OK… what did I miss this time?

OK, I’ve been digging in the PowerShell properties for scheduled tasks for a while now… How the heck do I set to run logged on or not via powershell?

Main answer, says to use a principal with type password, but it’s a service account? Second main answer says to use system, like no this is a gMSA and we need a domain account for the issues stated above. For shits I tried setting the principal logon type to S4U, as mentioned by one commenter, but it gave me access denied response, then I picked password type and it took it, somehow it is set now… what?! (See picture below)

I went to check the task history… It worked!

Holy Bloody Mary, it actually worked!

OK but it’s seem really stupid when you define the option to run when logged on or not it won’t deploy the task, but if you leave it as user as to logon it does, then you have to use powershell to set the proper logontype. So another powershell script… Ughhhh, there’s also the issue of installing the gMSA on the computer account, I wonder if I could have two additional tasks to run powershell commands to those needfuls.

Ahhh crap, if the GPO action is replace… and I just had to do manual steps I haven’t automated yet….

gpupdate /force… yup back to run when user is logged on crap! Normally the replace action is good if you want to make changes, in this case it’s not wanted, and would be kind of redic to have these multiple scripts to fix themselves go off every time there’s a gpupdate. In this case I changed the Action back to create. K that works, but how do I run these simple powershell commands right after that… automatically.

$principal = New-ScheduledTaskPrincipal -UserId domain\gMSA$ -LogonType password
Set-ScheduledTask -TaskName InstallDefinitionUpdates -Principal $principal

For the first issue, this was the closest I could find. The main answer of using LAPS is poop. The issue around credSSP could be the fact, but not sure if putting creds into a script is a great idea anyway if it is required. I wonder if the system account can run the command… or the “Computer account” maybe via a simplified startup script?

Sine the amount of systems I had to deploy on was small, I skipped this. But if this was wanting to be deployed on end machines, workstations, or laptops. This might be a required step in that case.

As for Issue number two. I ran the above commands manually after installing the gMSA manually. At this point it makes you wonder what was the point of automating the creation of the scheduled task, if I simply have to manually do the other steps. The only answer to that I have is, I didn’t know, I learnt as I go. However it only now required 2 more hurdles to resolve to actually fully automate the process.

Summary

This was another very painful learning experience, all cause definition updates were tied to MS updates, and couldn’t have their own install schedule or install action. I going to create a separate blog post to cover creating a Scheduled Task with a gMSA like this one did. but more specific to that task.

May I suggest you use a standard domain account and just deploy a script pointing to that, and store the creds somewhere if you really need to. This is a painful process.

 

Microsoft Exchange Vulns and Buggy Updates

I’ll keep this post short. If you are unaware, there’s been a big hack on exchange servers.

Microsoft Exchange hack, explained (cnbc.com)

I ran the IOC scripts from MS, was I affected, it appears I may have.

Initiated my own lab DRP/BCP. Informed myself that services would be down, and restored AD and Exchange from backups before the logged incidents. Took the OWA Rev proxy rule  down till the servers could be fully patched.

Booted restored VMs, patched, hopefully good to go.

Then doing patch Tuesday updates users laptops start failing to boot after installing KB5000802. All I could find was news of prints causing BSODs classic.. BSODs! In my case it was causing boot crashing, I did my usual trick, but I got a different error, then ran the Windows Start up repair process, which amazingly got it to boot but said it reverted an updated (the one above). i attempted a install again, but same problem. I didn’t want to re-image as it was an VIPs machine, and time was of the essence. I took a whim, and decided to install all the latest drivers from the laptop OEM vendor (In case some was using MS drivers instead), after that tried the update again, and got a successful install.  Phewwww!

VMware HA down after 6.5 patch

The Story

So the other day I tested the latest VMware patch that was released as blogged about here.

Then I ran the patch on a clients setup which was on 6.5 instead of 6.7. Didn’t think would be much different and in terms of steps to follow it wasn’t.

First thing to note though is validating the vCenter root password to ensure it isn’t expired. (On 6.7u1 a newer)Else the updater will tell you the upgrade can’t continue.

Logged into vCenter (SSH/Console) once in the shell:

passwd

To see the status of the account.

chage -l root

To set the root password to never expire (do so at your own risk, or if allowed by policies)

chage -I -1 -m 0 -M 99999 -E -1 root

Install patch update, and reboot vCenter.

All is good until…

ERROR: HA Down

So after I logged into the vCenter server, an older cluster was fine, but a newer cluster with newer hosts showed a couple errors.

For the cluster itself:

“cannot find vSphere HA master”

For the ESXi hosts

“Cannot install the vCenter Server agent service”

So off to the internet I go! I also ask people on  IRC if they have come across this, and crickets. I found this blog post, and all the troubleshooting steps lead to no real solution unfortunately. It was a bit annoying that “it could be due to many reason such as…” and list them off with vCenter update being one of them, but then goes throw common standard troubleshooting steps. Which is nice, but non of them are analytical to determine which of the root causes caused it, as to actual resolve it instead of “throwing darts at a dart board”.

Anyway I decided to create an SR with VMware, and uploaded the logs. While I kept looking for an answer, and found this VMware KB.

Which funny the resolution states… “This issue is resolved in vCenter Server 6.5.x, available at VMware Downloads.”

That’s ironic, I Just updated to cause this problem, hahaha.

Anyway, my Colleague notices the “work around”…

“To work around this issue in earlier versions, place the affected host(s) in maintenance mode and reboot them to clear the reboot request.”

I didn’t exactly check the logs and wasn’t sure if there actually was a pending reboot, but figured it was worth a shot.

The Reboot

So, vMotion all VMs off the host, no problem, put into maintenance mode, no problem, send host for reboot….

Watching screen, still at ESXi console login…. monitoring sensors indicate host is inaccessible, pings are still up and the Embedded Host Controller (EHC) is unresponsive…. ugghhhh ok…..

Press F2/F12 at console “direct management as been disabled” like uhhh ok…

I found this, a command to hard reboot, but I can’t SSH in, and I can’t access the Embedded Host Controller… so no way to enter it…

reboot -n -f

Then found this with the same problem… the solution… like computer in a stuck state, hard shutdown. So pressed the power button for 10-20 seconds, till the server was fully off. Then powered it back on.

The Unexpected

At this point I was figuring the usual, it comes back up, and shows up in vCenter. Nope, instead the server showed disconnected in vcenter, downed state. I managed to log into the Embedded Host Controller, but found the VMs I had vMotion still on it in a ghosted state. I figured this wouldn’t be a problem after reconnecting to vCenter it should pick up on the clean state of those VM’s being on the other hosts.

Click reconnect host…

Error: failed to login with the vim admin password

Not gonna lie, at  this point I got pretty upset. You know, HULK SMASH! Type deal. However instead of smashing my monitors, which wouldn’t have been helpful, I went back to Google.

I found this VMware KB, along with this thread post and pieced together a resolution from both. The main thing was the KB wanted to reinstall the agents, the thread post seemed most people just need the services restarted.

So I removed the host from vCenter (Remove from inventory), also removed the ghosted VM’s via the EHC, enabled SSH, restarted the VPXA and HOSTD services.

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

Then re-added the host to vCenter and to the cluster, and it worked just fine.

The Next Server

Alright now so now vMotion all the VMs to this now rebooted host. So we can do the same thing on the alternative ESXi host to make sure they are all good.

Go to set the host into maintenance mode, and reboot, this server sure enough hangs at the reboot just like the other host. I figured the process was going to be the same here, however the results actually were not.

This time the host actually did reconnect to vCenter after the reboot but it was not in Maintenance mode…. wait what?

I figured that was weird and would give it another reboot, when I went to put it into Maintenance Mode, it got stuck at 2%… I was like ughhhh wat? weird part was they even stated orphaned ghosted VM’s so I thought maybe it had them at this point.

Googling this, I didn’t find of an answer, and just when I was about to hard reboot the host again (after 20 minutes) it succeeded. I was like wat?

Then sent a reboot which I think took like 5 minutes to apply, all kinds of weird were happening. While it was rebooting I disconnected the host from vCenter (not removed), and waited for the reboot, then accessed this hosts EHC.

It was at this point I got a bit curious about how you determine if a host needs a reboot, since the vCenter didn’t tell, and the EHC didn’t tell… How was I suppose to know considering I didn’t install any additional VIBs after deployment… I found this reddit post with the same question.

Some weird answers the best being:

vim-cmd hostsvc/hostsummary|grep -i reboot

The real thing that made me raise my brow was this convo bit:

Like Wat?!?!?! hahaha Anyway, by this time I got an answer from VMware support, and they simply asked when the error happened, and if I had a snippet of the error, and if I rebooted the vCenter server….

Like really…. ok don’t look at the logs I provided. So ignoring the email for now to actually fix the problem. At this point I looked at the logs my self for the host I was currently working on and noticed one entry which should be shown at the summary page of the host.

“Scratch location not set”… well poop… you can see this KB so after correcting that, and rebooting the server again, it seemed to be working perfectly fine.

So removed from the inventory, ensured no VPXuser existed on the host, restarted the services, and re-added the host.

Moment of Truth

So after ALL that! I got down on my knees, I put my head down on my chair, I locked my hands together, and I prayed to some higher power to let this work.

I proceeded to enable HA on the cluster. The process of configuring HA on both host lingered @ 8% for a while. I took a short walk, in preparation for the failure, to my amazement it worked!

WOOOOOOOOO!!!

Summary

After this I’d almost recommend validating rebooting hosts before doing a vCenter update, but that’s also a bit excessive. So maybe at least try the commands on ESXi servers to ensure there’s no pending reboot on ESXi hosts before initiating a vCenter update.

I hope this blog posts helps anyone experiencing the same type of issue.

 

Creating Custom ESXi Image

Follow these steps

  1. Download Offline Bundle of ESXi Image
  2. Download Drivers E.G The Native ESXi USB NIC drivers
  3. Install PowerCLI (Set-ExecutionPolicy Remotesigned; Import-Module PowershellGet; Install-Module -Name VMware.PowerCLI)
  4. In PowerCLI connect the standard SoftwareDepot by typing:

    Add-EsxSoftwareDepot -DepotUrl <Path to zip>

  5. Get the ImageProfile list:

    Get-EsxImageProfile

  6. Clone standard ImageProfile:

    New-EsxImageProfile -CloneProfile ESXi-6.7.0-8169922-standard -Name MyProfile -Vendor <vendor>

  7.  [Only If Required] If your vib file has Acceptance Level – CommunitySupported, we need to set this Acceptance Level for our ImageProfile:

    Set-EsxImageProfile -ImageProfile MyProfile -AcceptanceLevel CommunitySupported

  8. Add our vib to SoftwareDepot:

    Get-EsxSoftwarePackage -PackageUrl <path to vib>

  9. Add our vib to ImageProfile:

    Add-EsxSoftwarePackage -PackageUrl

Error:

Search result.

Answer driver for specfic version (7.1, need 6.5)

So I downloaded the proper driver but I couldn’t figure out how to pick the right software package since the “get” command was actually already loaded the other driver, so it kept trying to add the 7.1 driver. Only thing I could think of was to close the powershell windows and start fresh…

10. Export ImageProfile to ISO image:

Export-EsxImageProfile -ImageProfile MyProfile -ExportToIso -FilePath

That was it! Sadly the laptop I wanted to use this on was still boot looping, and sadly the USB NIC “Insagnia” didn’t seem to work and was getting NFS4 client failed to load, and not network adapters found on the machine. But was worth a shot.

Using Flash in 2021

The Story

No one should have to use flash…. however, there have been some amazing things that were done with the framework at the height of its time.

Now my issue was more around the fact that VMware, as I use VMware a lot. Happened to choose this framework for their Management Web Interface with 6.0~6.5, with only having fully depreciated it in 6.7. Why did they do this? Cause they didn’t want to rely on a Windows based framework anyway, AKA .NET. Say what you will about Microsoft and Windows, but when you look at the two frame works, it’s pretty clear which is still supported and which is clearly not…

Anyway I digress, if you attempt to use the flash based interface, if you happen to still be on 6.0~6.5 and need access to the flash based interface. Well let’s see…

Wow, that was unreal useless…

Strange same results in new Edge Chromium (well ok that’s not to odd considering they are based on the same engine), I remember just seeing the weird new Flash logo with an information logo.

I thought this might be due cause I never configured the old GPO’s which were used to define allowed sites for flash.

I thought I remember setting it for another thing that used flash and if I navigate to that page…

Clicking Get Flash leads to the same online EOL page… alright. Not exactly the results I was expecting. I swear I remember it popping up that flash logo.. let me try one other machine to validate… Ok all browsers same results, IE simply doesn’t even load the page. I don’t use Firefox. Either way, one of two things happen you get the above snippets, or you get this Flash logo:

Hunt for an Answer

My first google search brought me to a ghacks blog post suggesting to use a web app called ruffle… no thank you.

However lucky for me reading the comments another guy with a way nicer site (no dang ads cluttered everywhere), the guys name is Charles Wilkinson. The rest of this blog post I’ll follow along with Charles’s to see how it plays out.

The Fix

He’s done such a good job with the basic detail I kind don’t even want to paraphrase it, so here’s a direct copy n paste of the How to Fix it from Charles’s blog:

“Reading the Flash Player Administrator’s Guide, in a section called: Administration > Enterprise Enablement we find the official solution.

On any device that we want to enable our legacy app on, we need to edit the mms.cfg file that holds the configuration for Flash Player.

This file can be found under:

  • /Library/Application Support/Macromedia/mms.cfg on OSX
  • C:\Windows\System32\Macromed\Flash\mms.cfg on 32bit Windows OS
  • C:\Windows\SysWOW64\Macromed\Flash\mms.cfg on 64bit Windows OS

This file needs to be replaced with the following content:

# Disable Automatic Updates
AutoUpdateDisable=1
SilentAutoUpdateEnable=0

# Disable prompts to uninstall Flash Player
EOLUninstallDisable = 1

# duplicate actionscript console output
# in browser's console for javascript
TraceOutputEcho=1

# Enable the AllowList feature
EnableAllowList=1

# Normally, the allow list blocks URL requests
# unless the url matches a pattern in the list.
# In preview mode, all requests go unblocked,
# but console output is written for each request
# indicating which pattern it matched or that
# no match was found.
AllowListPreview=0

# Pattern to enable Your Legacy Flash Web App:
AllowListUrlPattern=http://legacy.app.domain.name:8001/

Obviously, you need to replace http://legacy.app.domain.name:8001/ with the URL of your legacy app.

Once this file is saved, hit refresh in your browser and your legacy web app should load. You do not need to restart the browser (at least not when I tested this on OSX with Firefox) – Flash seems to pick these settings up next time you refresh the page.”

Doing the Needful

OK Let’s try it out, my machine is Windows 10 x64, lets navigate to the path mentioned.

I dunno about you, but, I don’t see no mms.cfg

OK, I can’t see much else as to if you need to create this file yourself, or what…

Wait a second….. double reading the Limitations section from Charles post….

Limitations

This fix allows Flash to continue to run, disables the prompts to uninstall and disables automatic updates, however, it does not prevent newer browser versions from removing Flash Support. Users who need to access your legacy app will need to use an older version of Chrome or Firefox with automatic updates disabled. The last versions of browsers supporting Flash are:

Firefox version 84
Microsoft Edge version 87
Chrome version 87

It also seems that Microsoft have released a Windows update that will uninstall Flash: Adobe Flash Removal Update for Windows 10 – KB4577586. Sysadmins will probably want to prevent this update from being installed.

Putting the Pieces Together

Oh.. I’m starting to think the reason I don’t see the flash logo above when I did before is cause I believe the update to remove flash was pushed onto this machine, also the browsers got updated, now on 88.0.705.74

So I literally have to have a system that doesn’t install one particular windows update (if I want to keep it “online”), or use an older machine that is fully offline to get any of these updates, either it be the OS KB updated mentioned or the browser itself updating. Both these requirements are pretty bad.

I should have suspected this, but it sort of slipped my mind, till right now. OK so what are my options…

Option 1

Old copy of a machine, and prevent it from reaching the internet, only access to the devices or URL’s it needs to manage/access. OK so I managed to find a backup/copy/vm of a system that has an older copy of Chrome (version 80) that’s well below the 87… OK but how do I stop it from updating if it does manage to connect to the internet… really just rename the folder update, neat. In my case when I went to go rename told me the file was locked by system… which leaves me to believe there’s a service.. and sure enough there were two, let me just disable these services and then rename the folder…

Weird, even after stopping those services it still won’t let me rename the folder saying its locked by the system…

So after creating a clone of the VM, and disabled the browser updates, and disable windows updates, I navigated to the page, got the “run one time” and it finally tried to load, and I finally got the logo as mentioned on Charles’s blog, that means it’s finally time to try the “hack”.

Open CMD as an admin, and create the file in question:

and fill the table.

 

Not sure if a reboot is required or what lets do one to be safe…

SOB… Chrome updated…. let me try that again…

Well even with Chrome 78, and enabled Flash in settings, and clicked allow on pop-up and I get Download Failed. Sigh… so I grabbed the PPAPI flash installer from the web archive linked in the comments of Charles blog. Installed it and sure enough again, got the logo I posted above, this time a file already existed in the c:\windows\sysWOW64\Macromed\Flash and I edited with the same options mentioned above again…

Same flash logo not sure if I need to reboot to apply or try like the other comments in that blog post stated, and put it in a special appdata location… I’ll try that first and then reboot as a last attempt.

Yes! The Flash based web interface finally loaded!

I have no idea what Option 2 even is at this point…

Soo Summary..

Summary

  1. You need to ensure a Chrome/Chromium based browser Pre-87
  2. If you have MS KB4577586, you need to install the PPAPI flash manually.
  3. Enable Flash within the Browser Settings
  4. Manually edit/create mms.cfg as shown above, and have in both C:\Windows\SysWOW64\Macromed/Flash as well as C:\Users\%Username%\AppData\Local\Google\Chrome\User Data\Default\Pepper Data\Shockwave Flash\System\

I hope someone finds this guide useful… cause I sure found this process painful. 😀

 

VMware vCenter Updates using VAMI

This is a quick post on the latest security release notification from VMware.

VMSA-2021-0002 (vmware.com)

If for whatever reason an update is not possible you can follow these workarounds.

While you can use VUM to distribute updates and patches to ESXi hosts.

You’ll have to use VAMI for updating vCenter.

You can download the latest patches here (vmware account required).

I did this on my lab vCenter,  took a lil while but not bad.

  1. Made a backup of the VCSA using Veeam
  2. Shutdown Veeam or any other backup solution that might use vCenter
  3. Notified anyone that might use vCenter that it would be inaccessible during update
  4. Attached ISO to VCSA VM (You can do as 4sysops did and upload to a datastore, or you can simply open the VCSA console via VMRC, and attach the ISO from your Downloads folder)
  5. Log into VAMI (https://vcsa:5480)
  6. Click Update on left nav, then Update -> Check CD-ROM
  7. The update should be available as the option, then click Stage and Install
  8. Accept the EULA, use/don’t use CEIP, Check I have a backup, Click Install.

It could take an hour or so, then everything is back to running state, here’s the summary page after completion:

You can read the alternative methods such as using CLI, or how to handle a vCenter HA cluster upgrade using the link above to 4sysops guide on upgrading vCenter.

Sorry this post is not as extensive as usual, just a heads up about the latest VMware patches. Stay Safe out there.

 

Get Windows Server out of Stuck Update State

I probably should be a bit more clear, this post will cover how I managed to get a Windows Server 2016 to “check for updates” when it had gone wrong and was stuck looping (checking) and failing where it replaces the “check for updates” button with nothing other than “retry”.

This happened after clicking “Search Microsoft Online for Updates” in which case it found a couple that were not approved by WSUS or not selected as category’s that WSUS actually downloads.

Funny in this case after I did what will be mentioned below, clicking retry did just start checking again, and then stated “Your device is up to date”.

So ok it worked that time, but what I discovered at the time, was that there’s a new command to use on the backend (command line) to do the needful when the UI doesn’t have the appropriate button available. Like usual Microsoft fashion, notifying stakeholders was poor, and so was an documentation.

Now this isn’t the first time I discussed issues around Windows update, in particular around the tool MS has given Syadmins to do the needful; WSUS. Such as this time, when clients are not showing up within WSUS after clearly showing they had applied the GPOs (registries) required and no network issues between them, or this time CU updates weren’t being downloaded by WSUS although clearly the types and categories were fully correct.

In this case however instead the issue was simply what commands to use, as stated within the original person asking the question in the TechNet link above “Since wuauclt has been depreciated in windows 10, I was googling what has replaced it.

I found that usoclient is what has replaced this command for windows update in the command line. ”

What authoritative source is there for this claim, well I found this

“The wuauclt.exe /detectnow command has been removed and is no longer supported. To trigger a scan for updates, do either of the following:

  • Run these PowerShell commands:
    $AutoUpdates = New-Object -ComObject "Microsoft.Update.AutoUpdate"
    $AutoUpdates.DetectNow()
    
  • Alternately, use this VBScript:
    Set automaticUpdates = CreateObject("Microsoft.Update.AutoUpdate")
    automaticUpdates.DetectNow()

Funny thing about this is I found that wuauclt /reportnow still works in Server 2016, as noted in my other blog posts. I generally didn’t use /detectnow. However what I found was that the new commands did work for me.

Such as these as mentioned from Spiceworks:

“Start checking for updates: UsoClient StartScan

Start downloading Updates: UsoClient StartDownload

Start installing the downloaded updates: UsoClient StartInstall

Restart your device after installing the updates: UsoClient RestartDevice

Check, Download and Install Updates: UsoClient ScanInstallWait”

Then of course these as mentioned in the TechNet post:

“RefreshSettings – used to quickly enact any settings changes
RestartDevice – as the name implies, it restarts the device. Can be used in a script to allow updates to finish installing on next boot.
ResumeUpdate – used to tell the tool to resume updating after a reboot.
StartDownload – initiates a full download (from Microsoft) of existing updates
StartInstall – kicks-off the installation of the downloaded updates
ScanInstallWait – Combined Scan Download Install
StartInteractiveScan – we’ve yet to get this one to work, but it suggests that the process may work in a GUI
StartScan – kicks-off a regular scan”

While it is nice to see something available, it would be nice if MS made a more formal announcement of the deprecation and the replacements.

Hope this helps someone.

SharePoint – Can’t Delete ContentType

The Story

You know every post has to start with a story. So it’s story time, It all started with a site that needed to be templated and used to create new sites. Now when the user went to actually deploy the new template via “create new site” link under the site contents area, it errored out stating that there was an error.

Create SharePoint Site Template

I wouldn’t have blog posts if everything work via the happy path, there’s other people to blog about that…

This of course required jumping through some hoops to even make the site savable as a template, in my case it was just the property to be set using PowerShell:

$web = Get-SPWeb http://your_ site
$web.AllProperties["SaveSiteAsTemplateEnabled"] = "true"
$web.Update()
$web.Dispose()

Else you’ll get the following error:

So once this is done, you can finally create a template.

However, now we have to actually deploy it.

*NOTE* when you create a template of a site, you are secretly creating and activating a “Solution” to the main site. So if you need to manage, or delete a template you first have to deactivate the solution then you can delete it.

Deploy New SharePoint Site Template!

Would you expect anything else form my blog post? 😛 OK this should be easy enough let’s just delete this old content type as it was a legacy one left behind from a migration.

So first since this template is trash, you’d figure these type of checks would take place at creation of the template.

Ahhh SharePoint never cease to piss me off… OK, let’s google this…

The first source is dead on the solution… However it required making a direct Database change. To keep SharePoint “supported” state, although obviously broken. The alternative solution is to either find the original feature package, and re-install it either via command line (stsadm.exe), or Powershell or the front end. Of course if this is a third party feature, and you only have installation for older SharePoint, then this would have to be cleaned up on the old environment before migration. If I find the link (didn’t save at the moment) 🙁 there apparently is a way to map this ContentType to “Dummy” features, delete the content type, then delete the dummy feature. This is the only alternative way while doing it via the front end to stay “supported”.

In the mean time, you can also spin up the site in a test environment, and do the needful on the content type in the database backend (connected to the instance, and Database for the site content (WSS_Content by default):

Update DBO.ContentTypes
Set IsFromFeature = 0
Where ContentTypeID = 0xIDNum

The content type ID can be extract from the address bar via the front end as it is known by the web parameter ctypeID:

Now you’d figure there be no problem delete the content type, until another error shows up with a different reason. (OK I remember it being different but until I run through these test again maybe they were the same, as the second source explains…)

[Insert Picture of error after DB change]

Googling I came across this guys very nice blog post about the same issue!

Really short version… the content type is still used/referenced by another SharePoint object within the environment. He does show and reference some really nice C# code to help track the issuing objects. However I have no interest in building an App, just to find these… there has to be another way!

Ohh stackexchange how beautiful you are

$site = Get-SPSite("your-site-url");
foreach ($web in $site.AllWebs) {
   $ctype = $web.ContentTypes["Your Content Type"]
   $usages = [Microsoft.Sharepoint.SPContentTypeUsage]::GetUsages($ctype)
   foreach ($usage in $usages) {
      Write-Host $usage.Url
   }
}

Which helped me track the objects, in my case Lists…

Turned out to be the list in all subsites called “Tasks” now this is a SharePoint created list object, however they were created after this particular feature was enabled on the site, thus all subsites inherited the issue.

Now there are some nice online references to delete content types, or lists and other objects via PowerShell.

However if you know the object model well enough you can pull one liners to do wonders…

$spsite = Get-SPSite http://yoursite
$webs = $spsite.AllWebs
($webs.Lists | ?{$_.Title -eq "Tasks"}).Delete()
$webs.Dispose()
$spsite.Dispose()

And just like that hundreds of old SharePoint lists that were no longer used were gone. If the lists you have contain data that is to be kept, you are going to have to migrate the data to a new list, then delete the offending list and migrate the data back.

OK, NOW you can create a template from the site, and deploy it and it should succeed without issue. You can now navigate to the site content area where the solution packages are stored and copy it out, and then upload it to your production environment and create new clean sites. However note that this won’t fix the issue in your production side.

So you’ll have trade offs to consider in which way you decide to handle the issue.

Summary

SharePoint is a beast of a designed machine, and can often include some bugs that were not expected. I hope to extent this blogs and provide more SharePoint related content in the future. Cheers, I hoped this helped someone out there.