Renewing expired certificates on vCenter 5.5

Do you follow best practice? Have you setup a VMware HA cluster with vCenter. Do you have your own PKI and certificates? Did you not have active monitoring on said certs? Then chance are you are in the exact same boat as me! This blog post assumes you are well advise in using the SSL Cert Automation Tool as well as creating certificates for use with the tool.

This one begins on a Monday after the weekend. I was getting alerts of failed backup jobs. I managed to configure Veeam at my work place and have been happy with the product and support from day 1. I also configured a cold site for backup retention in the event our primary site, you know…. implodes. Anyway, I was used to getting “failed” alerts when really there was simply a communication hiccup across my IPsec tunnel, which usually the job would complete successfully and just report the error. This time however it was different, the errors were for normal backup jobs and reported “incorrect username and password.” I knew the service account’s password, used by Veeam, never expired or changes. Instantly telling me something else is wrong. I then attempt to login into vSphere connecting to my vCenter server, and sure enough it says the same thing wrong username and password, to which another notice pops up saying all communications are untrusted due to expired certs. Doh!

At this point you’ll probably have done exactly what I did… check your installation documentation right?!?! I mean if you are running custom certs, I’m assuming you follow other best practices such as documenting. :P. But after that you are probably googling once you discover part of the SSL tool are not working!

Chances are you came across VMwares KB on renewing certs on a 5.5 version instance of vCenter only to discover that at step 5 a) that the tool reports the local machine doesn’t have the SSO service installed. This really comes down to what the “tool” really is, and that’s a batch script. Yeah you read that right a BATCH script, so you could imagine how ugly and how painful that must have been to code. Like seriously 5.5 was released in Sept 2013 and they were coding using PowerShell by then… shame on you VMware. Anyway, the most likely problem here is in the way this batch script actually checks for the installed service (I looked at the source code of the “tool” but I didn’t actually locate the part that handles this and I’m strictly making assumptions here) is that it probably has a more direct string to which it looks for, again assuming here a reg key or something of that nature and its probably using a version number to check against, if the version changes the script would reply a “can’t find this”. and thus you get the above error which you know is wrong. So how do you fix this, well you grab the exact version of the tool for the updated instance of vCenter you are on (this requires a valid VMware subscription to grab the version of the tool you need). I managed to update one form post in hopes it helps others at this stage of the game.

At this point I kept following through the tutorial, just an FYI I was going through all this with a VMware tech support, and they had to get another tech who specialized in these cases. I came across other issues as well such as in Step 5 d) I got a error similar to this. Sadly I’m writing this up several days after the event so I can’t remember what exactly we did to recover from this one.
At this point gotta keep pushing through the KB which has a total of 24 steps, so you could imagine how painful all this is to do. At the same time I’m not sure HA is even available, and all my backups couldn’t run and any management of VMs would have to be done manually till vCenter could be back up and running. I’ve talked to others and many people suggest to stick with self signed certs even though we all know its not best practice. Thanks VMware for making best practice really hard to implement and maintain.
Also at the very end steps I didn’t not actually have a listed service ID for web client but only the web logger, although you can have separate service ID instance for these, in my case I had to use the web logger service ID to complete the final step. Then after the Web Client wasn’t working properly which I fixed by reinstalling the service/feature via add/remove programs. The fact there is no repair option on this installer bugs me.

To Paraphrase to solution:

1) Ensure you are using the latest and correct version of the SSL tool *cough BATCH script*.
2) Create all your new certificates and chains.
3) Follow the KB article very carefully, specially when it says to do some steps manually vs using the "tool".
4) Google any errors along the way.
5) Bash your head in for following best practices.

Jan 2018 Updates

This brings back bad memories, It’ll soon be time to update to 6.5. We’ll see how VMware has handled internal PKI this time.

Reclaim unused space from VMDK

Let’s say you have a bunch of servers *Cough Server 2008 R2* that have been fairly well maintained and all running on VMware’s ESXi hypervisor system. As a regular server admin you’ve come to terms with updates and keeping systems for the most part on the latest n greatest. Now lets also say you happen to be the storage admin as well and you find you are running out of space on your SAN. What do you do? Usually buy more space. But lets get to the real heart of the matter… Systems, if not properly set up, get messy (don’t get me started on Windows registry.) we’re sticking with storage as the topic of the day. Well good news is I’m here to help you reorganize and re-claim all that space. Lets get started!

*NOTE* if you are running thick provisioned discs you’ll have to svmotion them to another datastore to convert them to thin first.

First and foremost you’re going to want to clean up your WINSXS folder. Don’t believe me, run windirstat to find out just how big it has become from all those updates.

How do you clean up your WINSXS? You may ask well, first ensure your server has Windows6.1-KB2852386-v2-x64 installed. Note these steps work for Windows 7 as well if anyone happens to need to save space on a client machine. You might be able to find cleanmgr.exe online, but your safer to copy it from another server. or try this. run cleanmgr.exe make sure you run it as an administrator and clean system files. Clean up the old update files. Reboot (Your HAVE to reboot to complete the update removals before moving to the next step!)

For the next part you may or may not want to do depending on what the app reports. Run Disk Defrag. In this case my servers were about %40 fragged; meaning that over time as files were added, used,and then deleted they were placed randomly throughout the disk depending on where the FAT (File Allocation Table) generally in this case NTFS telling which sections were free to overwrite. Yup when you delete a file it’s not actually deleted from the sections just from the table. So Defragging pretty much “shoves” all the actually still in use data nice and organized at the “front” of the disc. This is generally only required on spindle discs, if your system is using SSD, or a logical unit based on RAID this won’t matter.

Now if you’re simply clearing space on a phsyical device, barebone device. You’re pretty much good to go. However for the rest of us virtulaized guys who want to reclaim space on our SAN’s we still have a ways to go.

This is where I find the “fun” begins. if you attempt to look it up you’ll find some old articles from VMware about using vmware tools. Well #1) The GUI options are gone,if you attempt to find vmware tools under control panel, you won’t find it. #2) If you go ahead and try to use the cmdlets you’ll probably find it simply returns the disc can’t be shrunk. I personally say don’t waste your time attempting to do anything here with VMware tools. For Linux users you can accomplish this via dd very easily. For the rest of us Windows users we can thank the Great Mark Russinovic for sysinternals, in particular this time for sdelete. Grab it and run sdelete -z (important in v1.56 it was -c, in 1.61 use -z) If you don’t specify a drive it will use the drive you run the cmd from, I’m assuming.

Time for the last and final fun part. Read this and this. Once you’ve done that I’ll provide my findings:

1) You have to svmotion between datastore of different blocksizes (I found the 2 MB block size was the one that worked for me)

2) you can’t use the vmkstools holepunch option against a VMDK stored on a NFS datastore

To Paraphrase to solution:

1) Remove and delete temp files, unused profiles, and old update files.
2) Defrag to organize all the blockson the guest file system.
3) Use sdelete or dd to zero dirty blocks.
4) Hole punch or svmotion the VMDK to shrink used size.
5) Enjoy a beer and a bunch of recovered space.
6) You might even notice a performance increase from all the organized guest file systems

Jan 2018 Updates

2016 didn’t have many posts, but they sure are good ones, I forgot all about this stuff. haha.

Zoneminder on Acer Netbook

The story begins when I first got my IP camera; a WansCam PT with IR. I got a couple of them for around $50 bucks a pop, which was an amazing price. Still hard to match to this day. I had set it up on the overhang of my parents garage. This caused some issues with their garage door opener due to interference of the WiFi. This cheap camera amazingly having survived two winters without issue, it was however protected by the overhang.
The thing was it only had send a picture to a FTP server at most every second on motion sensor. It could also send email alerts, and a couple other basic features, and could supposedly monitor it remotely if you port forwarded the web hosting port… which I never did for security reasons, for one it only having basic forms authentication, and I setup a more secure SSH tunnel with key authentication to accomplish the same task, but way more secure.
So whenever I had to review an event, or even just check the images I had to scroll through hundreds of pictures, and it wasn’t very good at it either, WiFi, interference, etc.
Then I discovered Zoneminder. It blows my mind. I’m a huge open source advocate. This one… takes the cake; for me!

Installing Zoneminder on Acer Netbook x86.

1) grab Debain from here.

2) grabbed Rufus to place installer on to USb as notebooks don’t have dvd drives.

3) put Debain 8.2 x86 USB installer into notebook USB port. Powered on notebook pressing f12 for boot options.

4) Installed Debain, no Desktop Enviro. This is a server afterall.

5) remove cdrom from /etc/apt/sources.list using vi

6) Follow this: Install Zoneminder on Debian.

7) uncommment HandleLidSwitch=suspend and change suspend to ignore in /etc/systemd/logind.conf

8) Enjoy ZoneMinder on a Acer KA90 netbook, by navigating to http://netbookip/zm

9) I) adding a Wanscam IP camera; In Zoneminder Web GUI -> options -> ZM_OPT_CONTROL enabled. restart zone minder service.
II) ZoneMinder Web GUI -> Add New Monitor. *NOTE* The Zoneminder account here is a operator account I created on the Wanscam WebInterface Prior to setting up ZoneMinder.

Well Poops… I had 3 images here that got lost…

III) Nightvision IR control is controlled with the Wake/Sleep buttons under zoneminder control.
IV) Set your presets, and have fun moniorting and controlling your PT IP camera from zoneminder!
The big benifit here now is Zoneminder will track motion and record events, so no more needing FTP enabled on the IP camera. Plus all events images are kept together and more than 1 a second, I’ve been able to get about 4-5 fps all being saved a seperate jpeg images. My next goal witll be to get a IP camera of otherwise setup to see if I can get recorded video on motion.
I also want to eventually figure out scripted, timed presets for a patrolling type camera on the cheap. Also need to setup POE to reduce cables. I plan on getting that setup using a old Cisco 1811 POE router. I hope this post is helpful for someone out there.

Jan 2018 Update

This server has actually been P2Ved (Physical to Virtual migration). I’ll probably blog about how I accomplished this. The only other server on my hypervisor at this moment. Haha

Managing Software locally and remotely using CMD and PowerShell

Using CMD one uses wmic command…

1) Example to query listed applications on remote system running Windows
	wmic /NODE:RemoteHostName product get name, version
2) Example to uninstall application remotely using wmic
	wmic /NODE:RemoteHostName product where name="ApplicationName" call uninstall /nointeractive

*NOTE* these require WMI management to be allowed through the windows firewall.

That’s neat, this can be better achieved using powershell…

1) Example to query listed applications on remote system running Windows via PowerShell v2
	gwmi Win32_Product -co nb00647 | ft name, version
2) Example to uninstall application remotely using wmic
	(gwmi win32_Product -co Server1 | where {$_.Name -like '*ApplicationName*'}).Uninstall()

That’s amazing!! What’s the issue?

Well, first off, it’s not clear if this query runs agaist both known application registries (on any 64 bit based Windows system), those being… HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall (For 32 Bit Apps) HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall (For 64 Bit Apps)

At this point I wasn’t sure if this was querying both or just one of these locations..
I decided to test this with an old FireFox installation, that I had replaced a while ago (I initially used FrontMotion firefox to allow for configuring via GPO’s) This was made possible by later version of firefox via the mozilla.cfg file and this file could be pushed and enforced by GPO, anyway.

PS C:\Windows\system32> gwmi Win32_Product | where {$_.Name -like '*Fire*'}

IdentifyingNumber : {3F98D293-8219-4730-B49B-F223030021B8}
Name              : Mozilla Firefox (en-US)
Vendor            : FrontMotion
Version           :
Caption           : Mozilla Firefox (en-US)

Once I had ensured the correct object being returned, I called its uninstall function.

PS C:\Windows\system32> (gwmi Win32_Product | where {$_.Name -like '*Fire*'}).uninstall()

__GENUS          : 2
__CLASS          : __PARAMETERS
__RELPATH        :
__DERIVATION     : {}
__SERVER         :
__NAMESPACE      :
__PATH           :
ReturnValue      : 0
PSComputerName   :

Key thing here is the Return value, claims 0, so that be considered a success, lets check the returned value..
Sure enough, no returned objects, lets scan the registry for stale keys for that particular GUID/IdenitfierNumber

reg query HKLM /f "3F98D293-8219-4730-B49B-F223030021B8" /s
(This can take a long time, if local to the machine, searching via find in regedit can be quicker)
reg query HKCR /f "3F98D293-8219-4730-B49B-F223030021B8" /s

Both queries return no values, thus were cleanly removed from the registry..
However, I still have a firefox version 39 listed in my Programs and Features.
So, what gives? As I had mentioned before on what the wmic and qwmi commands query the Win32_Product class, from what I’ve seen so far it appears this is querying on a specific set of the registry and not all the applicable registry sections:
HKCR\Installer\Products HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall
Doing a quick reg query for the word firefox sure enough displayed the listed installation of Firefox 39, and not the old 29 listed above…

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\Mozilla Firefox 39.0 (x86 en-US)
    Comments    REG_SZ    Mozilla Firefox 39.0 (x86 en-US)
    DisplayIcon    REG_SZ    C:\Program Files (x86)\Mozilla Firefox\firefox.exe,0
    DisplayName    REG_SZ    Mozilla Firefox 39.0 (x86 en-US)
    InstallLocation    REG_SZ    C:\Program Files (x86)\Mozilla Firefox
    UninstallString    REG_SZ    "C:\Program Files (x86)\Mozilla Firefox\uninstall\helper.exe"
    URLUpdateInfo    REG_SZ

According to this stackflow post, there is no way to use wmi/qwmi to query 32 bit applications… I find this hard to believe and will update this blog should new news pop up.

Now here’s the kicker, Firefox was removed from my Program Files, but a Mozilla folder still exists in my Program files (x86), again seemingly like a lack of wmic application control for 32 bit applications. However I have no firefox in my search, and no firefox.exe avilable in the existing folder in PF(x86)… lets try to uninstall whats listed under programs and features… Would you look at that… says something happened during uninstall, and asked to remove the listing from programs list. Doing another “reg query HKLM /f “firefox” /s” shows it been removed from the keys mentioned above. However lots of plugin keys remain… oh well Deleted Profile Data, Program File Data and called it a night.

In order to build a more-or-less reliable list of applications that appear in the "Programs and Feautres" in the Control Panel, you have to consider that not all applications were installed using MSI. WMI only provides the ones installed with MSI.

Here is a short summary of what I've found out:

MSI applications always have a Product Code (GUID) subkey under HKLM\...\Uninstall and/or under HKLM\...\Installer\UserData\S-1-5-18\Products. In addition, they may have a key that looks like HKLM\...\Uninstall\NotAGuid.

Non-MSI applications do not have a product code, and therefore have keys like HKLM\...\Uninstall\NotAGuid or HKCU\...\Uninstall\NotAGuid.

Infro provided by Ilya Kogan

Jan 2018 Update

This brings back bad memories haha. I should find some time to play with this again on Windows 10, see if anythings changed since.

Schannel Fatal Alert (70) on Exchange Server 2010

So I use Zenoss for centralized system monitoring, including everything from network devices, ESXi hosts, all the way to end server such as Windows VM’s using WMI.

As I receive a flood of events from SharePoint and it’s child service a terrible workflow server add on called K2 Blackpearl, I ignored my Zenoss server quite a bit. I did clean up my other servers pretty well. So when I noticed this alert on my Exchange server, I wasn’t too happy. I like clean event logs in most of the servers I manage. (I’ve made an expectation to SharePoint and K2 since there a whole mixed bag of service accounts and permissions, and web parts… so many moving parts, I simply don’t care about their events.. Given there are no issues)

So I set out to figure out what was causing this event… usual googling came up with the usual TechNet articles of those claiming it probably just re-associating to another acceptable protocol and to accept it. And as per usual whenever people can’t figure out why a event is triggering but doesn’t seem to affect production: “you can just ignore it, or disable SChannel events” This is not good enough for me, as it clearly indicates an issue going on in the back end.

Digging further I came across this tid bit of info. Using this info I knew it was a protocol version issue, with SSL and since it’s on my exchange server I had an itchy suspicion it was ActiveSync related.

Installing Wireshark onto the server and running it with the SSL filter in place, I sure enough was able to pin point the device triggering the events. My boss’s Note 4 running Android 5.01 using the native mail app. At first I simple went into his exchange settings (just to note that it would work externally but not internally) and unchecked SSL (caused Auth to fail as expected), then re-enabled SSL. At first this seemed to make his ActiveSync work and I figured the events would go away, they did not, checking Wireshark it was still from his phone.

To Paraphrase to solution:

1) Remove the corporate email account from the device. (Completely)
2) Re-add the account to the device.

So that’s it! Since doing that I haven’t received any other SChannel fatal error (70). I hope this helps other that come across the same events in their Exchange environment. Just note this was on Exchange 2010 SP3 RU 10.

Jan 2018 Update

Got to love event logging. See so much, but sometimes, so much can drown you. Just have to take care of the ones you can when you can.

Remotely working with user profiles

Checking up on my daily emails, I see my usual AV report. I open it up to see a who’s the bad surfer, turns out only one system. And it turned out to be the new temp we hired. I didn’t expect him to actually go to any bad sites, didn’t seem to be the type, so I quickly viewed the infected file.

Turns out the file specified was from an old user profile, of someone who used the system before him… making me wonder how many old profiles are on his system. I’m a systems admin who prefers to get some things done without affecting other peoples work. I manage to do these with different remote applications. Most of these applications share the users screen and allow taking over of the system. This still interrupts the user, so often I resort to sysinternals psexec.

So I wanted to find out how I could enumerate a list of local profile accounts on a system via command. Doing a bit of research showed this to not be as easy as I had expected (was hoping for a simple wmic command). As it turns out sometimes I forget I’m stuck in the past and should really get with the times, old way

Which is from 2005, thankfully Since Vista they introduced a new class to handle these. 😀 Check out this post for more info.

Easy Peasy way using Win32_UserProfile class and PowerShell 😀

To Paraphrase to solution:

1) Open PS in elevated mode.
2) gwmi win32_userprofile -co $REMOTESYSTEM | ft sid, localpath -a
3) Where gwmi = Get-WinObject | Win32_UserProfile is the class | -co is the remote system attribute | |is the pipe | and ft is format table
4) Note as with all object based programming/applications This can be very easily used to manage user profiles as well, ex:
5) (gwmi win32_userprofile -co Server1 | where {$_.LocalPath -like '*\cjohn*'}).Delete()

So that’s it! Ever since Vista managing user profiles has become a breeze and no longer requires intensive scripting to be managed remotely! Thanks MS you finally did something right!

DC Demote failes due to ForestDNSZones

Scenario: You are about to remove the final Physical Domain Controller from your infrastructure, As you’ve done this before you figure it’ll be a piece of cake and will go about you day.. Instead you’re presented with this!

After some googlings, you’ll probably comes across this. Which will tell you that you need to edit the objects attribute to an active DC role owner.

If you decided to look in the dcpromo log file, you probably noticed that the fsmroleOwner is pointing to an old Server, which was probably the PDC at one point within the domain.

Even though it took a decent amount of time to troubleshoot, I’ll keep this post short. Just check out MS tech guy: Chris Davis blog about the issue.

Grab FixFSMO.vbs from MS support articles, or Davis’s blog and run it against the object in DN notation on a PDC or the DC you wish to have as the FSMO owner.
ex. cscript fixfsmo.vbs DC=DomainDnsZones,DC=Contoso,DC=com

Jan 2018 Update

Funny I don’t recall this one all that well, but great blog post by Chris which covers the nitty gritty pretty well, considering it’s a direct MS technet blog unlikely to go down. Good job.

Switching between Skype UI and Lync UI

To change All Users to Skype for Business UI:
Set-CsClientPolicy -Identity Global -EnableSkypeUI $true

To change All Users to Lync 2013 UI:
Set-CsClientPolicy -Identity Global -EnableSkypeUI $false

What if you only want to change the UI for a certain group of users?
It’ll only take 2 extra cmdlets, in the same sphere.

First you create a new client policy by which to identify this group of users. Let’s call them “SkypeTesters”.
The cmdlet will look like this:
New-CsClientPolicy -Identity SkypeTesters -EnableSkypeUI $true

Then you collect users & assign them to this new SkypeTesters policy. You can collect users via department, AD group, etc. I’ll use a Marketing Department for this example.

To collect users:
Get-CsUser -LDAPFilter “Department=Marketing”
To grant them the new client policy & enable Skype for Business UI:
Grant-CsClientPolicy -PolicyName SkypeTesters

(Of course you can pipe these two cmdlets together & save time. I split them up just for clarity’s sake.)

Information here was provided by The Lync Insider

If you wish to enable all users for Lync after you are done with your initial test group, remove users from the test group with the following piped cmdlet:
Get-CsUser -Filter {ClientPolicy -eq “SkypeTesters”} | Grant-CsClientPolicy -PolicyName “”
The key is specifying a blank PolicyName, this took me rather long time to figure out hahah.
Once that is comepleted you can run the inital cmdlet above to enable the skype UI on the global policy.
This is way better demo then my initial blog post, thanks Lync Insider for having a better write up then MS answers! Cheers!

FYI, to check the global policy and what it’s attribute is set to run the following cmdlet:
Get-CSClientPolicy -Identity Global | select Identity, EnableSkypeUI | fl

PowerShell: SkypeUIEnabled

Jan 2018 Update

Good ol’ Lync/Skype; Seems MS can never get their marketing choices right and all the Devs suffer for it.

Repair a corrupted Windows boot

This blog is going to be an interesting one. The points as to how I got to a windows machine with a bad boot is as interesting as how I managed to resolve the issue.

It started on a Friday, well it technically started before that, but long story short my colleague and I were planning to do a P2V of a physical SQL machine. I prepared the new ESXi host slowly during the week to prepare for this.

As I was destined to go camping that weekend and still needed a bit of work to be done to get ready for the progress, I’ll admit I was a bit rushed in telling my colleague the appropriate steps to take. Now at this point the host was ready, albeit was a bit on the small side when it came to the local datastore, (3 x 6Gb/s sata discs in a RAID 5 (I really wanted RAID 1 with a hot spare, but this was the best suitable option this HBA/RAID controller provided me)). Anyway while I was away camping, go figure in a location with no cell service, my colleague completed the P2V. Turns out it didn’t go 100% perfectly as planned, as he emailed me a bunch of error noted by the host ESXi. Turns out he had provisioned Thick Provision lazy zeroed discs normally I wouldn’t deny its a good choice under certain circumstances, however, in this case with the limited datastore space it wasn’t the greatest choice cause there wasn’t really room to spare on empty “zeroed” data.

So after I was informed of the situation I began to attempt to fix the issue, which happened to be that the host was unable to remove/consolidate snapshot due to the lmited space left on the datatstore. I began by adding the host into our SAN network and connecting it to our SAN storage. I did a svMotion after I had initially shutdown the guest OS. Shutting down the guest OS took over 20 minutes with the slowness it had been brought to by the issue, while it was still specifying shutting down I had got impatient and forced the system down.

After the initial svMotion, I checked the datastore and noticed that there was still a VM folder in there and all the space had not yet cleared, which I found a bit strange considering it should have migrated all the data to the new store.

I figured well lets see how the VM reacts now… and whomp whomp waaaaaaa was presented with this! (Imagine Source was removed, I can’t remember what it was lol)

At this point I wasn’t sure if this was realated to the snapshots and the VM folder set being not all in one place.. So I decided to delete all the snapshots. Even after completion and noting that all Data had indeed migrated to the SAN, I was still presented with the error shown above.
At this point I was starting to worry that I might have ruined 20 hours of P2V work, I was too tired to carry on for the day and decided to boot the physical back up to handle DB requests for the following work day.

The next day I jumped on fixing the issue at hand to recover this VM and save the 20 hours it took to P2V this thing. I initially started by mounting the Windows Server 2008 R2 installation DVD to the guest VM and adjusting the boot time to allow me to load the boot order and boot from the disc. Even though selecting repair my computer did see all the local discs including the installed OS, it would only give me the option of recovering from a system image (which I didn’t have), run diag tools (doesn’t help in this case) and command prompt. So I loaded command prompt. Now everything I tried in bootrec.exe options had failed:

/FixMBR (didn’t work)
/FixBoot (didn’t work)
/ScanOS (Found 0 installed instances)
/RebuildBCD (Found 0 installed instances)

At this point I felt it was pretty shot and unrecoverable, but like usual I felt to give one last google search on the issue of 0 found instances. Which lead me to this MS answers post, with the same question. To paraphrase the solution from Vijay B

To Paraphrase to solution:

1) bcdedit /export c:\bcdbackup (Backup the existing bcd)
2) attrib c:\boot\bcd -h -r -s  (Allow write/modify of the BCD file)
3) ren c:\boot\bcd bcd.old      (rename the BCD file, can also just be deleted, this is a backup solution)
4) bootrec /rebuildbcd

At this point it will catch the windows install and actually rebuild the BCD (/Rebuild BCD), believe it or not after that I was able to successfully boot the VM, and saved a 20 hour P2V. I can now freely vMotion and move this VM as required in my hypervised system!
Thanks ViJay!

Jan 2018 Update

Another good post, but sad didn’t write out the error message as clearly the outsourced image has been lost from the interwebs.

The User Profile Service failed the logon

It’s a beautiful Monday morning. I get up shower and get dressed for work. Hop on the bus, that happens to be crowded to the tits! As I stand silently enjoying the sun shine through the bus windows, I hear the annoying sounds of a child’s educational video game. Have you ever heard the sound FX from those things…. so repetitive it’ll drive you crazy!

I silently tough out my nightmarish bus ride, and walk into work. Pull up on my new standing desk, and begin to check on Backup status, and server updates. Then go to grab a coffee from the lunch room.

As I return to my office, I notice a Lync/Skype for Business communication from a colleague that works on another floor stating another user there can’t log into their system.

I quickly open cmd prompt to verify the workstation has network connectivity, it sure does. I remote in using our remote software, and watch the users login attempt, sure enough it fails.

The User Profile Service failed the logon

As this was something I had not seen before, although I had a real good assumption it was user profile based, I quickly googled the error as seen on the screen. Where would we be without google!? My first investigation brought me to this MS support page but it’s offered suggestions were a bit outlandish for me, considering it basically wanted me to manually re-create the profile and migrate the data. I don’t think so, as I went to go see how large the profile was via Advanced System Settings, I noticed the profile status was set to “backup”. Googling this issue brought me to this awesome blog page.

 To Paraphrase to solution:
	1) Open Regedit (�HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList)
	2) Find the profile SID with the .bak
	3) Rename SID profile (with ProfileImagePath = c:\users\temp) to ***.ba
	4) Rename SID profile with .bak by removing the .bak
	5) Set DWROD State and RefCount to 0
	6) Rename or delete (haven't tested) ***.ba to ***.bak
	7) Reboot and enjoy

Jan 2018 Update

Sourced links, paraphrased solutions in case links die, good story and to the point. Well done.