Everything IT
This is such a common problem it’s ridiculous…
Ensure you have a Windows Installation media available (Mounted ISO, inserted DVD, whatever it takes) then run this *Replace d: with your installation media mount label*:
DISM /Online /Enable-Feature /FeatureName:NetFx3 /All /LimitAccess /Source:d:\sources\sxs
I manage core servers often and more than not I forget the exact syntax and objects and parameters used when doing so. Like this one:
Netsh advfirewall firewall set rule group=”remote desktop” new enable=yes
If you’re reading this chances are you have read this or this (If you like Wizards) … and this only to ask your self… “What permissions does this “admin” account need?” I mean… it says specify an admin account right?!
“Under LDAP admin, specify the administrator account, the corresponding password and the base distinguished name”
After quick testing, turns out no, any regular domain user account worked here.
You might ask why, I recently completed the VMware training module for installation and management of VMware vSphere components, to start to play around I don’t exactly have a bunch of hardware kicking around. I do however, have my awesome gamming Rig which is massively over powered in terms of CPU, Memory however… err not so much, disk I/O… also meh, these will need to be expanded on, but I do at least have Windows 10 running on SSD and a 3 TB spindle disc for more regular storage needs, but everyone knows a 7200 RPM disk provides mediocre performance.
Anyway I’m choosing Hyper-v Since I already have windows, and it comes free with windows, there are other options such as Oracles Virtual Box, and VMware player (can only run one VM at a time though for the free version :S)
Besides that here’s the steps so far.
1) Activate your Windows 10 Pro (1607), as mentioned installed mine on a 120 GB SSD.
2) Ensure VT-x and VT-e and probably VT-d is enabled, and that you have a motherboard and CPU capable of doing virtualization.
3) make sure all your hardware drivers are up-to-date.
4) Install Hyper-v.
5) Configure server settings such as HDD location, CPU allocation, and networks. In my case, I want my ESXi hosts to be isolated from the internet, so I pick internal.
6) Grab the ESXi ISO installation media from VMware (login and subscription required).
7) Create VM, I had to pick Gen 1 with BIOS, Gen 2 with EUFI didn’t boot the ISO for me.
8) I noticed at first attempt at the ESXi VM, it was sitting at loading kernel for an awfully long time, sure enough a simple Google search and discovered this gem.
When ESXi installer runs hit tab; add ignoreHeadless=TRUE
9) Before I could go any further I came across the dreaded, there are no network adapters available. You can Google this, but you will probably get blog posts about people attempting to load the nested VM with only 2 GB of ram when ESXi minimum requires 4 GB of RAM, so you have to be very specific in your search. In this case it’s amazing the power of the open community these days:
Turns out (as usual) it’s a driver related issue (don’t worry I’ll talk about this a couple times throughout this guide).
Lucky enough some lad was genius enough to figure out a solution, not only that but also provide the direct VIB to inject into the ISO file.
I followed the instructions, discovering that the latest supported release was for 5.x including 5.5 for ESXi installation customizer.
A double whammy it didn’t run on my Windows 10 x64 box… dismay not, we’re playing with VMs here.
I quickly created another VM and install my old Windows XP ISO with custom Dark Vista theme imbedded.
The great part was getting the files into the VM was a breeze. Simply shutdown the VM, navigate to the VMs HDD folder, Right click the VHDX file, and select the mount context menu.
This mounts the system and C:\ as separate disks on my windows host, copied the files in, booted the VM, and followed the instructions using the provided VIB and the ESXi 6.0 installer from VMware (login required). Bam sure enough I got a new custom ESXi 6.0 installer ISO file. Moved it out in the same fashion. Mounted it as the ESXi VM’s disc, and booted it up!
Finally the installation moves on! (Make sure you choose a “Legacy Network adapter”)
*Note I do not discuss storage choice when setting up this test host, I simply chose to create a VHDX file of 1 TB for the Nested VMs and for ESXi to be installed on.
10) Once the installation completes, and reboots make sure to hit SHIFT + O and add “ignoreHeadless=TRUE”. Let ESXi boot in DCUI
11) At DCUI, navigate to “Troubleshooting Options”, Then “Enable Shell”
12) Press ALT + F1 (Not F2 as the source states, F2 is the DCUI, F1 is the console). Then Login with root.
13) Type in this command and you won’t have type the headless part of the boot.
“esxcfg-advcfg –set-kernel “TRUE” ignoreHeadless”
(Copy command then select the Menu item “Clipboard”, then “Type Clipboard Text”) (classic Ctrl + V works too)
14) I was finally able to manage set an IP address for managing the host, the virtualized ESXi host hahaha. Sadly the vSphere client failed to connect on my XP VM.
So I setup a Windows 7 x64 bit VM instead. I set this up on Hyper-V on my Windows ten machine, alongside my ESXi hosts to mimic having a laptop running Windows 7.
The vSphere phat client can be downloaded from VMware (login required). Creating my first test VM on my nested ESXi host seemed to have an issue, reading further in the communities shows others with the exact same issue.
Turns out one can simply add a line to the VM’s VMX file “vmx.allowNested = TRUE”. This can be done via SSH (if enabled) or direct console (ALT+F1) using vi.
15) Another thing I noticed was when I was using the Hyper-V Manager’s console to manage my Windows VM running vSphere, and then having it open up vSphere’s console that the Hyper-v console would hang.
My only option at this point was to change my Windows 7 mgmt VM’s network setup. Instead of it only being in the locked down management network, I added another NIC to the VM after creating an external vSwitch in Hyper-V.
Since I have a DHCP server in my local LAN, having the Windows 7 NIC setup to DHCP provided it from my DHCP pool. Using ipconfig (in VM) or checking my DHCP server’s pool I was able to find the IP to remote into.
This of course required setting up remote desktop permissions on the Windows 7 VM. This also allowed me to work in full screen mode, and didn’t crash when opening up vSphere consoles, including of course copy and paste abilities. :D.
16) Next sort of problem was kind of expected. No x64 VM’s in my Nested enviro. There’s topics on this. So I decided to grab the latest 32 bit version of windows that’s available… you guessed it; Server 2008 (Not R2).
Grabbing a couple different versions available from MSDN, gave me a tad bit of issues. First off, don’t use the Checked/Debug, I played with the standard and the SP2 versions. I found the issue was it was hanging at completing installation.
Checking the VM stats via vSphere phat client VM’s performance tab, showed MAX CPU (not always a sign of being hung as it could still just be processing, but definitely a sign on the less), then the big give away, Disk I/O and consumed memory.
Disk I/O was none, and the consumed memory was on a steady decline till it plateaued neared nothing, all signed of stuck or looped process. Since I felt like giving it a little benefit of the doubt, and I had two virtual ESXi hosts to play with,
I decided to bump the CPU on one from 2 to 4. This allowed me to create a VM with 4 virtual CPUs instead of 2. Not sure why this would make a diff, and not sure if it exactly was. So I mounted the same 2008 with SP2 ISO and load the full desktop standard.
This time it finally got into the desktop… guess I’ll try the Standard core now on my other host after upping the CPU as well… let’s see. Yay Server Core installed using the standard 2008 32bit ISO with 4 core CPU.
17) Next issue I came across was not being able to have the VM’s inside the nested ESXi servers communicate with any other device in the same flat layer 2 network. I was sure I had configured everything correctly.
If one Googles this they will find lots n lots of articles on it stating the importance of promiscuous mode. I was up super late trying to figure out this problem and was starting to get a bit crazy. Setting all forms of the settings I could possibly find.
Including attempting to set mirror ports on the ESXi’s VM NICs on Hyper-v hahaha. AS I mentioned you’ll find many references to it, but googling promiscuous mode hyper-v and you discover most people stating to add a line to the VM’s XML config file.
Well it probably won’t take you long before you discover you VM config location doesn’t contain XML files but rather vmcx files. Yeeeeapppp, good luck opening them up… they are now binary…. Wooooo! No admins playing around in here! Take that you tweakers!
This was a change in Hyper-V starting with Server 2016 / Windows 10. I spent a couple hours tumbling down this rabbit hole. To help other I’ll make this part as clear as mud!
IN Hyper-V, ON THE ESXi VMs NETWORK SETTING THAT IS THE LEGACY NETOWRK ADAPTER (the one used as the “physical” adapter in the ESXi vSwitch) EXPAND THE SETTING AND UNDER ADVANCED FEATURES SELECT “Enable MAC address spoofing”.
That’s it! That is Server 2016/Windows 10 Hyper-V’s work around for nested hypervisors. Although as usual support people on TechNet instead of giving an answer or a technical work around would rather dust their hands the classical “not supported” instead of “It’s possible, here’s how, but if something doesn’t work with these settings that’s all we can help with” which I feel would have been a far better response. Maybe these support people just aren’t aware, who knows here’s where I found my answer.
18) So now that I got my hosted ESXi servers up and running and communicating the next step is vCenter. vCenter will setup its own SSO domain, we can add a MS AD domain later and change the default SSO domain to be our active direct domain. However the default SSO domain created at vCenter deployment is the local configuration domain for all vCenter services. Grab vCenter Appliance from VMware. You might be wondering what gives when you discover under the download list for vCenter that there’s an ISO and an IMG file, but no OVA/OVF. This is cause in vSphere 6.0 the vCenter appliance is deployed via a client system using some weird system to communicate to the host to deploy via some web stuff… even I don’t know the exact details of what’s up, either way, if you attempt to create an VM and mount the ISO, you’ll find it’s not bootable. So mount it to the management VM. In my case my Windows 7 VM with vSphere installed. Since Windows 7 doesn’t have native ISO mounting features I had to install virtual clone drive. Then mount the ISO and navigate inside.
Oddly enough it almost seems as if you need a windows system to deploy a Linux appliance. Under the VCSA folder you should find an integration plugin exe installable… run the installable exe file. There seems to be a set that states installing certificates and service, this might be the start of the certs of the built in SSO domain. Not sure though. Once it’s done it sort of leaves you in the dark… as every just closes and there’s not complete window in the wizard…
Guess I’ll just run vsa-setup.html now… Since I have a native version of Windows 7 setup… looks like I’ll need IE 10/11 as the default IE 8 won’t suffice. Lucky for me the Windows 7 machine still had access to the internet, so I Googled the IE 11 installer and ran it, this may be a pre-requirement for the normal installer. As it seems to download and install required updates. You may need to find an offline installer file for IE 11 if you are in a test enviro where you Windows machine doesn’t have access to the internet.
Click Allow. Another pop-up will appear, click Allow.
Now we can finally click install: S
Accept the user agreement, then enter one of the hosts IP address. Member I installed and run this one the Windows 7 machine that can already access the hosts via SSH or the vSphere phat client. So I will enter the IP address here as I haven’t setup DNS at all yet in my environment, and one wouldn’t technically yet if the plan was to have nested DNS servers (The DNS the hosts point to are VMs it hosts).
I made a wrong IP entry, it alerted me as it couldn’t connect to the host, then corrected the IP, and got a cert warning.
Setup the Virtual Appliance OS’s Root password (I believe it uses openSUSE, so this would set the underlying openSUSE root password).
Now under the deployment type if one were never to connect any other vCenter server into the SSO domain for enhanced link mode, you can pick embedded, however for scalability, and the fact I need to setup a Windows Server vCenter to run Update Manger, I’ll create an external Platform Services Controller (PSE). This will require me to run through the wizard separately to actually deploy the vCenter server, in this case I’m actually just setting up the PSE Virtual Appliance (VA). Hence the options all make sense.
As I mentioned this will create the SSO domain for all vCenter services, do not make this the same as your AD domain, this will cause confusion between domains when you add and set your AD domain as the primary SSO identity source. I stick with VMware default vsphere.local, then add a site name (generally this would be some sort of regional reference). Also set the SSO admin password.
It complained about DNS a requirement and a System name, I’m assuming this is hostname, even though it requested it be either FQDN or IP like it was required for some sort of looked, I specified simply the hostname, and a DNS server IP that is not yet even setup for DNS (That which will be my PDC in my test AD setup) This allowed me to continue the setup. I’m thinking this might be what it enters as a common name or SAN for its cert. is my guess.
I went back and changed it to an IP address as I figured my first couple attempts to access it will be through its IP address and I didn’t want to deal with cert warnings. It did however warn me that FQDN is more preferred and this makes sense when a proper DNS system is already implemented.
Hahaha Sure enough had to go through all that IE 11 setup, and plugin installation for it simply to deploy an OVF file hahaha.
Cool, I guess it’s more that based on how you want to set vCenter up with the new PSE instead of having a bunch of Documentation to read through (While this is technically always best to do anyway) it sort of automates the templates to deploy and how to configure them using a questionnaire type setup. I believe in 6.5 this is maybe easier with some sort of HTML 5 based deployment system. Not sure though.
So I hit a couple snags on deployment. First off I thought I was stuck on not being able to do nest x64 virtualization on my nested ESXi hosts. Until the great lads in Freenodes #vmware told me to enable virtualization extensions to the ESXi VM.
“17:18 < genec> Zew: then did you forget to pass VT-x to the ESXi VM?” – Oh Neat! Thanks genec.
Since I was running all my stuff on Hyper-V I had to Google this. Did take long till I found my answer.
Set-VMProcessor -VMName -ExposeVirtualizationExtensions $true
The VM name being my ESXi hypervisor.
This however I only discovered after I enabled the whole vmx.allowNested = TRUE bit on the deployed VM after I saw that it failed with that usual error message. Luckily enough a bit of googling again and I was able to find my answer.
“You can add vmx.allowNested = “TRUE” to /etc/vmware/config in the ESXi VM to avoid having to put it in every nested VM’s configuration file.” –Thanks Matt
I’ll delete the existing vApp and try my deployment again.
Once I managed to mount the VSCA ISO and install the client plugin, and attempt to deploy the PSC/VCSA I got hung up. It appears all my x64 VMs within my Nested ESXi hosts failed to properly boot. All the different VCSA\PSC versions all went into a boot loop. Windows 7 x64 gave a fault screen after loading the installer files and attempting to boot the setup.exe. Server 2016 just showed a black screen. Looking into this I discovered this guy’s blog… looks like I may have to resort to VMware Workstation Pro!
I’ll post this blog post for now as it has become rather long. I will post my success or failure in the upcoming weeks. Stay tuned!
Jan 2018 Update
I remember this being extremely painful, but was so way easier on Workstation Pro.
So I recently re-installed my Windows OS on my desktop. Glad to say the experience is general a really good one; most driver install automagically.
I even let it grab the latest updates, as you might expect this includes the Windows 10 1607 version update; aka the anniversary edision.
to my dismay I noticed a Windows.old folder under my C:\ the drive I selected to install windows on. Being a fresh install I was a bit surprised to see this. as it’s usually from a complete Windows upgrade. I guess the 1607 version is a complete new version according to MS?
A quick google search however provided a nice blog by of course my fav HowToGeek .
Sadly even after this, I still had a Windows.old folder. 🙁
I couldn’t figure out why, so I decided to navigate into the folder. I discovered it was a particular file InstAud.sys or something of that nature. I opened up the properties of it and seized ownership of the file, I am the administrator after all.
To my dismay again, I still couldn’t add or remove permissions. I managed to use takeown and cacls to add permissions, but then it stated was locked by a process. Using sysinternals handle and procexp both came up empty handed for any locks.. odd :S.
autoruns also didn’t show anything linked to that file… mmm. Even after, disabling my intl video (I was using a gefore PCI-e video card anyway), disabled it in device manager. and uninstalled the software/drivers. Made sure the service was gone. Yet it pertained I coudn’t delete the file.
So, normally I’d use a linux live USB to mount the partition and wipe it out that way. However I wanted to keep it all Windows and attempt to do this without pointing user to any other site or tools.
And yes…. I did figure it out! hahah. Follow these steps.
Step 1) Click the Action Center Icon (or swipe from the right side on a tablet) and select All Settings -> Update and Security -> Recovery -> (Under Advanced Startup) Restart Now. Step 2) Select Troubleshoot -> Cmd prompt. Step 3) Change to the windows disk drive, generally C:
Now if you attempt to navigate directly to the file you might find the commands you set to provide you account permissions might not have been persistent.
You might also be surprised to find there is no cacls command. well fear not, you’ll just have to use icacls instead.
Step 4) Take Ownership: cd C:\Windows.old takeown /F .\*
Step 5) Grant Permissions: icacls .\ /grant Administrator:F
Step 5) Navigate into the folders, till you find the issue file, use del to delete, use takeown and icacls to take ownership and permissions. Step 6) Once that's done use rmdir /S /Q to delete C:\Windows.old
That’s it! your done, reboot. And See Windows.old is gone! That was tougher than it shoulda been. Now I can re-enable, and re-install my integrated Intel drivers, I’m not sure exactly what happened, it was an update that ran right after installing Windows. I did however use a dedicated Windows 10 USB stick that’s 6 months old or so. Just show how important to get the latest builds whenever possible.
I run my own domain. You’d be amazed to find out most my my services are hosted by a single router, a Asus RT-N16 running DD-WRT.
Lately I noticed my website wouldn’t load inside my own network. I saw that I had no actual records for my base domain, or it’s www counter-part. Although I had hoped that an external record lookup and a hairpin would be good enough, it didn’t seem to be the case, plus to aboved un-needed latency a direct lookup with the direct internal IP would be far more benificial.
To my dismay I haven’t managed the box in a while and I couldn’t seem to find any DHCP based records under /etc/hosts.
I was aware of where I set the file for DHCP records, however not sure if DNSmasq uses that for DNS lookups as well or not.
A quick google search however provided a nice blog with step on how to accomplish it via the Web Interface.
Log into the administration interface and go to the Services tab. Find the DNSMasq section and make sure the DNSMasq option is enabled. In the Additional DNSMasq Options box type in your local DNS configurations (one entry per line): address=/host-or-domain/ip-address where host-or-domain refers to the machine name (or domain name) you want to customize the address for and ip-address is the numeric IP address Save and Apply and you should be all set.
Thanks ZEDT
The reason for this blog post was due to a domain migration which involved a SharePoint 2010 server. These were the symptoms, and all the steps I took to resolve them.
To start ensure you have set a new farm admin account.
E.G. stsadm -o updatefarmcredentials -userlogin domain\farmadmin -password PASSWORD This is of course under the sharepoint bin directory and wont be part of the servers default path. All Tasks moving forward will be done with this account (in my case the account has local admin rights as well as being the sharepoint farm admin permissions)
You might have come across an error such as this when working with SharePoint. Might have been from an admin removing a service account in AD, or in my case a whole domain change.
If you’re new to SharePoint you might simply panic before thinking and simply google and you might come across this.
Which sadly leads to a dead end. The reason his leads to a dead end while others have answers I’ll get to soon.
Next the simple thing you’d figure is to do as it says, reconfigure the service to run under a different account. To your dismay you discover the Central Admin page to make this change gives you this!
You might be thinking you’re in a catch 22 here. But fret not, when in doubt PowerShell. I love PowerShell and it only keeps getting better, in this case it’s our savour.
Remember when I stated that SharePoint link was a dead end. It was a dead cause there are different component types, 4 to be exact and I’m gonna tell you how to fix them all! With Sources!
First up the 4 different types!
1) Service Instance:
Cmdlet: Get-SPServiceInstance
Use the comdlet to list all SharePoint Service instances, mark the ID. Place the service instance into a variable.
E.G. $WindowsTokenService=Get-SPServiceInstance -Identity ServiceGUID
Once you have the variable, it should have a bunch of subclasses. In this case .service.ProcessIdentity. You can see it attributes by typing.
E.G. $WindowsTokenService.service.ProcessIdentity
It should display the service account used to run the service. Simply change it via
E.G $WindowsTokenService.service.ProcessIdentity.Username="Domain\NewServiceAccount"
*NOTE* This accuont needs to already be registered as a serivce account in SharePoint, either via the Central Admin page or Powershell.
Then call the object'ssubclass update and deploy methods.
E.G. $WindowsTokenService.service.ProcessIdentity.Update()
E.G. $WindowsTokenService.service.ProcessIdentity.Deploy()
That's it for basic SharePoint Service Instances.
2) Service Applications:
Cmdlet: Set-SPServiceApplicationPool
The funny part about SharePoint Service instances is theres no Set type powershell comdlete for it. Thus the cmdlet used was a Get cmdlet.
The funnt part about Service Applications is not to use the ServiceApplication cmdlet, but rather the ServiceApplicationPool cmdlets.
E.G Get-SPServiceApplicationPool -Identity SercurityTokenServiceApplicationPool | Set-ServiceApplicationPool -Account "Domain\NewServiceAccount"
Don't forget to do an IIS reset. Then running the Get-SPServiceApplicationPool cmdlet the service application should have a set PrcoessAccountName.
3) Content Applications:
Cmdlet: [Microsoft.SharePoint.Administration.SPWebService]::ContentService.ApplicationPools | ft Name
Yeah you read that right, there's no direct PowerShell cmdlet for this one. You got to go deep... real deep. Anyway run the cmdlet to list all Content Applications.
Once you have determined the one you need to change the service account for place it in a variable.
E.G. $SPAppPool=[Microsoft.SharePoint.Administration.SPWebService]::ContentService.ApplicationPools | where {$_.Name -like "My Content App Pool Name"}
Calling this variable will result in an output very similar to a Service Instance's Service.ProcessIdentity subclass. So you guessed it.
E.G. $SPAppPool.Username="Domain\NewServiceAccount"
E.G. $SPAppPool.Update()
E.G. $SPAppPool.Deploy()
4) Search Service:
Cmdlet: Get-SPEnterpriseSearchService
Yup believe it another Get cmdlet to make a change, where are these Set counterparts you may ask, well that's a good fucking question. We should ask Microsoft.
Anyway, if this hasn't annoyed you enough already chances are you haven't been a SharePoint admin for long, cause it's a rabbit hole. So to finish up here.
E.G. $SSS=(Get-SPEnterpriseSearchService).get_ProcessIdentity()
E.G. $SSS.Username="Domain\NewServiceAccount"
E.G. $SSS.Update()
E.G. $SSS.Deploy()
Sources:
General SharePoint 2010 Managed Service Accounts
Service Instances Source
Service Application Source
Content Application Source
Search Service Source
Once the Get-SPServiceAccounts shows all good on password expiry and no bad accounts exist, there should be no issues opening the Configure Managed Service Accounts section in Central Administration page of SharePoint 2010.
Happy Configuring. 🙂
To Paraphrase:
0) There is no way to paraphrase this.
1) Don’t break SharePoint.
2) Don’t break SharePoint.
Have you ever experienced a Network Partitioned warning in vSphere 5? Hopefully not, but if you find yourself with this warning in vSphere. Don’t panic its not as bad as it could have been in 4.x.
This literally just means that the host can not communicate with any of the designated VMK’s checked off for management traffic. In my case it happened after making network changes to my infrastructure. In this case I still had bonded links at my switches, but somehow the VMK load balancing algo had switched to “route based on originating port ID”, this load balance algo doesn’t work with bonded NICs, and needs to be “route based on IP hash”. My end goal was to get off bonded links for my host and use the default load balance algo that VMware uses, as this can be down with non stacked switches and can be done with minimal switching knowledge (in case others need to manage the system in the future).
It took me a little bit to catch the issue, cause the symptoms were that each host could ping any device in their respected management subnets but NOT the other host, flat /24 subnet too, really had me baffled. As I couldn’t vMotion in this state either, but lucky the VMs on each host remained active (as they have separate communication VMPGs on dedicated physical connections).
Once I caught the error, I was able to verify vMotion worked again. That’s all there is to it!
To Paraphrase to solution:
1) Check which VMKs have management checked off. 2) Check those vSwitches physical connections. 3) If multiple ports check configs on physical switch and load balance algo. 4) Google any errors along the way. 5) Check host to host communication by consoling into host and using vmkping.
Jan 2018 Update
I remember this…
Have you ever restored a VM? Have you done your DR testing by actually doing a full recovery with AD? Did you find you had a couple odd things occur after restore, such as not being able to RDP into your recovered server? Chances are your network profile has changed to public, instead of Domain. This in turn causes certain firewall rules to trigger.
I remember coming across this issue multiple times, especially when people usually want private instead of public and vice versa. So chances are you’ve come across this, telling you to use PowerShell cmdlet to change its setting, which to my guess makes a registry change. The other option they specified was to use the GUI.
Well I find changing local security policies and all that other stuff rather annoying. Soo after a bit more googling I found a really nice answer, which worked and was very simple to implement. Very nicely written and easy to follow by a Evan A Barr. You can view his site here.
To Paraphrase to solution Using Network Connection Properties:
0) by adding a DNS suffix so that NLA can properly locate the domain controller. 1) Go to Network Connections. 2) Go to the properties of of the network adapter in the wrong location. 3) Go to the properties for IPv4. 4) Click the "Advanced..." button. 5) Select the DNS tab. 6) Enter your domain name into the text box for "DNS suffix for this connection:". 7) Disable and then enable the connection to get NLA to re-identify the location.
Do you follow best practice? Have you setup a VMware HA cluster with vCenter. Do you have your own PKI and certificates? Did you not have active monitoring on said certs? Then chance are you are in the exact same boat as me! This blog post assumes you are well advise in using the SSL Cert Automation Tool as well as creating certificates for use with the tool.
This one begins on a Monday after the weekend. I was getting alerts of failed backup jobs. I managed to configure Veeam at my work place and have been happy with the product and support from day 1. I also configured a cold site for backup retention in the event our primary site, you know…. implodes. Anyway, I was used to getting “failed” alerts when really there was simply a communication hiccup across my IPsec tunnel, which usually the job would complete successfully and just report the error. This time however it was different, the errors were for normal backup jobs and reported “incorrect username and password.” I knew the service account’s password, used by Veeam, never expired or changes. Instantly telling me something else is wrong. I then attempt to login into vSphere connecting to my vCenter server, and sure enough it says the same thing wrong username and password, to which another notice pops up saying all communications are untrusted due to expired certs. Doh!
At this point you’ll probably have done exactly what I did… check your installation documentation right?!?! I mean if you are running custom certs, I’m assuming you follow other best practices such as documenting. :P. But after that you are probably googling once you discover part of the SSL tool are not working!
Chances are you came across VMwares KB on renewing certs on a 5.5 version instance of vCenter only to discover that at step 5 a) that the tool reports the local machine doesn’t have the SSO service installed. This really comes down to what the “tool” really is, and that’s a batch script. Yeah you read that right a BATCH script, so you could imagine how ugly and how painful that must have been to code. Like seriously 5.5 was released in Sept 2013 and they were coding using PowerShell by then… shame on you VMware. Anyway, the most likely problem here is in the way this batch script actually checks for the installed service (I looked at the source code of the “tool” but I didn’t actually locate the part that handles this and I’m strictly making assumptions here) is that it probably has a more direct string to which it looks for, again assuming here a reg key or something of that nature and its probably using a version number to check against, if the version changes the script would reply a “can’t find this”. and thus you get the above error which you know is wrong. So how do you fix this, well you grab the exact version of the tool for the updated instance of vCenter you are on (this requires a valid VMware subscription to grab the version of the tool you need). I managed to update one form post in hopes it helps others at this stage of the game.
At this point I kept following through the tutorial, just an FYI I was going through all this with a VMware tech support, and they had to get another tech who specialized in these cases. I came across other issues as well such as in Step 5 d) I got a error similar to this. Sadly I’m writing this up several days after the event so I can’t remember what exactly we did to recover from this one.
At this point gotta keep pushing through the KB which has a total of 24 steps, so you could imagine how painful all this is to do. At the same time I’m not sure HA is even available, and all my backups couldn’t run and any management of VMs would have to be done manually till vCenter could be back up and running. I’ve talked to others and many people suggest to stick with self signed certs even though we all know its not best practice. Thanks VMware for making best practice really hard to implement and maintain.
Also at the very end steps I didn’t not actually have a listed service ID for web client but only the web logger, although you can have separate service ID instance for these, in my case I had to use the web logger service ID to complete the final step. Then after the Web Client wasn’t working properly which I fixed by reinstalling the service/feature via add/remove programs. The fact there is no repair option on this installer bugs me.
To Paraphrase to solution:
1) Ensure you are using the latest and correct version of the SSL tool *cough BATCH script*. 2) Create all your new certificates and chains. 3) Follow the KB article very carefully, specially when it says to do some steps manually vs using the "tool". 4) Google any errors along the way. 5) Bash your head in for following best practices.
Jan 2018 Updates
This brings back bad memories, It’ll soon be time to update to 6.5. We’ll see how VMware has handled internal PKI this time.