Remove Existing Enterprise Root CA (Part 2)

Intro

Continuing on from my source blog post. In this case he goes on to install and configure the role to be a subordinate enterprise CA. But what do you do if you already deployed an Enterprise Root CA? I’m going to go off a hunch and that something gets applied into AD somewhere to present this information to domain clients. I found this nice article from MS directly on the directions to take, it stated for Server 2012, so I hope the procedure on this hasn’t changed much in 2016.

*NOTE* All steps that state need to be done to AD objects, those commands are run as a Domain Admin, or Enterprise Admin directly logged onto those servers. Most other commands or steps will be done via a client system MMC Snap-in, or logged directly into the CA server.

Remove Existing Enterprise Root CA

Revoke Existing Certificates

Step 1: Revoke all active certificates that are issued by the enterprise CA

  1. Click Start, point to
    Administrative Tools, and then click Certification Authority.
  2. Expand your CA, and then click the Issued Certificates folder.
  3. In the right pane, click one of the issued certificates, and then press CTRL+A to select all issued certificates.
  4. Right-click the selected certificates, click All Tasks, and then click Revoke Certificate.
  5. In the Certificate Revocation dialog box, click to select Cease of Operation as the reason for revocation, and then click OK.

Simple enough…

Increase the CRL interval

Step 2: Increase the CRL publication interval

  1. In the Certification Authority Microsoft Management Console (MMC) snap-in, right-click the Revoked Certificates folder, and then click Properties.
  2. In the CRL Publication Interval box, type a suitably long value, and then click OK.

Note The lifetime of the Certificate Revocation List (CRL) should be longer than the lifetime that remains for certificates that have been revoked.

Easy enough, done and done.

Step 3: Publish a new CRL

  1. In the Certification Authority MMC snap-in, right-click the
    Revoked Certificates folder.
  2. Click All Tasks, and then click
    Publish.
  3. In the Publish CRL dialog box, click
    New CRL, and then click OK.

Again easy, done.

Deny Pending Requests

*DEFAULT, generally Not required.

Step 4: Deny any pending requests

By default, an enterprise CA does not store certificate requests. However, an administrator can change this default behavior. To deny any pending certificate requests, follow these steps:

  1. In the Certification Authority MMC snap-in, click the Pending Requests folder.
  2. In the right pane, click one of the pending requests, and then press CTRL+A to select all pending certificates.
  3. Right-click the selected requests, click All Tasks, and then click Deny Request.

Not the case for me.

Uninstall Certificate Services

Step 5: Uninstall Certificate Services from the server

  1. To stop Certificate Services, click Start, click Run, type cmd, and then click OK.
  2. At the command prompt, type certutil -shutdown, and then press Enter.
  3. At the command prompt, type
    certutil -key, and then press Enter. This command will display the names of all the installed cryptographic service providers (CSP) and the key stores that are associated with each provider. Listed among the listed key stores will be the name of your CA. The name will be listed several times, as shown in the following example:

(1)Microsoft Base Cryptographic Provider v1.0:
  1a3b2f44-2540-408b-8867-51bd6b6ed413
  MS IIS DCOM ClientSYSTEMS-1-5-18
  MS IIS DCOM Server
  Windows2000 Enterprise Root CA
  MS IIS DCOM ClientAdministratorS-1-5-21-436374069-839522115-1060284298-500

  1. Delete the private key that is associated with the CA. To do this, at a command prompt, type the following command, and then press Enter:

certutil -delkey CertificateAuthorityName

Note If your CA name contains spaces, enclose the name in quotation marks. 

In this example, the certificate authority name is “Windows2000 Enterprise Root CA.” Therefore, the command line in this example is as follows:

certutil -delkey “Windows2000 Enterprise Root CA”

* OK, this is where things got weird for me. For some reason I wasn’t getting back the same type of results as the guide, instead I got this:

C:\ProgramData\Microsoft\Crypto\RSA>certutil –key
Microsoft Strong Cryptographic Provider:
TSSecKeySet1
f686aace6942fb7f4566yh1212eef4a4_ae5889t-54c3-4b6f-8b60-f9f8471c0525
RSA
AT_KEYEXCHANGE

CertUtil: -key command completed successfully.

And any attempt to delete the key based on the known CA name just failed. I asked about this in TechNet under the security section, and was told basically what I figured and that the key either didn’t exist or was corrupted. So basically continue on with the steps. It was later answered by Mark Cooper.

Locating the CA Master Key

This one again got answered by Mark Cooper, include –csp ksp (keys are located under: %allusersprofile%\Microsoft\Crypto\Keys)

Viewing the Certificate store

Certutil –store  my

This made me start to wonder where the actual certificate files were stored, a google away and it turns out to be in the registry? Lol (HKLM\System\Microsoft\SystemCertificates)

You can see they key container name matches the certificate hash.

Nothing more than just a string of obfuscated code (much like opening up a CSR), so the only way to interact with them is using the Microsoft CryptoAPI (CertUtil), or the Snap-in.

Deleting the CA Master Key

From all the research I’ve done, it seems people are adamant that you delete the key before you remove the certs, why exactly I’m not sure…(From my testing if you deleted the certificate via certutil, it comes right back when restarting certsvc. It must be rebuilt from the registry?)

So: certutil –csp ksp –delkey <key>

Checking the keys directory show empty. Good stuff.

Deleting the CA Certificate

Certutil –delstore my <Serial>

Reopening regedit, and the cert is gone.

Delete Trusted Root CA Cert

Certutil –store ca
Certutil –delstore ca <serial>

So moving on…*

  1. List the key stores again to verify that the private key for your CA was deleted.
    Check
  2. After you delete the private key for your CA, uninstall Certificate Services. To do this, follow these steps, depending on the version of Windows Server that you are running.

    Uninstall-AdcsCertificationAuthority

    If the remaining role services, such as the Online Responder service, were configured to use data from the uninstalled CA, you must reconfigure these services to support a different CA. After a CA is uninstalled, the following information is left on the server:

    • The CA database (To be deleted see below)
    • The CA public and private keys (Deleted see above)
    • The CA’s certificates in the Personal store (Deleted See above)
    • The CA’s certificates in the shared folder, if a shared folder was specified during AD CS setup (N/A)
    • The CA chain’s root certificate in the Trusted Root Certification Authorities store (Deleted See Above)
    • The CA chain’s intermediate certificates in the Intermediate Certification Authorities store (none existed for me)
    • The CA’s CRL (yup)

By default, this information is kept on the server in case you are uninstalling and then reinstalling the CA. For example, you might uninstall and reinstall the CA if you want to change a stand-alone CA to an enterprise CA.

Known AD CA Objects

Step 6: Remove CA objects from Active Directory

When Microsoft Certificate Services is installed on a server that is a member of a domain, several objects are created in the configuration container in Active Directory. 

These objects are as follows:

  • certificateAuthority object
    • Located in CN=AIA,CN=Public Key Services,CN=Services,CN=Configuration,DC=ForestRootDomain.
    • Contains the CA certificate for the CA.
    • Published Authority Information Access (AIA) location.
  • crlDistributionPoint object
    • Located in CN=ServerName,CN=CDP,CN=Public Key Service,CN=Services,CN=Configuration,DC=ForestRoot,DC=com.
    • Contains the CRL periodically published by the CA.
    • Published CRL Distribution Point (CDP) location
  • certificationAuthority object
    • Located in CN=Certification Authorities,CN=Public Key Services,CN=Services,CN=Configuration,DC=ForestRoot,DC=com.
    • Contains the CA certificate for the CA.
  • pKIEnrollmentService object
    • Located in CN=Enrollment Services,CN=Public Key Services,CN=Services,CN=Configuration,DC=ForestRoot,DC=com.
    • Created by the enterprise CA.
    • Contains information about the types of certificates the CA has been configured to issue. Permissions on this object can control which security principals can enroll against this CA.

When the CA is uninstalled, only the pKIEnrollmentService object is removed. This prevents clients from trying to enroll against the decommissioned CA. The other objects are retained because certificates that are issued by the CA are probably still outstanding. These certificates must be revoked by following the procedure in the “Step 1: Revoke all active certificates that are issued by the enterprise CA” section. 

For Public Key Infrastructure (PKI) client computers to successfully process these outstanding certificates, the computers must locate the Authority Information Access (AIA) and CRL distribution point paths in Active Directory. It is a good idea to revoke all outstanding certificates, extend the lifetime of the CRL, and publish the CRL in Active Directory. If the outstanding certificates are processed by the various PKI clients, validation will fail, and those certificates will not be used. 

If it is not a priority to maintain the CRL distribution point and AIA in Active Directory, you can remove these objects. Do not remove these objects if you expect to process one or more of the formerly active digital certificates.

Remove all Certification Services objects from Active Directory

To remove all Certification Services objects from Active Directory, follow these steps:

  1. Know the CA common name (use CertUtil)
  2. Use Sites and Service MMC Snap-in from a client computer using a domain admin account with proper permissions, highlight the parent snap-in node -> View (from the toolbar) -> Show Services Node.
  3. Expand Services, expand Public Key Services, and then click the AIA folder.
  4. In the right pane, right-click the CertificationAuthority object for your CA, click Delete, and then click “Yes”.
  5. Left Nav, Click CDP folder.
  6. In the right pane, right-click the CertificationAuthority object for your CA, click Delete, and then click “Yes”.
  7. Left Nav, Click Certificate Authority.
  8. In the right pane, right-click the CertificationAuthority object for your CA, click Delete, and then click Yes.
  9. Left Nav, Click Enrollment Services (This should have been auto removed, in my case it was)
  10. If you did not locate all the objects, some objects may be left in the Active Directory after you perform these steps. To clean up after a CA that may have left objects in Active Directory, follow these steps to determine whether any AD objects remain:
    1. Type the following command at a command line, and then press ENTER:
      1. ldifde -r “cn=CACommonName” -d “CN=Public Key Services,CN=Services,CN=Configuration,DC=ForestRoot,DC=com” -f output.ldf
    2. In this command, CACommonName represents the Name value that you determined in step 1. For example, if the Name value is “CA1 Contoso,” type the following:
      1. ldifde -r “cn=CA1 Contoso” -d “cn=public key services,cn=services,cn=configuration,dc=contoso,dc=com” -f remainingCAobjects.ldf
    3. Open the remainingCAobjects.ldf file in Notepad. Replace the term “changetype: add” with “changetype: delete.” Then, verify whether the Active Directory objects that you will delete are legitimate.
    4. At a command prompt, type the following command, and then press ENTER to delete the remaining CA objects from Active Directory:
      1. ldifde -i -f remainingCAobjects.ldf

At this point I was having issues with the input command of the ldf file was failing. I posted these results in my Technet post. After a bit more research I noticed other examples online not having any other information appended after the “changetype: delete” line. So I simply followed along and did the same deleting all the lines after that one, leaving the base DN object in place and sure enough it finally succeeded.

Generate base object LDF file:

After editing line as specified in MS article:

New altered LDF file:

Same command after altering file:

Second run I simply deleted the object under the KRA folder, and it returns no values.

13) Delete the certificate templates if you are sure that all of the certificate authorities have been deleted. Repeat step 12 to determine whether any AD objects remain.

I did this via the Site and Service Snap-in, under the PKI section of the Services node.

Delete NTAuthCertificates Objects Published Certificates

Step 7: Delete certificates published to the NtAuthCertificates object

After you delete the CA objects, you have to delete the CA certificates that are published to the NtAuthCertificates object. Use either of the following commands to delete certificates from within the NTAuthCertificates store:

certutil -viewdelstore “ldap:///CN=NtAuthCertificates,CN=Public Key 
Services,…,DC=ForestRoot,DC=com?cACertificate?base?objectclass=certificationAuthority”

certutil -viewdelstore “ldap:///CN=NtAuthCertificates,CN=Public Key 
Services,…,DC=ForestRoot,DC=com?cACertificate?base?objectclass=pKIEnrollmentService”

Note You must have Enterprise Administrator permissions to perform this task.

The -viewdelstore action invokes the certificate selection UI on the set of certificates in the specified attibute. You can view the certificate details. You can cancel out of the selection dialog to make no changes. If you select a certificate, that certificate is deleted when the UI closes and the command is fully executed

Use the following command to see the full LDAP path to the NtAuthCertificates object in your Active Directory:

certutil store -? | findstr “CN=NTAuth”

Nice and easy, finally.

Delete the CA Database

Step 8: Delete the CA database

When Certification Services is uninstalled, the CA database is left intact so that the CA can be re-created on another server.

To remove the CA database, delete the %systemroot%\System32\Certlog folder.

Nice and easy, I like these steps.

Clean up the DC’s

Step 9: Clean up domain controllers

After the CA is uninstalled, the certificates that were issued to domain controllers must be removed. 

Which states for 2003 and up:

certutil -dcinfo deleteBad

My results:

With the same list of garbage for the DC, then rerunning Certutil –dcinfo, still reported the same certs… So I had to manually remove these, but again opening a MMC snap-in via a client system, add the certificate snap-in and point to the machine store on the DC’s. Then manually delete the certificates, once this was done for both DC’s. CertUtil –dcinfo finally reported clean…

Finally!!! What a gong show it is to remove an existing CA from an environment… even one that literally wasn’t used for anything outside its initial deployment as an enterprise root CA.

Setup Offline Root CA (Part 1)

Installing an Offline Root CA

Nothing special here, run through the windows installer as usual.

Source

  1. Configure VM with hardware settings as specified, Boot option EFI, Remove Floppy Device, attach Windows Server 2016 ISO, and connect at boot. Boot VM
  2. Server 2016, US English, Next, Install Now, Server 2016 Datacenter (Desktop Experience), Accept EULA, custom, select blank disk, next. Install Complete, Reboot. Install VMware tools.
  3. Change hostname and workgroup (Note complained about NetBOIS only supporting 15 characters, so the NetBIOS name was shortened to CORP-OFFLINE-RO)

Installing Certificate Services

Deploying Certificate Services on Windows Server 2016 is simple enough – open Server Manager, open the Add Roles and Features wizard and choose Active Directory Certificate Services under Server Roles. Ensure you choose only the Certificate Authority role for the Root CA.

To make installing Certificate Services simpler, do it via PowerShell instead via Add-WindowsFeature:

Add-WindowsFeature -Name ADCS-Cert-Authority -IncludeManagementTools

Which will look like this, no reboot required:

Configure Certificate Services

After Certificate Services is installed, start the configuration wizard from Server Manager:

Set the credentials to be used while configuring Certificate Services. In this case, we’re configuring CA on a standalone machine and I’m logged on with the local Administrator account.

For the Root CA, we have only on role to configure.

This certificate authority is being configured on a stand-alone server not a member of Active Directory, so we’ll only be able to configure a Standalone CA.

This is the first CA in our environment, so be sure to configure this as a root CA.

With the first CA in the environment, we’ll won’t have an existing private key, so must choose to create a new one.

Windows will no longer accept certificates signed with SHA1 after 1st of January 2017, so be sure to choose at least SHA256. (Default for Windows Server 2016)

*Note there are many cryptographic providers available, but generally most places should stick with RSA, I have seen certain cases where DSA has been selected, only choose this option if you have a specific reason for it. As well generally stick with a 2048 Key Length, you can go higher if you know your system resources can handle the additional computational load, or lower if you are running older hardware and don’t require has high of a security posture.

Specify a name for the new certificate authority. I’d recommend keeping this simple using the ANSI character set, using a meaningful name.

Select the validity period – perhaps the default is the best to choose; however, this can be customized based on your requirements. This is a topic that is a whole security conversation in itself; however, renewing CA certificates isn’t something that you want to be doing too often. Considerations for setting the validity period should include business risk, the size and complexity of the environment you are installing the PKI into, and how mature the IT organization is.

*Note pretty well stated, and in our case I don’t want to renew certs every 5 years, so 10 years sounds about good to me, and I’m hoping 2048 Key length with a SHA256 Hash will still be pretty common 10 years from now, but at least this gives us a very nice time buffer should things change.

On the next page of the wizard, you can choose the location of the certificate services database and logs location (C:\Windows\System32\Certlog), which can be changed depending on your specific environment.

On the last page, you will see a summary of the configuration before committing it to the local certificate services.

Verifying the Root CA

Now that certificate services has been installed and the base configuration is complete, a set of specific configuration changes is required to ensure that an offline Root CA will work for us.

Start – Windows Administrative Tools -> Certificate Authority

If you open the Certificate Authority management console, you can view the properties of the certificate authority and the Root CA’s certificate:

 

Configure the CA Extensions

Before we take any further steps, including deploying a subordinate CA for issuing certificates, we need to configure the Certificate Revocation List (CRL) Distribution Point. Because this CA will be offline and not a member of Active Directory, the default locations won’t work. For more granular information on configuring the CDP and AIA, see these sources:one and two.

CDP

In the properties of the CA, select the Extensions tab to view the CRL Distribution Points (CDP). By default, the ldap:// and file:// locations will be the default distribution points. These, of course, won’t work for the reasons I’ve just stated, and because these locations are embedded in the properties of certificates issued by this CA, we should change them.

Default values:

C:\windows\system32\CertSrv\CertEnroll\<CaName><CRLNameSuffix><DeltaCRLAllowed>.crl

ldap:///CN=<CATruncatedName><CRLNameSuffix>,CN=<ServerShortName>,CN=CDP,CN=Public Key Services,CN=Services,<ConfigurationContainer><CDPObjectClass>

http://<ServerDNSName>/CertEnroll/<CaName><CRLNameSuffix><DeltaCRLAllowed>.crl

file://<ServerDNSName>/CertEnroll/<CaName><CRLNameSuffix><DeltaCRLAllowed>.crl

To set up a CRL distribution point that will work with a location that’s online (so that clients can contact the CRL), we’ll add a new distribution point rather than modify an existing DP and use HTTP.

Before that we’ll want to do two things:

  1. Ensure that ‘Publish CRLs to this location’ and ‘Publish Delta CRLs to this location’ are selected on the default C:\Windows\System32\CertSrv\CertEnroll location. This should be the default setting.
  2. For each existing DP, remove any check marks enabled for ‘Include in CRLs’. (He failed to mention anything about the “Include in the CDP extension of issued Certs” So I’m going to assume to leave this as is for all DP’s)
    *Note* This was a mistake I had to fix manually. 

Now add a new CRL location, using the same HTTP location value included by default; however, change <ServerDNSName> for the FQDN for the host that will serve the CRL. In my example, I’ve changed:

http://<ServerDNSName>/CertEnroll/<CaName><CRLNameSuffix><DeltaCRLAllowed>.crl

to

http://ca.corp.com/CertEnroll/<CaName><CRLNameSuffix><DeltaCRLAllowed>.crl

This FQDN is an alias for the subordinate certificate authority that I’ll be deploying to actually issue certificates to clients. This CA will be online with IIS installed, so will be available to serve the CRLs. (Again he doesn’t provide a snippet of the completed entry, just a snippet of the creation of the entry, since by default all check boxes are unticked, and no mention of any changes to the added location, I’m again going to assume they are simply left untouched.) Here are my example setup:

and then adding the custom http CDP location that will be the Sub-CA with IIS.

AIA

Repeat the same process for the Authority Information Access (AIA) locations:

  1. Disable ‘Include in the AIA extensions of issued certificates’ for all existing locations. (In my case only one had it checked; file:// record)
  2. Copy the existing http:// location
  3. Add a new http:// location, changing <ServerDNSName> for the FQDN of the alias also used for the CRL distribution point. (Here I noticed he was nice enough to provide a snippet and I noticed the “Include in the AIA extension if issued certificates” is checked off, while by default it is not, so following this I will check this option off on the AIA record, I will also go back and check off The include in CDP extensions of issued certificates on the CDP record as well.)

Apply the changes, and you will be prompted to restart Active Directory Certificate Services. If you don’t remember to manually restart the service later.

Configure CRL Publishing

Before publishing the CRL, set the Publication Interval to something other than the default 1 week. Whatever you set the interval to, this will be the maximum amount of time that you’ll need to have the CA online, publish the CRL and copy it to you CRL publishing point.

Open the properties of the Revoked Certificates node and set the CRL publication interval to something suitable for the environment you have installed the CA into. Remember that you’ll need to boot the Root CA and publish a new CRL before the end of this interval.

Setting the CRL Publication Interval on the Root CA

Ensure that the Certificate Revocation list is published to the to the file system – right-click Revoked Certificates, select All Tasks / Publish. We will then copy these to the subordinate CA.

Browse to C:\Windows\System32\CertSrv\CertEnroll to view the CRL and the root CA certificate.

Setting the Issued Certificate Validity Period

The default validity period for certificates issued by this CA will be 1 year. Because this is a stand-alone certification authority, we don’t have templates available to use that we can use to define the validity period for issued certificates. So we need to set this in the registry.

As we’ll only be issuing subordinate CA certificates from this root CA, 1 year isn’t very long. If the subordinate CA certificate is only valid for 1 year, any certificates that it issues can only be valid for less than 1 year from the date of issue – not long indeed. Therefore, we should set the validity period on the root CA before we issue any certificates.

To change the validity period, open Registry Editor and navigate to the following key:

HKLM\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration\<certification authority name>

In my lab, this path is:

HKLM\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration\CORP-OFFLINE-ROOT-CA

Here I can see two values that define how long issued certificates are valid for – ValidityPeriod (defaults to 1) and ValidityPeriodUnits (defaults to “Years”).

Viewing the Root CA certificate validity lifetime

Open ValidityPeriodUnits and change this to the desired value. My recommendation would be to make this 1/2 the lifetime of Root CA’s certificate validity period, so if you’ve configured the Root CA for 10 years, set this to 5 years. You’ll need to restart the Certificate Authority service for this to take effect.

Setting the Root CA’s ValidityUnits

An alternative to editing the registry directly is to set this value to certutil.exe. To change the validity period to 5 years run:

certutil -setreg ca\ValidityPeriodUnits “5”

Conclusion

Yes this is pretty much a copy n paste of the source, it was so well written and nice to follow, there are just a couple additions I added in where things got a little confusing I hope those might help someone who comes across this.

Much like the source in my next post I’ll also cover setting up a Subordinate Root CA, however I will also cover removing an existing CA from an AD environment before replacing it with the new subordinate. As well as cover some errors and issues I faced along the way and how I managed to correct them. This was part was pretty straight cut so I didn’t have much reason to alter it from the source.

Thanks StealthPuppy.

PS – If you plan on publishing new CRL to be hosted by AD for domain systems, don’t forget to set the DSConfigDN setting on this offline CA.
certutil -setreg ca\DSConfigDN “CN=Configuration,DC=Your,DC=Domain”

Renewing expired certificates on vCenter 5.5

Do you follow best practice? Have you setup a VMware HA cluster with vCenter. Do you have your own PKI and certificates? Did you not have active monitoring on said certs? Then chance are you are in the exact same boat as me! This blog post assumes you are well advise in using the SSL Cert Automation Tool as well as creating certificates for use with the tool.

This one begins on a Monday after the weekend. I was getting alerts of failed backup jobs. I managed to configure Veeam at my work place and have been happy with the product and support from day 1. I also configured a cold site for backup retention in the event our primary site, you know…. implodes. Anyway, I was used to getting “failed” alerts when really there was simply a communication hiccup across my IPsec tunnel, which usually the job would complete successfully and just report the error. This time however it was different, the errors were for normal backup jobs and reported “incorrect username and password.” I knew the service account’s password, used by Veeam, never expired or changes. Instantly telling me something else is wrong. I then attempt to login into vSphere connecting to my vCenter server, and sure enough it says the same thing wrong username and password, to which another notice pops up saying all communications are untrusted due to expired certs. Doh!

At this point you’ll probably have done exactly what I did… check your installation documentation right?!?! I mean if you are running custom certs, I’m assuming you follow other best practices such as documenting. :P. But after that you are probably googling once you discover part of the SSL tool are not working!

Chances are you came across VMwares KB on renewing certs on a 5.5 version instance of vCenter only to discover that at step 5 a) that the tool reports the local machine doesn’t have the SSO service installed. This really comes down to what the “tool” really is, and that’s a batch script. Yeah you read that right a BATCH script, so you could imagine how ugly and how painful that must have been to code. Like seriously 5.5 was released in Sept 2013 and they were coding using PowerShell by then… shame on you VMware. Anyway, the most likely problem here is in the way this batch script actually checks for the installed service (I looked at the source code of the “tool” but I didn’t actually locate the part that handles this and I’m strictly making assumptions here) is that it probably has a more direct string to which it looks for, again assuming here a reg key or something of that nature and its probably using a version number to check against, if the version changes the script would reply a “can’t find this”. and thus you get the above error which you know is wrong. So how do you fix this, well you grab the exact version of the tool for the updated instance of vCenter you are on (this requires a valid VMware subscription to grab the version of the tool you need). I managed to update one form post in hopes it helps others at this stage of the game.

At this point I kept following through the tutorial, just an FYI I was going through all this with a VMware tech support, and they had to get another tech who specialized in these cases. I came across other issues as well such as in Step 5 d) I got a error similar to this. Sadly I’m writing this up several days after the event so I can’t remember what exactly we did to recover from this one.
At this point gotta keep pushing through the KB which has a total of 24 steps, so you could imagine how painful all this is to do. At the same time I’m not sure HA is even available, and all my backups couldn’t run and any management of VMs would have to be done manually till vCenter could be back up and running. I’ve talked to others and many people suggest to stick with self signed certs even though we all know its not best practice. Thanks VMware for making best practice really hard to implement and maintain.
Also at the very end steps I didn’t not actually have a listed service ID for web client but only the web logger, although you can have separate service ID instance for these, in my case I had to use the web logger service ID to complete the final step. Then after the Web Client wasn’t working properly which I fixed by reinstalling the service/feature via add/remove programs. The fact there is no repair option on this installer bugs me.

To Paraphrase to solution:

1) Ensure you are using the latest and correct version of the SSL tool *cough BATCH script*.
2) Create all your new certificates and chains.
3) Follow the KB article very carefully, specially when it says to do some steps manually vs using the "tool".
4) Google any errors along the way.
5) Bash your head in for following best practices.

Jan 2018 Updates

This brings back bad memories, It’ll soon be time to update to 6.5. We’ll see how VMware has handled internal PKI this time.