I had this got this to work with this requirement for an external A host record, redirects, negate rules. It was quite complex, and, in the end, it did work. I was excited, I got ready to write this long post, then I realized, I had somehow missed the obvious. I found this post on the forms with someone having the exact same issue what amazed me the most, was how simple their solution was.
So, I tested it…
The HTTP to HTTPS redirect condition:
and this will take any HTTP request and convert them into HTTPS. If you configured HTTP validation though this will be a problem when the request from ACME comes in to hit the backend created by the ACME plugin.
As stated by the guy, he simply made a clone of the condition, and made it a negate.
AD with a Enterprise CA
Why? For easier Certificate management, if you want step by step details using self sign, you can read this blog post by Tyler Muir. Thanks Tyler for your wonderful blog post it was really help to me.
Server Core (2016+)
A Certificate Template published and available to client machines
Now you *Technically* don’t need a template, if you were using self signed. However there are some prerequisites to the Certificate. According to the official Microsoft source it states:
“WinRM HTTPS requires a local computer Server Authentication certificate with a CN matching the hostname to be installed. The certificate mustn’t be expired, revoked, or self-signed.”
If you have a correct cert but not for the type of server auth you will get an error:
Which is super descriptive and to the point.
Implementation
Basic Implementation
If you don’t have a Server Authenticating certificate, consult your certificate administrator. If you have a Microsoft Certificate server, you may be able to request a certificate using the web certificate template from HTTPS://<MyDomainCertificateServer>/certsrv.
Once the certificate is installed type the following to configure WINRM to listen on HTTPS:
winrm quickconfig -transport:https
If you don’t have an appropriate certificate, you can run the following command with the authentication methods configured for WinRM. However, the data won’t be encrypted.
winrm quickconfig
Example:
On my Core Server domain joined, using a “Computer”/Machine Template certificate.
powershell
cd Cert:\LocalMahcine\My
Get-Certificate -Template Machine
ensure you exit out of powershell to run winrm commands
winrm quickconfig -transport:HTTPS
Congrats you’re done.
Advanced Implementation
Now remember in the above it stated “If you don’t have a Server Authenticating certificate, consult your certificate administrator. If you have a Microsoft Certificate server, you may be able to request a certificate using the web certificate template ”
That’s what this section hopes to cover.
There’s only one other pre-req I can think of besides the primary ones mentioned at the start of this blog post.
Once these are met, request a certificate from the CA and ensure it’s installed on the client machine you wish to configure WinRM on. Once installed grab the certificate Thumbprint.
Creating the listener using the certificate ThumbPrint:
When using the MMC snap in pointed to a ore server certificate store, and generated the cert request, and imported the certificate all using the MMC Snap cert plugin remotely. Whenever I would go to create the listener it would error out with “The function: “HttpSetServiceConfiguration” failed unexpectedly. Error=1312. ”
I could only find this guys blog post covering it where he seems to indicate that he wasn’t importing the key for the cert.
This reminded me of a similar issue using Microsoft User Migration Tool and the Cert store showing it had the cert key (little key icon in the cert mmc snap in) but not actually being available. I felt this was the same case. Creating the req from the client machine directly, copying to CA, signing, copying signed cert back to client machine and installing manually resolved the issue.
My might have been able to just use the cert I created via the MMC snap in by running
“The requested certificate template is not supported by this CA.
A valid certification authority (CA) configured to issue certificates based on this template cannot be located, or the CA does not support this operation, or the CA is not trusted.”
This one lead me down a rabbit hole for a long time. Whenever I would have everything in place and request the certificate via powershell I would get this error. If you Google it you will get endless posts how all you need to do is “Publish it to your CA”, such this and this…
“A certificate issued by the certificate authority cannot be installed. Contact your system administrator.
a certificate chain could not be built to a trusted root authority.”
I think checked, and sure enough (I have no clue how) my DC was missing the Offline Root Certificate in it’s Trusted Root Authority store.
Again all buggy, attempting to do it via the Certificate Snap in MMC remotely caused an error, so I had to manually copy the offline root cert file to the domain controller and install it manually with certutil.
This error can also stem from specifying a certificate template that doesn’t exist on the CA. Hence all the blog posts to “publish it”. HOWEVER, in my case I had assumed the “Computer” template (as seen in MMC Snap in Cert tool) is only the display name, the actual name for this template is actually “Machine”
Five
I just have to share this, cause this trick saved my bacon. If you use RDP to manage a core server, you can also use the same RDP to copy files to the core server. Since you know, server core doesn’t have a “GUI”.
Thus does not have secpol, or mmc.exe natively. To set settings either use Group Policy, or if testing on standalone instances or Server Core 2016, you’ll have to define to security policies via a system with a GUI installed, export them and import them into core using secedit.
What you might notice is how strange the UI is designed, you simple find the certificate, and in it’s settings check off to use SMTP.
Then in the connectors options, you simply check off TLS.
Any sensible person, might soon wonder… if you have multiple certificates, and they can all enable the check box for SMTP, and you can have multiple connectors with the checkbox enabled for TLS…. then… which cert is being used?
If you have any familiarity with IIS you know that you have multiple sites, then you go enable HTTPS per site, you define which cert to use (usually implying the use of SNI).
Anyway, from the first post seems my question was answered, in short “Magic”…
“The Exchange transport will pick the certificate that “fits” the best, based on the if its a third party certificate, the expiration date and if a subject name on the certificate matches what is set for the FQDN on the connector used.” -AndyDavid
You may have noticed a fair amount of chatter in that first thread about the default certificate. You may have even noticed some stern warnings:
“You can’t unless you remove the cert. Do not remove the built-in cert however. ” “Yikes. Ok, as I mentioned, do not delete that certificate.”-AndyDavid
So I followed along, and annoyingly you need an SMB shared path accessible to the Exchange server to accomplish this task. (I get it; for clustered deployments)
Anyway doing this and using the UI to assign the certificates to all the required services. Deleted the old Self Signed Cert, wait a bit, close the ECP, reopen it and….
The first main answer was to “wait n hour or more”, yeah I don’t think that’s going to fix it…
KarlIT700 – ”
Our cert is an externally signed cert that is due to expire next year so we wanted to keep using it and not have to generate a new self sign one.
We worked around this by just running the three PS commands below in Exchange PS
Set-AuthConfig -NewCertificateThumbprint <WE JUST USED OUR CURRENT CERT THUMPRINT HERE> -NewCertificateEffectiveDate (Get-Date)
Set-AuthConfig -PublishCertificate
Set-AuthConfig -ClearPreviousCertificate
Note: that we did have issues running the first command because our cert had been installed NOT allowing the export of the cert key. once we reinstalled the same cert back into the (local Computer) personal cert store but this time using the option to allow export of the cert key, the commands above worked fine.
We then just needed to restart ISS and everything was golden. :D”
Did return the certificate I renewed UI the ECP webUI… even then I decided to follow the rest of the steps, just as Karl has mentioned using the thumbprint from the only self signed cert that was there.
Which sure enough worked and everything was working again with the new self signed cert.
Anyway, figured maybe this post might help someone.
Yup had to use IE, sigh I’ll never get away from this browser. Same with locking down mixed content and blocking iframes using lower grade TLS 1.0 or 1.1. So in these cases I still have to tell people to use an older browser. How does this increase security when functionality is removed for perceived security risks. When lots of these systems can be in locked down networks where these risks of lower cypher suites are low?
Now we have to tell people to use older more insecure browsers to access resources or older web services, then they start browsing the internet inadvertently with a vulnerable browser.
Everyone loves a good story. Well today it begins with something I wanted to do for a while but haven’t got around to. I remember adjusting the certificates on 5.5 vCenter and it caused a lot of grief. Now it may have been my ignorance it also may have been due to poor documentation and guides, who knows. Now with VMware now going full linux (Photon OS) for the vCenter deployments (much more light weight) it’s still nice to see a green icon in your web browser when you navigate the nice new HTML5 based management interface. Funny that the guide I followed, even after applying their own certificate still had a “not secure” notification in their browser.
This might be because he didn’t install his Root CA certs into the computers trusted CA store on the machine he was navigating the web interface from. However I’m still going to thank RAJESH RADHAKRISHNAN for his post in VMArena. it helped. I will cover some alternatives however.
Not often I do this but I’m lazy and don’t feel like paraphrasing…
VCSA Certificate Overview
Before starting the procedure just a quick intro for managing vSphere Certificates, vSphere Certificates can manage in two different modes
VMCA Default Certificates
VMCA provides all the certificates for vCenter Server and ESXi hosts on the Virtual Infrastructure and it can manage the certificate lifecycle for vCenter Server and ESXi hosts. Using VMCA default the certificates is the simplest method and less overhead.
VMCA Default Certificates with External SSL Certificates (Hybrid Mode)
This method will replace the Platform Services Controller and vCenter Server Appliance SSL certificates, and allow VMCA to manage certificates for solution users and ESXi hosts. Also for high-security conscious deployments, you can replace the ESXi host SSL certificates as well. This method is Simple, VMCA manages the internal certificates and by using the method, you get the benefit of using your corporate-approved SSL certificates and these certificates trusted by your browsers.
Here we are discussing about the Hybrid mode, this the VMware’s recommended deployment model for certificates as it procures a good level of security. In this model only the Machine SSL certificate signed by the CA and replaced on the vCenter server and the solution user and ESXi host certificates are distributed by the VMCA.
I guess before I did the whole thing, were today I’m just going to be changing the cert that handles the web interface, which is all I really care about in this case.
Requirements
Working PKI based on Active directory Certificate Server.
Certificate Server should have a valid Template for vSphere environment
Note : He uses a custom template he creates. I simply use the Web Server template built in to ADCS.
vCenter Server Appliance with root Access
Requesting the Certificate
Now requesting the certificate requires shell access, I recommend to enable SSH for ease of copying data to and from the VCSA as well as commands.
To do this log into the physical Console of the VCSA, in my case it’s a VM so I opened up the console from the VCSA web interface. Press F2 to login.
Enable both SSH and BASH Shell
OK, now we can SSH into the host to make life easier (I used putty):
Run
/usr/lib/vmware-vmca/bin/certificate-manager
and select the operation option 1
Specify the following options:
Output directory path: path where will be generated the private key and the request
Country : your country in two letters
Name : The FQDN of your vCSA
Organization : an organization name
OrgUnit : type the name of your unit
State : country name
Locality : your city
IPAddess : provide the vCSA IP address
Email : provide your E-mail address
Hostname : the FQDN of your vCSA
VMCA Name: the FQDN where is located your VMCA. Usually the vCSA FQDN
Once the private key and the request is generated select Option 2 to exit
Next we have to export the Request and key from the location.
There are several options on how to compete this. Option 1 is how our source did it…
Option 1 (WinSCP)
using WinSCP for this operation .
To perform export we need additional permission on VCSA , type the following command for same
chsh -s /bin/bash root
Once connected to vCSA from winscp tool navigate the path you have mentioned on the request and download the vmca_issued_csr.csr file.
Option 2 (cat)
Simple Cat the CSR file, and use the mouse to highlight the contents. Then paste it into ADCS Request textbox field.
Signing The Request
Now you simply Navigate to your signing certificate authorizes web interface. usually you hope that the PKI admin has secured this with TLS and is not just using http like our source, but instead uses HTTPS://FQDN/certsrv or just HTTPS://hostname/certsrv.
Now we want to request a certificate, an advanced certificated…
Now simply, submit and from the next page select the Base 64 encoded option and Download the Certificate and Certificate Chain.
Note :- You have to export the Chain certificate to .cer extension , by default it will be PKCS#7
Open Chain file by right click or double click navigate the certificate -> right click -> All Tasks -> export and save it as filename.cer
Now that we have our signed certificate and chains lets get to importing them back into the VCSA.
Importing the Certificates
Again there are two options here:
Option 1 (WinSCP)
using WinSCP for this operation .
To perform export we need additional permission on VCSA , type the following command for same
chsh -s /bin/bash root
Once connected to vCSA from winscp tool navigate the path you have mentioned on the request and upload the certnew.cer file. Along with any chain CA certs.
Option 2 (cat)
Simply open the CER file in notepad, and use the mouse to highlight the contents. Then paste it into any file on the VCSA over the putty session.
E.G
vim /tmp/certnew.cer
Press I for insert mode. Right click to paste. ESC to change modes, :wq to save.
Run
/usr/lib/vmware-vmca/bin/certificate-manager
and select the operation option 1
Enter administrator credentials and enter option number 2
Add the exported certificate and generated key path from previous steps and Press Y to confirm the change
Custom certificate for machine SSL: Path to the chain of certificate (srv.cer here)
Valid custom key for machine SSL: Path to the .key file generated earlier.
Signing certificate of the machine SSL certificate: Path to the certificate of the Root CA (root.cer , generated base64 encoded certificate).
Piss what did I miss…
That doesn’t mean shit to me.. “PC Load letter, wtf does that mean!?”
Since I have an intermediate CA, and I was trying either the Intermediate or the offline it would fail.. I needed them both in one file. So opened each .cer and pasted them into one file “signedca.cer”
Now this did take a while, mostly around 70% and 85% but then it did complete!
Checking out the web interface…
Look at that green lock, seeing even IP listed in the SAN.. mhm does that mean…
Awwww yeah!!! even navigating the VCSA by IP and it still secure! Woop!
Conclusion
Changing the certificate in vCenter 6.7 is much more flexable and easier using the hybird approach and I say thumbs up. 😀 Thanks VMware.
Ohhh yea! Make sure you update your inventory hosts in your backup software with the new certificate else you may get error attempting backup and restore operations, as I did with Veeam. It was super easy to fix just validate the host under the inventory area, by going through the wizard for host configuration.
First off I wanted to address that this wasn’t the intended post of today, this was suppose to be part 2 of the Global Protect post, which is the second dependency; Certificates. However I recently had to get a new cert at work, and discovered this same issue as I had deployed my CA following a decent blog tutorial online by StealthPuppy, and sure enough the same sight provides a follow up on how to fix a mistake made, we all make them and these are great opportunities to learn, so lets get learning!
So you might happen to open up the Certificate Authority Snap and point it to your CA server, to find this…. it’s shutdown….
You might find your certificate authority, in this case, a subordinate certificate authority that is not started, perhaps after a server reboot. Attempting to start the CA, results in this message:
The revocation function was unable to check revocation because the revocation server was offline.
0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE)
Which looks like this:
In the Application log on the subordinate CA, I can see event id 100 from source CertificationAuthority:
Active Directory Certificate Services did not start: Could not load or verify the current CA certificate. stealthpuppy Issuing CA The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE).
As well as, event id 48 from the same source, CertificationAuthority:
Revocation status for a certificate in the chain for CA certificate 0 for stealthpuppy Issuing CA could not be verified because a server is currently unavailable. The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE).
Certificate 0 is the subordinate CA’s certificate, issued by the offline Root CA.
I bypassed this portion of the blog as I didn’t want to have pictures of before the next required step soooo….
The Workaround
Of course, you probably want to get the CA up and running as quickly as possible. The easy way to do that is to disable CRL checking with the following command on the CA server:
Run this from an elevated command prompt and you should now be able to start the CA and get on with the business of troubleshooting.
Perfect, now lets fix this!
The Cause
My CRL was online as it is available in Active Directory (for domain joined machines) and via HTTP at subca.zewwy.ca, an alias of the subordinate CA. I’ve tested that I can retrieve the CRL by putting the HTTP path into a browser and I’m prompted to download a file.
Through having spent some time recently with setting up an Enterprise PKI in my lab and for a project, I’ve come to know the command line tool certutil.exe. This tool is available in all versions of Windows and should be the first tool to use to troubleshoot and manage certificates and certificate authorities on Windows.
Certutil can be used to perform many functions, one of which is to verify a CRL. I know the path to the CRL file because I can view the CRLs on the file system (in C:\Windows\System32\certsrv\CertEnroll) and I’ve previously configured CRLs for both CAs.
To verify the CRL, use the -URL switch with the HTTP (or LDAP) path to the CRL:
This will display the URL Retrieval Tool that shows that the CRLs are able to be contacted and show a status of OK.
*NOTE* I discovered this works directly on the Windows Core Server, if you happened to be running Core (I do as I love optimization, specially when you have to work in lab environments). Except as you might have noticed from the screenshot, the radio buttons are all messed (wonder what library handles that, that wasn’t in core…)
However, if we load a target certificate, in this case, the subordinate CA’s cert, we can start to see why we have an issue with the CRL.
Select the certificate for the subordinate CA that has been previously exported to the file system (in C:\Windows\System32\certsrv\CertEnroll) – click Select, open the certificate and click Retrieve again. This time, we can see a new line that shows that the base CRL for the subordinate CA’s certificate is Expired. (Unfortunetly I had to simply use his source images as in my case I had to also correct my CDP locations on my sign Sub CA Certificate as I had mentioned in my initial Setup Offline Root CA blog post, but I guess I didn’t do it in my lab, so I sort of had to fix to birds with one stone, the CPD locations in my cert by reissuing that, and the offline root ca CRL file, so my additional steps were a bit beyond these exact steps, I also didn’t take snapshots as I wanted to get back on pace with my blog post)
The CRL for the subordinate CA’s certificate will come from the root CA, so we’ll need to check that CRL. Open the CRL file (C:\windows\system32\certsrv\CertEnroll\stealthpuppy Offline Root CA.crl) – double-click or right-click and Open. Here we can see the CRL information, including the next publishing time (Next CRL Publish).
At the time of troubleshooting, this date was in the past and because the Root CA is offline and the CRL is hosted on a different server (the subordinate CA), this particular CRL will never receive an update. So, when the subordinate CA has rebooted, it has checked the Root CA’s CRL and found it expired. Hence the certification authority service won’t start.
How To Fix It
Now we know why the certification authority service won’t start and an understanding of why the CRL is offline, even if the wording doesn’t match the symptoms. If the error message had told me the CRL had expired instead of being offline, I might have saved some troubleshooting time. We now know that we need to re-publish the CRL from the Root CA.
Start the offline Root CA, log into it and open the Certification Authority console. We will first want to ensure that the CRL publication interval is extended so that we don’t run into the same problem in the near future. Open the properties of the Revoked Certificates node to view and set the publication interval. The default interval is 1 week, obviously too often for an offline Root CA.
Instead, set this value to something suitable for the environment you have installed the CA into. Remember that you’ll need to boot the Root CA and publish a new CRL before the end of this interval, otherwise, you’ll have exactly the same issue.
Now publish a new CRL – right-click the Revoked Certificates node and click All Tasks /Publish.
Copy the updated CRL (from C:\Windows\System32\certsrv\CertEnroll by default) from the Root CA to the CRL distribution point and overwrite the existing CRL file (C:\Windows\System32\certsrv\CertEnroll again on my subordinate CA).
Now if we again use certutil.exe to verify the CRL, it comes up roses:
To ensure that the subordinate CA’s certification authority service will start, re-enable CRL checking:
If you have re-published the CRL from the Root CA correctly, the service should start and you can then shut down the Root CA. Then open Outlook and put a reminder in the calendar for a week before the CRL expires again.
Conclusion
I’ve had this issue with an Offline CRL a few times now and not really understood what the issue is until I took the time to troubleshoot the issue properly. I don’t spend that much time with an enterprise PKI and it’s easy to underestimate the complexity of setting up AD Certificate Services correctly.
Install the subordinate certificate authority
Request and approve a CA certificate from the offline root CA
Configure the subordinate CA for the CRL to work correctly
Required Permissions
You need to be a member of the Enterprise Admins group to complete these tasks.
Procedure
Installing Certificate Services
Just as with the offline Root CA, deploying Certificate Services on Windows Server 2012 R2 is simple, I stuck with PowerShell, view source blog for step-by-step GUI tutorial. Instal-ADCSCertificateAuthority?
In the source guide he talk about creating a CNAME record, since he set his offline Root CA CRL to point to “clr.blah.domain” in my case I specified the direct hostname of the CA. Maybe he did this for obfuscation security reasons, I’m not sure, either way I skipped this since an A host record already exists for the path I entered in the CRL information for the offline root ca.
Configuring Certificate Services
After the Certificate Services roles are installed, start the configuration wizard from Server Manager – click the flag and yellow icon and click the Configure Active Directory Certificate Services… link.
My CA server is core thus no GUI, thus no direct Server Manager. Connect to a client system that has required network access to run Server Manager and point it to the CA server. In this case I’ll be using a Windows 10 client machine. Run Server Manager, and add CA server as Domain/Enterprise Admin account.
Then just like the source blog guide, you should notice a notification at the top right requiring post ADCS configuration deployment.
Use a proper admin account:
Click Next, then select CA and CA Web Enrollment.
Click Next, Configure this subordinate certificate authority as an Enterprise CA. The server is a member of a domain and an Enterprise CA allows more flexibility in certificate management, including supporting certificate auto enrollment with domain authentication.
Click Next, Configure this CA as a subordinate CA. After configuration, we will submit a CA certificate request to the offline root CA.
Click Next, Create a new private key for this CA as this is the first time we’re configuring it. Now I’m curious to see what CertUtil reports after this wizard and what the RSA directories on the CA will contain. They should contain the keys, right?!
Click Next, leave the defaults, again simply going to use RSA @ 2048 Key Length, with a SHA256 hash checksum. Should remain the standard for hopefully the next 10 years.
Click Next,
Click Next, because this is a subordinate CA, we’ll need to send a CA certificate request to the offline root CA. Save the request locally which will be used later to manually request and approve the certificate. This is saved to the root of C: by default. Again oddly the initial Common name is auto generated into the request name with no option to alter it…
Moving along, click next, and specify the DB location. Generally leave the defaults.
Finally Summary and confirmation.
Click Configure and the wizard will configure the certificate services roles. Note the warning that the configuration for this CA is not complete, as we still need to request, approve and import the CA certificate.
Configuring the CRL Distribution Point
Before configuring the Certification Authority itself, we’ll first copy across the certificate and CRL from the root CA.
Ensure the root CA virtual machine is running and copy the contents of C:\Windows\System32\certsrv\CertEnroll from the root CA to the same folder on the subordinate CA. This is the default location to which certificates and CRLs are published. Keeping the default locations will require the minimum amount of configuration for the CRL and AIA distribution points.
The result on the subordinate certificate authority will look something like this – note that the CRL for the root CA is located here:
In my case it’s an offline (non domain joined) making a shareable UNC path is slightly painful in these cases, and for the most part, it is completely offline, and no NIC settings are even defined on the VM, heck I could remove the vNic completely :D. Anyway to complete this task I did the usual vUSB (a VMDK I mount to different VMs as needed to transfer files) and copied the resulting files specified above into this VMDK, then attached to the Sub CA VM, and moved files to their appropriate path. Again in this case the Sub CA is Core, so either use diskpart, or Server Manager from the client machine to bring the disk online and mount it.
Issuing the Subordinate CA Certificate
Next, we will request, approve the certificate request for the subordinate CA. At this point, the subordinate CA is un-configured because it does not yet have a valid CA certificate.
Copy the initial request created by the confiz wizard to the movable VMDK.
Now, remove the disk from the Sub CA and attach it to the Off-line root CA, then open up the CA tool to request a new certificate. (Don’t worry about taking the disk offline, it’ll unmount automagically).
Browse to where the certificate request for the subordinate certificate authority is located and open the file.
The certificate request will then be listed under Pending Requests on the root CA. Right-click the request, choose All Tasks and Issue.
The subordinate CA’s certificate will now be issued and we can copy it to that CA. View the certificate under Issued Certificates. Right-click the certificate, click Open and choose Copy to File… from the Details tab on the certificate properties.
Export the new certificate to a file in PKCS format. Copy the file back to the subordinate certificate authority, so that it can be imported and enable certificate services on that machine.
Configuring the Subordinate CA
With the certificate file stored locally to the subordinate CA, open the Certificate Authority console – note that the certificate service is stopped. Right-click the CA, select All Tasks and choose Install CA Certificate…
So from a client system open the CA snap-in, point to the new sub CA…
This is where I got stuck for a good while, all the guides I found online were using CA’s with Desktop experience enabled allowing them to run the CA MMC Snap-in locally, and from all my testing against a Server 2016 Core server running the CA role, the snap-in simply wouldn’t load the input wizard…
So I posted the bug on Technet, and Mark saved my bacon!
“You will need to use the commandline to do this. On the CA itself:
1) Open a command prompt
2) Navigate to where your certificate file is located
3) certutil -installcert <your certificate file name here>”
WOOOOOO! We have a working Enterprise Sub-CA… Now the question on if CRL works, and how to deploy the chain properly to servers and clients so things come up with a trusted chain and a green check mark!
“If the CRL is online correctly, the service should start without issues.
Continuing on from my source blog post. In this case he goes on to install and configure the role to be a subordinate enterprise CA. But what do you do if you already deployed an Enterprise Root CA? I’m going to go off a hunch and that something gets applied into AD somewhere to present this information to domain clients. I found this nice article from MS directly on the directions to take, it stated for Server 2012, so I hope the procedure on this hasn’t changed much in 2016.
*NOTE* All steps that state need to be done to AD objects, those commands are run as a Domain Admin, or Enterprise Admin directly logged onto those servers. Most other commands or steps will be done via a client system MMC Snap-in, or logged directly into the CA server.
Remove Existing Enterprise Root CA
Revoke Existing Certificates
Step 1: Revoke all active certificates that are issued by the enterprise CA
Click Start, point to Administrative Tools, and then click Certification Authority.
Expand your CA, and then click the Issued Certificates folder.
In the right pane, click one of the issued certificates, and then press CTRL+A to select all issued certificates.
Right-click the selected certificates, click All Tasks, and then click Revoke Certificate.
In the Certificate Revocation dialog box, click to select Cease of Operation as the reason for revocation, and then click OK.
Simple enough…
Increase the CRL interval
Step 2: Increase the CRL publication interval
In the Certification Authority Microsoft Management Console (MMC) snap-in, right-click the Revoked Certificates folder, and then click Properties.
In the CRL Publication Interval box, type a suitably long value, and then click OK.
Note The lifetime of the Certificate Revocation List (CRL) should be longer than the lifetime that remains for certificates that have been revoked.
Easy enough, done and done.
Step 3: Publish a new CRL
In the Certification Authority MMC snap-in, right-click the Revoked Certificates folder.
Click All Tasks, and then click Publish.
In the Publish CRL dialog box, click New CRL, and then click OK.
Again easy, done.
Deny Pending Requests
*DEFAULT, generally Not required.
Step 4: Deny any pending requests
By default, an enterprise CA does not store certificate requests. However, an administrator can change this default behavior. To deny any pending certificate requests, follow these steps:
In the Certification Authority MMC snap-in, click the Pending Requests folder.
In the right pane, click one of the pending requests, and then press CTRL+A to select all pending certificates.
Right-click the selected requests, click All Tasks, and then click Deny Request.
Not the case for me.
Uninstall Certificate Services
Step 5: Uninstall Certificate Services from the server
To stop Certificate Services, click Start, click Run, type cmd, and then click OK.
At the command prompt, type certutil -shutdown, and then press Enter.
At the command prompt, type certutil -key, and then press Enter. This command will display the names of all the installed cryptographic service providers (CSP) and the key stores that are associated with each provider. Listed among the listed key stores will be the name of your CA. The name will be listed several times, as shown in the following example:
(1)Microsoft Base Cryptographic Provider v1.0: 1a3b2f44-2540-408b-8867-51bd6b6ed413 MS IIS DCOM ClientSYSTEMS-1-5-18 MS IIS DCOM Server Windows2000 Enterprise Root CA MS IIS DCOM ClientAdministratorS-1-5-21-436374069-839522115-1060284298-500
Delete the private key that is associated with the CA. To do this, at a command prompt, type the following command, and then press Enter:
certutil -delkey CertificateAuthorityName
Note If your CA name contains spaces, enclose the name in quotation marks.
In this example, the certificate authority name is “Windows2000 Enterprise Root CA.” Therefore, the command line in this example is as follows:
certutil -delkey “Windows2000 Enterprise Root CA”
* OK, this is where things got weird for me. For some reason I wasn’t getting back the same type of results as the guide, instead I got this:
C:\ProgramData\Microsoft\Crypto\RSA>certutil –key
Microsoft Strong Cryptographic Provider:
TSSecKeySet1
f686aace6942fb7f4566yh1212eef4a4_ae5889t-54c3-4b6f-8b60-f9f8471c0525
RSA
AT_KEYEXCHANGE
CertUtil: -key command completed successfully.
And any attempt to delete the key based on the known CA name just failed. I asked about this in TechNet under the security section, and was told basically what I figured and that the key either didn’t exist or was corrupted. So basically continue on with the steps. It was later answered by Mark Cooper.
Locating the CA Master Key
This one again got answered by Mark Cooper, include –csp ksp (keys are located under: %allusersprofile%\Microsoft\Crypto\Keys)
Deleting the CA Master Key
From all the research I’ve done, it seems people are adamant that you delete the key before you remove the certs, why exactly I’m not sure…(From my testing if you deleted the certificate via certutil, it comes right back when restarting certsvc. It must be rebuilt from the registry?)
So: certutil –csp ksp –delkey <key>
Checking the keys directory show empty. Good stuff.
Viewing the Certificate store
Certutil –store my
This made me start to wonder where the actual certificate files were stored, a google away and it turns out to be in the registry? Lol (HKLM\System\Microsoft\SystemCertificates)
You can see they key container name matches the certificate hash.
Nothing more than just a string of obfuscated code (much like opening up a CSR), so the only way to interact with them is using the Microsoft CryptoAPI (CertUtil), or the Snap-in.
Deleting the CA Certificate
Certutil –delstore my <Serial>
Reopening regedit, and the cert is gone.
Delete Trusted Root CA Cert
Certutil –store ca
Certutil –delstore ca <serial>
So moving on…*
List the key stores again to verify that the private key for your CA was deleted.
Check
After you delete the private key for your CA, uninstall Certificate Services. To do this, follow these steps, depending on the version of Windows Server that you are running.
Uninstall-AdcsCertificationAuthority
If the remaining role services, such as the Online Responder service, were configured to use data from the uninstalled CA, you must reconfigure these services to support a different CA. After a CA is uninstalled, the following information is left on the server:
The CA database (To be deleted see below)
The CA public and private keys (Deleted see above)
The CA’s certificates in the Personal store (Deleted See above)
The CA’s certificates in the shared folder, if a shared folder was specified during AD CS setup (N/A)
The CA chain’s root certificate in the Trusted Root Certification Authorities store (Deleted See Above)
The CA chain’s intermediate certificates in the Intermediate Certification Authorities store (none existed for me)
The CA’s CRL (yup)
By default, this information is kept on the server in case you are uninstalling and then reinstalling the CA. For example, you might uninstall and reinstall the CA if you want to change a stand-alone CA to an enterprise CA.
Known AD CA Objects
Step 6: Remove CA objects from Active Directory
When Microsoft Certificate Services is installed on a server that is a member of a domain, several objects are created in the configuration container in Active Directory.
These objects are as follows:
certificateAuthority object
Located in CN=AIA,CN=Public Key Services,CN=Services,CN=Configuration,DC=ForestRootDomain.
Contains the CA certificate for the CA.
Published Authority Information Access (AIA) location.
crlDistributionPoint object
Located in CN=ServerName,CN=CDP,CN=Public Key Service,CN=Services,CN=Configuration,DC=ForestRoot,DC=com.
Contains the CRL periodically published by the CA.
Published CRL Distribution Point (CDP) location
certificationAuthority object
Located in CN=Certification Authorities,CN=Public Key Services,CN=Services,CN=Configuration,DC=ForestRoot,DC=com.
Contains the CA certificate for the CA.
pKIEnrollmentService object
Located in CN=Enrollment Services,CN=Public Key Services,CN=Services,CN=Configuration,DC=ForestRoot,DC=com.
Created by the enterprise CA.
Contains information about the types of certificates the CA has been configured to issue. Permissions on this object can control which security principals can enroll against this CA.
When the CA is uninstalled, only the pKIEnrollmentService object is removed. This prevents clients from trying to enroll against the decommissioned CA. The other objects are retained because certificates that are issued by the CA are probably still outstanding. These certificates must be revoked by following the procedure in the “Step 1: Revoke all active certificates that are issued by the enterprise CA” section.
For Public Key Infrastructure (PKI) client computers to successfully process these outstanding certificates, the computers must locate the Authority Information Access (AIA) and CRL distribution point paths in Active Directory. It is a good idea to revoke all outstanding certificates, extend the lifetime of the CRL, and publish the CRL in Active Directory. If the outstanding certificates are processed by the various PKI clients, validation will fail, and those certificates will not be used.
If it is not a priority to maintain the CRL distribution point and AIA in Active Directory, you can remove these objects. Do not remove these objects if you expect to process one or more of the formerly active digital certificates.
Remove all Certification Services objects from Active Directory
To remove all Certification Services objects from Active Directory, follow these steps:
Know the CA common name (use CertUtil)
Use Sites and Service MMC Snap-in from a client computer using a domain admin account with proper permissions, highlight the parent snap-in node -> View (from the toolbar) -> Show Services Node.
Expand Services, expand Public Key Services, and then click the AIA folder.
In the right pane, right-click the CertificationAuthority object for your CA, click Delete, and then click “Yes”.
Left Nav, Click CDP folder.
In the right pane, right-click the CertificationAuthority object for your CA, click Delete, and then click “Yes”.
Left Nav, Click Certificate Authority.
In the right pane, right-click the CertificationAuthority object for your CA, click Delete, and then click Yes.
Left Nav, Click Enrollment Services (This should have been auto removed, in my case it was)
If you did not locate all the objects, some objects may be left in the Active Directory after you perform these steps. To clean up after a CA that may have left objects in Active Directory, follow these steps to determine whether any AD objects remain:
Type the following command at a command line, and then press ENTER:
In this command, CACommonName represents the Name value that you determined in step 1. For example, if the Name value is “CA1 Contoso,” type the following:
Open the remainingCAobjects.ldf file in Notepad. Replace the term “changetype: add” with “changetype: delete.” Then, verify whether the Active Directory objects that you will delete are legitimate.
At a command prompt, type the following command, and then press ENTER to delete the remaining CA objects from Active Directory:
ldifde -i -f remainingCAobjects.ldf
At this point I was having issues with the input command of the ldf file was failing. I posted these results in my Technet post. After a bit more research I noticed other examples online not having any other information appended after the “changetype: delete” line. So I simply followed along and did the same deleting all the lines after that one, leaving the base DN object in place and sure enough it finally succeeded.
Generate base object LDF file:
After editing line as specified in MS article:
New altered LDF file:
Same command after altering file:
Second run I simply deleted the object under the KRA folder, and it returns no values.
13) Delete the certificate templates if you are sure that all of the certificate authorities have been deleted. Repeat step 12 to determine whether any AD objects remain.
I did this via the Site and Service Snap-in, under the PKI section of the Services node.
Delete NTAuthCertificates Objects Published Certificates
Step 7: Delete certificates published to the NtAuthCertificates object
After you delete the CA objects, you have to delete the CA certificates that are published to the NtAuthCertificates object. Use either of the following commands to delete certificates from within the NTAuthCertificates store:
Note You must have Enterprise Administrator permissions to perform this task.
The -viewdelstore action invokes the certificate selection UI on the set of certificates in the specified attibute. You can view the certificate details. You can cancel out of the selection dialog to make no changes. If you select a certificate, that certificate is deleted when the UI closes and the command is fully executed
Use the following command to see the full LDAP path to the NtAuthCertificates object in your Active Directory:
certutil store -? | findstr “CN=NTAuth”
Nice and easy, finally.
Delete the CA Database
Step 8: Delete the CA database
When Certification Services is uninstalled, the CA database is left intact so that the CA can be re-created on another server.
To remove the CA database, delete the %systemroot%\System32\Certlog folder.
Nice and easy, I like these steps.
Clean up the DC’s
Step 9: Clean up domain controllers
After the CA is uninstalled, the certificates that were issued to domain controllers must be removed.
Which states for 2003 and up:
certutil -dcinfo deleteBad
My results:
With the same list of garbage for the DC, then rerunning Certutil –dcinfo, still reported the same certs… So I had to manually remove these, but again opening a MMC snap-in via a client system, add the certificate snap-in and point to the machine store on the DC’s. Then manually delete the certificates, once this was done for both DC’s. CertUtil –dcinfo finally reported clean…
Summary
Finally!!! What a gong show it is to remove an existing CA from an environment… even one that literally wasn’t used for anything outside its initial deployment as an enterprise root CA.
Configure VM with hardware settings as specified, Boot option EFI, Remove Floppy Device, attach Windows Server 2016 ISO, and connect at boot. Boot VM
Server 2016, US English, Next, Install Now, Server 2016 Datacenter (Desktop Experience), Accept EULA, custom, select blank disk, next. Install Complete, Reboot. Install VMware tools.
Change hostname and workgroup (Note complained about NetBOIS only supporting 15 characters, so the NetBIOS name was shortened to CORP-OFFLINE-RO)
Installing Certificate Services
Deploying Certificate Services on Windows Server 2016 is simple enough – open Server Manager, open the Add Roles and Features wizard and choose Active Directory Certificate Services under Server Roles. Ensure you choose only the Certificate Authority role for the Root CA.
To make installing Certificate Services simpler, do it via PowerShell instead via Add-WindowsFeature:
After Certificate Services is installed, start the configuration wizard from Server Manager:
Set the credentials to be used while configuring Certificate Services. In this case, we’re configuring CA on a standalone machine and I’m logged on with the local Administrator account.
For the Root CA, we have only on role to configure.
This certificate authority is being configured on a stand-alone server not a member of Active Directory, so we’ll only be able to configure a Standalone CA.
This is the first CA in our environment, so be sure to configure this as a root CA.
With the first CA in the environment, we’ll won’t have an existing private key, so must choose to create a new one.
*Note there are many cryptographic providers available, but generally most places should stick with RSA, I have seen certain cases where DSA has been selected, only choose this option if you have a specific reason for it. As well generally stick with a 2048 Key Length, you can go higher if you know your system resources can handle the additional computational load, or lower if you are running older hardware and don’t require has high of a security posture.
Specify a name for the new certificate authority. I’d recommend keeping this simple using the ANSI character set, using a meaningful name.
Select the validity period – perhaps the default is the best to choose; however, this can be customized based on your requirements. This is a topic that is a whole security conversation in itself; however, renewing CA certificates isn’t something that you want to be doing too often. Considerations for setting the validity period should include business risk, the size and complexity of the environment you are installing the PKI into, and how mature the IT organization is.
*Note pretty well stated, and in our case I don’t want to renew certs every 5 years, so 10 years sounds about good to me, and I’m hoping 2048 Key length with a SHA256 Hash will still be pretty common 10 years from now, but at least this gives us a very nice time buffer should things change.
On the next page of the wizard, you can choose the location of the certificate services database and logs location (C:\Windows\System32\Certlog), which can be changed depending on your specific environment.
On the last page, you will see a summary of the configuration before committing it to the local certificate services.
Verifying the Root CA
Now that certificate services has been installed and the base configuration is complete, a set of specific configuration changes is required to ensure that an offline Root CA will work for us.
Start – Windows Administrative Tools -> Certificate Authority
If you open the Certificate Authority management console, you can view the properties of the certificate authority and the Root CA’s certificate:
Configure the CA Extensions
Before we take any further steps, including deploying a subordinate CA for issuing certificates, we need to configure the Certificate Revocation List (CRL) Distribution Point. Because this CA will be offline and not a member of Active Directory, the default locations won’t work. For more granular information on configuring the CDP and AIA, see these sources:one and two.
CDP
In the properties of the CA, select the Extensions tab to view the CRL Distribution Points (CDP). By default, the ldap:// and file:// locations will be the default distribution points. These, of course, won’t work for the reasons I’ve just stated, and because these locations are embedded in the properties of certificates issued by this CA, we should change them.
To set up a CRL distribution point that will work with a location that’s online (so that clients can contact the CRL), we’ll add a new distribution point rather than modify an existing DP and use HTTP.
Before that we’ll want to do two things:
Ensure that ‘Publish CRLs to this location’ and ‘Publish Delta CRLs to this location’ are selected on the default C:\Windows\System32\CertSrv\CertEnroll location. This should be the default setting.
For each existing DP, remove any check marks enabled for ‘Include in CRLs’. (He failed to mention anything about the “Include in the CDP extension of issued Certs” So I’m going to assume to leave this as is for all DP’s)
*Note* This was a mistake I had to fix manually.
Now add a new CRL location, using the same HTTP location value included by default; however, change <ServerDNSName> for the FQDN for the host that will serve the CRL. In my example, I’ve changed:
This FQDN is an alias for the subordinate certificate authority that I’ll be deploying to actually issue certificates to clients. This CA will be online with IIS installed, so will be available to serve the CRLs. (Again he doesn’t provide a snippet of the completed entry, just a snippet of the creation of the entry, since by default all check boxes are unticked, and no mention of any changes to the added location, I’m again going to assume they are simply left untouched.) Here are my example setup:
and then adding the custom http CDP location that will be the Sub-CA with IIS.
*NOTE* UNCHECK ALL CHECKBOXES from the LDAP and FILE, the above picture for those settings are wrong.
Disable ‘Include in the AIA extensions of issued certificates’ for all existing locations. (In my case only one had it checked; file:// record)
Copy the existing http:// location
Add a new http:// location, changing <ServerDNSName> for the FQDN of the alias also used for the CRL distribution point. (Here I noticed he was nice enough to provide a snippet and I noticed the “Include in the AIA extension if issued certificates” is checked off, while by default it is not, so following this I will check this option off on the AIA record, I will also go back and check off The include in CDP extensions of issued certificates on the CDP record as well.)
Apply the changes, and you will be prompted to restart Active Directory Certificate Services. If you don’t remember to manually restart the service later.
Configure CRL Publishing
Before publishing the CRL, set the Publication Interval to something other than the default 1 week. Whatever you set the interval to, this will be the maximum amount of time that you’ll need to have the CA online, publish the CRL and copy it to you CRL publishing point.
Open the properties of the Revoked Certificates node and set the CRL publication interval to something suitable for the environment you have installed the CA into. Remember that you’ll need to boot the Root CA and publish a new CRL before the end of this interval.
Setting the CRL Publication Interval on the Root CA
Ensure that the Certificate Revocation list is published to the to the file system – right-click Revoked Certificates, select All Tasks / Publish. We will then copy these to the subordinate CA.
Browse to C:\Windows\System32\CertSrv\CertEnroll to view the CRL and the root CA certificate.
Setting the Issued Certificate Validity Period
The default validity period for certificates issued by this CA will be 1 year. Because this is a stand-alone certification authority, we don’t have templates available to use that we can use to define the validity period for issued certificates. So we need to set this in the registry.
As we’ll only be issuing subordinate CA certificates from this root CA, 1 year isn’t very long. If the subordinate CA certificate is only valid for 1 year, any certificates that it issues can only be valid for less than 1 year from the date of issue – not long indeed. Therefore, we should set the validity period on the root CA before we issue any certificates.
To change the validity period, open Registry Editor and navigate to the following key:
Here I can see two values that define how long issued certificates are valid for – ValidityPeriod (defaults to 1) and ValidityPeriodUnits (defaults to “Years”).
Viewing the Root CA certificate validity lifetime
Open ValidityPeriodUnits and change this to the desired value. My recommendation would be to make this 1/2 the lifetime of Root CA’s certificate validity period, so if you’ve configured the Root CA for 10 years, set this to 5 years. You’ll need to restart the Certificate Authority service for this to take effect.
Setting the Root CA’s ValidityUnits
An alternative to editing the registry directly is to set this value to certutil.exe. To change the validity period to 5 years run:
certutil -setreg ca\ValidityPeriodUnits “5”
Conclusion
Yes this is pretty much a copy n paste of the source, it was so well written and nice to follow, there are just a couple additions I added in where things got a little confusing I hope those might help someone who comes across this.
Much like the source in my next post I’ll also cover setting up a Subordinate Root CA, however I will also cover removing an existing CA from an AD environment before replacing it with the new subordinate. As well as cover some errors and issues I faced along the way and how I managed to correct them. This was part was pretty straight cut so I didn’t have much reason to alter it from the source.
Thanks StealthPuppy.
PS – If you plan on publishing new CRL to be hosted by AD for domain systems, don’t forget to set the DSConfigDN setting on this offline CA.
Open Certificates Snap-in from Remote Management computer.
Connect to remote computers Machine store (Core-IIS).
Right click the Personal Store object (folder) -> All tasks -> Advanced Operations -> Create Custom Request…
Pick Enrollment Policy (Custom Request for 3rd Party CAs) I have internal CA, So I picked AD Enrollment Policy.
Template – Web Server | PKCS #10
Click Details to expand the AD EP – Web Server template properties (Make sure you enter a common name of the host name of the server, and SAN names for any sites you plan to use this certificate for)
Save the Request file.
A) Navigate to your CA sign the certificate. (In my case I have web enrollment, so I simply navigated to my CA web page and picked custom request, then opened the request file in notepad and copied the request contents into the text box of the web request, after issued, saved a copy in Base64 format)
OR
B) Open the CA snap-in, Right click the CA -> All tasks -> Submit a new Request, Save the issued cert file.
Right click the Personal store on the remote Certificates Snap-in again, select All Tasks -> Import.
Your Certificate is ready for use on the Website binding section of IIS.
Creating an INF file to set the certificate properties
[Version]
Signature="$Windows NT$"[NewRequest]
Subject = "CN=SERVER.CONTOSO.COM" ; For a wildcard use "CN=*.CONTOSO.COM" for example
; For an empty subject use the following line instead or remove the Subject line entierely
; Subject =
Exportable = FALSE ; Private key is not exportable
KeyLength = 2048 ; Common key sizes: 512, 1024, 2048, 4096, 8192, 16384
KeySpec = 1 ; AT_KEYEXCHANGE
KeyUsage = 0xA0 ; Digital Signature, Key Encipherment
MachineKeySet = True ; The key belongs to the local computer account
ProviderName = "Microsoft RSA SChannel Cryptographic Provider"
ProviderType = 12
SMIME = FALSE
RequestType = CMC; At least certreq.exe shipping with Windows Vista/Server 2008 is required to interpret the [Strings] and [Extensions] sections below[Strings]
szOID_SUBJECT_ALT_NAME2 = "2.5.29.17"
szOID_ENHANCED_KEY_USAGE = "2.5.29.37"
szOID_PKIX_KP_SERVER_AUTH = "1.3.6.1.5.5.7.3.1"
szOID_PKIX_KP_CLIENT_AUTH = "1.3.6.1.5.5.7.3.2"[Extensions]
%szOID_SUBJECT_ALT_NAME2% = "{text}dns=computer1.contoso.com&dns=computer2.contoso.com"
%szOID_ENHANCED_KEY_USAGE% = "{text}%szOID_PKIX_KP_SERVER_AUTH%,%szOID_PKIX_KP_CLIENT_AUTH%"[RequestAttributes]
CertificateTemplate= WebServer
Compiling the INF file into a REQ fileThe following command-line command will generate key material and turn the INF file into a certificate request.
certreq –new ssl.inf ssl.req
Once the certificate request was created you can verify the request with the following command:
certutil ssl.req
Submitting the REQ file to the CA: if the CA is reachable via RPC over the network, use the following command to submit the certificate request to the CA:
certreq –submit ssl.req
You will get a selection dialog to select the CA from. If the CA is configured to issue certificates based on the template settings, the CA may issue the certificate immediately.If RPC traffic is not allowed between the computer where the certificate request was created and the CA, transfer the certificate request to the CA and perform the above command locally at the CA.
If the certificate template name was not specified in the certificate request above, you can specify it as part of the submission command:
Installing the certificate at the IIS or ISA computer: Once the certificate was issued and is available as a file on the target computer, use the following command to install it.
certreq –accept ssl.cer
The installation actually puts the certificate into the computer’s personal store, links it with the key material created in step #1 and builds the certificate property. The certificate property stores information such as the friendly name which is not part of a certificate.After performing steps 1 to 4 the certificate will show up in the IIS or ISA management interface and can be bound to a web site or a SSL listener.
*UPDATE* Powershell – I love PowerShell. If you’d like some more automated scripts to help with such a task. Please see this Blog Post by Adam Bertram in which he provides a link to his gitHub page with the required scripts.
Thanks Adam for this fine work. I might just make a pull request to make some changes/tweaks to the scripts. Amazing the neat little tricks I learn from reading other peoples code. 🙂
