Basic Setup of a PAN VM 50

Quick Intro

Heyo! so on my last post we went through a basic install and update of a Palo Alto Firewall VM. Now it’s time to setup a dataplane NIC, some zones, some rules to allow some basic internet.

I decided to do some very basic setup of one NIC and was surprised to find I could not get any ping responses either from the firewall, or the firewall making any requests. I had a memory of talking to a smart fellow once before about this, and sure enough…

A Caveat

You have to enable Promiscuous mode on the VMPG the NIC is a member of…

I know it sounds ridiculous and it is, but without it, nothing flows through the PA VM. Quick Update on this, I didn’t like this idea one bit, so to ease the risk I did find something rather interesting: according to this (requires a PA login) this hasn’t been needed since PAN OS 7, I disabled it on my Test network

and the pings dropped… ugh… ok… According to the post it says PAN OS 7 and onward uses this setting by default but can be changed under:

Device > Setup > Management > General Settings

Enabled by default huh… doesn’t seem to be enabled to me…

enable it, commit. Now MAC address changes will take place in this case I did loose connection to my external IP, but pinging from my PA VM to my gateway managed to fix that quickly.

And now sure enough with Promiscuous mode rejected on my vSwitch settings…

Oh thank goodness I can go to bed knowing I didn’t suggest a terrible practice!

Basic Setup

Look at this test network… was using an OPNsense router/firewall, but all these guys are currently shutdown. Lets spin one up and make the PA VM 50 it’s new gateway…

Adding the required Virtual NICs

Then add a new NIC to the PA VM (since it only came with two by default (the first being the mgmt NIC, and the second I connected to my DC)

This should be the second Interface under the PA VM Network Tab.

K looks like we should be good, power on the PA VM again.

Configuring the Interface

Once in the PA Web interface, navigate to Network -> Interfaces.

Again this will be Ethernet 1/2, although it is the third NIC on the VM.

Once we click on Eth1/2 and configure it properly it should show up green as well. I have configured a interface mgmt profile already under Network -> Network Profiles -> Interface Mgmt. Ping checked off, open subnet permitted.

Also a simple Zone, simply named Test.

First thing we have to define is the type (Layer 3), we want a dedicated collision domain please. ๐Ÿ˜€ In this case I’m simply interested in PA to client connection in the dataplane to be confirmed. We will place the NIC in the default router as well as the Test zone.

Then we click on IPv4 to set an IP address up for this layer 3 NIC.

specifying /24 is important here. else any ip address without a defined subnet is treated as a /32. Then under the advanced tab select the interface mgmt profile to allow it to be pingable.

Once committed it should come up green.

and should be reachable by VMs in the same subnet….

Yay it is, but alas this is not enough to give this VM an internet connection. Remember that default router we connected the NIC to, well it has no default route defined, or well any routes for that matter, however because I connected both NICs (my ZewwyDC and Test) into the same router, even without any routes defined, the XP VM can ping the ZewwyDC IP of the PA VM

Security rules and the fact the server and VMs use a different gateway then what the PA VM has for its test IP in that subnet, the responses would never come back to the PA VM anyway, never mind that we didn’t define any security rules to allow it, it was simply because I had the “allow ping all” interface mgmt profiles on all the NICs and connected to the same router that made those ping requests work.

Since I’m not interested at this every moment to move the DCs internet right now, I’ll provide the PA VM a public IP address of it’s own and then create a NAT rule to allow the Test XP VM an internet connection.

The Internet Interface

Also since I don’t want to keep having to “system is shutdown” my PA, I guess this time I’ll populate it with all the VNICs it will ever be able to use… (8)

I did this mainly cause I wanted the last interface on the Web UI to be used for this internet connection

So you might remember my blog post on getting another NIC in my hypervisor host I was going to use it with OPNsense, but since my physical PA has become more useless than online multiplayer only game with all its servers shutdown. So this is to become a replacement as I re-purpose it’s chassis for another epic build I plan to blog about this summer :D!

Interface Mappings:

Well now that I got my MS paint fun out of the way you can get an idea of which NIC I want this PA VM to have one of it’s internet connections on: Eth9

I created a new Zone: Deadly Internet, and connected it to our default router:

Then I configured the public IP I had originally configured for my OPN VM by clicking on the IPv4 tab… and to help make sense of this, some more paint fun ๐Ÿ˜€

I also applied my Allow Ping All Interface Mgmt Profile so I can verify that the interface is not only up (green) but actually reachable, sure enough after a commit… the interface shows green (Also checked off Connected and connected at boot under the VM settings).

Mhmmmm not reachable…. ohhh right, the routers default gateway….

Default Route

Since we are configuring this statically and not via DHCP by our ISP this info is also provided to you.

Network -> Virtual Routers -> Default (in my case) -> Static Routes

So As you can see, anything it doesn’t know, next hop, the IP my ISP gave me as my default gateway.

Commit.

Alright, my attempts to ping it are not successful, which has happened to me the last time I configured all this and I had to reboot the modem, but just before I do that. I’m going to login into the PA VM via SSH and attempt to ping out via that interface:

Alright well last time I got up to this point were I had everything tripple checked, I contract my ISP support and we ended up rebooting the modem which is in bridge mode, Since I assume the MAC address table isn’t being update accordingly or unno its stuck with the old MAC… I suppose I could test this theory by spoofing that NIC with the other NICs MAC…. mhmmmmmm I think I’m gonna pleasure my thought here teehee…

dang it won’t let me change the MAC while it’s on, power off PA VM… set MAC… Spoofed from old OPN sense VM… Power on VM… and…. nope I can’t manually assign it, it’s a dedicate MAC that ESXi won’t allow me to manually assign… so set back to automatic, and boot, if no pings after this rebooting the modem… sigh.

Alright, so pinging my IP still no work even after reboot, I created a firewall rule assuming it was that… nope still no ping response even after commit that, odd cause I didn’t see anything under my traffic log on the firewall itself… so I logged into the firewall again via ssh, but this time I did mange to get a response from my gateway device, wooo yay… ok… so let me try pinging it again externally…. Yes! There it is! had me worried a bit, I had all bases covered so it should have worked, and now it is, w00t!

This is all well and good, however my test VM, on the test switch still won’t be able to reach out, however, it should be able to reach what will become it’s NATed IP address when it comes time to roam the interwebs.

Whoops that wasn’t possible till I expanded the scope of my security rule:

Firewall is very finicky about allowing packets through zones and subnets, so ensure you create rules accordingly. Normally I like to have a deny all rule at the bottom of my list, these would be however above the built in rules:

However there are some Caveats that comes from around doing that which I hope to cover at some point in my Palo Alto series blog posts. For now we won’t go there yet, just be aware of these rules, any packets that reach them are not shown under the traffic tab (IIRC).

However now that we have got all that out of the way, we can finally create the NAT rule (as well as a security rule) we need for getting internet access to our test subnet.

NATing

It’s time to get into the baby potatoes… mhmm who doesn’t move some baby potatoes…. anyway I won’t be covering all the possible NATs that can be accomplished (although I do plan on covering a whole post on those in this series as well), we will do a basic internet NAT here to get us started.

Policies -> NAT -> Add

Pretty straight forward configuration here, anyone from my test subnet from my test zone, will be NATed out my internet connection on Eth9 using the IP address I have assigned it which came from my ISP.

Security Rules!

I hope you liked my pun there, if not, alls good lets setup some security rules…

Policies -> Security -> add

To do this more salable instead of adding the subnet IP by IP range every time, I added an object…

User tab is passed, as we won’t get into that meat today…

Application: Web Browsing, DNS, Ping, ICMP

Service: Application Default

Now Commit, we should hopefully be able to ping out to an external DNS provider, like 8.8.8.8 from our Test subject VMs… muhahahahaha

Boo yea! There we go.. and internet… whoops… forgot to allow DNS lol….

mhmm connection reset ehh, well I guess we need another application defined… or right SSL

finally….

Update

for some reason a couple days later, I noticed I was unable to access Google, even though I had accessed it before, as the above screen shot shows.

Then I created an open rule and i was able to access google, and found out for google to work it’s defined as it’s own App ID (Google-base), I like granular control, but I should be able to select web browsing and that should be able to group sub apps to make my web browsing experience work… On top of that I noticed the same reset connection errors going to Youtube, and reddit… ok this is getting a bit redic…

Here’s my new ridiculous rule just to go to Palo altos own site that referenced a youtube video, google itself, and one reddit result I was interested in… Holy eff man…

Setting the Host Name

Device -> Setup -> General Settings

Here you can enter, the host name, domain name, login banner, timezone and a couple other general settings:

Awesome even though it appeared squished after pasting. It still applied ๐Ÿ˜€

DHCP

It be ridiculous to expect those systems in the Test network to configure themselves, let’s give them a hand with good ol’ DHCP.

Network -> DHCP -> add

Select Interface (in our case Eth2), enter a range in the IP Pools, and Click OK.

Commit, it’s that easy, once created there’s a link to show the IP allocations. ๐Ÿ˜€

If you need to add custom DHCP options, just click the options tab. Which you will for things like the gateway and DNS servers ๐Ÿ˜›

Summary

Well I hoped you enjoyed this blog post, we got some basic things done, some zones, some policies, some new interfaces, objects, yet we haven’t even got into the real meat and potatoes, like wild fire profiles, and URL cat profiles and all those other fun things we will get to soon.

The idea behind the basic first couple blog posts is to just get our baseline going so when it comes to the more complex stuff I have some reference material already available for those that need some reference as to exactly “how I got here“.

In my next post I’ll cover using some of the great features, some of these features will be provided with a standard license, other are license separately for your needs and requirements. Since I got a whole lab bundle for educational purposes I’ll get to post about all the goodies soon. ๐Ÿ˜€

Stay Tuned!

Setting up a Palo Alto Networks VM 50

Intro

Heyo! It would seem the awesomeness of spring has sprung on to us, and that delightful sun’s warm and longer days just feel so awesome in the wake of a cold long winter.

Anyway…. PAN TIME. so I finally got my auth codes I’ve been waiting on. To start you need to get a deploy-able image from a Value added reseller (VAR). Since Palo Alto has no public download for their VM series firewalls. Not a huge fan of their tatics on this one, honestly I believe education should be free and easily accessible. SO this is one area where I do tend to have to give PAN a thumbs down. However when it comes to security, and granular control of said security it is really nice.

Installing PAN VM 50

Deploy the OVA

For my Lab I’ll be using ESXi and an OVA deployment file. So on the vSphere Management, File -> Deploy OVF template. (If you are using the web management, follow this)

In this case my A drive is a mapped drive of all my applications and images, although I did request a newer image than 7.1 as that is rather old and I was hoping for 8.x for 9 even, but I’m hoping I can just update the VM software with my auth codes once I get the VM up and running.

Next you’ll get some details about how the VM will be deployed, simply ensure you have enough resources available to meet the deployment needs.

Click next to assign and name and location for the VM info and VHDD.

I gave it a generic name then the PAN OS number as again, I’m hoping to upgrade it with my auth codes. After that select the datastore to use, I used the local datastore for this VM, and stuck with thin provisioning after that, click next to begin the deployment. depending on your network connections and datastore selection, this time may vary.

Not sure if the copy of the file to my network share got messed, but every-time I deployed it from the share it failed, so I grabbed my IODD device where I had the initial copy, deployed it from there, and it worked.

Yay! Alright time to check its settings.

Alright a couple NICs I was expecting more than that… Anyway normally PAN devices are headless and you can’t see the boot process unless you connect to a serial port, but VMs have direct console, soo I’ll set the NICs not to be connected at the moment as I don’t want them to be in my home NATed network.

Powering on the VM

So disconnected the virtual NICs and booted the VM:

Then I got a login prompt, rather quickly, but don’t be fooled, you have to wait…

After a couple minutes, you’ll get the real login prompt.

Set Admin Password

Now that we got the VM up and running we should change the password:

As you can see it’s not cisco, so short wording doesn’t work. Also just to show that you don’t enter a password at the cli, you enter the word password and it will ask you for them without printing them back to the screen (thumbs up).

Don’t forget to commit. Now we need to figure out how to configure the mgmt IP address… mhmm

Set Management IP Address

So since I wanted to be able to manage this VM easily in my current home network “VM Network” vSwitch on my ESXi host, first I pinged an IP and ensured it was available. Then on the PA VM I ran:

Configure (get into configuration mode)

set deviceconfig system ip-address 192.168.0.55 netmask 255.255.255.0 default-gateway 192.168.0.1

commit

Then I opened the VM settings and enabled the connect:

Then tested my pings again, and success ๐Ÿ˜€

K, so now that we can ping the management IP let’s see if we can access the web interface, and if so hopefully that should be all we need to do at the CLI. I love CLI commands and stuff, but for most management I like GUI’s unless it becomes doing something x number of times, then scripting via the CLI is a necessity.

Access the Web Interface

Once you access the VM’s IP in a local browser you shouldn’t be surprised to be presented with this:

Usual certificate security and warning of un-trusted due to self signed.. yada yada, advanced, proceed….

Mhmmmm I really miss that 7.x Web look, just the right amount of color…

If my upgrades go successfully I’ll be able to show you the new login, a tad more bland….

Awww man, just look at that delightful dashboard, the system info, haha unknown serial in VM mode with no license (yet) ๐Ÿ˜› I like how it even shows my two login sessions (CLI and Web).

As well as of course the usual, PAN Tabs (ACC, Monitor, Policies, Objects, Network and Device) mhmmmm so delightful.

Now my main goal of today and this post is simply to get the VM booted up, but also updated. Now I can’t do that without a license, which I got just a couple days ago. Now sadly I can’t share these with you, but I can tell you how to accomplish the task.

Managing Licenses

Click on the Device Tab -> Licenses

In my case I can’t remember if I had uploaded it to my usual PA login account online, so for now I will be using #2 Activate via Auth Codes.

First things first though, set the DNS servers.. :S whoops lol

Device -> Setup -> Services -> edit -> Primary and secondary DNS servers

So even after that I kept getting communication error message, so I googled.

After that I figured they are doing their usual ways, and locking this down in some other form that doesn’t provide any nice error message to try and stop use of these images if they leak, and it’s extremely frustrating for legit users… not gonna lie.

So I decided after I got my DNS up n running to apply the Auth code again and this time I got a different error, that my auth codes have to be registered to my support account before i can create and register the VM… ughhhhhhh

This as you can see is the real annoying side to any DRM. Let me jump through these hoops and come back to this post in a little bit… :S

Alright, so I logged into the online suport portal, found the section to register my auth codes, did that, then jumped back into the VM web and entered the auth codes again, this time it didn’t complain, the VM showed it was rebooting while the web interface stayed at the licenses section… odd haha I was going to take a snippet of that happening but the reboot was rather quick.

Since I knew the VM had rebooted as I saw it via the vSphere console window, I gave it a couple minutes before navigating to the web interface.

Sure enough after logging in again, I know have a serial number defined on my PA VM. ๐Ÿ˜€ I hope now I can actually check for updates without getting a generic, false error message…

Yes! So many PAN OS’s to choose from…. but sadly no PAN OS 9… or 8.1.x for that matter… Well that sucks I was hoping to be able to play around with TLS 1.3… oh boy… maybe I have to upgrade first?

Upgrading PAN OS on PA VM 50

Sooo I selected 8.0, downloaded and configured into software manager successfully awesome! Install failed, not enough memory…. nice.

Well considering it’s a VM which are amazingly salable in this regard I won’t blame them here, the message is to the point. I’ll just shutdown the VM and up it’s memory…

Device -> Setup -> Operations -> Shutdown Device

Yeap… System is shutdown. lol

Bammmm more memory like that!

You got me again, you can code for the validation, but you can’t code the process to do that for me eh…. they could, they just didn’t want to.. so let me jump through some more hoops…

Dynamic Updates -> Check Now -> Apps n Threats -> Download (8136-5163 at the time of this writing) -> Install

Yay, at least that worked without some issue to overcome. Let’s try that software upgrade for a third time. Third times the charm right?

SO far so good, device needs a reboot, OK. ๐Ÿ™‚

And here it is.. the bland 8.0 login .. ๐Ÿ™

Just no color, no life… just go look and compere the login before and this one, I even liked that they had a soft indent of their logo in the background, made it feel so elegant to this… so minimalist…

As for the software, upgrading to 8.0 did make 8.1 available… but still no 9.0 errr lets upgrade again and see… ooo yeah…. there it is… 9.0!

So I can… Yeeee, I dunno if I’ll do it just yet, but good to know I can when I want to.

Summary

Overall the deployment and use of the PA VM is very good. I’m rather excited to get my SSL inspection rules setup for some stuff… ๐Ÿ˜€ as well as cover other blog posts covering some more in-depth setups and configurations.

In my next post I’ll cover actually setting up some zones and network configurations. or I might even just show how to migrate a physical configuration. In this case since I won’t have a 1 for 1 NIC assignment there would probably be some tweaking required, maybe even before the firewall would accept the config file. but we’ll cover that when we get there. ๐Ÿ˜€

 

OPNSense for Exchange Reverse Proxy

OPNsense and Exchange

Unlike the German blog I reference below, I use a Palo Alto as my main device to handle normal NAT for the OPNsense box’s internet, as well as the NAT rule to allow HTTP Validation (which I covered in my last blog as it was causing me some issues). Another notable difference is I have a dedicated Datacenter zone which has it’s own dedicated NAT rules for internet access, but not direct NAT rules from the outside world (as it should be), which means no dirty double NAT (like it should be). Then once certs are setup, the OPNsense will reverse proxy the HTTPS requests for OWA, and hopefully Active Sync.

First however, I’m going to add a new VMPG network in this I called it (DMZ) and assigned it a VLAN (70). Since this is ESXi running on an old desktop with only 1 NIC (initially) I have to utilize VLAN to make the most out of the lack of physical adapters. Then I’ll need to create a sub interface on my Palo Alto, with the same VLAN tag of 70, and give it an IP address of 192.168.16.1/24. This will be the subnet of the DMZ. Now you maybe wondering why I’m putting the subinterface and IP on my Palo Alto and not on the OPNsense VM, the reason for this is I use Palo Alto firewall to manage all the other networks in my environment. so all known routes will take place there.

The whole idea here is to get Active Sync to work, and the PANs do not support reverse proxying. So the idea is to have a NAT rule allow port 443 (HTTPS) from the internet to the OPNsense vm. so after the redesign I have 1 OPNsense VM (192.168.16.10/24 – VLAN 70) and a new DMZ VR, with a new subinterface on the PAN (192.168.16.1/24 – VLAN 70)

 

and the PAN…

So I added static routes between my Zewwy network and my new DMZ, as you can also tell based on the mgmt-interface profiles, I only allowed pinging the gateway, so the OPNsense ICMP request shown above to succeed.

I had to set the default gateway on the OPNsense VM via the CLI first in order to gain access to the OPNsense web UI

route add default 192.168.16.1

change IP based on your gateway. Then once in the UI go to:

System : Gateways : Single : Add

This was required to keep the default route persistent after reboots.

 

Well I was getting a bit stuck so decided to google a bit and sure enough a blog to the rescue, odd enough, it’s a German blog. Ich can ein klein beste duetch aber nicht sehr gut. So I picked translate…

I thought… oooo he’s on a VM on ESXi too, and installing VMtools nice… goto plugins… don’t see a list like him, and thought… Shiiiit, my OPNsense have no internet…

Sooo, I decided to give my OPN VM internet access to get updates and plugins (best move). I won’t cover this but basically required me to add a default route to the DMZ VR, create NAT rule and Sec rule, test pinging internet IP from OPN, and success.

OK so.. Now that the PAN is all setup, and we have tested our NAT rule for internet for the OPNsense VM… let’s just go over the OPNsense install…

OPNsense Install

On your Hypervisor or Hardware of choice, in my case ESXi New VM. ๐Ÿ™‚

In this case I know I/O is not a big deal so the local ESXi datastore will suffice for this VM:

Pick VM V8 (cause I’m still on ESXi 5.5)

FreeBSD 64Bit (for some reason we won’t be able to pick EUFI)

CPU: 2, Mem: 2GB, 1 E1000 Nic in the DMZ

LSI Logic Parallel SCSI, New 20 Gig Thin Prov Disk, Create VM.

Edit VM settings, remove floppy, Boot Options Force BIOS.

Open Console, and Boot VM. Disable Disekette A:

Advanced, IO Device Config, Disable All (its a VM we don’t need these)

Now, Select the disc part and mount the OPNsense ISO for booting:

Boot it! by Pressing F10 in the VM and save BIOS settings:

Mhmmmmm so delightful…. and now we let it load the live instance, while this live instance is good enough to start using, I don’t exactly feel like loosing my settings every-time it boots and having to remount my ISO from my local machine… so we’ll install OPNsense by logging in with the installer account:

As you can see it’s assigned our one and only NIC the LAN settings, to ease our deployment and the above section I striked out, we’ll be assigning the interface the WAN value. ๐Ÿ˜› anyway logging in the with opnsense password.

Mhmmm just look at the old style look, make me juicy…

*NOTE* if installing EFI based the input here may freeze… googling it quickly I only found one reference to the issue by a comment by eugine-chow

  1. Press CTRL + C (This exists the installer)
  2. re-logon on as installer account (This resumes the install with keyboard control

OK, Let’s go! Accept, Guided instillation! Pick Disk, for simplicity and low disk, we’ll just pick MBR… and look at that installation go… mhmmm humbling…

Set a root password:

Now reboot and unmount the ISO, now the boots quicker and our settings will be saved! First things first, assigning NICs… or should I say our one NIC, login in as root via the console. Press 1 to assign interfaces. Even though I showed VLAN assigning above that is used by the ESXi hypervisor and thus I select no to VLAN tagging here, and then specify em0 as my WAN NIC:

Now in my case it wait a long while at Configuring WAN interface, cauuse it’s defaulting to DHCP, and there’s no DHCP in the subnet… ugh, I don’t know why they don’t ask for IP assignment type in this part of the wizard…

now Select option 2 to set IP which should have been part of the wizard in part 1…

Now that is out of the way, we can access the OPNsense web UI from our Datacenter Laptop/VM… you won’t be able to ping it, but the anti-lockout rule will be created on the WAN rules so…

Follow the config guide… only important part being the upstream gateway:

And of course in my case since it’s being NATed the RFC1918 Networks will be unblocked as it’s using one ๐Ÿ˜› and NO LAN IP.

First order of business is going to be moving th eport off of port 80 as that will be needed for Lets Encrypt Validation (only cause my DNS provider doesn’t have the API for DNS validation yet).

Finally time for OPNpackages

OPN packages

Bammmmm that was easy!

OK, Firewall, since my OPNsense only has WAN, and it’s open, all security will be handled by the Pal alto, so I don’t want to open HTTPS from the internet to my the OPN sense just yet, till we create the other requirements.

HAPRoxy

Create a Real Server, in this case this will be our Exchange server as in the topology.

Now for a Backend Pool

He doesn’t mention any other settings so I just clicked save… I probably should have named the Backend pool better but meh.

Following the German guide I was a lil upset cause I was running OPNsense 19.1, it seems they changed the HAProxy options, however I did manage to figure it out after a while…

ACLs now Conditions

Go to Services -> HAProxy -> Rules & Checks -> Conditions

Add a condition, for testing I kept it simple as the blog I was following:

and then…

Actions are now Rules

Go to Services -> HAProxy -> Rules & Checks -> Rules

add a rule:

Frontends are now Public Services

Go to Services -> HAProxy -> Virtual Services -> Public Services

Add a public service:

Enable The HAProxy Service:

OPNsense Firewall Settings

Even though this VM wasn’t routing any traffic, I still had to create an allow rule under the firewall area before my PA firewall would see completed packets:

first attempts, gave site unavailable and my PA logs showed…

On OPNsense:

Firewall -> Rules -> WAN -> Add -> TCP (HTTPS) Allow + TCP (HTTP) Allow

 

basically allowing all TCP packets, after applying I was able to get the OWA page from my Windows 10 VM in the datacenter:

so now it’s going to basically be creating a NAT rule on the PA to see it from the internet… but before I get to that…

Certificates!

Now that I covered getting Let’s Encrypt to work behind a Palo Alto firewall I should be able to complete this part!

Lets Encrypt

Enable the service, and the extension to HAproxy, hit apply
Create an Account

I did select my exchange front end, even though I didn’t show it here, then I created a Lets Encrypt Frontend as exchange won’t deal with HTTP:

LetEncrypt FrontEnd

Well lets test this out… Create a Certificate..

Click save changes, but just before we click Issue Certificates, lets tail the log (/var/log/acme.sh.log) to see the process… If you try to open it before you click issue it will fail cause the file only gets created on first run… so click issue and then quickly open the log file with tail command… if it gets stuck at ACCOUNT_THUMBPRINT something went wrong… and of course… something went wrong… ugh……

Mhmmm sure enough… Domain Key error on second try…

But if I alter my HTTP validation to…

and attempt to issue the certificate then I see in my acme.sh.log its success…

but the UI will still show validation error even though it was issued successfully…

Let me see if I can at least assign this cert even though it may not be automatic…

seems like it… lets test…

Well at least that’s something… I’m not sure if the auto renewall will still work… if so I’m not sure exactly what the point of the HA plugin really is… I mean if you can specify the normal WAN and port 80 to validate the certs and seclt the cert to use on the public service… figured it work none-the-less right?

Well I guess well find out… now there one last thing I want to cover… but I’ll do that when I get it figured out again…

For now I’ll post this blog post as is casue it is getting rather long.

Cheers! OK NM I did it quickly…

Blocking the ECP

Under OPNsense HAProxy go to Conditions:

Then Rules:

Then Edit your Public Service settings and add the rules:

Finally test access to ECP via the Proxy…

Ahhhh much better… ๐Ÿ˜€ something not mentioned by the German blogger makes me wonder if I can access his ECP.. mhmmm

Alright that’s all for tonight. ๐Ÿ˜€

Lets Encrypt HTTP Validation
And the Palo Alto Firewall

The Story

This…… this one…. this one drove me NUTS! for almost a week…. it was a lil mix of a perfect storm I guess… but lets start from the beginning shall we..

So a couple weeks ago i wanted to get active sync setup for my exchange server (Checking OWA sucks)… so I was sought after OPNsense for my open source firewall of choice.

I started following this German blog post, and I hope to have that blog post up very soon as well (sorry I don’t usually get hung up like this).

My setup was pretty much exactly the same however I was getting hung up on the plugin not validating my scripts over HTTP. See the full pain details here on github, anyway, I did finally manage to get my OPNsense server behind the NAT rule to finally succeeded behind my Palo Alto Firewall (by basically opening up the rule way more then I ever wanted to) so I knew! I knew it was the Palo Alto blocking still somehow… but how I couldn’t make sense so I wasn’t sure how to create my Security rule.

First try

My first try was exactly like the github issue describes, was failing on domain key creation, this failed even on my OPNsense with a Public IP and all rules exactly as the OPNsense basic guide states to set it up.

When Neilpang (the main script writer/contributor) said ti was fixed and no commit was applied, I tried again and it worked, I can only assume this was due to the fact DNS may not have replicated to the external DNS servers lets encrypt servers are configured to use when I first made my attempts at a cert validation.

That didnt’ explain why every attempt behind my Palo Alto with a NAT and security rule would fail…

The Palo Alto

I love these things, but they can also be very finicky. to verify my rule I had used my IIS Core VM (That I’ve used in previous posts on how to manage Windows Server Core) along with the HAProxy plugin on OPNsense to basically move the requests from the NAT rule of the Palo Alto but really serve up the IIS website of my IIS server. Not to my amazement, but sure enough I was able to access the IIS website from the internet, so my security rules and nat rules on the Palo ALto are working fine, as well as the security rules on the OPNsense server…. so what gives? Why are these HTTP Validation requests failing??

Again, as stated above I knew it was the Palo Alto from opening up the rule completely and it working, but I figured it was the issue even before I did that… but opening up the security rule completely is not the answer here… like it works but its far to insecure…

So I managed to talk to a friend of mine who happens to be realllllly good at deploying Palo Alto as he does it for a living. I basically describe my issue to him, and ask him if there’s anything he can think of that might be a problem. (I’ll hopefully be having a couple more Palo Alto blog posts as soon as I can get my proper licensed VM) To my actual amazement he goes on about this one setting you can use inside security rules and about a story about when it caused him grief…. go figure, he’s experienced it all!

What was it?!?!?!

Alright so here’s my rule I intially had, which was causing failures of the let’s encrypt OPNsense plugin…

AS you can see nothing really special, until he told me about… PAN DSRI or Palo Alto’s Disable Server Response Inspection you can check the link for more details. Now the funny part is that post covers better performance…. in my case, it was simply needed to work! And all it was, was a checkbox….

once that checkbox was selected, the rule adds a icon to it.

I was able to click Issue certificates on the OPNsense Lets Encrypt plugin, and I got some certs! I’m ready to now add the Let’s Encrypt HAProxy plugin integration and set these certificates for backend services… like my ActiveSync… or OWA… Ohhh exciting stuff!

Man that feels good to finally have that sorted! Wooooo!

WMI and the WBEMTEST

WMI and the WBEMTEST

I’ll try and keep this post short, as I have many things to catch up on, and this just happened to be one of those things I haven’t done in a while and had to do today for some newer servers that have been configured.

Now since I hadn’t blogged about this myself I went out to the interests to give me a good reminder on how to accomplish this. My first hit was, Sysops… and I usually really like this site…. well till i read this…

“Access denied should be self-explanatory. The credentials you use must have administrator rights.”

Ughhhhh I’m sorry what did you just say? No I don’t think so, WMI maybe, by default, restricted, but it doesn’t require such drastic permissions to utilize.

My second find was a lot nicer, in particular telling you how to manage those permissions, without ahem need administrator access lol.

So lets follow along shall we! so much for short..

First order of busy-nas is creating a user:

Of course WMI being Windows Management Interface, means I’m making obviously a windows domain user. Nothing special, especially no admin.. ๐Ÿ˜›

Again, nothing special here. Alright now I need two servers, well I guess in this case the server being monitored is sort of like a client… ugh anyway…

I guess fo r now I’ll just login to my exchange server and wmi query another server to test out first off… mhmm all I have besides that are core servers, oh boy ok… I think I’m going to need to spin up a new testing server one second…

OK all basic settings…

remove floppy boot into EUFI:

Boot system… attach disc from local host…

lets find us some windows erver 2016…. bug CD-ROM stuck “connecting”…
Close vSphere, reopen console, try again…

always loved this trick over uploading a ISO to a datastore….

Ahh modern Windows still giving off that great nostalgic feel.. ๐Ÿ˜€

yada yada, setup, vmware tools, and join domain, you get the jist of it.

Ping and the Firewall

First order of Business Ping and the Firewall!

Ahh yes connectivity verified (I knew it was good cause I joined the system to the domain, but I like ping… just nothing like a good ICMP) good thing that m is not a u….

Anyway time to run WBEMTEST, bet the first attempt fails cause the firewall again…. hour glass… and (not responding) yeah…. sounds like a stupid firewall…

What?! no way RPC error… lol I totally saw this coming cause again a default server installation doesn’t allow these connections through the firewall by default.

This is a bit old, but lets see if it still works…

Amazing it worked… but yes this was just to verify connectivity through the firewall… so…

WBEMTEST Testing WMI with Least Privileges

OK now that we verified connectivity to the wmi stack with wbemtest using our admin account, lets do it again as a normal domain user. Just to validate these credentials were OK as a standard user i logged into a normal workstation with it, if you want to protect this even further you’d use GPOs to disallow this account local logon. Anyway…

What?! Access denied… lol again expected.. now instead of granting this account admin access, which is overkill, lets grant it the basic enable and remote access on the WMI object… so back on the server we want to be monitored via WMI…

Hope that was easy enough to follow without even saying anything.. anyway lets try that connection again…

Try 2, Scale-able

Mhmmm access still denied… lets see here

This is how I normally do it for a monitoring account anyway cause it usually needs more permissions when mointoring a server so lets try it that way… revert the direct permissions… and grant performance group access…

Now lets add wmi reader account to the dcom groujps and the performance monitor group and reboot the server…

Server rebooting, back up, and lets test that connection again on wbemtest!

and….

Bazzaaaaaa! An account thats not a admin anywhere with permissions needed to monitor your server with WMI! Use these accounts on software such as PRTG, Splunk, Zenoss, etc etc.

Hope everyone enjoyed this tutorial on WMI configuration and testing. ๐Ÿ˜€

Palo Alto VPN (GlobalProtect)
Part 5 โ€“ Rules, Testing, Troubleshooting

Intro

In this 5 Part series I covered all the requirements to configure Palo Alto Network’s GlobalProtect VPN:

1) Authentication, Auth Profiles and testing them.

2) Certificates, Cert Profiles, SSL/TLS Profiles and creating them.

3) Portals, what they do and how to configure them.

4) Gateways, what they do and how to configure them.

This part will cover the security rule required, and a little troubleshooting steps along the way.

Things not Covered

I didn’t cover creating DNS records, as again, these come down to your own DNS provider and whatever tools and portals they offer to manage those.

I don’t cover configuring the interfaces (public facing or internal), I don’t cover the virtual router and routes. All these are assumed to be handled by the administrator reading these guides.

I don’t cover installing the client software, if you have the certificates installed on the client devices (Required), it’s simply navigating to the portal address with a supported browser and downloading the installation packages (.exe for windows).

For giggles, I tested navigating my portal from my phone, it did prompt me for my certificate (the VPN was working well) yet after selecting my certificate I got a connection reset error on my browser and checking the Palo Alto Firewall logs (Monitor tab -> traffic) I indeed saw the Deny traffic and action reset-both action… why this is, even though the application was identified correctly as web-browsing and that was enabled in the rule, it wasn’t being allowed by my rule and instead was being denied by my deny all rule. I”m not sure exactly why this is, however I don’t have intentions of accessing my portal web page anytime soon, so for now I’ll ignore this as I use IPsec XAuth RSA on my android device.

I have also noticed that for some reason with Samsung Android I can’t seem to get this VPN setup to work, from quick google searches people seem to say it’s due to packet fragmentation somehow. I haven’t yet had the chance to look into the nitty gritty of this issue just yet, but when I do it will be it’s own blog post!

I also don’t cover installing the completed certificates onto end devices as again this comes down to the end devices being supported by the administrator configuring Global Protect and is outside the scope of this guide.

The Security Rule

As you can tell pretty simple, anyone from the internet (I could be connecting from anywhere, and my IP address changes on my phone all the time, random access points etc) to my public IP address which hosts my portal and gateway, and the required applications (IKE, ipsec-esp-udp, and the SSL and web-browsing) again I haven’t exactly figured out the portal web-page loading issue just yet.

My Phone Config

In my case I do run an Android phone, running : 8.0.0: Kernel 4.4.78

The OS is some H93320g couldn’t find much but this about it

For the most part I install both my Offline-Root-CA and my Sub-CA certificates on my phone. Which can be found under (General -> Lock Screen & Security -> Encryption & Credentials -> Trusted Credentials (Instead of CA’s who knows?) -> User (Both Should be listed here)

Then Installed the User certificate with the private key, which then shows up under (General -> Lock Screen & Security -> Encryption & Credentials -> User Credentials (Instead of User Certificates?)) The other annoying part is once you have the certificate installed, this area doesn’t allow you to see the certificate details, you can see them under the area mentioned above, but this area…. nope.. :@

Once the certificates are installed, it simply comes down to configuring the VPN settings. (Settings -> Network -> VPN -> BasicVPN -> Click the plus in the upper right hand corner. Then)

Name: Give it a meaningful name

Type: IPSec XAuth RSA

Server Address: The Address defined in Part 3 -> Agents -> External Gateways

IPSec User Cert: The User Certificate you installed and verified above

IPSec CA Certificate: Don’t verify server (Which is probably why I didn’t need the above server address in the gateway certs as a SAN)

IPSec Server Certificate: Receive from server

Then enter a username and password for a user you defined to be allowed per your Authentication Profile you created in Part 1.

You shouldn’t have to define the advanced settings as those should defined to the client from the gateway config we created in Part 4.

Summary

If done correctly you should have a successfully, you should be able to see all the parts play out in both the traffic logs, and the system logs…

System:

Traffic:

That is pretty much it, if you have a failed connection do the usual step by step troubleshooting starting with connectivity, you should be able to see the access attempt from the device in the traffic logs, if they are being blocked by rules, adjust them accordingly.

If you verified all other things, it maybe your chain, or you are enabling extra security like verifying the server certificate than you chain would have to be different then presented here, probably all certificate including the portal and gateway certs being signed by the sub CA completely, then all certs will be trusted by all devices. I’ll admit this isn’t the cleanest setup, but it’s the closest to a bare minimum install of Global Protect using your own internal PKI.

I hope this guide helps someone. ๐Ÿ˜€

Palo Alto VPN (GlobalProtect)
Part 4 โ€“ Gateways

Intro

The Gateway is pretty much exactly as it is named, the gateway where you get a virtual connection to tunnel into the network.

Requirements:

1) And Interface with a Public IP address.
2) Certificates (Covered in Part 2)
3) Authentication Profile (Covered in Part 1)

Configuration

On the Palo Alto Firewall go to Network -> GlobalProtect -> Gateway

Under General give it a Name and define the interface in which has your Public IP address. *Note* The IP address can be left as none, this will work fine if your interface gets its IP address via DHCP, if you have static the static IP address should be populated from the drop down and can be selected. The Appearance section allows you to alter the web login portal that can be used to download the GlobalProtect client software.

Under Authentication Select a SSL/TLS Profile which contains the certificate which will secure this portal)

Then click add under Client Authentication and add the Auth Profile which states which users are going to be allowed to authenticate through this portal. Then select a Certificate Profile. (Covered in Part 2)

This is the first section that actually looks different than the portal configurations, under the Agent section the first area is the tunnel settings. This is where you define which tunnel interface (i picked the default, you may need to create additional tunnel interfaces if doing multiple portal/gateway configurations). In my case I was setting up tunnel based IPsec type VPN.

I left all the Timeout Settings as default, then moved onto client settings. Here we define any particular users, what OS they are allowed, and what IP addresses they are to be assigned (basically acts as a dedicated DHCP for the virtual tunnel interface when the VPN is established).

 

Next, under Network Services define the internal DNS server and WINS servers, as well as the DNS suffix users who connect will use, this will allow them to work as if they were locally at work.

In my case I didnโ€™t have to deal with HIP Notifications or the Satellites section. ๐Ÿ˜€

Thatโ€™s it for the Gateway, this unfortunately is not enough and we still need to define our Security Rules. Luckily since the Portal utilizes a public facing interfaces, we don’t have to deal with any NAT rules as connections are routed through the virtual tunnels that get created pretty much through the settings we defined in this part. ๐Ÿ˜€

As you can tell these post are a lot shorter as the hardest parts is building the pre-requisites.

Till Part 5, Cheers!

Palo Alto VPN (GlobalProtect)
Part 3 โ€“ Portals

Intro

The Portal is pretty much exactly as it is named, the portal where you fist connect to, validate you have the certificate to establish a secure communication to send your credentials over and tell your device what gateway to establish to tunnel connection with.

Requirements:

1) And Interface with a Public IP address.
2) Certificates (Covered in Part 2)
3) Authentication Profile (Covered in Part 1)

Configuration

On the Palo Alto Firewall go to Network -> GlobalProtect -> Portals

Under General give it a Name and define the interface in which has your Public IP address. *Note* The IP address can be left as none, this will work fine if your interface gets its IP address via DHCP, if you have static the static IP address should be populated from the drop down and can be selected. The Appearance section allows you to alter the web login portal that can be used to download the GlobalProtect client software.

Under Authentication Select a SSL/TLS Profile which contains the certificate which will secure this portal)

Then click add under Client Authentication and add the Auth Profile which states which users are going to be allowed to authenticate through this portal.


Under the Agent section is where you define the which users group use which gateways. As well as which CA they use. *NOTE* The address defined as the gateway should created on your external DNS provider. Also it seem it is not required as a SAN on the certificate.

In my case I didn’t have to deal with Satellites section. ๐Ÿ˜€

That’s it for the Portal, this unfortunately is not enough and we still need to define our gateway as well, which ironically in a simple setup such as in my case and examples as a lot of the same steps.

As you can tell these post are a lot shorter as the hardest parts is building the pre-requisites. I also don’t cover creating your external DNS records as that comes down to your own DNS provider and the tools and services they provide.

Till Part 4, Cheers!

Palo Alto VPN (GlobalProtect)
Part 2 โ€“ Certificates

Certificates

In my previous post I covered recovering a downed CA, cause it will be needed for this section of the GlobalProtect tutorial.

Step 1) Importing the CA Certs

We need to add all the CA certs that are involved in completing the chain, so this includes, the Offline-Root-Ca, as well as the Sub Ca.

Adding the Sub CA cert:

Device -> Certs -> Import -> Base64 cer file

Step 2) Generating a CSR

Generate a a Sub CA Key for the PA to handle the Gateway certs, afterwards generate a Gateway certificate as well.

Click generate:

Click Generate

export the CSR, for some reason the latest Chrome causes a constant refresh, argggg had to export the CSR via IE, gross….

Navigate to your CA’s signing Web page (the Sub CA in this case), open the CSR in notepad and paste the results, and select Sub CA for the template:

Then save as Base64 type cert, and import back into the PA firewall, if successful will look like this:

Also import Offline-root-ca cert to complete the chain

Step 3) Certificate Profiles

Alright time for Certificate Profiles

Add all the Certs

Step 4) SSL/TLS Profiles

Create a SSL/TLS Profile:

Name it whatever, pick TLS 1.2 as min and max, and select the PA Sub CA we created earlier.

Step 5) Create User Certificate

Step 5.1) Create Template on CA

Then under Cert Templates, right click it, and duplicate

5 Years, i don’t like doing this often

Signature and encryption, check off include symmetric allowed by subject, min key size of 2048 and key is exportable

Along with the default, check off MS RSA and AES, and RSA SChannel

Subject Name, Supply in the Request, it will complain about the security risk, accept them. (Normally you’d create the certificates at the client machines, but in this case I am doint it the “wrong way” by having a global user certificate)

Click Apply.

If you require additional permissions apply them now, by default domain admins have full control, and domain users have enroll rights.

Step 5.2) Generate User CSR

With the Template configured, lets create the User Cert for the VPN, in this case we generate the CSR on the PA, but since we made the key exportable, we can export the certificate with key to be installed on the end device (instead of the CSR being generated on the device and then signed, and the public key being installed on the portal, which is the right way… hopefully I can get that, but the toughest part is generating certificates on phones, have to learn each devices OS on how to do it)

On the PA Device, Certs, Generate

*NOTE* I noticed that with the latest Chrome that when you attempt to export any certificate it just seems to refresh the page, sadly the only work around I have is to use IE… Ugh….

Open the CSR in Notepad, navigate to your Sub CA’s certificate signing page, sign the certificate.

*Secrete enable remote management on IIS Core*

lol, I was wondering why i couldn’t see my Template in the web interface, so I looked up my own very old blog post (3rd one I believe) and I realized I forgot to publish it, like I did the Authentication Session Template. Durrrr, then it kept complaining about https for cert destro (makes sense) but since I had a core subca, I couldn’t connect to the IIS remotely, then I found this, saved my bacon, and followed this to enable HTTPS, Then finally…

then Import it on to the Firewall,

it should look like this

In the next section I’ll cover configuring the Portal and Gateway settings. ๐Ÿ˜€

Resolving a down CA

Downed CA!

First off I wanted to address that this wasn’t the intended post of today, this was suppose to be part 2 of the Global Protect post, which is the second dependency; Certificates. However I recently had to get a new cert at work, and discovered this same issue as I had deployed my CA following a decent blog tutorial online by StealthPuppy, and sure enough the same sight provides a follow up on how to fix a mistake made, we all make them and these are great opportunities to learn, so lets get learning!

So you might happen to open up the Certificate Authority Snap and point it to your CA server, to find this…. it’s shutdown….

Fear not StealthPuppy has given us some helpful tips to resolve this!

The Issue

You might find your certificate authority, in this case, a subordinate certificate authority that is not started, perhaps after a server reboot. Attempting to start the CA, results in this message:

The revocation function was unable to check revocation because the revocation server was offline.
0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE)

Which looks like this:

In the Application log on the subordinate CA, I can see event id 100 from source CertificationAuthority:

Active Directory Certificate Services did not start: Could not load or verify the current CA certificate. stealthpuppy Issuing CA The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE).

As well as, event id 48 from the same source, CertificationAuthority:

Revocation status for a certificate in the chain for CA certificate 0 for stealthpuppy Issuing CA could not be verified because a server is currently unavailable. The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE).

Certificate 0 is the subordinate CAโ€™s certificate, issued by the offline Root CA.

I bypassed this portion of the blog as I didn’t want to have pictures of before the next required step soooo….

The Workaround

Of course, you probably want to get the CA up and running as quickly as possible. The easy way to do that is to disable CRL checking with the following command on the CA server:

certutil โ€“setreg ca\CRLFlags +CRLF_REVCHECK_IGNORE_OFFLINE

Run this from an elevated command prompt and you should now be able to start the CA and get on with the business of troubleshooting.

Perfect, now lets fix this!

The Cause

My CRL was online as it is available in Active Directory (for domain joined machines) and via HTTP at subca.zewwy.ca, an alias of the subordinate CA. Iโ€™ve tested that I can retrieve the CRL by putting the HTTP path into a browser and Iโ€™m prompted to download a file.

Through having spent some time recently with setting up an Enterprise PKI in my lab and for a project, Iโ€™ve come to know the command line tool certutil.exe. This tool is available in all versions of Windows and should be the first tool to use to troubleshoot and manage certificates and certificate authorities on Windows.

Certutil can be used to perform many functions, one of which is to verify a CRL. I know the path to the CRL file because I can view the CRLs on the file system (in C:\Windows\System32\certsrv\CertEnroll) and Iโ€™ve previously configured CRLs for both CAs.

To verify the CRL, use the -URL switch with the HTTP (or LDAP) path to the CRL:

certutil -URL "http://subca.zewwy.ca/CertEnroll/OFFLINE-ROOT-CA.crl"

This will display the URL Retrieval Tool that shows that the CRLs are able to be contacted and show a status of OK.

*NOTE* I discovered this works directly on the Windows Core Server, if you happened to be running Core (I do as I love optimization, specially when you have to work in lab environments). Except as you might have noticed from the screenshot, the radio buttons are all messed (wonder what library handles that, that wasn’t in core…)

However, if we load a target certificate, in this case, the subordinate CAโ€™s cert, we can start to see why we have an issue with the CRL.

Select the certificate for the subordinate CA that has been previously exported to the file system (in C:\Windows\System32\certsrv\CertEnroll) โ€“ click Select, open the certificate and click Retrieve again. This time, we can see a new line that shows that the base CRL for the subordinate CAโ€™s certificate is Expired. (Unfortunetly I had to simply use his source images as in my case I had to also correct my CDP locations on my sign Sub CA Certificate as I had mentioned in my initial Setup Offline Root CA blog post, but I guess I didn’t do it in my lab, so I sort of had to fix to birds with one stone, the CPD locations in my cert by reissuing that, and the offline root ca CRL file, so my additional steps were a bit beyond these exact steps, I also didn’t take snapshots as I wanted to get back on pace with my blog post)

The CRL for the subordinate CAโ€™s certificate will come from the root CA, so weโ€™ll need to check that CRL. Open the CRL file (C:\windows\system32\certsrv\CertEnroll\stealthpuppy Offline Root CA.crl) โ€“ double-click or right-click and Open. Here we can see the CRL information, including the next publishing time (Next CRL Publish).

At the time of troubleshooting, this date was in the past and because the Root CA is offline and the CRL is hosted on a different server (the subordinate CA), this particular CRL will never receive an update. So, when the subordinate CA has rebooted, it has checked the Root CAโ€™s CRL and found it expired. Hence the certification authority service wonโ€™t start.

How To Fix It

Now we know why the certification authority service wonโ€™t start and an understanding of why the CRL is offline, even if the wording doesnโ€™t match the symptoms. If the error message had told me the CRL had expired instead of being offline, I might have saved some troubleshooting time. We now know that we need to re-publish the CRL from the Root CA.

Start the offline Root CA, log into it and open the Certification Authority console. We will first want to ensure that the CRL publication interval is extended so that we donโ€™t run into the same problem in the near future. Open the properties of the Revoked Certificates node to view and set the publication interval. The default interval is 1 week, obviously too often for an offline Root CA.

Instead, set this value to something suitable for the environment you have installed the CA into. Remember that youโ€™ll need to boot the Root CA and publish a new CRL before the end of this interval, otherwise, youโ€™ll have exactly the same issue.

Now publish a new CRL โ€“ right-click the Revoked Certificates node and click All Tasks /Publish.

Copy the updated CRL (from C:\Windows\System32\certsrv\CertEnroll by default) from the Root CA to the CRL distribution point and overwrite the existing CRL file (C:\Windows\System32\certsrv\CertEnroll again on my subordinate CA).

Now if we again use certutil.exe to verify the CRL, it comes up roses:

To ensure that the subordinate CAโ€™s certification authority service will start, re-enable CRL checking:

certutil โ€“setreg ca\CRLFlags -CRLF_REVCHECK_IGNORE_OFFLINE

If you have re-published the CRL from the Root CA correctly, the service should start and you can then shut down the Root CA. Then open Outlook and put a reminder in the calendar for a week before the CRL expires again.

Conclusion

Iโ€™ve had this issue with an Offline CRL a few times now and not really understood what the issue is until I took the time to troubleshoot the issue properly. I donโ€™t spend that much time with an enterprise PKI and itโ€™s easy to underestimate the complexity of setting up AD Certificate Services correctly.