Exporting OPNsense HAProxy Let’s Encrypt Certificates

You know… in case you need it for the backend service… or a front end IDS inspection… whatever suits your needs for the export.

Step 1) Locate the Key and certificate, use the ACME logs!

cat /var/log/acme.sh.log | grep “Your cert”

*No that is not a variable for your cert, actually use the line as is

Step 2) Identify your Certificate and Key

Step 3) run the openssl command to create your file:

openssl pkcs12 –export –out certificate.pfx –inkey privateKey.key –in certificate.crt

Step 4) use WinSCP to copy your files to your workstation

*Note use SFTP when connecting to OPNsense, for some reason SCP just no worky

April 2, 2019April 2, 2019

Zewwy has not one but two Epiphanies

The Story

Nothing goes better together than a couple moments of realization, and fine blog story. It was a fine brisk morning, on the shallow tides of the Canadian West… as the sun light gazed upon his glorious cheek… wait wait wait… wrong story telling.

The First Epiphany

First to get some reference see my blog post here on setting up OPNsense as a reverse proxy, in this case I had no authentication and my backend pool was a single server so nothing oo-lala going on here. I did however re-design my network to encompass my old dynamic IP for my static one. One itsy bitsy problem I’m restricted on physical adapters, which isn’t a big deal, with trunking and VLAN tagging and all that stuff… however, I am limited on public IP addresses, and the amount of ports that can listen on the standard ports… which is well one for one… If it wasn’t for security, host headers would solve this issue with ease at the application layer (the web server or load balancer) with the requirement of HTTPS there’s just one more hurdle to overcome… but with the introduction of TLS 1.2 (over ten years now, man time flys) we can use Server Name Indication (SNI) to provide individual certs for each host header being served. Mhmmm yeah.

This of course is not the epiphany… no no, it was simply how to get HAproxy plugin on OPNsense configured to use SNI. All the research I did, which wasn’t too much just some quick Googling… revealed that most configurations were manual via a conf file. Not that I have anything against that *cough Human error due to specialized syntax requirements*… it’s just that UIs are sort of good for these sort of things….

The light bulb on what to do didn’t click (my epiphany) till I read this blog post… from stuff-things.net … how original haha

It was this line when the light-bulb went off…

“All you need to do to enable SNI is to be give HAProxy multiple SSL certificates” also note the following he states… “In pass-through mode SSL, HAProxy doesn’t have a certificate because it’s not going to decrypt the traffic and that means it’s never going to see the Host header. Instead it needs to be told to wait for the SSL hello so it can sniff the SNI request and switch on that” this is a lil hint of the SSL inspection can of worms I’ll be touching on later. Also I was not able to specifically figure out how to configure pass-though SSL using SNI… Might be another post however at this time I don’t have a need for that type of configuration.

Sure enough, since I had multiple Certificates already created via the Let’s encrypt plugin… All I had to do was specify multiple certificates… then based on my “Rules/Actions/Conditions” (I used host based rules to trigger different backend pools) zewwy.ca -> WordPress and owa.zewwy.ca -> exchange server

and just like that I was getting proper certificates for each service, using unique certs… on OPNsense 19.1 and HAProxy Plugin, with alternative back-end services… now that’s some oo-lala.

My happiness was sort lived when a new issue presented it’self when I went to check my site via HTTPS:

The Second Epiphany

I let this go the first night as I accepted my SNI results as a victory. Even the next day this issue was already starting to bother me… and I wanted to know what the root of the issue was.

At first I started looking at the Chrome debug console… notice it complaining about some of the plugins I was using and that they were seem as unsafe

but the point is it was not the droids I was actually after… but it was the line (blocked:mixed-content) that set off the light bulb…

So since I was doing SNI on the SSL listener, but I I was specifying my “Rule/Action” that was pointing to my Backend Server that was using the normal HTTP real server. I however wanted to keep regular HTTP access open to my site not just for a HTTP->HTTPS redirect. I had however another listener available for exactly just that. At this point it was all just assumptions, even though from some post I read you can have a HTTPS load balancer hosting a web page over HTTPS while the back-end server is just HTTP. So Not sure on that one, but I figured I’d give it a shot.

So first I went back to my old blog post on getting HTTPS setup on my WordPress website but without the load balancer… turns out it was still working just fine!

Then I simply created a new physical server in HAProxy plugin,

created a new back-end Pool for my secure WordPress connection

created a new “Rule/Action” using my existing host header based condition

and applied it to my listener instead of the standard HTTP rule (Rules on the SSL listener shown in the first snippet):

Now when we access our site via HTTPS this time…

Clean baby clean! Next up some IDS rules and inspection to prevent brute force attempts, SQL injections… Cross site scripting.. yada yada, all the other dirty stuff hackers do. Also those 6 cookies, where did those come from? Maybe I’ll also be a cookie monster next post… who knows!

I hope you enjoyed my stories of “ah-ha moments”. Please share your stories in the comments. 😀

March 28, 2019March 28, 2019

PAN ACC and WildFire

The ACC

For more in depth detail check Palo Alto Networks Page on the topic. Since the Palo Alto are very good Layer 7 based firewalls which allow for amazing granular controls as well as the use of objects and profiles to proliferate amazing scale-ability.

However, if you been following along with this series all I did was setup a basic test network with a single VM, going to a couple simple websites. Yet when I checked my ACC section I had a rating of 3.5…. why would my rating be so high, well according to the charts it was the riskiest thing of all the Internets…. DNS. While there have been DNS tunneling techniques discussed, one would hope PAN has cataloged most DNS sources attempting to utilize this. Guess I can test another time…

You may notice the user is undefined and that’s because we have no User ID servers specified, or User ID agents created. Until then that’s one area in the granular control we won’t be able to utilize till that’s done, which will also be covered under yet another post.

I did some quick search to see why DNS was marked so high, but the main thing I found was this reddit post.

“akrob – Partner · 5 months ago – Drop the risk of applications like DNS ;)”

Hardy har har, well can’t find much for that, but I guess the stuff I was talking about above would be the main reasons I can think of at this time.

The better answer came slightly further down which I will share cause I find it will be more of value…

so we got the power, it just takes a lot of time to tweak and adjust for personal needs. For now I’ll simply monitor my active risk with normal use and see how it adjusts.

For now I just want to enable WildFire on the XP VMs internet rule to enable the default protection.

The WildFire

Has such a nice ring to it… even though wild fires are destructive in nature… anyway… this feature requires yet another dedicated license, so ensure you have all your auth codes in place and enabled under Device -> Licenses before moving on.

Now this is similar to the PAN URL categories I covered in my last post. Yes, these are coming out at a rather quicker than normal pace, as I wish to get to some more detailed stuff, but need these baselines again for reference sake. 😀

Go under Objects -> Security Profiles -> WildFire Analsysis

You will again see a default rule you can use:

Names self explanatory, the location I’m not sure what that exactly is about, the apps and file types are covered under more details here.

to use it again you simply have to select which profile to use under whatever rules you choose under the security rules section. Policies -> Security

Now you can see that lil shield under the profile column thats the PAN URL filter we applied. now after we apply the wild fire…

we get a new icon 😀

Don’t forget to commit…. and now we have the default protection of wild fire. Now this won’t help when users browse websites and download content when those sites are secured with HTTPS. The Palo Alto is unable to determine what content is being generated or passed over those connections, all the PAN FW knows are the URLs being used.

Testing

Following this site, which has links to download test file which are generate uniquely each time to provide a new signature as to trigger the submission. It’s the collaborative work through these submissions that make this system good.

Checking the Wildfire Submissions section under the Monitor Tab.

There they are they have been submitted to Palo Alto WildFire for analysis, which I’m sure they probably have some algo to ignore these test files in some way, or maybe they use to analyze to see how many people test, who knows what things can all be done with all that meta data…. mhmmm

Anyway, you may have noticed that the test VM is now Windows 7, and that the user is till not defined, as there’s no user agent, or LDAP servers since this machine is not domain joined that wouldn’t help anyway and an agent would be required AFAIK to get the user details. I may have a couple features to cover before I get to that fun stuff.

Summary

As you may have noticed the file was still downloaded on the client machine, so even though it was submitted there was nothing stopping the user from executing the download file, well at least trying to. It would all come down to the possibility of the executable and what version of Windows is being used when it was clicked, etc, etc. Which at that point you’d have to rely on another layer of security, Anti Virus software for example. Oh yeah, we all love A/V right? 😛

You may have also noticed that there was 3 downloads but only 2 submissions, in this case since there is no SSL decryption rules (another whole can of worms I will also eventually cover in this series… there’s a lot to cover haha) when the test file was downloaded via HTTPS, again the firewall could not see that traffic and inspect the downloaded contents for any validity for signatures (cause privacy). Another reason you’d have to again rely on another layer of security here, again A/V or Updates if a certain Vulnerability is attempted to be exploited.

So for now no wild fire submissions will take place until I can snoop on that secure traffic (which I think you can already see why there’s a controversy around this).

Till my next post! Stay Secure!

March 27, 2019March 28, 2019

PAN URL Categories

Heyo! So today I’m gonna cover URL category’s. Obviously Uniform Resource Locations are nothing new and even more so categories hahah. So when you know existing ones and have classified them, you can do some amazing things, what’s the hardest part…. Yes… proper classification of every possible URL, near impossible, but with collaboration feasible. In this post I’m going to cover how to set this up on a Palo Alto Networks firewall, cover some benefits, a couple annoyances, and ways to resolve them when possible…. Let’s get started!

License Stuff

Now when I first started with Palo Alto Networks Firewalls, they were using Brightcloud… here’s a bit of details from here

Palo Alto Networks firewalls support two URL filtering vendors:

PAN-DB—A Palo Alto Networks developed URL filtering database that is tightly integrated into PAN-OS and the Palo Alto Networks threat intelligence cloud. PAN-DB provides high-performance local caching for maximum inline performance on URL lookups, and offers coverage against malicious URLs and IP addresses. As WildFire, which is a part of the Palo Alto Networks threat intelligence cloud, identifies unknown malware, zero-day exploits, and advanced persistent threats (APTs), the PAN-DB database is updated with information on malicious URLs so that you can block malware downloads, and disable Command and Control (C2) communications to protect your network from cyber threats.

BrightCloud—A third-party URL database that is owned by Webroot, Inc. that is integrated into PAN-OS firewalls. For information on the BrightCloud URL database, visit http://brightcloud.com.

I’m not exactly sure if Brightcloud is going to continued to be supported or not and they have instead stuck more with their own in house URL DB, which of course requires a license so under Device -> Licenses ensure you have an active PAN URL-DB license.

For a list of all the class types you can use see here. (PAN login required)

Once you get this out of the way lets get into the good stuff.

Still under the Licenses area, Click the Download Now link under the area.

Considering I have nothing… Yes…

Not sure why they have a region selection… but alright…

Yay!

Now we are ready to start using them!

Objective Profiles… I mean Object Profiles

Yeah… click on the Objects tab… look under Security Profiles… URL Filtering.

There lies a default profile, which allows 57 categories while blocking only 9. For a simple test I’ll use this, the blocked categories are:

abused-drugs (LOL, cause other poisons like Tobacco and alcohol are allowed, cause laws)
adult (I’m assuming this is a business friendly term for porn)
command-and-control (duh)
gambling (duh)
hacking (interesting class definition)
malware (duh)
phishing (duh)
questionable (duh)
weapons (awwwww)

Well that seems like a fairly reasonable list. Creating your own allow and block listing is just as easy as creating a new profile and defining each class accordingly, and yes you can easily clone an existing profile and change one or two categories as required.

The Allow and Block lists are specified under the overrides areas if you happen to need to allow or block a URL before it can be officially re-classed by PAN DB. As quoted by the wizard, “For the block list and allow list enter one entry per row, separating the rows with a newline. Each entry should be in the form of “www.example.com” and without quotes or an IP address (http:// or https:// should not be included). Use separators to specify match criteria – for example, “www.example.com/” will match “www.example.com/test” but not match “www.example.com.hk”” Which makes sense it’s will determine what is allowed as for proctols under the security rules area, this simply states which addresses (DNS or IP based) to allow or block. In the case of DNS till proper classification.

Checking a URL for a Category

To check a address class, check PANs site for it here. If you find a site is mis-classed you can send an email to Palo Alto Networks team and they will test the verification of the re-class and re-class the PAN DB accordingly. As far as I can tell I don’t think this one actually requires a login.

Using IT!

Alright, alright, lets actually get to some uses. Now if you were following my series see my last two posts here, and here for reference material. Under the Security Rule Test Internet, the final tab, actions, we did not define any profile settings, this is where the rubber hits the road for the first time.

Pick Profiles, We’ll cover groups a bit later (its just a group of profiles, who’d of thought).

As you can see this expands the window to show all the profiles you saw under the Objects -> Security Profiles area, in this case we are just going to play with the URL filtering.

Now once I apply this on the internet rule.. productive for my Test XP machine should go up… muahahah and…

HAHAHAHA you lazy mid 2000’s virtual worker… you can’t go gambling get back to work!

Summary

As you can see how useful URL categories can be, unfortunately I did want to cover more granular examples; such as only allowing a server to access it’s known update server URL’s. Hopefully I can update this post to cover that as well.

For now I hope you enjoyed this quick blog post. In my next post I hope to cover how this isn’t an IDS of any kind at this point, but a single layer of the multi-layer security onion. Stay tuned for more. 🙂

March 26, 2019March 30, 2019

Basic Setup of a PAN VM 50

Quick Intro

Heyo! so on my last post we went through a basic install and update of a Palo Alto Firewall VM. Now it’s time to setup a dataplane NIC, some zones, some rules to allow some basic internet.

I decided to do some very basic setup of one NIC and was surprised to find I could not get any ping responses either from the firewall, or the firewall making any requests. I had a memory of talking to a smart fellow once before about this, and sure enough…

A Caveat

~~You have to enable Promiscuous mode on the VMPG the NIC is a member of…~~

I know it sounds ridiculous and it is, but without it, nothing flows through the PA VM. Quick Update on this, I didn’t like this idea one bit, so to ease the risk I did find something rather interesting: according to this (requires a PA login) this hasn’t been needed since PAN OS 7, I disabled it on my Test network

and the pings dropped… ugh… ok… According to the post it says PAN OS 7 and onward uses this setting by default but can be changed under:

Device > Setup > Management > General Settings

Enabled by default huh… doesn’t seem to be enabled to me…

enable it, commit. Now MAC address changes will take place in this case I did loose connection to my external IP, but pinging from my PA VM to my gateway managed to fix that quickly.

And now sure enough with Promiscuous mode rejected on my vSwitch settings…

Oh thank goodness I can go to bed knowing I didn’t suggest a terrible practice!

Basic Setup

Look at this test network… was using an OPNsense router/firewall, but all these guys are currently shutdown. Lets spin one up and make the PA VM 50 it’s new gateway…

Adding the required Virtual NICs

Then add a new NIC to the PA VM (since it only came with two by default (the first being the mgmt NIC, and the second I connected to my DC)

This should be the second Interface under the PA VM Network Tab.

K looks like we should be good, power on the PA VM again.

Configuring the Interface

Once in the PA Web interface, navigate to Network -> Interfaces.

Again this will be Ethernet 1/2, although it is the third NIC on the VM.

Once we click on Eth1/2 and configure it properly it should show up green as well. I have configured a interface mgmt profile already under Network -> Network Profiles -> Interface Mgmt. Ping checked off, open subnet permitted.

Also a simple Zone, simply named Test.

First thing we have to define is the type (Layer 3), we want a dedicated collision domain please. 😀 In this case I’m simply interested in PA to client connection in the dataplane to be confirmed. We will place the NIC in the default router as well as the Test zone.

Then we click on IPv4 to set an IP address up for this layer 3 NIC.

specifying /24 is important here. else any ip address without a defined subnet is treated as a /32. Then under the advanced tab select the interface mgmt profile to allow it to be pingable.

Once committed it should come up green.

and should be reachable by VMs in the same subnet….

Yay it is, but alas this is not enough to give this VM an internet connection. Remember that default router we connected the NIC to, well it has no default route defined, or well any routes for that matter, however because I connected both NICs (my ZewwyDC and Test) into the same router, even without any routes defined, the XP VM can ping the ZewwyDC IP of the PA VM

Security rules and the fact the server and VMs use a different gateway then what the PA VM has for its test IP in that subnet, the responses would never come back to the PA VM anyway, never mind that we didn’t define any security rules to allow it, it was simply because I had the “allow ping all” interface mgmt profiles on all the NICs and connected to the same router that made those ping requests work.

Since I’m not interested at this every moment to move the DCs internet right now, I’ll provide the PA VM a public IP address of it’s own and then create a NAT rule to allow the Test XP VM an internet connection.

The Internet Interface

Also since I don’t want to keep having to “system is shutdown” my PA, I guess this time I’ll populate it with all the VNICs it will ever be able to use… (8)

I did this mainly cause I wanted the last interface on the Web UI to be used for this internet connection

So you might remember my blog post on getting another NIC in my hypervisor host I was going to use it with OPNsense, but since my physical PA has become more useless than online multiplayer only game with all its servers shutdown. So this is to become a replacement as I re-purpose it’s chassis for another epic build I plan to blog about this summer :D!

Interface Mappings:

Well now that I got my MS paint fun out of the way you can get an idea of which NIC I want this PA VM to have one of it’s internet connections on: Eth9

I created a new Zone: Deadly Internet, and connected it to our default router:

Then I configured the public IP I had originally configured for my OPN VM by clicking on the IPv4 tab… and to help make sense of this, some more paint fun 😀

I also applied my Allow Ping All Interface Mgmt Profile so I can verify that the interface is not only up (green) but actually reachable, sure enough after a commit… the interface shows green (Also checked off Connected and connected at boot under the VM settings).

Mhmmmm not reachable…. ohhh right, the routers default gateway….

Default Route

Since we are configuring this statically and not via DHCP by our ISP this info is also provided to you.

Network -> Virtual Routers -> Default (in my case) -> Static Routes

So As you can see, anything it doesn’t know, next hop, the IP my ISP gave me as my default gateway.

Commit.

Alright, my attempts to ping it are not successful, which has happened to me the last time I configured all this and I had to reboot the modem, but just before I do that. I’m going to login into the PA VM via SSH and attempt to ping out via that interface:

Alright well last time I got up to this point were I had everything tripple checked, I contract my ISP support and we ended up rebooting the modem which is in bridge mode, Since I assume the MAC address table isn’t being update accordingly or unno its stuck with the old MAC… I suppose I could test this theory by spoofing that NIC with the other NICs MAC…. mhmmmmmm I think I’m gonna pleasure my thought here teehee…

dang it won’t let me change the MAC while it’s on, power off PA VM… set MAC… Spoofed from old OPN sense VM… Power on VM… and…. nope I can’t manually assign it, it’s a dedicate MAC that ESXi won’t allow me to manually assign… so set back to automatic, and boot, if no pings after this rebooting the modem… sigh.

Alright, so pinging my IP still no work even after reboot, I created a firewall rule assuming it was that… nope still no ping response even after commit that, odd cause I didn’t see anything under my traffic log on the firewall itself… so I logged into the firewall again via ssh, but this time I did mange to get a response from my gateway device, wooo yay… ok… so let me try pinging it again externally…. Yes! There it is! had me worried a bit, I had all bases covered so it should have worked, and now it is, w00t!

This is all well and good, however my test VM, on the test switch still won’t be able to reach out, however, it should be able to reach what will become it’s NATed IP address when it comes time to roam the interwebs.

Whoops that wasn’t possible till I expanded the scope of my security rule:

Firewall is very finicky about allowing packets through zones and subnets, so ensure you create rules accordingly. Normally I like to have a deny all rule at the bottom of my list, these would be however above the built in rules:

However there are some Caveats that comes from around doing that which I hope to cover at some point in my Palo Alto series blog posts. For now we won’t go there yet, just be aware of these rules, any packets that reach them are not shown under the traffic tab (IIRC).

However now that we have got all that out of the way, we can finally create the NAT rule (as well as a security rule) we need for getting internet access to our test subnet.

NATing

It’s time to get into the baby potatoes… mhmm who doesn’t move some baby potatoes…. anyway I won’t be covering all the possible NATs that can be accomplished (although I do plan on covering a whole post on those in this series as well), we will do a basic internet NAT here to get us started.

Policies -> NAT -> Add

Pretty straight forward configuration here, anyone from my test subnet from my test zone, will be NATed out my internet connection on Eth9 using the IP address I have assigned it which came from my ISP.

Security Rules!

I hope you liked my pun there, if not, alls good lets setup some security rules…

Policies -> Security -> add

To do this more salable instead of adding the subnet IP by IP range every time, I added an object…

User tab is passed, as we won’t get into that meat today…

Application: Web Browsing, DNS, Ping, ICMP

Service: Application Default

Now Commit, we should hopefully be able to ping out to an external DNS provider, like 8.8.8.8 from our Test subject VMs… muhahahahaha

Boo yea! There we go.. and internet… whoops… forgot to allow DNS lol….

mhmm connection reset ehh, well I guess we need another application defined… or right SSL

finally….

Update

for some reason a couple days later, I noticed I was unable to access Google, even though I had accessed it before, as the above screen shot shows.

Then I created an open rule and i was able to access google, and found out for google to work it’s defined as it’s own App ID (Google-base), I like granular control, but I should be able to select web browsing and that should be able to group sub apps to make my web browsing experience work… On top of that I noticed the same reset connection errors going to Youtube, and reddit… ok this is getting a bit redic…

Here’s my new ridiculous rule just to go to Palo altos own site that referenced a youtube video, google itself, and one reddit result I was interested in… Holy eff man…

Setting the Host Name

Device -> Setup -> General Settings

Here you can enter, the host name, domain name, login banner, timezone and a couple other general settings:

Awesome even though it appeared squished after pasting. It still applied 😀

DHCP

It be ridiculous to expect those systems in the Test network to configure themselves, let’s give them a hand with good ol’ DHCP.

Network -> DHCP -> add

Select Interface (in our case Eth2), enter a range in the IP Pools, and Click OK.

Commit, it’s that easy, once created there’s a link to show the IP allocations. 😀

If you need to add custom DHCP options, just click the options tab. Which you will for things like the gateway and DNS servers 😛

Summary

Well I hoped you enjoyed this blog post, we got some basic things done, some zones, some policies, some new interfaces, objects, yet we haven’t even got into the real meat and potatoes, like wild fire profiles, and URL cat profiles and all those other fun things we will get to soon.

The idea behind the basic first couple blog posts is to just get our baseline going so when it comes to the more complex stuff I have some reference material already available for those that need some reference as to exactly “how I got here“.

In my next post I’ll cover using some of the great features, some of these features will be provided with a standard license, other are license separately for your needs and requirements. Since I got a whole lab bundle for educational purposes I’ll get to post about all the goodies soon. 😀

Stay Tuned!

March 23, 2019

Setting up a Palo Alto Networks VM 50

Intro

Heyo! It would seem the awesomeness of spring has sprung on to us, and that delightful sun’s warm and longer days just feel so awesome in the wake of a cold long winter.

Anyway…. PAN TIME. so I finally got my auth codes I’ve been waiting on. To start you need to get a deploy-able image from a Value added reseller (VAR). Since Palo Alto has no public download for their VM series firewalls. Not a huge fan of their tatics on this one, honestly I believe education should be free and easily accessible. SO this is one area where I do tend to have to give PAN a thumbs down. However when it comes to security, and granular control of said security it is really nice.

Installing PAN VM 50

Deploy the OVA

For my Lab I’ll be using ESXi and an OVA deployment file. So on the vSphere Management, File -> Deploy OVF template. (If you are using the web management, follow this)

In this case my A drive is a mapped drive of all my applications and images, although I did request a newer image than 7.1 as that is rather old and I was hoping for 8.x for 9 even, but I’m hoping I can just update the VM software with my auth codes once I get the VM up and running.

Next you’ll get some details about how the VM will be deployed, simply ensure you have enough resources available to meet the deployment needs.

Click next to assign and name and location for the VM info and VHDD.

I gave it a generic name then the PAN OS number as again, I’m hoping to upgrade it with my auth codes. After that select the datastore to use, I used the local datastore for this VM, and stuck with thin provisioning after that, click next to begin the deployment. depending on your network connections and datastore selection, this time may vary.

Not sure if the copy of the file to my network share got messed, but every-time I deployed it from the share it failed, so I grabbed my IODD device where I had the initial copy, deployed it from there, and it worked.

Yay! Alright time to check its settings.

Alright a couple NICs I was expecting more than that… Anyway normally PAN devices are headless and you can’t see the boot process unless you connect to a serial port, but VMs have direct console, soo I’ll set the NICs not to be connected at the moment as I don’t want them to be in my home NATed network.

Powering on the VM

So disconnected the virtual NICs and booted the VM:

Then I got a login prompt, rather quickly, but don’t be fooled, you have to wait…

After a couple minutes, you’ll get the real login prompt.

Set Admin Password

Now that we got the VM up and running we should change the password:

As you can see it’s not cisco, so short wording doesn’t work. Also just to show that you don’t enter a password at the cli, you enter the word password and it will ask you for them without printing them back to the screen (thumbs up).

Don’t forget to commit. Now we need to figure out how to configure the mgmt IP address… mhmm

Set Management IP Address

So since I wanted to be able to manage this VM easily in my current home network “VM Network” vSwitch on my ESXi host, first I pinged an IP and ensured it was available. Then on the PA VM I ran:

Configure (get into configuration mode)

set deviceconfig system ip-address 192.168.0.55 netmask 255.255.255.0 default-gateway 192.168.0.1

commit

Then I opened the VM settings and enabled the connect:

Then tested my pings again, and success 😀

K, so now that we can ping the management IP let’s see if we can access the web interface, and if so hopefully that should be all we need to do at the CLI. I love CLI commands and stuff, but for most management I like GUI’s unless it becomes doing something x number of times, then scripting via the CLI is a necessity.

Access the Web Interface

Once you access the VM’s IP in a local browser you shouldn’t be surprised to be presented with this:

Usual certificate security and warning of un-trusted due to self signed.. yada yada, advanced, proceed….

Mhmmmm I really miss that 7.x Web look, just the right amount of color…

If my upgrades go successfully I’ll be able to show you the new login, a tad more bland….

Awww man, just look at that delightful dashboard, the system info, haha unknown serial in VM mode with no license (yet) 😛 I like how it even shows my two login sessions (CLI and Web).

As well as of course the usual, PAN Tabs (ACC, Monitor, Policies, Objects, Network and Device) mhmmmm so delightful.

Now my main goal of today and this post is simply to get the VM booted up, but also updated. Now I can’t do that without a license, which I got just a couple days ago. Now sadly I can’t share these with you, but I can tell you how to accomplish the task.

Managing Licenses

Click on the Device Tab -> Licenses

In my case I can’t remember if I had uploaded it to my usual PA login account online, so for now I will be using #2 Activate via Auth Codes.

First things first though, set the DNS servers.. :S whoops lol

Device -> Setup -> Services -> edit -> Primary and secondary DNS servers

So even after that I kept getting communication error message, so I googled.

After that I figured they are doing their usual ways, and locking this down in some other form that doesn’t provide any nice error message to try and stop use of these images if they leak, and it’s extremely frustrating for legit users… not gonna lie.

So I decided after I got my DNS up n running to apply the Auth code again and this time I got a different error, that my auth codes have to be registered to my support account before i can create and register the VM… ughhhhhhh

This as you can see is the real annoying side to any DRM. Let me jump through these hoops and come back to this post in a little bit… :S

Alright, so I logged into the online suport portal, found the section to register my auth codes, did that, then jumped back into the VM web and entered the auth codes again, this time it didn’t complain, the VM showed it was rebooting while the web interface stayed at the licenses section… odd haha I was going to take a snippet of that happening but the reboot was rather quick.

Since I knew the VM had rebooted as I saw it via the vSphere console window, I gave it a couple minutes before navigating to the web interface.

Sure enough after logging in again, I know have a serial number defined on my PA VM. 😀 I hope now I can actually check for updates without getting a generic, false error message…

Yes! So many PAN OS’s to choose from…. but sadly no PAN OS 9… or 8.1.x for that matter… Well that sucks I was hoping to be able to play around with TLS 1.3… oh boy… maybe I have to upgrade first?

Upgrading PAN OS on PA VM 50

Sooo I selected 8.0, downloaded and configured into software manager successfully awesome! Install failed, not enough memory…. nice.

Well considering it’s a VM which are amazingly salable in this regard I won’t blame them here, the message is to the point. I’ll just shutdown the VM and up it’s memory…

Device -> Setup -> Operations -> Shutdown Device

Yeap… System is shutdown. lol

Bammmm more memory like that!

You got me again, you can code for the validation, but you can’t code the process to do that for me eh…. they could, they just didn’t want to.. so let me jump through some more hoops…

Dynamic Updates -> Check Now -> Apps n Threats -> Download (8136-5163 at the time of this writing) -> Install

Yay, at least that worked without some issue to overcome. Let’s try that software upgrade for a third time. Third times the charm right?

SO far so good, device needs a reboot, OK. 🙂

And here it is.. the bland 8.0 login .. 🙁

Just no color, no life… just go look and compere the login before and this one, I even liked that they had a soft indent of their logo in the background, made it feel so elegant to this… so minimalist…

As for the software, upgrading to 8.0 did make 8.1 available… but still no 9.0 errr lets upgrade again and see… ooo yeah…. there it is… 9.0!

So I can… Yeeee, I dunno if I’ll do it just yet, but good to know I can when I want to.

Summary

Overall the deployment and use of the PA VM is very good. I’m rather excited to get my SSL inspection rules setup for some stuff… 😀 as well as cover other blog posts covering some more in-depth setups and configurations.

In my next post I’ll cover actually setting up some zones and network configurations. or I might even just show how to migrate a physical configuration. In this case since I won’t have a 1 for 1 NIC assignment there would probably be some tweaking required, maybe even before the firewall would accept the config file. but we’ll cover that when we get there. 😀

February 24, 2019March 23, 2020

OPNSense for Exchange Reverse Proxy

OPNsense and Exchange

Unlike the German blog I reference below, I use a Palo Alto as my main device to handle normal NAT for the OPNsense box’s internet, as well as the NAT rule to allow HTTP Validation (which I covered in my last blog as it was causing me some issues). Another notable difference is I have a dedicated Datacenter zone which has it’s own dedicated NAT rules for internet access, but not direct NAT rules from the outside world (as it should be), which means no dirty double NAT (like it should be). Then once certs are setup, the OPNsense will reverse proxy the HTTPS requests for OWA, and hopefully Active Sync.

First however, I’m going to add a new VMPG network in this I called it (DMZ) and assigned it a VLAN (70). Since this is ESXi running on an old desktop with only 1 NIC (initially) I have to utilize VLAN to make the most out of the lack of physical adapters. Then I’ll need to create a sub interface on my Palo Alto, with the same VLAN tag of 70, and give it an IP address of 192.168.16.1/24. This will be the subnet of the DMZ. Now you maybe wondering why I’m putting the subinterface and IP on my Palo Alto and not on the OPNsense VM, the reason for this is I use Palo Alto firewall to manage all the other networks in my environment. so all known routes will take place there.

The whole idea here is to get Active Sync to work, and the PANs do not support reverse proxying. So the idea is to have a NAT rule allow port 443 (HTTPS) from the internet to the OPNsense vm. so after the redesign I have 1 OPNsense VM (192.168.16.10/24 – VLAN 70) and a new DMZ VR, with a new subinterface on the PAN (192.168.16.1/24 – VLAN 70)

and the PAN…

So I added static routes between my Zewwy network and my new DMZ, as you can also tell based on the mgmt-interface profiles, I only allowed pinging the gateway, so the OPNsense ICMP request shown above to succeed.

~~I had to set the default gateway on the OPNsense VM via the CLI first in order to gain access to the OPNsense web UI~~

route add default 192.168.16.1

~~change IP based on your gateway. Then once in the UI go to:~~

~~System : Gateways : Single : Add~~

~~This was required to keep the default route persistent after reboots.~~

Well I was getting a bit stuck so decided to google a bit and sure enough a blog to the rescue, odd enough, it’s a German blog. Ich can ein klein beste duetch aber nicht sehr gut. So I picked translate…

I thought… oooo he’s on a VM on ESXi too, and installing VMtools nice… goto plugins… don’t see a list like him, and thought… Shiiiit, my OPNsense have no internet…

Sooo, I decided to give my OPN VM internet access to get updates and plugins (best move). I won’t cover this but basically required me to add a default route to the DMZ VR, create NAT rule and Sec rule, test pinging internet IP from OPN, and success.

OK so.. Now that the PAN is all setup, and we have tested our NAT rule for internet for the OPNsense VM… let’s just go over the OPNsense install…

OPNsense Install

On your Hypervisor or Hardware of choice, in my case ESXi New VM. 🙂

In this case I know I/O is not a big deal so the local ESXi datastore will suffice for this VM:

Pick VM V8 (cause I’m still on ESXi 5.5)

FreeBSD 64Bit (for some reason we won’t be able to pick EUFI)

CPU: 2, Mem: 2GB, 1 E1000 Nic in the DMZ

LSI Logic Parallel SCSI, New 20 Gig Thin Prov Disk, Create VM.

Edit VM settings, remove floppy, Boot Options Force BIOS.

Open Console, and Boot VM. Disable Disekette A:

Advanced, IO Device Config, Disable All (its a VM we don’t need these)

Now, Select the disc part and mount the OPNsense ISO for booting:

Boot it! by Pressing F10 in the VM and save BIOS settings:

Mhmmmmm so delightful…. and now we let it load the live instance, while this live instance is good enough to start using, I don’t exactly feel like loosing my settings every-time it boots and having to remount my ISO from my local machine… so we’ll install OPNsense by logging in with the installer account:

As you can see it’s assigned our one and only NIC the LAN settings, to ease our deployment and the above section I striked out, we’ll be assigning the interface the WAN value. 😛 anyway logging in the with opnsense password.

Mhmmm just look at the old style look, make me juicy…

*NOTE* if installing EFI based the input here may freeze… googling it quickly I only found one reference to the issue by a comment by eugine-chow

Press CTRL + C (This exists the installer)
re-logon on as installer account (This resumes the install with keyboard control

OK, Let’s go! Accept, Guided instillation! Pick Disk, for simplicity and low disk, we’ll just pick MBR… and look at that installation go… mhmmm humbling…

Set a root password:

Now reboot and unmount the ISO, now the boots quicker and our settings will be saved! First things first, assigning NICs… or should I say our one NIC, login in as root via the console. Press 1 to assign interfaces. Even though I showed VLAN assigning above that is used by the ESXi hypervisor and thus I select no to VLAN tagging here, and then specify em0 as my WAN NIC:

Now in my case it wait a long while at Configuring WAN interface, cauuse it’s defaulting to DHCP, and there’s no DHCP in the subnet… ugh, I don’t know why they don’t ask for IP assignment type in this part of the wizard…

now Select option 2 to set IP which should have been part of the wizard in part 1…

Now that is out of the way, we can access the OPNsense web UI from our Datacenter Laptop/VM… you won’t be able to ping it, but the anti-lockout rule will be created on the WAN rules so…

Follow the config guide… only important part being the upstream gateway:

And of course in my case since it’s being NATed the RFC1918 Networks will be unblocked as it’s using one 😛 and NO LAN IP.

First order of business is going to be moving th eport off of port 80 as that will be needed for Lets Encrypt Validation (only cause my DNS provider doesn’t have the API for DNS validation yet).

Finally time for OPNpackages

OPN packages

Bammmmm that was easy!

OK, Firewall, since my OPNsense only has WAN, and it’s open, all security will be handled by the Pal alto, so I don’t want to open HTTPS from the internet to my the OPN sense just yet, till we create the other requirements.

HAPRoxy

Create a Real Server, in this case this will be our Exchange server as in the topology.

Now for a Backend Pool

He doesn’t mention any other settings so I just clicked save… I probably should have named the Backend pool better but meh.

Following the German guide I was a lil upset cause I was running OPNsense 19.1, it seems they changed the HAProxy options, however I did manage to figure it out after a while…

ACLs now Conditions

Go to Services -> HAProxy -> Rules & Checks -> Conditions

Add a condition, for testing I kept it simple as the blog I was following:

and then…

Actions are now Rules

Go to Services -> HAProxy -> Rules & Checks -> Rules

add a rule:

Frontends are now Public Services

Go to Services -> HAProxy -> Virtual Services -> Public Services

Add a public service:

Enable The HAProxy Service:

OPNsense Firewall Settings

Even though this VM wasn’t routing any traffic, I still had to create an allow rule under the firewall area before my PA firewall would see completed packets:

first attempts, gave site unavailable and my PA logs showed…

On OPNsense:

Firewall -> Rules -> WAN -> Add -> TCP (HTTPS) Allow + TCP (HTTP) Allow

basically allowing all TCP packets, after applying I was able to get the OWA page from my Windows 10 VM in the datacenter:

so now it’s going to basically be creating a NAT rule on the PA to see it from the internet… but before I get to that…

Certificates!

Now that I covered getting Let’s Encrypt to work behind a Palo Alto firewall I should be able to complete this part!

Lets Encrypt

Enable the service, and the extension to HAproxy, hit apply

I did select my exchange front end, even though I didn’t show it here, then I created a Lets Encrypt Frontend as exchange won’t deal with HTTP:

Well lets test this out… Create a Certificate..

Click save changes, but just before we click Issue Certificates, lets tail the log (/var/log/acme.sh.log) to see the process… If you try to open it before you click issue it will fail cause the file only gets created on first run… so click issue and then quickly open the log file with tail command… if it gets stuck at ACCOUNT_THUMBPRINT something went wrong… and of course… something went wrong… ugh……

Mhmmm sure enough… Domain Key error on second try…

But if I alter my HTTP validation to…

and attempt to issue the certificate then I see in my acme.sh.log its success…

but the UI will still show validation error even though it was issued successfully…

Let me see if I can at least assign this cert even though it may not be automatic…

seems like it… lets test…

Well at least that’s something… I’m not sure if the auto renewall will still work… if so I’m not sure exactly what the point of the HA plugin really is… I mean if you can specify the normal WAN and port 80 to validate the certs and seclt the cert to use on the public service… figured it work none-the-less right?

Well I guess well find out… now there one last thing I want to cover… but I’ll do that when I get it figured out again…

For now I’ll post this blog post as is casue it is getting rather long.

Cheers! OK NM I did it quickly…

Blocking the ECP

Under OPNsense HAProxy go to Conditions:

Then Rules:

Then Edit your Public Service settings and add the rules:

Finally test access to ECP via the Proxy…

Ahhhh much better… 😀 something not mentioned by the German blogger makes me wonder if I can access his ECP.. mhmmm

Alright that’s all for tonight. 😀

February 21, 2019

Lets Encrypt HTTP Validation
And the Palo Alto Firewall

The Story

This…… this one…. this one drove me NUTS! for almost a week…. it was a lil mix of a perfect storm I guess… but lets start from the beginning shall we..

So a couple weeks ago i wanted to get active sync setup for my exchange server (Checking OWA sucks)… so I was sought after OPNsense for my open source firewall of choice.

I started following this German blog post, and I hope to have that blog post up very soon as well (sorry I don’t usually get hung up like this).

My setup was pretty much exactly the same however I was getting hung up on the plugin not validating my scripts over HTTP. See the full pain details here on github, anyway, I did finally manage to get my OPNsense server behind the NAT rule to finally succeeded behind my Palo Alto Firewall (by basically opening up the rule way more then I ever wanted to) so I knew! I knew it was the Palo Alto blocking still somehow… but how I couldn’t make sense so I wasn’t sure how to create my Security rule.

First try

My first try was exactly like the github issue describes, was failing on domain key creation, this failed even on my OPNsense with a Public IP and all rules exactly as the OPNsense basic guide states to set it up.

When Neilpang (the main script writer/contributor) said ti was fixed and no commit was applied, I tried again and it worked, I can only assume this was due to the fact DNS may not have replicated to the external DNS servers lets encrypt servers are configured to use when I first made my attempts at a cert validation.

That didnt’ explain why every attempt behind my Palo Alto with a NAT and security rule would fail…

The Palo Alto

I love these things, but they can also be very finicky. to verify my rule I had used my IIS Core VM (That I’ve used in previous posts on how to manage Windows Server Core) along with the HAProxy plugin on OPNsense to basically move the requests from the NAT rule of the Palo Alto but really serve up the IIS website of my IIS server. Not to my amazement, but sure enough I was able to access the IIS website from the internet, so my security rules and nat rules on the Palo ALto are working fine, as well as the security rules on the OPNsense server…. so what gives? Why are these HTTP Validation requests failing??

Again, as stated above I knew it was the Palo Alto from opening up the rule completely and it working, but I figured it was the issue even before I did that… but opening up the security rule completely is not the answer here… like it works but its far to insecure…

So I managed to talk to a friend of mine who happens to be realllllly good at deploying Palo Alto as he does it for a living. I basically describe my issue to him, and ask him if there’s anything he can think of that might be a problem. (I’ll hopefully be having a couple more Palo Alto blog posts as soon as I can get my proper licensed VM) To my actual amazement he goes on about this one setting you can use inside security rules and about a story about when it caused him grief…. go figure, he’s experienced it all!

What was it?!?!?!

Alright so here’s my rule I intially had, which was causing failures of the let’s encrypt OPNsense plugin…

AS you can see nothing really special, until he told me about… PAN DSRI or Palo Alto’s Disable Server Response Inspection you can check the link for more details. Now the funny part is that post covers better performance…. in my case, it was simply needed to work! And all it was, was a checkbox….

once that checkbox was selected, the rule adds a icon to it.

I was able to click Issue certificates on the OPNsense Lets Encrypt plugin, and I got some certs! I’m ready to now add the Let’s Encrypt HAProxy plugin integration and set these certificates for backend services… like my ActiveSync… or OWA… Ohhh exciting stuff!

Man that feels good to finally have that sorted! Wooooo!

February 21, 2019

WMI and the WBEMTEST

I’ll try and keep this post short, as I have many things to catch up on, and this just happened to be one of those things I haven’t done in a while and had to do today for some newer servers that have been configured.

Now since I hadn’t blogged about this myself I went out to the interests to give me a good reminder on how to accomplish this. My first hit was, Sysops… and I usually really like this site…. well till i read this…

“Access denied should be self-explanatory. The credentials you use must have administrator rights.”

Ughhhhh I’m sorry what did you just say? No I don’t think so, WMI maybe, by default, restricted, but it doesn’t require such drastic permissions to utilize.

My second find was a lot nicer, in particular telling you how to manage those permissions, without ahem need administrator access lol.

So lets follow along shall we! so much for short..

First order of busy-nas is creating a user:

Of course WMI being Windows Management Interface, means I’m making obviously a windows domain user. Nothing special, especially no admin.. 😛

Again, nothing special here. Alright now I need two servers, well I guess in this case the server being monitored is sort of like a client… ugh anyway…

I guess fo r now I’ll just login to my exchange server and wmi query another server to test out first off… mhmm all I have besides that are core servers, oh boy ok… I think I’m going to need to spin up a new testing server one second…

OK all basic settings…

remove floppy boot into EUFI:

Boot system… attach disc from local host…

lets find us some windows erver 2016…. bug CD-ROM stuck “connecting”…
Close vSphere, reopen console, try again…

always loved this trick over uploading a ISO to a datastore….

Ahh modern Windows still giving off that great nostalgic feel.. 😀

yada yada, setup, vmware tools, and join domain, you get the jist of it.

Ping and the Firewall

First order of Business Ping and the Firewall!

Ahh yes connectivity verified (I knew it was good cause I joined the system to the domain, but I like ping… just nothing like a good ICMP) good thing that m is not a u….

Anyway time to run WBEMTEST, bet the first attempt fails cause the firewall again…. hour glass… and (not responding) yeah…. sounds like a stupid firewall…

What?! no way RPC error… lol I totally saw this coming cause again a default server installation doesn’t allow these connections through the firewall by default.

This is a bit old, but lets see if it still works…

Amazing it worked… but yes this was just to verify connectivity through the firewall… so…

WBEMTEST Testing WMI with Least Privileges

OK now that we verified connectivity to the wmi stack with wbemtest using our admin account, lets do it again as a normal domain user. Just to validate these credentials were OK as a standard user i logged into a normal workstation with it, if you want to protect this even further you’d use GPOs to disallow this account local logon. Anyway…

What?! Access denied… lol again expected.. now instead of granting this account admin access, which is overkill, lets grant it the basic enable and remote access on the WMI object… so back on the server we want to be monitored via WMI…

Hope that was easy enough to follow without even saying anything.. anyway lets try that connection again…

Try 2, Scale-able

Mhmmm access still denied… lets see here

This is how I normally do it for a monitoring account anyway cause it usually needs more permissions when mointoring a server so lets try it that way… revert the direct permissions… and grant performance group access…

Now lets add wmi reader account to the dcom groujps and the performance monitor group and reboot the server…

Server rebooting, back up, and lets test that connection again on wbemtest!

and….

Bazzaaaaaa! An account thats not a admin anywhere with permissions needed to monitor your server with WMI! Use these accounts on software such as PRTG, Splunk, Zenoss, etc etc.

Hope everyone enjoyed this tutorial on WMI configuration and testing. 😀

November 18, 2018May 4, 2021

Palo Alto VPN (GlobalProtect)
Part 5 – Rules, Testing, Troubleshooting

Intro

In this 5 Part series I covered all the requirements to configure Palo Alto Network’s GlobalProtect VPN:

1) Authentication, Auth Profiles and testing them.

2) Certificates, Cert Profiles, SSL/TLS Profiles and creating them.

3) Portals, what they do and how to configure them.

4) Gateways, what they do and how to configure them.

This part will cover the security rule required, and a little troubleshooting steps along the way.

Things not Covered

I didn’t cover creating DNS records, as again, these come down to your own DNS provider and whatever tools and portals they offer to manage those.

I don’t cover configuring the interfaces (public facing or internal), I don’t cover the virtual router and routes. All these are assumed to be handled by the administrator reading these guides.

I don’t cover installing the client software, if you have the certificates installed on the client devices (Required), it’s simply navigating to the portal address with a supported browser and downloading the installation packages (.exe for windows).

For giggles, I tested navigating my portal from my phone, it did prompt me for my certificate (the VPN was working well) yet after selecting my certificate I got a connection reset error on my browser and checking the Palo Alto Firewall logs (Monitor tab -> traffic) I indeed saw the Deny traffic and action reset-both action… why this is, even though the application was identified correctly as web-browsing and that was enabled in the rule, it wasn’t being allowed by my rule and instead was being denied by my deny all rule. I”m not sure exactly why this is, however I don’t have intentions of accessing my portal web page anytime soon, so for now I’ll ignore this as I use IPsec XAuth RSA on my android device.

I have also noticed that for some reason with Samsung Android I can’t seem to get this VPN setup to work, from quick google searches people seem to say it’s due to packet fragmentation somehow. I haven’t yet had the chance to look into the nitty gritty of this issue just yet, but when I do it will be it’s own blog post!

I also don’t cover installing the completed certificates onto end devices as again this comes down to the end devices being supported by the administrator configuring Global Protect and is outside the scope of this guide.

The Security Rule

As you can tell pretty simple, anyone from the internet (I could be connecting from anywhere, and my IP address changes on my phone all the time, random access points etc) to my public IP address which hosts my portal and gateway, and the required applications (IKE, ipsec-esp-udp, and the SSL and web-browsing) again I haven’t exactly figured out the portal web-page loading issue just yet.

*UPDATE* ensure to add panos-global-protect application type, else only X-auth RSA connection will succeeded, that does not rely on the Global Protect Portals.

Failure to add panos-global-protect applicatin results in end client getting “No Network Available” error on the Global Protect App.

My Phone Config

In my case I do run an Android phone, running : 8.0.0: Kernel 4.4.78

The OS is some H93320g couldn’t find much but this about it…

For the most part I install both my Offline-Root-CA and my Sub-CA certificates on my phone. Which can be found under (General -> Lock Screen & Security -> Encryption & Credentials -> Trusted Credentials (Instead of CA’s who knows?) -> User (Both Should be listed here)

Then Installed the User certificate with the private key, which then shows up under (General -> Lock Screen & Security -> Encryption & Credentials -> User Credentials (Instead of User Certificates?)) The other annoying part is once you have the certificate installed, this area doesn’t allow you to see the certificate details, you can see them under the area mentioned above, but this area…. nope.. :@

Once the certificates are installed, it simply comes down to configuring the VPN settings. (Settings -> Network -> VPN -> BasicVPN -> Click the plus in the upper right hand corner. Then)

Name: Give it a meaningful name

Type: IPSec XAuth RSA

Server Address: The Address defined in Part 3 -> Agents -> External Gateways

IPSec User Cert: The User Certificate you installed and verified above

IPSec CA Certificate: Don’t verify server (Which is probably why I didn’t need the above server address in the gateway certs as a SAN)

IPSec Server Certificate: Receive from server

Then enter a username and password for a user you defined to be allowed per your Authentication Profile you created in Part 1.

You shouldn’t have to define the advanced settings as those should defined to the client from the gateway config we created in Part 4.

Summary

If done correctly you should have a successfully, you should be able to see all the parts play out in both the traffic logs, and the system logs…

System:

Traffic:

That is pretty much it, if you have a failed connection do the usual step by step troubleshooting starting with connectivity, you should be able to see the access attempt from the device in the traffic logs, if they are being blocked by rules, adjust them accordingly.

If you verified all other things, it maybe your chain, or you are enabling extra security like verifying the server certificate than you chain would have to be different then presented here, probably all certificate including the portal and gateway certs being signed by the sub CA completely, then all certs will be trusted by all devices. I’ll admit this isn’t the cleanest setup, but it’s the closest to a bare minimum install of Global Protect using your own internal PKI.

I hope this guide helps someone. 😀