Palo Alto VPN (GlobalProtect)
Part 5 – Rules, Testing, Troubleshooting

Intro

In this 5 Part series I covered all the requirements to configure Palo Alto Network’s GlobalProtect VPN:

1)  Authentication, Auth Profiles and testing them. 

2) Certificates, Cert Profiles, SSL/TLS Profiles and creating them.

3) Portals, what they do and how to configure them.

4) Gateways, what they do and how to configure them.

This part will cover the security rule required, and a little troubleshooting steps along the way.

Thing not Covered

I didn’t cover creating DNS records, as again, these come down to your own DNS provider and whatever tools and portals they offer to manage those.

I don’t cover configuring the interfaces (public facing or internal), I don’t cover the virtual router and routes. All these are assumed to be handled by the administrator reading these guides.

I don’t cover installing the client software, if you have the certificates installed on the client devices (Required), it’s simply navigating to the portal address with a supported browser and downloading the installation packages (.exe for windows).

For giggles, I tested navigating my portal from my phone, it did prompt me for my certificate (the VPN was working well) yet after selecting my certificate I got a connection reset error on my browser and checking the Palo Alto Firewall logs (Monitor tab -> traffic) I indeed saw the Deny traffic and action reset-both action… why this is, even though the application was identified correctly as web-browsing and that was enabled in the rule, it wasn’t being allowed by my rule and instead was being denied by my deny all rule. I”m not sure exactly why this is, however I don’t have intentions of accessing my portal web page anytime soon, so for now I’ll ignore this as I use IPsec XAuth RSA on my android device.

I have also noticed that for some reason with Samsung Android I can’t seem to get this VPN setup to work, from quick google searches people seem to say it’s due to packet fragmentation somehow. I haven’t yet had the chance to look into the nitty gritty of this issue just yet, but when I do it will be it’s own blog post!

I also don’t cover installing the completed certificates onto end devices as again this comes down to the end devices being supported by the administrator configuring Global Protect and is outside the scope of this guide.

The Security Rule

As you can tell pretty simple, anyone from the internet (I could be connecting from anywhere, and my IP address changes on my phone all the time, random access points etc) to my public IP address which hosts my portal and gateway, and the required applications (IKE, ipsec-esp-udp, and the SSL and web-browsing) again I haven’t exactly figured out the portal web-page loading issue just yet.

My Phone Config

In my case I do run an Android phone, running : 8.0.0: Kernel 4.4.78

The OS is some H93320g couldn’t find much but this about it

For the most part I install both my Offline-Root-CA and my Sub-CA certificates on my phone. Which can be found under (General -> Lock Screen & Security -> Encryption & Credentials -> Trusted Credentials (Instead of CA’s who knows?)  -> User (Both Should be listed here)

Then Installed the User certificate with the private key, which then shows up under (General -> Lock Screen & Security -> Encryption & Credentials -> User Credentials (Instead of User Certificates?)) The other annoying part is once you have the certificate installed, this area doesn’t allow you to see the certificate details, you can see them under the area mentioned above, but this area…. nope.. :@

Once the certificates are installed, it simply comes down to configuring the VPN settings. (Settings -> Network -> VPN -> BasicVPN -> Click the plus in the upper right hand corner. Then)

Name: Give it a meaningful name

Type: IPSec XAuth RSA

Server Address: The Address defined in Part 3 -> Agents -> External Gateways

IPSec User Cert: The User Certificate you installed and verified above

IPSec CA Certificate: Don’t verify server (Which is probably why I didn’t need the above server address in the gateway certs as a SAN)

IPSec Server Certificate: Receive from server

Then enter a username and password for a user you defined to be allowed per your Authentication Profile you created in Part 1.

You shouldn’t have to define the advanced settings as those should defined to the client from the gateway config we created in Part 4.

Summary

If done correctly you should have a successfully, you should be able to see all the parts play out in both the traffic logs, and the system logs…

System:

Traffic:

That is pretty much it, if you have a failed connection do the usual step by step troubleshooting starting with connectivity, you should be able to see the access attempt from the device in the traffic logs, if they are being blocked by rules, adjust them accordingly.

If you verified all other things, it maybe your chain, or you are enabling extra security like verifying the server certificate than you chain would have to be different then presented here, probably all certificate including the portal and gateway certs being signed by the sub CA completely, then all certs will be trusted by all devices. I’ll admit this isn’t the cleanest setup, but it’s the closest to a bare minimum install of Global Protect using your own internal PKI.

I hope this guide helps someone. 😀

Palo Alto VPN (GlobalProtect)
Part 4 – Gateways

Intro

The Gateway is pretty much exactly as it is named, the gateway where you get a virtual connection to tunnel into the network.

Requirements:

1)  And Interface with a Public IP address.
2) Certificates (Covered in Part 2)
3) Authentication Profile (Covered in Part 1)

Configuration

On the Palo Alto Firewall go to Network -> GlobalProtect -> Gateway

Under General give it a Name and define the interface in which has your Public IP address. *Note* The IP address can be left as none, this will work fine if your interface gets its IP address via DHCP, if you have static the static IP address should be populated from the drop down and can be selected. The Appearance section allows you to alter the web login portal that can be used to download the GlobalProtect client software.

Under Authentication Select a SSL/TLS Profile which contains the certificate which will secure this portal)

Then click add under Client Authentication and add the Auth Profile which states which users are going to be allowed to authenticate through this portal. Then select a Certificate Profile. (Covered in Part 2)

This is the first section that actually looks different than the portal configurations, under the Agent section the first area is the tunnel settings. This is where you define which tunnel interface (i picked the default, you may need to create additional tunnel interfaces if doing multiple portal/gateway configurations). In my case I was setting up tunnel based IPsec type VPN.

I left all the Timeout Settings as default, then moved onto client settings. Here we define any particular users, what OS they are allowed, and what IP addresses they are to be assigned (basically acts as a dedicated DHCP for the virtual tunnel interface when the VPN is established).

 

Next, under Network Services define the internal DNS server and WINS servers, as well as the DNS suffix users who connect will use, this will allow them to work as if they were locally at work.

In my case I didn’t have to deal with HIP Notifications or the Satellites section. 😀

That’s it for the Gateway, this unfortunately is not enough and we still need to define our Security Rules. Luckily since the Portal utilizes a public facing interfaces, we don’t have to deal with any NAT rules as connections are routed through the virtual tunnels that get created pretty much through the settings we defined in this part. 😀

As you can tell these post are a lot shorter as the hardest parts is building the pre-requisites.

Till Part 5, Cheers!

Palo Alto VPN (GlobalProtect)
Part 3 – Portals

Intro

The Portal is pretty much exactly as it is named, the portal where you fist connect to, validate you have the certificate to establish a secure communication to send your credentials over and tell your device what gateway to establish to tunnel connection with.

Requirements:

1)  And Interface with a Public IP address.
2) Certificates (Covered in Part 2)
3) Authentication Profile (Covered in Part 1)

Configuration

On the Palo Alto Firewall go to Network -> GlobalProtect -> Portals

Under General give it a Name and define the interface in which has your Public IP address. *Note* The IP address can be left as none, this will work fine if your interface gets its IP address via DHCP, if you have static the static IP address should be populated from the drop down and can be selected. The Appearance section allows you to alter the web login portal that can be used to download the GlobalProtect client software.

Under Authentication Select a SSL/TLS Profile which contains the certificate which will secure this portal)

Then click add under Client Authentication and add the Auth Profile which states which users are going to be allowed to authenticate through this portal.


Under the Agent section is where you define the which users group use which gateways. As well as which CA they use. *NOTE* The address defined as the gateway should created on your external DNS provider. Also it seem it is not required as a SAN on the certificate.

In my case I didn’t have to deal with Satellites section. 😀

That’s it for the Portal, this unfortunately is not enough and we still need to define our gateway as well, which ironically in a simple setup such as in my case and examples as a lot of the same steps.

As you can tell these post are a lot shorter as the hardest parts is building the pre-requisites. I also don’t cover creating your external DNS records as that comes down to your own DNS provider and the tools and services they provide.

Till Part 4, Cheers!

Palo Alto VPN (GlobalProtect)
Part 2 – Certificates

Certificates

In my previous post I covered recovering a downed CA, cause it will be needed for this section of the GlobalProtect tutorial.

Step 1) Importing the CA Certs

We need to add all the CA certs that are involved in completing the chain, so this includes, the Offline-Root-Ca, as well as the Sub Ca.

Adding the Sub CA cert:

Device -> Certs -> Import -> Base64 cer file

Step 2) Generating a CSR

Generate a a Sub CA Key for the PA to handle the Gateway certs, afterwards generate a Gateway certificate as well.

Click generate:

Click Generate

export the CSR, for some reason the latest Chrome causes a constant refresh, argggg had to export the CSR via IE, gross….

Navigate to your CA’s signing Web page (the Sub CA in this case), open the CSR in notepad and paste the results, and select Sub CA for the template:

Then save as Base64 type cert, and import back into the PA firewall, if successful will look like this:

Also import Offline-root-ca cert to complete the chain

Step 3) Certificate Profiles

Alright time for Certificate Profiles

Add all the Certs

Step 4) SSL/TLS Profiles

Create a SSL/TLS Profile:

Name it whatever, pick TLS 1.2 as min and max, and select the PA Sub CA we created earlier.

Step 5) Create User Certificate

Step 5.1) Create Template on CA

Then under Cert Templates, right click it, and duplicate

5 Years, i don’t like doing this often

Signature and encryption, check off include symmetric allowed by subject, min key size of 2048 and key is exportable

Along with the default, check off MS RSA and AES, and RSA SChannel

Subject Name, Supply in the Request, it will complain about the security risk, accept them. (Normally you’d create the certificates at the client machines, but in this case I am doint it the “wrong way” by having a global user certificate)

Click Apply.

If you require additional permissions apply them now, by default domain admins have full control, and domain users have enroll rights.

Step 5.2) Generate User CSR

With the Template configured, lets create the User Cert for the VPN, in this case we generate the CSR on the PA, but since we made the key exportable, we can export the certificate with key to be installed on the end device (instead of the CSR being generated on the device and then signed, and the public key being installed on the portal, which is the right way… hopefully I can get that, but the toughest part is generating certificates on phones, have to learn each devices OS on how to do it)

On the PA Device, Certs, Generate

*NOTE* I noticed that with the latest Chrome that when you attempt to export any certificate it just seems to refresh the page, sadly the only work around I have is to use IE… Ugh….

Open the CSR in Notepad, navigate to your Sub CA’s certificate signing page, sign the certificate.

*Secrete enable remote management on IIS Core*

lol, I was wondering why i couldn’t see my Template in the web interface, so I looked up my own very old blog post (3rd one I believe) and I realized I forgot to publish it, like I did the Authentication Session Template. Durrrr, then it kept complaining about https for cert destro (makes sense) but since I had a core subca, I couldn’t connect to the IIS remotely, then I found this, saved my bacon, and followed this to enable HTTPS, Then finally…

then Import it on to the Firewall,

it should look like this

In the next section I’ll cover configuring the Portal and Gateway settings. 😀

Resolving a down CA

Downed CA!

First off I wanted to address that this wasn’t the intended post of today, this was suppose to be part 2 of the Global Protect post, which is the second dependency; Certificates. However I recently had to get a new cert at work, and discovered this same issue as I had deployed my CA following a decent blog tutorial online by StealthPuppy, and sure enough the same sight provides a follow up on how to fix a mistake made, we all make them and these are great opportunities to learn, so lets get learning!

So you might happen to open up the Certificate Authority Snap and point it to your CA server, to find this…. it’s shutdown….

Fear not StealthPuppy has given us some helpful tips to resolve this!

The Issue

You might find your certificate authority, in this case, a subordinate certificate authority that is not started, perhaps after a server reboot. Attempting to start the CA, results in this message:

The revocation function was unable to check revocation because the revocation server was offline.
0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE)

Which looks like this:

In the Application log on the subordinate CA, I can see event id 100 from source CertificationAuthority:

Active Directory Certificate Services did not start: Could not load or verify the current CA certificate.  stealthpuppy Issuing CA The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE).

As well as, event id 48 from the same source, CertificationAuthority:

Revocation status for a certificate in the chain for CA certificate 0 for stealthpuppy Issuing CA could not be verified because a server is currently unavailable.  The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE).

Certificate 0 is the subordinate CA’s certificate, issued by the offline Root CA.

I bypassed this portion of the blog as I didn’t want to have pictures of before the next required step soooo….

The Workaround

Of course, you probably want to get the CA up and running as quickly as possible. The easy way to do that is to disable CRL checking with the following command on the CA server:

certutil –setreg ca\CRLFlags +CRLF_REVCHECK_IGNORE_OFFLINE

Run this from an elevated command prompt and you should now be able to start the CA and get on with the business of troubleshooting.

Perfect, now lets fix this!

The Cause

My CRL was online as it is available in Active Directory (for domain joined machines) and via HTTP at crl.home.stealthpuppy.com, an alias of the subordinate CA. I’ve tested that I can retrieve the CRL by putting the HTTP path into a browser and I’m prompted to download a file.

Through having spent some time recently with setting up an Enterprise PKI in my lab and for a project, I’ve come to know the command line tool certutil.exe. This tool is available in all versions of Windows and should be the first tool to use to troubleshoot and manage certificates and certificate authorities on Windows.

Certutil can be used to perform many functions, one of which is to verify a CRL. I know the path to the CRL file because I can view the CRLs on the file system (in C:\Windows\System32\certsrv\CertEnroll) and I’ve previously configured CRLs for both CAs.

To verify the CRL, use the -URL switch with the HTTP (or LDAP) path to the CRL:

certutil -URL "http://subca.zewwy.ca/CertEnroll/OFFLINE-ROOT-CA.crl"

This will display the URL Retrieval Tool that shows that the CRLs are able to be contacted and show a status of OK.

*NOTE* I discovered this works directly on the Windows Core Server, if you happened to be running Core (I do as I love optimization, specially when you have to work in lab environments). Except as you might have noticed from the screenshot, the radio buttons are all messed (wonder what library handles that, that wasn’t in core…)

However, if we load a target certificate, in this case, the subordinate CA’s cert, we can start to see why we have an issue with the CRL.

Select the certificate for the subordinate CA that has been previously exported to the file system (in C:\Windows\System32\certsrv\CertEnroll) – click Select, open the certificate and click Retrieve again. This time, we can see a new line that shows that the base CRL for the subordinate CA’s certificate is Expired. (Unfortunetly I had to simply use his source images as in my case I had to also correct my CDP locations on my sign Sub CA Certificate as I had mentioned in my initial Setup Offline Root CA blog post, but I guess I didn’t do it in my lab, so I sort of had to fix to birds with one stone, the CPD locations in my cert by reissuing that, and the offline root ca CRL file, so my additional steps were a bit beyond these exact steps, I also didn’t take snapshots as I wanted to get back on pace with my blog post)

An expired base CRL on the Subordinate CA

An expired base CRL on the Subordinate CA

The CRL for the subordinate CA’s certificate will come from the root CA, so we’ll need to check that CRL. Open the CRL file (C:\windows\system32\certsrv\CertEnroll\stealthpuppy Offline Root CA.crl) – double-click or right-click and Open. Here we can see the CRL information, including the next publishing time (Next CRL Publish).

Viewing the properties of the Root CA's CRL

Viewing the properties of the Root CA’s CRL

At the time of troubleshooting, this date was in the past and because the Root CA is offline and the CRL is hosted on a different server (the subordinate CA), this particular CRL will never receive an update. So, when the subordinate CA has rebooted, it has checked the Root CA’s CRL and found it expired. Hence the certification authority service won’t start.

How To Fix It

Now we know why the certification authority service won’t start and an understanding of why the CRL is offline, even if the wording doesn’t match the symptoms. If the error message had told me the CRL had expired instead of being offline, I might have saved some troubleshooting time. We now know that we need to re-publish the CRL from the Root CA.

Start the offline Root CA, log into it and open the Certification Authority console. We will first want to ensure that the CRL publication interval is extended so that we don’t run into the same problem in the near future. Open the properties of the Revoked Certificates node to view and set the publication interval. The default interval is 1 week, obviously too often for an offline Root CA.

Instead, set this value to something suitable for the environment you have installed the CA into. Remember that you’ll need to boot the Root CA and publish a new CRL before the end of this interval, otherwise, you’ll have exactly the same issue.

Setting the CRL Publication Interval on the Root CA

Setting the CRL Publication Interval on the Root CA

Now publish a new CRL – right-click the Revoked Certificates node and click All Tasks /Publish.

Publishing a new CRL from the Root CA

Publishing a new CRL from the Root CA

Copy the updated CRL (from C:\Windows\System32\certsrv\CertEnroll by default) from the Root CA to the CRL distribution point and overwrite the existing CRL file (C:\Windows\System32\certsrv\CertEnroll again on my subordinate CA).

Now if we again use certutil.exe to verify the CRL, it comes up roses:

Viewing the verified CRL with certutil.exe

Viewing the verified CRL with certutil.exe

To ensure that the subordinate CA’s certification authority service will start, re-enable CRL checking:

certutil –setreg ca\CRLFlags -CRLF_REVCHECK_IGNORE_OFFLINE

If you have re-published the CRL from the Root CA correctly, the service should start and you can then shut down the Root CA. Then open Outlook and put a reminder in the calendar for a week before the CRL expires again.

Conclusion

I’ve had this issue with an Offline CRL a few times now and not really understood what the issue is until I took the time to troubleshoot the issue properly. I don’t spend that much time with an enterprise PKI and it’s easy to underestimate the complexity of setting up AD Certificate Services correctly.

SSH Banners

I love SSH… like I really love it. It is pretty surprising that Windows only first had the ability to naively ssh only in the recent Windows 10 1803 build. That’s pretty sad. Just teste don my 1803 build.. nope… well I know I have done it before… anyway teh point I wanted to get here was more about SSH servers.

Now normally allowing SSH in to a system basically enables an SSH services on that system, making it the SSH server. Then you usually utilize a workstation, like the computer you usually use to navigate websites, like this one, to connect to that server with a piece of software (with Windows that’s usually Putty)… if I can get that dang native ssh to work (shakes fist)… anyway…

When you enable this service it is pretty powerful, depending on how you configure it, and what application is running the service. There are plenty of flavors to choose from (this is pretty common with Linux and open source). This is usually a good thing cause each one is scoped for a certain target audience. In my case I wanted to bring some old life back to my old Asus Router. I’ve been running DDWRT on it for a long time, and utilizing the simple command line interface (embedded linux) on a decent lil system only working as an AP otherwise, is a fun lil place to use IRC. 🙂 Find me on #Freenode (#Windows-Server, VMware, Cisco, Skullspace, FreeNAS) .

Now I figured if I was going to use this again, why not have some fun and re-do my loggin banner. Now in this case there are two things to consider:

1)  The Message of the Day (MOTD) – This displays as soon as a client connects before it asks for a username. In most cases this is a great place to place your unauthorized message. (In my case the MOTD was tied to a Read-Only FileSystem file, and I had no intentions of compiling my own build, so I decided to utilize the option to not display this).

2) The Login Banner – This message displays after you have specified a user name.

Now there can be many ways to customize your login banner, you may need to google the based on the SSH server you are using. In my case my router was utilizing dropbear lucky for me they have decent documentation.

In my case I simply had to create a simple text file pointing anywhere using the -b option: dropbear -b /somepath/banner.file

After I created my file I configured my startup script to point to my new banner file. Sure enough now when I log on I see this:

Boo yeah! Now that’s sweet.

BitLocker Can’t find the file

This was an interesting one, created a new Windows image to deploy recently. Then after deployment went to enable bitlocker and was prompted with the error “The system cannot not find the file specified”. Since this was new to new, what other than to do a web search to see if anyone else had experienced this, and sure enough, yup.

Short answer: rename REAgent.xml file (in C:\Windows\System32\Recovery) to REAgent.xml.old (or dlete it but I haven’t tested that).

and it worked, apparently….

“Sooooo, what we have found is that when we captured the image, since we had already opened the Bitlocker console (even though we hadn’t actually Bitlocked the unit), the REAgent.xml file (in C:\Windows\System32\Recovery) had been populated with the specific GUIDs for both WinreBCD and WinreLocation path.” – Borch25

I like borch, can’t wait for more.

Resetting Local Admin Password in Windows
Bypassing Windows Login

I’m getting ready soon to do a presentation on hacking my works laptops. I was giving the green light on a spare laptop we had purchased for corporate use. So in this case my test bed was a HP Folio 9470m, decent little guy for most basic office work. Like most places we run Windows, staying with the latest updates it was configured for Windows 10.

I won’t get into to much technical details as I’ll save that for the Long Con coming up this November. However, like most security and hacks there are many layers involved and the windows login just happens to be one of them.

Now I am already an admin on these systems, however I assume the role of a perpetrator and choose to find ways to break in as if I was not a current admin. This is where my mind got blown!

Alright, so the basic thing you need to break into windows is…. Windows! lol, well ok, you can actually pull this trick off with other… (Yup, just tested with lubuntu 17 against my Windows 7 imaged laptop)… OSes, but in this case we’ll stick with using your windows install media, as you likely already have this on hand in whatever form you may need it (CD/DVD, USB… or ughhhh unno whatever).

Once in the System32 Directory simply run the following:

move Utilman.exe Utilman.exe.bak

copy cmd.exe Utilman.exe

That is pretty much it, reboot. When you are prompted in login, press the Windows Key + U and marvel at how you get a elevated command prompt, then type lusrmgr in the command window to get the local users and groups windows and alter account, create accounts, change passwords to your hearts content.

Remember the bare minimum you need is any OS that can read NTFS (Usually for most Windows installations) and physical access to the system you are attempting to get into (given the boot options are not locked down by the UEFI/BIOS) which is the next layer I’ll talk about in my next blog post.

If you need a bit more “hold my hand” guidance in pulling this off here is a good source. This of course was using the Windows 7 installation media, against a Windows 7 machine, but the general concept of the trick, replacing UtilMan.exe with the cmd.exe can be done in many ways, then when the Windows image boots and you are at the login screen literally clicking the accessibility icon, or CTRL + U will open an elevated command prompt.

Util next time… Keep fit and have fun. 😛

I Spike You!

Ahhh the internet….

Publish my own personal mx record in hopes to get my own email going….
I decided to see why my email outbound wasn’t working (sigh even following Paul Cunningham’s post seems I’m missing something) seems all my out-bound based SMTP connections to external mail servers seems to be failing. According to my firewall (Palo Alto) The rule is allowing it out but the application shows incomplete… like it’s never establishing a connection. So from my previous posts, I use telnet to attempt a connection on external known IPs for SMTP mail server, and sure enough no connections can be established (I know I’ll eventually have to create a receive connector from outbound sources and create a security rule to allow email from outside in, but I wanted to tackle email going out first).

I decided to attempt the same port 25 connection to the new record I created (I have multiple internet connection to utilize to actually test connections from “outside” instead of having to rely on a loop back NAT rule or anything). to my dismay it showed failed to connect (I already expected this as I created a NAT rule but I never created a security rule to allow the connection). I decided to go to my Monitor tab to see if I could see the attempted connection, I indeed did see it. However what surprised me more was the failed attempts from others in the short time I created this record (considering I had the IP for a long time and pretty much all ports were blocked forever, I didn’t expect there to be much attempts) these were either crawlers or something else…. but guess who the every first was….

141.212.122.227
University of Michigan (AS36375)

Not once, but twice from two sequential IP addresses…. Mhmmm what are those Michigans up to?

185.35.62.150… unknown, someone remaining anonymous, Michigian Hookup? occurred 3 minutes after.

Then Hours Later….

107.170.227.216
Digital Ocean, Inc. (AS14061)

Not sure who they are, might have to check em out..

Couple hours later…

46.29.161.101…. Anonymous

I guess it only makes sense after Americans, and Anonymous it be nothing other than the Russians right…. To be fair I don’t actually known wtf thi sis lol, Japanese mixed with Russian or something pile of who knows what.

95.181.178.182
FOP ILIUSHENKO VOLODYMYR OLEXANDROVUCH (AS57311)

They are least tried three times in a row from same IP (Good thought idea, if it doesn’t work once, heck try again a couple times)

Then my attempt… pretty funny what you can hear if you just listen…

This isn’t actually I Spike You! Like from the old school GoldenEye movie, but this is what you’d actually do if you wanted to “Spike” someone online, this is my actual server I plan to use of course, but if I actually wanted to find out what people are up to I’d create a honeypot. Maybe now that I post this, they’ll think my mx record is a honeypot, but it’ll secretly become in use… sometime…. lol