Palo Alto VPN (GlobalProtect)
Part 4 – Gateways

Intro

The Gateway is pretty much exactly as it is named, the gateway where you get a virtual connection to tunnel into the network.

Requirements:

1) And Interface with a Public IP address.
2) Certificates (Covered in Part 2)
3) Authentication Profile (Covered in Part 1)

Configuration

On the Palo Alto Firewall go to Network -> GlobalProtect -> Gateway

Under General give it a Name and define the interface in which has your Public IP address. *Note* The IP address can be left as none, this will work fine if your interface gets its IP address via DHCP, if you have static the static IP address should be populated from the drop down and can be selected. The Appearance section allows you to alter the web login portal that can be used to download the GlobalProtect client software.

Under Authentication Select a SSL/TLS Profile which contains the certificate which will secure this portal)

Then click add under Client Authentication and add the Auth Profile which states which users are going to be allowed to authenticate through this portal. Then select a Certificate Profile. (Covered in Part 2)

This is the first section that actually looks different than the portal configurations, under the Agent section the first area is the tunnel settings. This is where you define which tunnel interface (i picked the default, you may need to create additional tunnel interfaces if doing multiple portal/gateway configurations). In my case I was setting up tunnel based IPsec type VPN.

I left all the Timeout Settings as default, then moved onto client settings. Here we define any particular users, what OS they are allowed, and what IP addresses they are to be assigned (basically acts as a dedicated DHCP for the virtual tunnel interface when the VPN is established).

 

Next, under Network Services define the internal DNS server and WINS servers, as well as the DNS suffix users who connect will use, this will allow them to work as if they were locally at work.

In my case I didn’t have to deal with HIP Notifications or the Satellites section. 😀

That’s it for the Gateway, this unfortunately is not enough and we still need to define our Security Rules. Luckily since the Portal utilizes a public facing interfaces, we don’t have to deal with any NAT rules as connections are routed through the virtual tunnels that get created pretty much through the settings we defined in this part. 😀

As you can tell these post are a lot shorter as the hardest parts is building the pre-requisites.

Till Part 5, Cheers!

Palo Alto VPN (GlobalProtect)
Part 3 – Portals

Intro

The Portal is pretty much exactly as it is named, the portal where you fist connect to, validate you have the certificate to establish a secure communication to send your credentials over and tell your device what gateway to establish to tunnel connection with.

Requirements:

1) And Interface with a Public IP address.
2) Certificates (Covered in Part 2)
3) Authentication Profile (Covered in Part 1)

Configuration

On the Palo Alto Firewall go to Network -> GlobalProtect -> Portals

Under General give it a Name and define the interface in which has your Public IP address. *Note* The IP address can be left as none, this will work fine if your interface gets its IP address via DHCP, if you have static the static IP address should be populated from the drop down and can be selected. The Appearance section allows you to alter the web login portal that can be used to download the GlobalProtect client software.

Under Authentication Select a SSL/TLS Profile which contains the certificate which will secure this portal)

Then click add under Client Authentication and add the Auth Profile which states which users are going to be allowed to authenticate through this portal.


Under the Agent section is where you define the which users group use which gateways. As well as which CA they use. *NOTE* The address defined as the gateway should created on your external DNS provider. Also it seem it is not required as a SAN on the certificate.

In my case I didn’t have to deal with Satellites section. 😀

That’s it for the Portal, this unfortunately is not enough and we still need to define our gateway as well, which ironically in a simple setup such as in my case and examples as a lot of the same steps.

As you can tell these post are a lot shorter as the hardest parts is building the pre-requisites. I also don’t cover creating your external DNS records as that comes down to your own DNS provider and the tools and services they provide.

Till Part 4, Cheers!

Resolving a down CA

Downed CA!

First off I wanted to address that this wasn’t the intended post of today, this was suppose to be part 2 of the Global Protect post, which is the second dependency; Certificates. However I recently had to get a new cert at work, and discovered this same issue as I had deployed my CA following a decent blog tutorial online by StealthPuppy, and sure enough the same sight provides a follow up on how to fix a mistake made, we all make them and these are great opportunities to learn, so lets get learning!

So you might happen to open up the Certificate Authority Snap and point it to your CA server, to find this…. it’s shutdown….

Fear not StealthPuppy has given us some helpful tips to resolve this!

The Issue

You might find your certificate authority, in this case, a subordinate certificate authority that is not started, perhaps after a server reboot. Attempting to start the CA, results in this message:

The revocation function was unable to check revocation because the revocation server was offline.
0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE)

Which looks like this:

In the Application log on the subordinate CA, I can see event id 100 from source CertificationAuthority:

Active Directory Certificate Services did not start: Could not load or verify the current CA certificate. stealthpuppy Issuing CA The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE).

As well as, event id 48 from the same source, CertificationAuthority:

Revocation status for a certificate in the chain for CA certificate 0 for stealthpuppy Issuing CA could not be verified because a server is currently unavailable. The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE).

Certificate 0 is the subordinate CA’s certificate, issued by the offline Root CA.

I bypassed this portion of the blog as I didn’t want to have pictures of before the next required step soooo….

The Workaround

Of course, you probably want to get the CA up and running as quickly as possible. The easy way to do that is to disable CRL checking with the following command on the CA server:

certutil –setreg ca\CRLFlags +CRLF_REVCHECK_IGNORE_OFFLINE

Run this from an elevated command prompt and you should now be able to start the CA and get on with the business of troubleshooting.

Perfect, now lets fix this!

The Cause

My CRL was online as it is available in Active Directory (for domain joined machines) and via HTTP at subca.zewwy.ca, an alias of the subordinate CA. I’ve tested that I can retrieve the CRL by putting the HTTP path into a browser and I’m prompted to download a file.

Through having spent some time recently with setting up an Enterprise PKI in my lab and for a project, I’ve come to know the command line tool certutil.exe. This tool is available in all versions of Windows and should be the first tool to use to troubleshoot and manage certificates and certificate authorities on Windows.

Certutil can be used to perform many functions, one of which is to verify a CRL. I know the path to the CRL file because I can view the CRLs on the file system (in C:\Windows\System32\certsrv\CertEnroll) and I’ve previously configured CRLs for both CAs.

To verify the CRL, use the -URL switch with the HTTP (or LDAP) path to the CRL:

certutil -URL "http://subca.zewwy.ca/CertEnroll/OFFLINE-ROOT-CA.crl"

This will display the URL Retrieval Tool that shows that the CRLs are able to be contacted and show a status of OK.

*NOTE* I discovered this works directly on the Windows Core Server, if you happened to be running Core (I do as I love optimization, specially when you have to work in lab environments). Except as you might have noticed from the screenshot, the radio buttons are all messed (wonder what library handles that, that wasn’t in core…)

However, if we load a target certificate, in this case, the subordinate CA’s cert, we can start to see why we have an issue with the CRL.

Select the certificate for the subordinate CA that has been previously exported to the file system (in C:\Windows\System32\certsrv\CertEnroll) – click Select, open the certificate and click Retrieve again. This time, we can see a new line that shows that the base CRL for the subordinate CA’s certificate is Expired. (Unfortunetly I had to simply use his source images as in my case I had to also correct my CDP locations on my sign Sub CA Certificate as I had mentioned in my initial Setup Offline Root CA blog post, but I guess I didn’t do it in my lab, so I sort of had to fix to birds with one stone, the CPD locations in my cert by reissuing that, and the offline root ca CRL file, so my additional steps were a bit beyond these exact steps, I also didn’t take snapshots as I wanted to get back on pace with my blog post)

The CRL for the subordinate CA’s certificate will come from the root CA, so we’ll need to check that CRL. Open the CRL file (C:\windows\system32\certsrv\CertEnroll\stealthpuppy Offline Root CA.crl) – double-click or right-click and Open. Here we can see the CRL information, including the next publishing time (Next CRL Publish).

At the time of troubleshooting, this date was in the past and because the Root CA is offline and the CRL is hosted on a different server (the subordinate CA), this particular CRL will never receive an update. So, when the subordinate CA has rebooted, it has checked the Root CA’s CRL and found it expired. Hence the certification authority service won’t start.

How To Fix It

Now we know why the certification authority service won’t start and an understanding of why the CRL is offline, even if the wording doesn’t match the symptoms. If the error message had told me the CRL had expired instead of being offline, I might have saved some troubleshooting time. We now know that we need to re-publish the CRL from the Root CA.

Start the offline Root CA, log into it and open the Certification Authority console. We will first want to ensure that the CRL publication interval is extended so that we don’t run into the same problem in the near future. Open the properties of the Revoked Certificates node to view and set the publication interval. The default interval is 1 week, obviously too often for an offline Root CA.

Instead, set this value to something suitable for the environment you have installed the CA into. Remember that you’ll need to boot the Root CA and publish a new CRL before the end of this interval, otherwise, you’ll have exactly the same issue.

Now publish a new CRL – right-click the Revoked Certificates node and click All Tasks /Publish.

Copy the updated CRL (from C:\Windows\System32\certsrv\CertEnroll by default) from the Root CA to the CRL distribution point and overwrite the existing CRL file (C:\Windows\System32\certsrv\CertEnroll again on my subordinate CA).

Now if we again use certutil.exe to verify the CRL, it comes up roses:

To ensure that the subordinate CA’s certification authority service will start, re-enable CRL checking:

certutil –setreg ca\CRLFlags -CRLF_REVCHECK_IGNORE_OFFLINE

If you have re-published the CRL from the Root CA correctly, the service should start and you can then shut down the Root CA. Then open Outlook and put a reminder in the calendar for a week before the CRL expires again.

Conclusion

I’ve had this issue with an Offline CRL a few times now and not really understood what the issue is until I took the time to troubleshoot the issue properly. I don’t spend that much time with an enterprise PKI and it’s easy to underestimate the complexity of setting up AD Certificate Services correctly.

Palo Alto VPN (GlobalProtect)
Part 1 – Authentication

Hey all,

I’d figure I’d spin up a really quick guide on setting up client type VPN using Palo Alto Firewalls. In this case it will use GlobalProtect, so will require appropriate licenses to work. I won’t exactly cover the license aspect in my blog as I find that stuff a bore, and can leave that for the good ol’ VARs to handle. Anyway let’s get into this.

Authentication

Palo Alto firewalls support a wide range of authentication sources (which is awesome) however, to start my lab utilizes LDAP. There are ways to secure LDAP with certs and TLS for LDAPS however again that is beyond the scope of this blog post, and will stick with plaintext LDAP connection son port 389. (In my case all my servers reside on one hypervisor, thus the chances of the vswitch being man in the middle is highly unlikely ;))

Step 1) Add a Server Profile

So to start on the Palo Alto (My Examples utilize PAN OS 7.1.x, however, 5-6, as well as 8 are very similar) Web interface go to Device -> Server Profiles -> LDAP then click add.

Give the Profile a meaningful name, then click on the add button under server list.

Couple Notes here:

1) The Name doesn’t have to be the server name, just give it a meaninful name, or give it the server name, whatever you choose (it’s not a DNS lookup)

2) I was having some issues with testing the Auth profile (Which will be covered a bit further down in the blog post) and I thought I may have had issues with my LDAP settings, however “During LDAP server configuration, the device automatically pulls the Base DN if the connection is successful.

Since I didn’t have LDAPS setup, I had no cert to use, so I had to uncheck “Require SSL/TLS secured connection” only then would the Base DN auto-populate, verifying that my LDAP connection was indeed successful.

Also note in my lab I had a separate NIC dedicated to my domain, which was separate from the usual home network, thus I had to setup a “Service Routes” for LDAP specifically.

I decided to also create a standard domain account and use it for the Bind DN. This may or may not be required.

Now this is where things get a little weird….

According to step 2 from this Palo Alto post you setup the authentication profile.

Step 2) Group Mapping (Optional)

Which is technically the bare minimum, however, I was hoping to have some group filtering at the Auth Profile, in order to do this one has to first setup a “Group Mapping” and followed this to do it:

Device -> User Identification -> Group Mapping

Click Add, Give it a meaningful name, and optionally fill in the domain

Helpful Commands

  • show user group-mapping state all
    Lists all group mappings on the firewall, which shows the Server Profile used and all groups found from it
  • show user group list
    Also lists all groups, but without ordering via the group mapping info
  • show user group name <group name>
    Lists users within the group specified
  • debug user-id refresh group-mapping all
    If new groups or users added to groups in AD, refresh the info on the PA

*Update* anonymous connection won’t error but will not populate groups when creating a group mapping:

If you do not setup a group mapping, you can’t filter by groups in the Authentication Profile, under the “allow list” which from testing might not be a bad thing… you’ll see…

Step 3) Create an Authentication Profile

This is where my results get weird, which I finally discovered was all due to the “allow list” filtering in which I was attempting to use. Let me show you what I mean, using my first authentication profile as an example:

So nothing special, select LDAP, selected my LDAP Server Profile, then click the next tab is it is required “Advanced” tab.

Now if you setup a group mapping as I specified under the option step 2, you will see all the groups in the domain (CN=xxxx,OU=xxxx,DC=xxxx) format. If you didn’t you will get only all (and that is probably a good thing, not exactly sure yet,…)

So in my first case I decided to use my “domain users” as my filter group, as it’s pretty common group and I don’t have many users in it.

Simple, so now we have all the basics to now test it.

Step 4) Testing the Authentication Profile

Test and fail and test and fail, and smash your face in for using group mappings…. (yet to be determined why)… but for now, log in via SSH into the Palo Alto firewall and test it:

What the heck?!?! I swear everything was correct, I also knew I was entering my password correctly and everything. No matter how you change the settings in the Auth Profile nothing works (I double and triple checked proper group membership in AD, and ensured the firewall could see it all “show user group” and “show user group name “CN=domain users,OU=users,DC=zewwy,DC=ca””) until I noticed from the link in this steps title that his simple test was using that “all” group that is used when there is no group mapping defined, or just simple at the top of the list.

So creating a new Authentication Profile not specifying my AD group..

and testing…

BAMMMMMMMM….

And that concludes the Authentication part well at least the dependency, in part 2 I’ll cover the certificates requirements. I currently have a ticket open with Palo ALto as to why the “Allow List” section of the Authentication Profile doesn’t pick up the users in which should be in those account as shown by the “show user group name “CN group name”, so if this commands shows users based on a group, the allow list should be able to validate that as well.

Until Part 2, Adios!

*UPDATE* According to PAN support the Test Authentication *BUG* although he didn’t want to admit it as a bug is claimed to be fixed in PAN OS 8.1.3. I have yet to upgrade some units to this version and verify that claim.

*UPDATE 2* At the time, I wasn’t sure what version I was running, and on my new system, I was now on 8.1.0, and PAN changed the layout to the group mapping area by adding a new tab (User and Group Attributes)

which in 8.1.0 did not auto populate and since Primary Username was the only required field I did enter “sAMAccountName” but I guess it wasn’t good enough and every time I attempted a test it fail.. and it was driving me nuts…

then I noticed something, in this post… even though it was different, and I noticed in my pictures afterwards too, and I had set my profile to other instead of active-directory.

even after that it was still failing!!

and this was even after updating to 8.1.10 after reading my own previous update that it had been patched, so I decided to create my auth profile and group mapping from scratch and that’s when I noticed it had now auto populated the new tab and I didn’t have to intervene and thus it worked now one 8.1.0 and following all the steps as before carefully:

I decided to quickly test to see if they had fixed the problem I had specified before now in 8.1.10:

nope double checking the group mapping:

I’m not sure right at this moment why the list shows in this style fullyqualfied zewwy.ca\user vs NetBIOS style zewwy\user

then I decided to test trying these:

it appears only matching the fully qualified name entered succeeds, I’ll have to play around to see if I can get that cleaned up to be auto filled so a user only needs to enter a username, or maybe this is required, figured a primary domain can be specified so it doesn’t need to be entered all the time….

well apparently I won’t be able to use a users UPN for this, not yet anyway…

*Update 2023* I updated to PAN-OS 10.2.4 and the “bug” still existed, for tested I removed the group mapping, then tested auth again. Test auth against the auth profile with “any” defined still worked, the auth profile that was restricted to the AD group didn’t work no matter how I formatted the user name. This makes me believe the Allow Group requires group mappings to work.

IIS redirect HTTP to HTTPS

The Requirement

Recently I posted about creating and using a certificate for use on and IIS website. I did this recently for my Dev to deploy his new web app, which required use of a devices camera for QR scanning purposes, well it runed out that the camera could not be used unless the app was secured with TLS (a certificate). So we created a cert and created a new secure binding.

However, it was soon pretty apparent every time I pressed the down arrow key on my browsers URL it would use the regular HTTP (at first we removed the port 80 binding which caused site unavailable issue). So after re-adding the standard port 80 binding for regular HTTP, I decided to use a rewrite rule to handle the redirection.

It wasn’t as intuitive as I thought it would be so a little google search and I was on my way…

As the source states, after you have your certificate and port binding for HTTPS, as well as one for regular HTTP.

In order to force a secure connection on your website, it is necessary to set up a certain HTTP/HTTPS redirection rule. This way, anyone who enters your site using a link like “yourdomain.com” will be redirected to “https://yourdomain.com” or “https://www.yourdomain.com” (depending on your choice) making the traffic encrypted between the server and the client side.

Below you can find the steps for setting up the required redirect:

The Source

    1. Download and install the “URL Rewrite” module.
    2. Open the “IIS Manager” console and select the website you would like to apply the redirection to in the left-side menu:iisred1
    3. Double-click on the “URL Rewrite” icon.
    4. Click “Add Rule(s)” in the right-side menu.
    5. Select “Blank Rule” in the “Inbound” section, then press “OK”:iisred2
    6. Enter any rule name you wish.
    7. In the “Match URL” section:- Select “Matches the Pattern” in the “Requested URL” drop-down menu
      – Select “Regular Expressions” in the “Using” drop-down menu
      – Enter the following pattern in the “Match URL” section: “(.*)”
      – Check the “Ignore case” box

      iisred3

    8. In the “Conditions” section, select “Match all” under the “Logical Grouping” drop-down menu and press “Add”.
    9. In the prompted window:
      – Enter “{HTTPS}” as a condition input
      – Select “Matches the Pattern” from the drop-down menu
      – Enter “^OFF$” as a pattern
      – Press “OK”

      iisred4

    10. In the “Action” section, select “Redirect” as the action type and specify the following for “Redirect URL”:https://{HTTP_HOST}/{R:1}
    11. Check the “Append query string” box.
    12. Select the Redirection Type of your choice. The whole “Action” section should look like this:iisred5

The Note

NOTE: There are 4 redirect types of the redirect rule that can be selected in that menu:
– Permanent (301) – preferable type in this case, which tells clients that the content of the site is permanently moved to the HTTPS version. Good for SEO, as it brings all the traffic to your HTTPS website making a positive effect on its ranking in search engines.
– Found (302) – should be used only if you moved the content of certain pages to a new place *temporarily*. This way the SEO traffic goes in favour of the previous content’s location. This option is generally not recommended for a HTTP/HTTPS redirect.
– See Other (303) – specific redirect type for GET requests. Not recommended for HTTP/HTTPS.
– Temporary (307) – HTTP/1.1 successor of 302 redirect type. Not recommended for HTTP/HTTPS.

  1. Click on “Apply” on the right side of the “Actions” menu.

The redirect can be checked by accessing your site via http:// specified in the URL. To make sure that your browser displays not the cached version of your site, you can use anonymous mode of the browser.

The Helping Hand

The rule is created in IIS, but the site is still not redirected to https://

Normally, the redirection rule gets written into the web.config file located in the document root directory of your website. If the redirection does not work for some reason, make sure that web.config exists and check if it contains the appropriate rule.

To do this, follow these steps:

  1. In the sites list of IIS, right-click on your site. Choose the “Explore” option:iisred6
  2. “Explore” will open the document root directory of the site. Check if the web.config file is there.
  3. The web.config file must have the following code block:
    <configuration>
    <system.webServer>
    <rewrite>
    <rules>
    <rule name=”HTTPS force” enabled=”true” stopProcessing=”true”>
    <match url=”(.*)” />
    <conditions>
    <add input=”{HTTPS}” pattern=”^OFF$” />
    </conditions>
    <action type=”Redirect” url=”https://{HTTP_HOST}/{R:1}” redirectType=”Permanent” />
    </rule>
    </rules>
    </rewrite>
    </system.webServer>
    </configuration>
  4. If the web.config file is missing, you can create a new .txt file, put the aforementioned code there, save and then rename the file to web.config.

This one, much like my Offline Root CA post is a direct copy of the source. For the same reason, I felt I had no need to re-word it as it was very well written.

Thanks Namecheap, although it would have been nice to thank the actual author who took the time to do the beautiful write up with snippets. 🙂

MSI installer – Network Resource Unavailable

The Start

The other day my boss came in with a colleagues laptop and told me that the VPN software failed to update, and by fail to update completely removed the old version and didn’t update to the latest version. Checking the laptops ‘Programs and Features’ sure enough showed no signs of the application.

I simply grabbed the latest version of the software installer and attempt to reinstall the application, yet to my dismay the latest installer complains with the following error ‘The feature you are trying to use is on a network resource that is unavailable.’ How insightful…

I figured it was a registry issue, as the registry is known to hold old settings from applications and do not get cleaned up properly. This place can be a cease pool on old machines that have been constantly upgraded.

The Dig

There are plenty of references online when it comes to this error. The main one people reference is the HKLM/Software/Classes/installer/products.

Even though I cleaned everything in this based on the application I was installing, it was still failing with with same error.

The Answer

Lucky for me it kept specifying what it was expecting for a network path, I decided to search the registry based on this, and found there was a key.

After removing the parent GUID based key, the installer ran successfully.

If my memory serves me correctly I believe the problematic key was under:

HKEY_CLASSES_ROOT\Installer\Products

SSH Banners

I love SSH… like I really love it. It is pretty surprising that Windows only first had the ability to naively ssh only in the recent Windows 10 1803 build. That’s pretty sad. Just teste don my 1803 build.. nope… well I know I have done it before… anyway teh point I wanted to get here was more about SSH servers.

Now normally allowing SSH in to a system basically enables an SSH services on that system, making it the SSH server. Then you usually utilize a workstation, like the computer you usually use to navigate websites, like this one, to connect to that server with a piece of software (with Windows that’s usually Putty)… if I can get that dang native ssh to work (shakes fist)… anyway…

When you enable this service it is pretty powerful, depending on how you configure it, and what application is running the service. There are plenty of flavors to choose from (this is pretty common with Linux and open source). This is usually a good thing cause each one is scoped for a certain target audience. In my case I wanted to bring some old life back to my old Asus Router. I’ve been running DDWRT on it for a long time, and utilizing the simple command line interface (embedded linux) on a decent lil system only working as an AP otherwise, is a fun lil place to use IRC. 🙂 Find me on #Freenode (#Windows-Server, VMware, Cisco, Skullspace, FreeNAS) .

Now I figured if I was going to use this again, why not have some fun and re-do my loggin banner. Now in this case there are two things to consider:

1) The Message of the Day (MOTD) – This displays as soon as a client connects before it asks for a username. In most cases this is a great place to place your unauthorized message. (In my case the MOTD was tied to a Read-Only FileSystem file, and I had no intentions of compiling my own build, so I decided to utilize the option to not display this).

2) The Login Banner – This message displays after you have specified a user name.

Now there can be many ways to customize your login banner, you may need to google the based on the SSH server you are using. In my case my router was utilizing dropbear lucky for me they have decent documentation.

In my case I simply had to create a simple text file pointing anywhere using the -b option: dropbear -b /somepath/banner.file

After I created my file I configured my startup script to point to my new banner file. Sure enough now when I log on I see this:

Boo yeah! Now that’s sweet.

BitLocker Can’t find the file

This was an interesting one, created a new Windows image to deploy recently. Then after deployment went to enable bitlocker and was prompted with the error “The system cannot not find the file specified”. Since this was new to new, what other than to do a web search to see if anyone else had experienced this, and sure enough, yup.

Short answer: rename REAgent.xml file (in C:\Windows\System32\Recovery) to REAgent.xml.old (or dlete it but I haven’t tested that).

and it worked, apparently….

“Sooooo, what we have found is that when we captured the image, since we had already opened the Bitlocker console (even though we hadn’t actually Bitlocked the unit), the REAgent.xml file (in C:\Windows\System32\Recovery) had been populated with the specific GUIDs for both WinreBCD and WinreLocation path.” – Borch25

I like borch, can’t wait for more.