Fixing SSL Decryption Issues in Palo Alto PAN-OS 10

Palo Alto Networks has recently introduced PAN-OS 10, and added some pretty nifty features such as Machine Learning (ML) baked into the core of the operating system for better prevention of web-based attacks, improved IoT security and device identification, and the ability to secure containerized applications across Kubernetes deployments.

But one feature that perhaps excites me the most is the improved SSL decryption troubleshooting.

The Many Woes of SSL Decryption

Any system or network administrator that has provisioned SSL decryption on any firewall knows that they'll be spending the next few days (weeks/months?) fixing web pages that don't load properly, applications that don't connect, SaaS logins that no longer work etc. It's a headache.

The main cause for all of these issues is that certain applications have the certificate of the server(s) it needs to connect to embedded or 'pinned' into them. So when a firewall intercepts the traffic and re-signs it with its own certificate, the application says Nope! and prevents the connection.

The severity of this can vary. Perhaps it means the application can't load images. Or perhaps you can't log into the application. Or maybe, as in the case of Dropbox, it fails to sync.

How to Fix SSL Decryption Issues

The solution to all this is to find the SNI (Server Name Identification) of the certificate being used by the application and excluding it from your firewall's SSL decryption feature.

Before PAN-OS v10, this was easier said than done in Palo Alto firewalls. The failing traffic would not even make it into the Threat, URL or Traffic logs as it failed before it reached that point. So identifying the SNI you needed to exclude required crawling through forum posts (hopefully someone has had the issue before you!) random guessing with trial and error, or poking through horrible packet captures hoping you'll stumble across what you need.

So it is with open arms that I welcome the new Decryption Failure Reasons widget in PAN-OS 10.

This section lists out all the SNIs associated with SSL decryption failures. Most times, the SNI is easy to recognise. For example, api.dropboxapi.com is obviously related to the Dropbox application, occasionally it requires a quick web search. For example, guzzoni.apple.com in the screenshot above is what Apple's Siri needs to function (although some would argue Siri needs a lot more than an SSL decryption exclusion to be truly functional, but anyway...).

You can also jump into the Decryption Logs to see the IP addresses affected by the SSL decryption failure.

Assuming you want the application to work and have accepted the risk of excluding its traffic from SSL decryption, you can copy the SNI (one thing I wish was a little easier in the Palo Alto UI) and paste it into Devices | Certificates | SSL Decryption Exclusions. Commit the changes, and your application is working again! No forum crawling, packet capturing or sacrificial goats required!

Here's a quick video on how to use the new SSL Decryption Failure Reasons in PAN-OS 10 to get Dropbox syncing again.

Do You Even Need SSL Decryption?

Even though it presents some headaches, implementing SSL decryption is as must-have in today's day and age. It is so easy for a bad actor to spin up a HTTPS website using a free Let's Encrypt certificate, load it with malware and direct traffic to it. Without SSL decryption, that malware served over HTTPS will sail straight through your firewall.

SSL decryption is also useful for reporting on internet traffic. If you want to know the 'real' websites your users are visiting as opposed to all the CDNs and advertising pixels that come along for the ride, or if you want to know search terms entered into search engines, or the YouTube videos visited, then SSL decryption is needed in order to for your Palo Alto firewall to log those details.

You'll also need a good reporting tool that knows how to correlate all those background sites into the 'real' site visited, extract those search terms and YouTube video URLs, and deliver internet usage reports to the right person in your organization. Unfortunately, Palo Alto Panorama won't do this for you, which is why we've created Fastvue Reporter for Palo Alto ;)

What have been your SSL decryption nightmares? Let us know in the comments!