How to decrypt HTTPS traffic using SSL Proxy

Introduction to a historical problem

Are you a Developer, DevOps Engineer, Infrastructure Engineer, Middleware admin then you might have had this question at least once in your career?

How to see what is being transmitted between the Server to Server [or] Client to Server over HTTP or HTTPS traffic like request, response headers, content etc.

In a less secure environment where data is transmitted over HTTP protocol, If you want to examine the HTTP GET and POST request or response data or headers. it can simply be done with tools like tcpdump (or) Wireshark etc.

I have written a brief researching article on how to read HTTP GET and POST data using tcpdump in my previous article

Well, Now HTTPS is a basic standard of web traffic where the data is an end to end encrypted and it makes the debugging the hardest job. I know it is secure and it is the way it should be.

But at least for debugging there has to be a way to see what is being transmitted over HTTPS like headers, request and responses etc.

When I almost convinced myself it is not possible. I found hope in a promising terminology and idea named SSL Proxying

So this article is all about how to perform WEB Debugging and see the GET and POST data and headers sent over secure HTTPS traffic and I hope you are using this for ethical reasons.

Let us begin

What is SSL Proxy and How it works

SSL proxy is a transparent proxy that performs Secure Sockets Layer encryption (SSL) and decryption between the client and the server. Neither the server nor the client can detect its presence.

SSL Proxy's main purpose is to handle the SSL certificates and Encryption and Decryption.

SSL Proxy handles both encryption and decryption as an acting client or server by being in the middle.

Simply put, for Client, it acts as a server and for a server, it acts as a client.

Besides web Debugging, there are other few things we could accomplish with a help of SSL Proxy

• Changing the SSL Ciphers from Weak to High and make sure that the SSL/TLS traffic is robust
• It provides visibility and protection against SSL related attacks like Poodle
• As the intermediate layer between server and client, It enforces a certain level of security to the server and the client
• Helps in Debugging the HTTPS sessions by decrypting the application data

Hope you have got some idea now on how SSL Proxy can help us in web Debugging. Let us see it in practice.

 

Charles - A Web Debugging SSL Proxy

While there are a lot of proprietary SSL Proxy products available in the market.  I have tested a product named Charles Proxy and found it to be a great Product.

While Charles is not open source you can try the product for 30 days for free and make you a decision of purchase, If you often want to sniff HTTPS traffic as part of your job or development I would say its worth a buy.

A quick intro to Charles Debugging SSL Proxy

Charles is an HTTP proxy / HTTP monitor / Reverse Proxy that enables a developer to view all of the HTTP and SSL / HTTPS traffic between their machine and the Internet. This includes requests, responses and the HTTP headers (which contain the cookies and caching information).

 

Salient Features of Charles Proxy

• Records all traffic between your browser and the Internet

• Reveals the contents of all requests, responses, cookies and headers

• Supports SSL and HTTPS

• Saves valuable time

• Simulates slower internet connections

• Download statistics

• Configurable

• Quickly find and eliminate bugs

   

Downloading Charles SSL Proxy

You can download the latest version of Charles from here  https://www.charlesproxy.com/download/

 

Testing the Charles SSL Proxy

I presume that you have installed the Charles Web Debugging Proxy and It is a time to start testing.

 

Launch the Charles Proxy and Configure SSL Proxy Settings

Once you have launched the Charles Proxy go to  Proxy -> SSL Proxy Settings and add the domain or the URL with port number ( 443 for all normal HTTPS unless otherwise required )

In my case it is medium.com

 

Add Root Certificate of Charles into your browser

As we have discussed earlier, the SSL renegotiation takes place in Charles as it acts as a man in the middle and handling the encryption and decryption.

If you are not adding the Certificate of Charles into your browser your browser would consider it as an actual Man-in-middle attack and throw an error. To avoid this, you have to tell the browser to trust Charles Root Certificate and allow it to trust websites.

The browser-specific instructions including, where to download the certificate would be found on this link

https://www.charlesproxy.com/documentation/using-charles/ssl-certificates/

In my case, It is firefox I just have to download the cert and import it into Firefox Certificates under Authorities  and select the check box to Trust this CA to identify websites

Now you have added Charles as new Certificate Authority.

 

Change the browser Proxy settings to point to Charles Proxy

Now in your browser settings update the Proxy settings to point to localhost:8888 which is a default address and port number of Charles proxy

 

Visit the website you have added to SSLProxy

Now, Hit the URL in the browser,  you have added earlier to the SSL Proxy settings. in my case it is medium.com

If you can connect to medium without any certificate error then your setup is good and you should be able to see the logs coming to Charles proxy.

If you select the host on the left and in the overview if you are seeing which means that the SSL Proxy has started monitoring the IN/OUT packets on this domain.

Refer to the following image.

By default, Charles proxy would start monitoring other HTTP requests and responses which are going through the firefox browser by now. But not all the domain would be added for SSL Proxying unless it was explicitly defined in the SSL Proxy settings like medium.com in our case.

Now If you traverse on the files beneath the medium.com domain you would be able to see more information behind each file being served from the domain. like request, response headers, body message etc.

Screenshot showing the request headers and the response HTML

 

Screenshot showing the request headers and the response headers

So there are numerous possibilities and scenarios you can fulfil with this and perform high-level network and application debugging with this.

In fact. Charles proxy can even be used for Mobile App Debugging and to sniff HTTPS traffic between Mobile App and to the Server.

I will cover that topic in detail on my next post. Stay Connected and bookmark DevOps Junction.

 

Cheers

Sarav AK

---