Click here to Skip to main content
15,867,308 members
Articles / General Programming / Threads

Implementing a Multithreaded HTTP/HTTPS Debugging Proxy Server in C#

Rate me:
Please Sign up or sign in to vote.
4.86/5 (35 votes)
3 Feb 2011Ms-PL4 min read 312.3K   15.6K   109   55
A complete proxy server except instead of SSL Tunneling, will perform a "man-in-the-middle" decryption on SSL traffic allowing you to inspect the encrypted traffic

Introduction

This article will show you how to implement a multithreaded HTTP proxy server in C# with a non-standard proxy server feature of terminating and then proxying HTTPS traffic. I've added a simple caching mechanism, and have simplified the code by ignoring http/1.1 requests for keeping connections alive, etc.

Disclaimer: Understand that this code is for debugging and testing purposes only. The author does not intend for this code or the executable to be used in any way that may compromise someone's sensitive information. Do not use this server in any environment which has users that are unaware of its use. By using this code or the executable found in this article, you are taking responsibility for the data which may be collected through its use.

Background

If you are familiar with fiddler, then you already know how this proxy server works. It essentially performs a "man-in-the-middle" on the HTTP client to dump and debug HTTP traffic. The System.Net.Security.SslStream class is utilized to handle all the heavy lifting.

Using the Code

The most important part about this code is that when the client asks for a CONNECT, instead of just passing TCP traffic, we're going to handle an SSL handshake and establish an SSL session and receive a request from the client. In the mean time, we'll send the same request to the destination HTTPS server.

First, let's look at creating a server that can handle multiple concurrent TCP connections. We'll use the System.Threading.Thread object to start listening for connections in a separate thread. This thread's job will be to listen for incoming connections, and then spawn a new thread to handle processing, thus allowing the listening thread to continue listening for new connections without blocking while one client is processed.

C#
public sealed class ProxyServer
{
   private TcpListener _listener;
   private Thread _listenerThread;

   public void Start()
   {
      _listener = new TcpListener(IPAddress.Loopback, 8888);
      _listenerThread = new Thread(new ParameterizedThreadStart(Listen));
      _listenerThread.Start(_listener);
   }
        
   public void Stop()
   {
      //stop listening for incoming connections
      _listener.Stop();
      //wait for server to finish processing current connections...
      _listenerThread.Abort();
      _listenerThread.Join();
   }

   private static void Listen(Object obj)
   {
      TcpListener listener = (TcpListener)obj;
      try
      {
         while (true)
         {
            TcpClient client = listener.AcceptTcpClient();
            while (!ThreadPool.QueueUserWorkItem
		(new WaitCallback(ProxyServer.ProcessClient), client)) ;
         }
      }
      catch (ThreadAbortException) { }
      catch (SocketException) { }
   }

   private static void ProcessClient(Object obj)
   {
      TcpClient client = (TcpClient)obj;
      try
      {
         //do your processing here
      }
      catch(Exception ex)
      {
         //handle exception
      }
      finally
      {
         client.Close();
      }
   }
}

And that's the beginning of the code to handle concurrent TCP clients in a multithreaded manner. Nothing really special there. The interesting bit is when we use SslStream to act as an HTTPS server and "trick" the client into believing it's talking to the destination server. Note that the browser should not actually be tricked because of the SSL certificate chain, but depending on their browser, it may or may not be apparent that the server's identity is in question.

Now let's take a look at the actual processing of the SSL request. Assume that we are somewhere inside the try block of the ProcessClient method shown above.

C#
//read the first line HTTP command
Stream clientStream = client.GetStream();
StreamReader clientStreamReader = new StreamReader(clientStream);
String httpCmd = clientStreamReader.ReadLine();

//break up the line into three components
String[] splitBuffer = httpCmd.Split(spaceSplit, 3);
String method = splitBuffer[0];
String remoteUri = splitBuffer[1];
Version version = new Version(1, 0); //force everything to HTTP/1.0

//this will be the web request issued on behalf of the client
HttpWebRequest webReq;

if (method == "CONNECT")
{
   //Browser wants to create a secure tunnel
   //instead = we are going to perform a man in the middle
   //the user's browser should warn them of the certification errors however.
   //Please note: THIS IS ONLY FOR TESTING PURPOSES - 
   //you are responsible for the use of this code
   //this is the URI we'll request on behalf of the client
   remoteUri = "https://" + splitBuffer[1];
   //read and ignore headers
   while (!String.IsNullOrEmpty(clientStreamReader.ReadLine())) ;

   //tell the client that a tunnel has been established
   StreamWriter connectStreamWriter = new StreamWriter(clientStream);
   connectStreamWriter.WriteLine("HTTP/1.0 200 Connection established");
   connectStreamWriter.WriteLine
	(String.Format("Timestamp: {0}", DateTime.Now.ToString()));
   connectStreamWriter.WriteLine("Proxy-agent: matt-dot-net");
   connectStreamWriter.WriteLine();
   connectStreamWriter.Flush();

   //now-create an https "server"
   sslStream = new SslStream(clientStream, false);
   sslStream.AuthenticateAsServer(_certificate, 
	false, SslProtocols.Tls | SslProtocols.Ssl3 | SslProtocols.Ssl2, true);

   //HTTPS server created - we can now decrypt the client's traffic
   //.... 
}

Points of Interest

You can see that I have an X509Certificate2 _certificate defined elsewhere in the code. To get a certificate for this, you need to use a tool like makecert.exe to create a self-signed certificate. I found makecert.exe in the Windows SDK and it also comes with Fiddler. I have included a certificate file in the source files to allow the server to run, but because it does not include the private key, to actually handle SSL traffic, You will need to run makecert.exe

Here is the syntax I used for makecert.exe:

makecert.exe cert.cer -a sha1 -n "CN=matt-dot-net" -sr LocalMachine -ss My -sky signature -pe -len 2048

One thing to note is that on your HttpWebRequest object, you will need to set the Proxy property to null if you are using Windows internet options to specify using your proxy server. This is because your HttpWebRequest will default to the Windows internet settings and you'll have a proxy server that is trying to use itself as a proxy server!

Another interesting and frustrating hang-up was the handling of cookies. When I thought that I was processing requests/responses perfectly, I found that I could not maintain state with any websites because cookies were not properly being sent by the client. I determined (from using fiddler and firefox live HTTP headers) that cookies needed to be set individually. The server was returning several cookies in one Set-Cookie header, but I needed to parse them out and return an individual Set-Cookie header for each one. I haven't researched to find out why this is, or what is the proper handling of cookies in HTTP, but all I can assume was that when a browser is set to use a proxy server, it expects the proxy server to process cookies into individual Set-Cookie headers and if not configured to use a proxy, the browser does this itself.

History

I threw together this code on a Sunday afternoon, so I expect it to have errors and problems. Also, I make no claim that this is properly handling errors, cleaning up objects, or is in any way the best way to do anything.

License

This article, along with any associated source code and files, is licensed under The Microsoft Public License (Ms-PL)


Written By
Software Developer (Senior)
United States United States
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
GeneralRe: certificate Pin
matt-dot-net11-May-11 7:15
matt-dot-net11-May-11 7:15 
GeneralAuthentication Exception Pin
Mark Woan9-Feb-11 2:58
Mark Woan9-Feb-11 2:58 
GeneralRe: Authentication Exception Pin
matt-dot-net14-Feb-11 4:10
matt-dot-net14-Feb-11 4:10 
GeneralMy vote of 5 Pin
Martin Thwaites4-Feb-11 13:10
Martin Thwaites4-Feb-11 13:10 
GeneralArticle Update Pin
matt-dot-net1-Feb-11 7:57
matt-dot-net1-Feb-11 7:57 
GeneralProblems with POST data Pin
Martin Thwaites1-Feb-11 5:49
Martin Thwaites1-Feb-11 5:49 
GeneralRe: Problems with POST data Pin
matt-dot-net1-Feb-11 6:34
matt-dot-net1-Feb-11 6:34 
GeneralRe: Problems with POST data Pin
Martin Thwaites4-Feb-11 13:09
Martin Thwaites4-Feb-11 13:09 
I must apologise, I was planning on getting back to you sooner...

It was actually my hack job that caused the problem, I removed the caching as I don't need that part of the app, all the requests will be drastically different.

Seems I was a little too... liberal... when removing code...

I'm hoping to have something in place over the weekend, and I'll probably post it up, maybe it will be useful to some others.

Features I've added are:
ability to specify multiple different certificates and their corresponding hostnames in the app.config (to allow for creation certificates for specific hosts, meaning that I can test applications by adding the CA cert into the trusted authorities, then generate certs for specific hosts).
Listen site list in the app.config (all other sites will not be logged, allows me to keep the logging threads to a minimum)
Database logging of the information from the request/response data (includes information on the client too).

I've compiled to .NET 2.0 so that it will easily work on mono, had to do some refactoring to make it work though (Tuples changed to KVPs, etc.).

Again, thanks for a great app, it was exactly what I was looking for to start me off!

Martin
GeneralError on ssl web sites. Pin
jlandrian26-Jul-10 13:17
jlandrian26-Jul-10 13:17 
GeneralRe: Error on ssl web sites. Pin
matt-dot-net26-Jul-10 14:42
matt-dot-net26-Jul-10 14:42 
GeneralMy vote of 5 Pin
Member 432084420-Jul-10 22:01
Member 432084420-Jul-10 22:01 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.