Using a NetworkStream with raw serialization, GZipStream, and CryptoStream

• Download demo project - 46 Kb

Introduction

After writing the raw serializer article a while ago, the next logical step was to get the whole process working with a NetworkStream. Ah, did that turn into a major problem? It turns out that the CryptoStream does not tie in well with a NetworkStream. At the end of the article is a variety of links on the subject, but the basic explanation is as follows, from a post by Stephen Martin:

When you don't use a CryptoStream, the Deserialize call can read the data from the network stream. As you say, the serialized data includes headers with the data length, and so it knows when the stream has finished retrieving data. But when you introduce the CryptoStream, it's the CryptoStream reading from the network stream not the deserializer. The CryptoStream knows nothing about the length expected, and so cannot know that it has reached the end of the data (and call TransformFinalBlock to receive the last decrypted buffer) until it receives the zero byte signal from the stream. You may be able to use a stream cipher for this, but I haven't looked into any of the stream cipher implementations.

So, the problem is, the NetworkStream is a byte stream while the CryptoStream is a block stream. As such, you can't connect the two because the NetworkStream never tells the CryptoStream that it has read the last byte of the last block. However, if you use an intermediate MemoryStream, then everything works well, at the sacrifice of having to buffer the entire packet in memory before shipping it off the NetworkStream. That may be a big sacrifice, however, it's the implementation I chose to illustrate in this article.

Implementation

This section describes the implementation, building on the Simple TCP Server, by adding a pooled TCP stream service and an "RCS" (Raw, Compressed, Secure) stream chain.

The PooledStreamTcpService

Using the Simple TcpServer I had written about last, I'm adding a PooledStreamTcpService class and an RCSConnectionService class. The PooledStreamTcpService class uses Stephen Toub's managed thread pool to manage the server's worker threads, as I discussed in this article. The PooledStreamTcpService instantiates the simple TcpServer class and hooks the Connected event. The event handler packages up the socket's NetworkStream and the TcpServer's ConnectionState instances, then queues a user work item which, very similar to the TcpServer, invokes any listeners attached to the Connected event. To illustrate:

The following code illustrates the core implementation:

/// <summary>
/// Constructor. The port on which to listen.
/// </summary>
/// <param name="port"></param>
public PooledStreamTcpService(int port)
{
  tcpLib = new TcpServer(port);
  tcpLib.Connected += new TcpServer.TcpServerEventDlgt(OnConnected);
}

/// <summary>
/// Starts listening for connections.
/// </summary>
public void Start()
{
  tcpLib.StartListening();
}

/// <summary>
/// Stops listening for connections.
/// </summary>
public void Stop()
{
  tcpLib.StopListening();
}

/// <summary>
/// When a connection is made, get a network stream and start 
/// the worker thread.
/// The worker thread uses the managed thread pool, which is
/// safe for handling
/// work that takes a lot of time. 
/// </summary>
/// <param name="sender"></param>
/// <param name="e"></param>
protected void OnConnected(object sender, TcpServerEventArgs e)
{
  ConnectionState cs = e.ConnectionState;
  NetworkStream ns = new NetworkStream(cs.Connection, true);
  ns.ReadTimeout = readTimeout;
  ns.WriteTimeout = writeTimeout;
  NetworkStreamConnection conn = new NetworkStreamConnection(ns, cs);
  ManagedThreadPool.QueueUserWorkItem(ConnectionProcess, conn);
}

/// <summary>
/// Inform the application that we are connected. This event 
/// fires in a worker thread.
/// </summary>
/// <param name="state"></param>
protected void ConnectionProcess(object state)
{
  NetworkStreamConnection conn = (NetworkStreamConnection)state;

  try
  {
    OnConnected(new TcpServiceEventArgs(conn));
  }
  catch (Exception e)
  {
    // Report exception not caught by the application.
    try
    {
      OnHandleApplicationException(new TcpLibApplicationExceptionEventArgs(e));
    }
    catch (Exception ex2)
    {
      // Oh great the app's handler threw an exception!
      System.Diagnostics.Trace.WriteLine(ex2.Message);
    }
    finally
    {
      // In any case, close the connection.
      conn.Close();
    } 
  }
}

/// <summary>
/// Invokes the Connected handler, if exists.
/// </summary>
/// <param name="e"></param>
protected virtual void OnConnected(TcpServiceEventArgs e)
{
  if (Connected != null)
  {
    Connected(this, e);
  }
}

The RCSConnectionService

To this stack is added the RCSConnectionService (Raw-Compressed-Secure). The RCSConnectionService builds the stream chain and manages the interaction with the NetworkStream, as illustrated here:

Note though that the application is responsible for connecting the PooledStreamTcpService to the RCSConnectionService, as I'll illustrate later. I did this primarily to decouple the RCSConnectionService from the pool service--if you don't want to use Stoub's managed pool, you can easily use something else instead.

The Serialization Chain

The serialization stream is initialized this way:

protected void InitializeSerializer(byte[] key, byte[] iv)
{
  writeBuffer = new MemoryStream();
  EncryptTransformer et = new EncryptTransformer(EncryptionAlgorithm.Rijndael);
  et.IV = iv;
  ICryptoTransform ict = et.GetCryptoServiceProvider(key);
  encStream = new CryptoStream(writeBuffer, ict, CryptoStreamMode.Write);
  comp = new GZipStream(encStream, CompressionMode.Compress, true); 
  serializer = new RawSerializer(comp);
}

The Deserialization Chain

And the deserialization stream is initialized as:

protected void InitializeDeserializer(Stream stream, byte[] key, byte[] iv)
{
  DecryptTransformer dt = new DecryptTransformer(EncryptionAlgorithm.Rijndael);
  dt.IV = iv;
  ICryptoTransform ict = dt.GetCryptoServiceProvider(key);
  decStream = new CryptoStream(stream, ict, CryptoStreamMode.Read);
  decomp = new GZipStream(decStream, CompressionMode.Decompress);
  deserializer = new RawDeserializer(decomp);
}

Begin/End Write

Because the NetworkStream is buffered by the MemoryStream, it is necessary to explicitly begin and end read/write operations, as the NetworkStream must be worked with as if it were a packet stream. The BeginWrite method:

public void BeginWrite()
{
  try
  {
    InitializeSerializer(key, iv);
  }
  catch (Exception e)
  {
    throw new TcpLibException(e.Message);
  }
}

initializes the serializer, which is straightforward enough. The EndWrite method is where the work is done, transferring the MemoryStream to the NetworkStream. The EndWrite method is used to indicate that the packet has been completely serialized and is ready to be transferred across the network.

public void EndWrite()
{
  try
  {
    comp.Close();
    encStream.FlushFinalBlock();
    byte[] length = BitConverter.GetBytes((int)writeBuffer.Length);
    networkStream.Write(length, 0, length.Length);
    networkStream.Write(writeBuffer.GetBuffer(), 0, (int)writeBuffer.Length);
    encStream.Close();
  }
  catch (Exception e)
  {
    throw new TcpLibException(e.Message);
  }
}

The above code:

• closes the GZipStream,
• flushes the final block from the encryption stream,
• gets the MemoryStream buffer length and converts it to a byte array,
• writes the buffer length,
• writes the buffer data,
• closes the encryption stream (which also closes the memory stream).

Begin/End Read

The begin/end read operations work oppositely to the begin/end write operations. Again, it is necessary to explicitly begin a packet read operation. The BeginRead method:

public void BeginRead()
{
  try
  {
    // Get packet length length.
    byte[] plength = new byte[sizeof(Int32)];
    // Read the length.
    networkStream.Read(plength, 0, plength.Length);
    // Convert to an Int32
    int l = BitConverter.ToInt32(plength, 0);
    // Initialize the buffer.
    byte[] buffer = new byte[l];
    // Read the packet data.
    networkStream.Read(buffer, 0, l);
    MemoryStream stream = new MemoryStream(buffer);
    InitializeDeserializer(stream, key, iv);
  }
  catch (Exception e)
  {
    throw new TcpLibException(e.Message);
  }
}

• reads the packet length,
• reads the packet data,
• puts the data into a MemoryStream,
• initializes the deserializer.

Whereas the EndRead method:

public void EndRead()
{
  try
  {
    decomp.Close(); // Close GZip stream.
    decStream.Close(); // Close crypto stream.
  }
  catch (Exception e)
  {
    throw new TcpLibException(e.Message);
  }
}

closes the GZipStream and the decryption stream.

Example

The following is a simple loop-back example in which a single application acts both as the server and the client. In this particular example (and please don't use this as a baseline), the 400 or so bytes of data are compressed to a 192 byte packet.

class Program
{
  // The crypto key and initial vector.
  static byte[] key = new byte[] 
     { 10, 20, 30, 40, 50, 60, 70, 80, 11, 22, 33, 44, 55, 66, 77, 88 };
  static byte[] iv = new byte[] 
     { 11, 22, 33, 44, 55, 66, 77, 88, 10, 20, 30, 40, 50, 60, 70, 80 };

  static void Main(string[] args)
  {
    // Set up a local loopback.
    PooledStreamTcpService psServ = 
        new PooledStreamTcpService("127.0.0.1", 14000);
    // Wire up the connected event.
    psServ.Connected += 
        new PooledStreamTcpService.ConnectedDlgt(OnServerConnected);
    // Start listening
    psServ.Start();

    // Setup the client socket.
    TcpClient tcpClient = new TcpClient();
    // Connect.
    tcpClient.Connect("127.0.0.1", 14000);
    // Get the network stream.
    NetworkStream ns = tcpClient.GetStream();
    // Create a NetworkStreamConnection helper class.
    NetworkStreamConnection nsc = new NetworkStreamConnection(ns, null);
    // Create the RCSConnectionService for the client.
    RCSConnectionService connClient = new RCSConnectionService(nsc, key, iv);

    // Setup something to write.
    string s = "The quick brown fox jumps over the lazy dog.\r\n";
    StringBuilder sb = new StringBuilder();
    for (int i = 0; i < 10; i++)
    {
      sb.Append(s);
    }

    // Begin write.
    connClient.BeginWrite();
    // Write the test string.
    connClient.Write(sb.ToString());
    // End of packet.
    connClient.EndWrite();

    // Wait for the server thread to run.
    Console.ReadLine();
    nsc.Close();
  }

  static void OnServerConnected(object sender, TcpServiceEventArgs e)
  {
    Console.WriteLine("Connected.");
    // Create the RCS service.
    RCSConnectionService conn = 
      new RCSConnectionService(e.Connection, key, iv);
    // Begin the packet read process.
    conn.BeginRead();
    // Read the string.
    string s = conn.ReadString();
    // Show it on the console.
    Console.WriteLine(s);
    e.Connection.Close();
    Console.WriteLine("Disconnecting.");
  }
}

Resulting in this display:

Further Reading

It took me a while to understand why a NetworkStream couldn't directly work with a CryptoStream. I made notes of the links I found on the issue. Here they are:

• Something to read...
• Block cypher and the read problem
• More on the CryptoStream/NetworkStream bug
• A NetworkStream example from Microsoft
• The reason!
• CBC stream cipher in C#

Conclusion

I hope this library helps people who have faced similar problems working with encrypted and compressed data. There's a lot more that can be done here, including changing the cryptographic algorithms to use public/private keys, support for nullable data types, and so forth. Feel free to modify the library to suite your needs, but please adhere to the BSD license terms in the source code files.