Serialize - Deserialize multiple Objects

Introduction

Most articles on Internet about Serialization / Deserialization describe two situations:

a) Serialize exactly one Object to file and Deserialize this one Object back.

b) Create a Collection of objects in a List and Serialize / Deserialize this List Object.

So, what is new about this article. In the Background I will describe a third approach and the requirement for this article.

Background

Imagine if we are subscribed to a webservice sending millions of objects (messages or datacontracts), and if we are the subscriber who would want to dump them in a file (for some requirement).

With option a), if we dump one object to one file, soon we will witness a file explosion.

With approach b) above we will have to keep these multiple objects in memory in a collection and once we get all of them, dump this collection in one shot to a file. In such case we will end up consuming a lot of memory.

The approach which you already have thought of by now is to have one file and write multiple objects to it. All the code on Internet that I could search for was either based on a) or b), so this article discusses this third approach.

This article is a simple code with a simple design which will write multiple object at a time to a same file and deserialize these when required.

The simple design

For every object that arrives, we will convert it into a Base64Encoded string and store it as one line in a text file. So, in this file, every row will have a serialized object per line. While reading we will read the file one line at a time and deserialize this Base64 encoded string into our Object. Easy.. so lets try out the code.

Using the code

This is very basic code and presents an idea of writing objects to file. This might not be great on performance (I have yet not looked into that aspect for this code).

So, lets begin by declaring some objects:

		StreamWriter _wr; // This will write to the file
		StreamReader _r; // This will read from the file
		

We can initialize this in our constructor, before we start listening for objects.

		_w = new StreamWriter(@"C:\MyData.txt"); // The objects will be written to this file
		

Here is the object that we will serialize.

            // Custom Object that will be Serialized and Deserialized
            // Nothing important here, no need to bother about this
            [Serializable()]
            public class ResultRow
            {
                public int id;
                public string Name;
                public float amount;
            }
        

[Serializable()] we all know about. For basics you may check msdn.

Now, we will create our method to serialize published object. To simulate this you can make a button on your form and on click call this method 10 times in a loop (create our ResultRow object and pass it to the Serialize method).

             private void Serialize(ResultRow result)
            {
                MemoryStream memoryStream = new MemoryStream();
                BinaryFormatter binaryFormatter = new BinaryFormatter();
                binaryFormatter.Serialize(memoryStream, result);
                string str = System.Convert.ToBase64String(memoryStream.ToArray());

                _w.WriteLine(str);
            }
		

A simple explanation: The above method converts the object passed to a Base64 string using BinaryFormatter and writes it into our file.

Idea here is that in our file we will store each object as a Base64 encoded string. So, each line will store one Object. Now that we are done with publishing. Lets close this file. We will now open it to read all our objects line by line.

             //All publishing done close and release handle
            _w.Close();

            //Time to read the same file
            _r = new StreamReader(@"C:\MyData.txt");
        

Now we will read one line at a time from the file, deserialize this one line and here comes our Object back to life.

            private void DeSerialize()
            {
                while(!_wr.EndOfStream)
                {
                    //Read one line at a time
                    string objectStream = _wr.ReadLine();
                    //Convert the Base64 string into byte array
                    byte[] memorydata = Convert.FromBase64String(objectStream);
                    MemoryStream rs = new MemoryStream(memorydata); 
                    BinaryFormatter sf = new BinaryFormatter();
                    //Create object using BinaryFormatter
                    ResultRow objResult = (ResultRow)sf.Deserialize(rs);
                    
                    //Do somthing with your object
                }
            }
        

Simple. Isn't it?

LINQ Thought: We should be able to use a yield return in the while loop and make use of LINQ to Objects to get one object at a time from the file. Am new to LINQ, so I might not be correct, but will try that in next article.

Points of Interest

While searching for solution to my problem, I found that Java has a ObjectOutputStream and ObjectInputStream class which solves this problem easily. We shall wait for C# to have a same abstraction for us someday.

History

I will try and make this article as a part series of a bigger solution where we can use this code. Till then.. have a great time!!