Click here to Skip to main content
15,891,204 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
I know how to serialize an object into a byte array, but it serializes it as a whole, is there a way to serialize it in chunks of say, a kb? For example, I have an object, that would serialize into a large byte array of say 4mb, instead of serializing the whole object at the same time, I want to serialize it 1 kb chunk at a time, so it would serialize 1kb worth and pass it on, then move on to the next one until it is completely serialized. How might I go about doing this?

My current serialization/de-serialization class:
C#
using System;
using System.Windows.Forms;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Runtime.Serialization.Formatters.Binary;
using System.IO;

namespace TCP_Handler
{
    public class SerialMediator
    {

        /// <summary>
        /// Serializes a packet into a byte array
        /// </summary>
        /// <param name="pkt"></param>
        /// <returns></returns>
        public static Byte[] Serialize(Packet pkt)
        {
            BinaryFormatter bf = new BinaryFormatter();
            MemoryStream ms = new MemoryStream();
            bf.Serialize(ms, (Object)pkt);
            return ms.ToArray();
        }

        /// <summary>
        /// Deserializes a packet from a byte array
        /// </summary>
        /// <param name="arr"></param>
        /// <returns></returns>
        public static Packet Deserialize(Byte[] arr)
        {
            BinaryFormatter bf = new BinaryFormatter();
            MemoryStream ms = new MemoryStream();
            ms.Write(arr, 0, arr.Length);
            ms.Seek(0, SeekOrigin.Begin);
            try
            {
                return (Packet)bf.Deserialize(ms);
            }
            catch (Exception ex)
            {
                MessageBox.Show(ex.Message);
                throw new Exception("Byte array did not represent a packet");
            }
        }
    }
}
Posted
Comments
Ron Beyer 21-Oct-13 14:25pm    
Why not serialize it then read it out and "pass it on" in chunks? There is no way to make the built-in serialization do it piece by piece, you would have to implement a custom serializer to do that.
Sergey Alexandrovich Kryukov 21-Oct-13 14:28pm    
Exactly. The request as is does not make a lot of sense...
—SA
Sicppy 21-Oct-13 14:31pm    
I want to avoid the large number of bytes that would be stored in RAM.
Sergey Alexandrovich Kryukov 21-Oct-13 14:39pm    
You cannot possibly operate with a "half-object". Rather, you should review the bigger picture of your project.
—SA
Sicppy 21-Oct-13 14:43pm    
I want to send it over a TCP stream, so I would like to serialize 1KB, send it over a TCP stream, and then delete it from ram and continue until it has sent the entire object

Streams hold the answer. The way you are doing at the moment you serialize the whole thing into a MemoryStream and then convert that to a byte array.

If you want to send the thing via Tcp, drop the MemoryStream and use the NetworkStream you get when you connect directly. There will be some buffering but usually you won't have to worry about this.
 
Share this answer
 
Comments
Sicppy 21-Oct-13 15:31pm    
Thanks! I'll try that!
Ron Beyer 21-Oct-13 15:34pm    
Streams allow you to read it piece-meal, but it still serializes the entire object at once into a buffer, then the stream reads out of that buffer. There is no way to serialize a "chunk" at a time without custom serialization.
Rob Philpott 21-Oct-13 15:38pm    
I dispute that. Yes, the serialisation happens as a continuous operation but the streams chunk it up by their very nature. That is why I can start deserialising data on a client which is still being serialised on a server.
Ron Beyer 21-Oct-13 15:39pm    
Right, but when you serialize to the memorystream it does it all in one operation. If you have a 4mb object, as soon as you serialize it makes a byte[] buffer in the background that has the entire serialized object. Yes you can read it out in chunks, but it doesn't partially serialize like the OP is asking.
Rob Philpott 21-Oct-13 15:40pm    
Yes, well if you read what I said I suggested not using the MemoryStream.
To avoid large memory footprint during serialization, you either need to break your big object up into smaller objects (i.e rethink design) or make the object a pure struct which can be zero copy read from using Marshal.StructToPtr (this has it's own drawbacks)
 
Share this answer
 
v2
Comments
Sicppy 21-Oct-13 15:04pm    
I'd just like to point out that I have an actual uncle named Paul haha.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900