Click here to Skip to main content
15,887,244 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
Hi I am using Zip archive[System.IO.Compression] for collect the multiple xml memory stream and convert into zip memory stream then sent to remote server [where I am storing zip file].

My intention is not to store any files[including xml file,zip file] in local machine,
and not to use any 3rd party dll for compression.

Below code is working fine and giving result as I expected. But still i am worrying about performance[In production env, we will get more records to zip storage]

kindly advise the best solution in terms of performance and memory leak.

What I have tried:

Extension Method to convert List<any object> into XML
/// <summary>
       /// Get the List of T and serialize into XML Memeory stream
       /// </summary>
       /// <typeparam name="T"></typeparam>
       /// <param name="dataToSerialize"></param>
       /// <returns></returns>
       public static MemoryStream Serialize<T>(this T dataToSerialize)
       {
           try
           {
               if (dataToSerialize == null) throw new ArgumentNullException();
               var serializer = new XmlSerializer(dataToSerialize.GetType());
               MemoryStream memstream = new MemoryStream();
               serializer.Serialize(memstream, dataToSerialize);

               return memstream;

           }
           catch
           {
               throw;
           }

       }


Extension method to convert into zip memory stream from List of different record.
Input value like dir["xmlfilename",List<object data for xml conversion>]

/// <summary>
     /// Combine all file memory stream and convert into Zip memory stream
     /// </summary>
     /// <typeparam name="T"></typeparam>
     /// <param name="dataToSerialize"></param>
     /// <returns></returns>
     public static MemoryStream SerializeIntoZip<T>(this Dictionary<string,T> dataToSerialize)
     {
         var outStream = new MemoryStream();
         try
         {
             if (dataToSerialize == null) throw new ArgumentNullException();
             using (var archive = new ZipArchive(outStream, ZipArchiveMode.Create, true))
             {
                 foreach(var data in dataToSerialize)
                 {
                     var fileInArchive = archive.CreateEntry(data.Key, CompressionLevel.Optimal);
                     using (var entryStream = fileInArchive.Open())
                     {

                         using (var fileToCompressStream = data.Value.Serialize()) // Calling existing file stream method
                         {
                             entryStream.Flush();
                             fileToCompressStream.Seek(0, SeekOrigin.Begin);
                             fileToCompressStream.CopyTo(entryStream);
                             fileToCompressStream.Flush();
                         }
                     }
                 }

             }

             outStream.Seek(0, SeekOrigin.Begin);
             return outStream;
         }
         catch
         {

             throw;
         }


     }


My main method
Dictionary<string, IList> listofobj =
                new Dictionary<string, IList>() {
                   {"fileName_XX.xml", GetXXList1()}, 
                   {"fileName_YY.xml", GetYYList()}};

var mem = listofobj.SerializeIntoZip();

//{
// code to upload zip memory stream to S3 bucket
//}
Posted
Updated 4-Jul-17 5:12am

1 solution

Normally isnt the RAM the problem on the machines, but the disk I/O and so you should optimize the disk read and write operations. It would make sense to make a file I/O thread and a data compression thread, so both operation wont hurt each other performance. The file I/O should only work on one file a time, it should first write one pending and than read one ahead. The memory stream than gets the compressing thread and the I/O thread can read the next file. For the remote transfer is some buffering very useful, but check the free memory.

At best you make some realistic tests of it to find some optimization. For instance if the files are small than read, compress and write a bunch of files to a temp to transfer them later.
 
Share this answer
 

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900