Re: Fastest way to write a text file - C# Discussion Boards

Fastest way to write a text file

19-Mar-07 22:48

Will this do the trick?

Adding binary data to a table[^]

public object BufferOverFlow<br />
        {<br />
__get { return BufferOverFlow; }<br />
__set { BufferOverFlow = value; }<br />
}

MSBuild Question

Member 390685019-Mar-07 20:53

Member 3906850

19-Mar-07 20:53

Code is simple as below:

Process process = Process.Start("MSBUILD.EXE", "D:\\Common\\Common.sln /p:Configuration=Debug");
process.WaitForExit();

When I ran above code in C#, I got "ExitCode = 1" error, couldn't figure out why. Anybody please help?

Thanks a bunch.

Hazy

blackjack215019-Mar-07 20:51

blackjack2150

19-Mar-07 20:51

Hi.

I need to write a large text file from my application. Large means something like 200+ Megabytes.

What is the fastest way to do it? Using the static methods of the "File" class or using streams?? I am aware that either way, it'll take a long, long tine, but ...time is precious.

And another related question. It helps in terms of performance to write as much data as possible at a time, doesn't it?

Thanks.

Russell Jones19-Mar-07 22:08

Russell Jones

19-Mar-07 22:08

blackjack2150 wrote:

And another related question. It helps in terms of performance to write as much data as possible at a time, doesn't it?

kind of depends what you mean by performance. but often yes.

1. The user starts the file save procedure and starts looking at email. The PC will look faster if your process is slightly less fast so smaller packets that don't block the disk queue might help.

2. The hard disk has issues. The program continually tries to write and fails. Because your packets are huge it fails more often as it's hard to find unbroken space that big and it has to attempt each write many times. (This would more likely be a problem on a network share for instance).

3. The hard disk is fast, is 0% fragmented and there's more ram and CPU power in the box than you could shake a stick at. This machine isn't going to slow down for anything so you might as well max the packet size and get the job done fast.

Do you need this file save to complete before the user continues? If not i would suggest sticking the save process in a background thread and maybe put a progress bar in the UI.

Russ

blackjack215019-Mar-07 22:33

blackjack2150

19-Mar-07 22:33

User interaction is not important for me. In fact is almost absent. I've converted my program to a console application, as I noticed that during the execution the UI got blocked. I know there are ways to avoid this, but I decided that it's not worth the effort.

If it helps I'll briefly describe what the program does:

I have a folder containing more than 7000 xml files which are news articles. I must read everyone of them and write to a file every distinct word in these xmls. It adds up to about 42000 words. I another file I must write the frequency of appearance of every word in every xml file, so there are 42000*7000 = a s...load of numbers to be written.

So, the user just starts the program, sits back and begins to read a book on programming. Smile | :)

It takes about 5 minutes to read the files and store the words in memory, another 5 to write the 42000 words to the first file, and writing the last file....well God knows, as I never had the patience to wait that much.
This file should have: (7000 * 42000 * 2) / 1024 /1024 ~= 560 MB. OMG | :OMG:

The data structures I use are just basic arrays and matrixes. Does the word "matrixes" exist? Or should it be matrices?

So, what do you suggest?

m@u19-Mar-07 23:18

m@u

19-Mar-07 23:18

Hi
well that does not sound like a I/O Performance problem. writing to streams is pretty fast.
i have an application that writes every day a 70GB Zip - File and it doesnt take more than 4 hours. and it gets slower the bigger the file gets. the first gb takes about 15 minutes.

is the whole data processing finished when your app writes the first file?
if not, i would say, the creation of the data used for your second file takes so long and not the write process of it..

greets
m@u

PIEBALDconsult23-Feb-10 10:46

PIEBALDconsult

23-Feb-10 10:46

It seems to me that you can write the first file as you go, unless you want to write them alphabetically or something.

If you have a reasonable idea of the maximum number of digits for the frequency, you could write the second file as you go as well, just keep a Dictionary containing the word, the current count, and the offset to the position of the frequency.

blackjack2150 wrote:
The data structures I use are just basic arrays and matrixes

I recommend a Dictionary.

blackjack2150 wrote:
matrices

That is correct.

joon _vh.19-Mar-07 22:22

19-Mar-07 22:22

I mocked up a quick perf. test. You can run it yourself to see the results (in VS 2005).

Just make an empty console app. and past this code:

    class Program<br />
    {<br />
        static string printme =<br />
            "Fastest way to write a text file Fastest way to write a text file Fastest way to write a text file";<br />
<br />
        static void Main(string[] args)<br />
        {<br />
            // start the first stopwatch <br />
<br />
            Stopwatch sw = new Stopwatch();<br />
            sw.Start();<br />
<br />
            // run the first test (stream)<br />
<br />
            test1();<br />
<br />
            // stop stopwatch and print<br />
<br />
            sw.Stop();<br />
            Console.WriteLine(sw.Elapsed);<br />
<br />
            // reset <br />
<br />
            sw.Reset();<br />
<br />
            // start second stopwatch<br />
            <br />
            sw.Start();<br />
<br />
            // run second test<br />
<br />
            test2();<br />
<br />
            // stop stopwatch and print<br />
<br />
            sw.Stop();<br />
            Console.WriteLine(sw.Elapsed);<br />
<br />
            // hold for laughs<br />
<br />
            Console.Read();<br />
        }<br />
<br />
<br />
        /// <summary><br />
        /// Method that writes 5000 lines of text with a streamwriter<br />
        /// </summary><br />
        static void test1()<br />
        {<br />
            TextWriter tw = new StreamWriter("test1.txt");<br />
            for (int i = 0; i < 50000; i++)<br />
            {<br />
                tw.WriteLine(i + " " + printme);<br />
            }<br />
            tw.Close();<br />
        }<br />
<br />
        /// <summary><br />
        /// Method that writes 5000 lines of text with the static methods from the File class<br />
        /// </summary><br />
        static void test2()<br />
        {<br />
            for (int i = 0; i < 50000; i++)<br />
            {<br />
                 File.AppendAllText("test2.txt",i + " " + printme + "\n\r");<br />
            }<br />
        }<br />
    }

I hope my logic is right, and if it is, you will hopefulyl see the same result as me. Streamwriter completes in a few seconds, and File.AppendAllText... takes forever.
There is also AppendText, but that returns a streamwriter and gives you the same result as test1.

Hope I helped.

public object BufferOverFlow<br />
        {<br />
__get { return BufferOverFlow; }<br />
__set { BufferOverFlow = value; }<br />
}

Guffa19-Mar-07 22:30

Guffa

19-Mar-07 22:30

blackjack2150 wrote:
What is the fastest way to do it? Using the static methods of the "File" class or using streams?? I am aware that either way, it'll take a long, long tine, but ...time is precious.

I don't think that there is such a big difference. Most of the static methods in the File class are using streams anyway, so then the difference is just how you use the stream.

There might be a difference in writing a text stream, and encoding the text first and writing it as a binary stream, so that might be worth testing.

blackjack2150 wrote:
It helps in terms of performance to write as much data as possible at a time, doesn't it?

That depends. The difference is hardly noticable unless the buffer size is very small or very big.

You should be aware of the large objects heap, though. Any object that is larger than 85 kb is allocated in the largs objects heap instead of the regular heap. The large objects heap grows when needed but never shrinks, so if you allocate a lot of large objects the program may allocate a lot of memory that is not returned to the system until the program ends.

---
single minded; short sighted; long gone;

joon _vh.19-Mar-07 23:49

19-Mar-07 23:49

Guffa wrote:
There might be a difference in writing a text stream, and encoding the text first and writing it as a binary stream, so that might be worth testing.

Here is some code. Again, make a console app and paste this testcode. A Bytestream is obviously faster, but I can't seem to debug it properly (it generates a character that shouldn't be there at the beginning of each write, so the file is bigger).

<br />
        static string printme =<br />
            "Fastest way to write a text file Fastest way to write a text file Fastest way to write a text file";<br />
<br />
        private static int linecount = 50000;<br />
<br />
        static void Main(string[] args)<br />
        {<br />
            // start the first stopwatch <br />
<br />
            Stopwatch sw = new Stopwatch();<br />
            sw.Start();<br />
<br />
            // run the first test (stream)<br />
<br />
            test1();<br />
<br />
            // stop stopwatch and print<br />
<br />
            sw.Stop();<br />
            Console.WriteLine(sw.Elapsed);<br />
<br />
            // reset <br />
<br />
            sw.Reset();<br />
<br />
            // start second stopwatch<br />
            <br />
            sw.Start();<br />
<br />
            // run second test<br />
<br />
            test2();<br />
<br />
            // stop stopwatch and print<br />
<br />
            sw.Stop();<br />
            Console.WriteLine(sw.Elapsed);<br />
<br />
            // hold for laughs<br />
<br />
            Console.Read();<br />
        }<br />
<br />
<br />
        /// <summary><br />
        /// Method that writes 5000 lines of text with a streamwriter<br />
        /// </summary><br />
        static void test1()<br />
        {<br />
            TextWriter tw = new StreamWriter("test1.txt");<br />
            for (int i = 0; i < linecount; i++)<br />
            {<br />
                tw.WriteLine(i + " " + printme);<br />
            }<br />
            tw.Close();<br />
        }<br />
<br />
        /// <summary><br />
        /// Method that writes 5000 lines of text with a bytestream<br />
        /// </summary><br />
        static void test2()<br />
        {<br />
            FileStream fs = File.Create("test2.txt");<br />
            BinaryWriter bw = new BinaryWriter(fs);<br />
            for (int i = 0; i < linecount; i++)<br />
            {<br />
                string toprint = i + " " + printme + "\r\n";<br />
                bw.Write(toprint);<br />
            }<br />
        }<br />
    }<br />

public object BufferOverFlow<br />
        {<br />
__get { return BufferOverFlow; }<br />
__set { BufferOverFlow = value; }<br />
}

Guffa20-Mar-07 2:10

Guffa

20-Mar-07 2:10

A BinaryWriter is not used to write text to a binary file, it's used to write data in binary format to a binary file. When you write a string using a BinaryWriter, it writes the length of the string, then the binary representation of the string. The string is not encoded either, so it will be in UTF-16, which is the internal format for strings.

You don't need a writer to write a binary file. A FileStream can do that.

static void test2() {
   using(FileStream fs = File.Create("test2.txt")) {
      for (int i = 0; i < linecount; i++) {
         string toprint = i.ToString() + " " + printme + "\r\n";
         byte[] buffer = Encoding.UTF8.GetBytes(toprint);
         bw.Write(buffer, 0, buffer.Length);
      }
   }
}

---
single minded; short sighted; long gone;

joon _vh.20-Mar-07 2:50

Placing RadioButton in Nodes of TreeView

20-Mar-07 2:50

I just discoverd that my quick test isn't very conclusive. It should be run in a closed environment, no influences from other programs, and probably run a 100 times before you can draw any real conclusions. Frown | :(

I think using the filestream or textwriter is the fastest, can't see any difference.
Also I forgot about the using clause, it's very important to dispose of writers and streams. Thanks Wink | ;)

Visual Studio can't evaluate this, can you?

public object moo<br />
        {<br />
__get { return moo; }<br />
__set { moo = value; }<br />
}

SakthiSurya19-Mar-07 20:42

SakthiSurya

19-Mar-07 20:42

Hi,

I want to use radioButton in Each Parent Node of TreeView.

How can i do that. if any one knows please help me.

With Thanks,

sakthi.

Re: Placing RadioButton in Nodes of TreeView

stancrm19-Mar-07 21:03

stancrm

19-Mar-07 21:03

http://www.codeproject.com/cs/miscctrl/TreeViewRadioBox.asp

datagrid with checkbox for pocket pc 2003

mohdmeraj19-Mar-07 20:17

mohdmeraj

19-Mar-07 20:17

hello

i want to use System DataGrid in pocket pc 2003 with checkbox feature

please help me

raj

code

how to get create query of msaccess table through c#

joerozario19-Mar-07 19:18

Re: how to get create query of msaccess table through c#

19-Mar-07 19:18

I have the database already with the tables.I want the create query of each table.
Is it possible to get the query through c#.or is it possible in msaccess?

sujithkumarsl19-Mar-07 19:59

sujithkumarsl

19-Mar-07 19:59

HI,

You can use Msaccess for generating query
go to Queries->Create query by using wizard
You can select the desirable tables for the queries

My small attempt...

Re: how to get create query of msaccess table through c#

joerozario19-Mar-07 20:21

Re: how to get create query of msaccess table through c#

19-Mar-07 20:21

thank u for u r reply.

Actually i have the tables already. your solution gives me the select query for the table but not the "create query". i want create query of the existing table.

is it posible to generate (create Query) query as we generate in sql server.

sujithkumarsl19-Mar-07 20:34

sujithkumarsl

19-Mar-07 20:34

I think your question is not clear
do you want to create table using query?
What you mean by "Create query" ?

My small attempt...

Re: I want to get the create query of existing table

joerozario19-Mar-07 21:10

Re: I want to get the create query of existing table

19-Mar-07 21:10

I have already created the tables.but i need create query of the existing table.

joerozario19-Mar-07 21:26

Re: I want to get the create query of existing table

19-Mar-07 21:26

i have the tables already.

i want the create table syntax of the existing table.How can i see?

say for example sample is the table that i have created already.

now i want the syntax like this
create table sample(.....);

Where can i go and find this systax in access.

If the search i could get the syntax like this only(Select .......)

But i want the create table syntax.

Russell Jones19-Mar-07 21:56

Russell Jones

19-Mar-07 21:56

i think you need a function called createquerydef() on the access database object.

Russ

Re: I want to get the create query of existing table

joerozario19-Mar-07 22:42