|
Hey,
I'm suffering an issue of reading and displaying a huge text file (>=300MB) in a textbox.
I found that the maximum string I can read into memory is about 250MB because of the limitation of the RAM and also likely related to the limitation of the .net component. (when the text is very large to display in a textbox, out of memory exception will be thrown)
So in this case, the only way we can do is to show the text file on demand, which is likely what the UltraEdit does.
Could anyone give me some tips or solutions about this issue?
Thanks in advance.
dengbo---------------------------------
Believe what you saw!
|
|
|
|
|
Use your own scrollbars (HScrollBar, VScrollBar). Set the Maximum on the HScrollBar to the size of the textfile / 80 (or whatever average line length you think would be appropriate). When somebody scrolls, you can approximate where they are in the file and load up that text into a fixed-size TextBox. When you change the text as they scroll, you'll want to remember the position of the caret and the selected text so you can reapply them after you change the text. Also, intercept key presses for up/down/left/right/page down/page up/etc so you can implement those functions yourself. For the HScrollBar Maximum, default it to 80. Like the VScrollBar, you should use the position of the HScrollBar to scroll through the fixed-size textbox (which really means changing the text in the textbox.
In a background thread, scan through the entire text file to find line breaks and actual line widths and the number of lines in the file. Store the positions of the line breaks in a file (text file, binary file, SQL Server CE file) so that you can get information about a given line by looking at that file (this helps you avoid using RAM). Once you have scanned the entire file, you can adjust the Maximum for the HScrollBar and VScrollBar to the actual values (i.e., the number of lines in the file and the number of characters in the widest line). Having the offset of the line beginnings stored in a file will allow you to instantly determine where in the text file the data is for a given line number. This will allow you to create perfect scrolling that is virtually instantaneous (the scrolling will be instantaneous and approximate up until you have loaded everything from the text file).
It gets a little more complicated if you want to allow the user to make edits. In that case, you'd probably want to asynchronously load each line of the file into a SQL Server CE database. Whenever an edit is made, you'd have to intercept it (e.g., via keypress) and made the corresponding edit to that line in the database. If you want to save the file, you'd then store each of those lines in the database back to the text file. Or, if you want all edits to immediately be saved to the text file, you could just avoid the database altogether.
There are other details you'll probably have to worry about, but I trust you can figure them out. Note that SQL Server CE has a maximum size of about 4GB. If you need larger than that, you can choose another database format or use your own file format.
|
|
|
|
|
Hi aspdotnetdev,
Thank you very much for giving me so detailed tips.
Yes, that's what I'm trying to implement.
A good news is that the textbox will be readonly, user only has the read, find, print options for this huge text.
dengbo.---------------------------------
Believe what you saw!
|
|
|
|
|
Dengbo wrote: reading and displaying a huge text file (>=300MB) in a textbox.
that sounds like extreme non-sense; who in his right mind is going to read one huge page of text like that?
 Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]
I only read code that is properly formatted, adding PRE tags is the easiest way to obtain that. All Toronto weekends should be extremely wet until we get it automated in regular forums, not just QA.
|
|
|
|
|
There are instances where this is necessary. For example, log files. A person might want to start reading a log file, starting with the last entry (at the end of the text file) and moving up to older entries. Or maybe search through the text to find what they are looking for.
|
|
|
|
|
that certainly does not make sense to me; if I were to need that amount of information, I would never show all of it on screen at once, I would instead provide means to view parts of it, have it searched, sorted, whatever is relevant.
When you have a modern PC with 1+ GHHz CPU and a 2+ gigabytes of memory, you can and must do better than just dump hundreds of megabytes of text in the user's face and leave him at a loss.
 Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]
I only read code that is properly formatted, adding PRE tags is the easiest way to obtain that. All Toronto weekends should be extremely wet until we get it automated in regular forums, not just QA.
|
|
|
|
|
Well it would be impossible to "show all of it on screen at once"... there would have to be some way to page through the text (like a scroll bar). Searching is another example of how to page through the data (you only jump to lines you are interested in).
Luc Pattyn wrote: you can and must do better than just dump hundreds of megabytes of text in the user's face and leave him at a loss
Just because the user wants to display the text in a text box does not mean that is all he/she wants to do. There could be additional features (such as search).
|
|
|
|
|
with 300MB of text, you get some 5M lines; when the screen is 1000 pixels high, one scrollbar pixel corresponds to 5000 lines. Are you ever going to touch the elevator and get to see what you are hoping for?
Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]
I only read code that is properly formatted, adding PRE tags is the easiest way to obtain that. All Toronto weekends should be extremely wet until we get it automated in regular forums, not just QA.
|
|
|
|
|
So you have 2 vertical scrollbars. One which scrolls the full height, one which scrolls 5000 lines at a time. Or you have a scrollbar and a slider so the user can control the amount the scrollbar scrolls. Or you let the user use page up/page down. Or you assume they will probably be using the search function. Or you take the horizontal position of the mouse into account when scrolling vertically... more to the right and you scroll more and more to the left means you scroll smaller increments. Or you allow the user to jump to a specific line or jump a certain number of lines down/up. Could even have a textbox (or slider) that controls the number of lines a page down/page up will jump. There are many ways this could be solved. No reason to put an artificial limit on the size file that can be viewed.
|
|
|
|
|
Luc Pattyn wrote: Dengbo wrote:
reading and displaying a huge text file (>=300MB) in a textbox.
that sounds like extreme non-sense; who in his right mind is going to read one huge page of text like that?
you know, some routing reported files (shop order, job cost, etc) in ERP can be very huge...---------------------------------
Believe what you saw!
|
|
|
|
|
Dengbo wrote: some routing reported files (shop order, job cost, etc) in ERP can be very huge...
I know, but that does not mean I want to see all their content, I want the app to present a decent user interface, not a truck load of data.
 Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]
I only read code that is properly formatted, adding PRE tags is the easiest way to obtain that. All Toronto weekends should be extremely wet until we get it automated in regular forums, not just QA.
|
|
|
|
|
If it is a log file, I suggest saving it with xml node info. This can then be hooked to some database (file to xml) technique to enable search features. Also, it will be more organized data.
If this log file is not created by you, then it sounds like a daunting task.
Dengbo wrote: So you have 2 vertical scrollbars. One which scrolls the full height, one which scrolls 5000 lines at a time
This gives me an idea. What you can do is not to have text in the textbox. Show only the info that the user wants.
Instead of adding ScrollBars, add buttons to scroll page up or down. This will enable a paging possibility and it will be easier to handle too.
|
|
|
|
|
FYI, you misattributed that quote... I wrote that.
Som Shekhar wrote: add buttons to scroll page up or down
That is what the PAGE UP/PAGE DOWN buttons would do. It would be, however, quite slow when paging 5,000 lines or so.
Som Shekhar wrote: Instead of adding ScrollBars
Why not have both ScrollBars and PAGE UP/PAGE DOWN functionality?
|
|
|
|
|
Well, I was thinking of having 5+ string builder objects.
1 for current text in textbox
1 for next page
1 for previous page
1 for 5000 lines ahead page
1 for 5000 lines behind page
you can add more if you wish in the same way.
You need to maintain few indexes in your program to read only those blocks using StringBuilder.ReadBlock method.
Searching is something I am not sure of.
|
|
|
|
|
The problem is not the buffer... the speed issue is from the user... it will take many pages before they traverse 5,000 lines of text. The text can be loaded from the file on-demand without the user noticing. The trick is to reduce the amount the user must interact with the UI.
|
|
|
|
|
Now, this confuses me. What exactly is your question then?
Reading your question, it sounded like displaying it in Textbox is the problem.
Are you talking about Search? or some other thing?
|
|
|
|
|
I am not the one with the question. Look at my username. I was merely correcting you.
|
|
|
|
|
HA HA HA... This was funny... I am so sorry!!!!! HA HA HA...
Now I see, I misquoted you, Misunderstood you and everything... just because you replied me first... Sorry Buddy.
|
|
|
|
|
aspdotnetdev wrote: The trick is to reduce the amount the user must interact with the UI.
this is where we agree.
Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]
I only read code that is properly formatted, adding PRE tags is the easiest way to obtain that. All Toronto weekends should be extremely wet until we get it automated in regular forums, not just QA.
|
|
|
|
|
hiwave is a external .exe, and it has a Component Object Model (COM) interface.
I add a COM reference of hiwave library in my C#.
When use "HiwaveLib.Hiwave aaa = new HiwaveLib.HiwaveClass();", I can get a new instance of hiwave and control it.
But I want to realize below function:
first, user run the hiwave, and then my C# connects to this existent hiwave instance through COM interface and control it.
What should I do?
Thanks
|
|
|
|
|
You haven't provided enough details to narrow down an answer.
Has this existing instance been created inside your own app, or was it created in another app? Did you write this other app or is it a "off-the-shelf" piece of software? Does the object you're trying to use have built in singleton support?
|
|
|
|
|
The existing instance is created in another app. My own app want to control the existing instance.
This other app is a "off-the-shelf" software.
|
|
|
|
|
Then you're very limited in what you can do.
If the COM object that app creates is a singleton AND is exposed to the rest of the system, then you can probably use the Activator.GetObject() method to get at the instance of the COM object (you still need to import the COM object in your project references.) If not, you're pretty much SOL.
|
|
|
|
|
I received the following error (see line *********error***********) when trying to transfer data from excel to sql using the code below. The sql table is in the database and columns all match the store procedure, does anyone have any idea what creates this problem.
Thanks in advance, Michael
Failed to obtain column collation information for the destination table. If the table is not in the current database the name must be qualified using the database name (e.g. [mydb]..[mytable](e.g. [mydb]..[mytable]); this also applies to temporary-tables (e.g. #mytable would be specified as tempdb..#mytable).
void ExcelToSqlBulkCopy(String strPath, String strFile, String strTableName)
{
strExcelStoredProcedure = "SELECT ID,City,State FROM [Cities$]";
String excelConnectionString = y;
String sqlConnectionString = z;
// Create Connection to Excel Workbook
using (OleDbConnection connection = new OleDbConnection(excelConnectionString))
{
OleDbCommand command = new OleDbCommand(strExcelStoredProcedure, connection);
connection.Open();
// Create DbDataReader to Data Worksheet
using (DbDataReader dr = command.ExecuteReader())
{
// Bulk Copy to SQL Server
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(sqlConnectionString))
{
bulkCopy.DestinationTableName = "[MyDatabase].[dbo].[" + strTableName + "]";
bulkCopy.WriteToServer(dr); *********error***********
}
}
connection.Close();
}
}
|
|
|
|
|
I never use a data reader so how do you inspect the contents of a datareader to check that the info in it is correct before you try to bulk copy.
I would change the datareader to a datatable and inspect the contents before the bulkcopy statement comparing the columns and the datatypes, excel is a crappy data source! Collation makes me suspect unicode or suspect characters in the data.Never underestimate the power of human stupidity
RAH
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
