|
I am in need for reading huge files (*.txt, *.dbf, *.csv) of around 4+ gb and displaying the contents of
the file page by page. Application also requires to display scrolling data. Functions like ShowLongestString() which would search entire file and display the string.
The current method of reading the files is very slow(using CFile). I have two options to improve the performance.
1) Using CMemFile.
2) Reading the file in chunks at a time and displaying the same. But I wonder how would ShowLongestString() functions perform. Also if the file is in a network it might make the overall system slower for every chunk of data read.
I am confused into wich way to go. Or of there is any better way. Any help with this will be very much appreciated.
Thanks and Regards,
Arti Gujare
-- modified at 10:01 Wednesday 21st June, 2006
|
|
|
|
|
Also please note that I read the whole file into a memory buffer and so its a slow one time read operation.
I have not used CMemfile before. So I am not sure whether CMemfile will satisfy all the app requirements.
|
|
|
|
|
ArtiGujare wrote: Also please note that I read the whole file into a memory buffer...
How are you reading a 4GB file into memory?
"The largest fire starts but with the smallest spark." - David Crow
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
The File size read is less right now. around 20kb. But application might have to read huge files around 4GB.
|
|
|
|
|
Reading a 20KB file into memory is fine. In fact, reading several MB into memory will not cause a noticeable impact on performance.
"The largest fire starts but with the smallest spark." - David Crow
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
I'm hoping you meant 4 MB and not 4 GB. Loading a 4 GB file into memory would be extremely painful and slow your system down to a crawl (if the system even lets you allocate the buffer to begin with). Not to mention that displaying such a file would be insane since no one in their right mind would be able to read 4 GB of data.
That said, if you did mean 4 MB, just load it into a memory buffer. The load operation will be slow, but all operations after that will be much faster. If you are doing this for a lot of files, I would try to come up with some sort of bufferring scheme to minimize the amount of memory needed to hold all the files data (e.g. cache the most recent data and serialize the rest to a temporary file until they become active again).
If you decide to become a software engineer, you are signing up to have a 1/2" piece of silicon tell you exactly how stupid you really are for 8 hours a day, 5 days a week
Zac
|
|
|
|
|
Use file mapping (MapViewOfFile() etc...) to only map part of the file into memory at a time.
Ryan "Punctuality is only a virtue for those who aren't smart enough to think of good excuses for being late" John Nichol "Point Of Impact"
|
|
|
|
|
I would suggest memory mapping sections of the file, like 5MB at a time, if you are going to be paging through the file. As you approach the end of a 5MB segment, look for an end of page marker and then reset the map starting at that marker. You can do the same thing paging backwards, too.
Remember that if you want to map the entire file in memory, you need a contigious address range available, which starts to get tricky when you get to sizes that are around 1GB+, let alone much larger.
You will not be able to map files that approach 2GB in size because IIRC, by default a Win32 process only gets 2GB of user address space allocated to it, and your application has likely already used some portion of that space. To map entire files that are 2GB and above, you will likely have to research the /3GB linker switch and how to write for and work with PAE and/or AWE. Such things are generally far beyond the scope of this message board! 
Peace!
-=- James
If you think it costs a lot to do it right, just wait until you find out how much it costs to do it wrong!
Avoid driving a vehicle taller than you and remember that Professional Driver on Closed Course does not mean your Dumb Ass on a Public Road!
DeleteFXPFiles & CheckFavorites (Please rate this post!)
|
|
|
|
|
Hello everyone,
What does FARPROC* mean on Windows platform? I can not find detailed descriptions from MSDN.
Another question, what should I do if I am going to port source code containing such keyword to other platforms -- like Linux?
thanks in advance,
George
|
|
|
|
|
FARPROC is defined in the WINDOWS.H file as follows:
typedef int (CALLBACK* FARPROC)();
|
|
|
|
|
Thank you zouchao1112!
I think it means a function pointer without any input parameters and returns int type, right?
What means CALLBACK?
regards,
George
|
|
|
|
|
|
Hi man,
It is a very good article. Actually, I know what means __cdecl, __stdcall or something similar. The problem is that, there seems not enough documents about what means FARPROC -- I can not find related information from this article.
regards,
George
|
|
|
|
|
George_George wrote: I can not find detailed descriptions from MSDN.
Did you look in any of the .h files? Set your cursor somewhere in FARPROC and press F12. You may be asked for permission to build the browser information file.
"The largest fire starts but with the smallest spark." - David Crow
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
Hi DavidCrow,
I have used Go to Definition function and find that it is a function pointer, which can be pointed to a function without any input parameters and returns int, right?
But in the definition, there is something like FAR and WINAPI. Do you mean what do they mean?
regards,
George
|
|
|
|
|
George_George wrote: But in the definition, there is something like FAR and WINAPI. Do you mean what do they mean?
Look them up in the .h files the same way (i.e., F12).
"The largest fire starts but with the smallest spark." - David Crow
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
Thank you DavidCrow!
I have followed your suggestion and I have solved all the problems except what means far and near. For example, in the following code segment of windef.h.
<br />
#define far<br />
#define near<br />
#if (!defined(_MAC)) && ((_MSC_VER >= 800) || defined(_STDCALL_SUPPORTED))<br />
#define pascal __stdcall<br />
#else<br />
#define pascal<br />
#endif<br />
Can you help please?
regards,
George
|
|
|
|
|
George_George wrote: ...what means far and near.
With 16-bit Windows and its segmented memory model, you had 16-bit near pointers and 32-bit far pointers. A far reference refers to a function or data object (or a pointer to a function or data object) that is in a different segment than the current one. Memory was referenced like:
segment:offset
Accessing code or data with a near reference is much quicker than accessing it with a far reference. When you use a far reference, your program must first find the segment and then find the code or data within that segment. When you use a near reference, your program only needs to find the code or data.
"The largest fire starts but with the smallest spark." - David Crow
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
Thank you DavidCrow!
In nowadays, whether memory is still working in the same way?
regards,
George
|
|
|
|
|
16-bit Windows will always behave in this fashion. 32-bit Windows does not. It uses a simple, 32-bit address.
"The largest fire starts but with the smallest spark." - David Crow
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
Thank you DavidCrow!
I think in nowadays, there is no 16-bit Windows. Right?
regards,
George
|
|
|
|
|
George_George wrote: I think in nowadays, there is no 16-bit Windows. Right?
How could I possibly know? Whether something exists and whether it is used are two different things.
"The largest fire starts but with the smallest spark." - David Crow
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
Thank you DavidCrow!
A good answer!
regards,
George
|
|
|
|
|
|
Hi Sarath,
The link describes something like __stdcall. There is no information about FARPROC.
regards,
George
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
