|
Hi,
I'm working on an application where it loads a huge file (like 4-6GB) and parses the file, searching for regular expressions, converting from Hex to Human readable form, filters etc.
Since the file is of very large size, I'm facing lot of performance issues where it takes hours to process and show the file information in a window.
To overcome this, what I did is, break the entire file into logical segments of 1 MB each and process 3 segments at a time. When the user is scrolling, the next segments will be processed. With this, a certain amount of improvements were done. But it is limited to files which are less than 300MB or so. For files which are over GBs, it is just crawling and pathetic.
Wondering if there is any better way to handle these large files in a better way so that the user need not wait for hours?
Will a buffered read and caching the decoded information in a temporary files using a thread would help? How can we make sure that the GUI doesn't hang during these intensive parsing process?
Really appreciate your suggestions.
Thanks
Vikas
|
|
|
|
|
by the way, the application is written in MFC
|
|
|
|
|
You could do memory mapping for the file segments.
You should get hold of a good profiler and check where the actual bottle neck is.
«_Superman_»
I love work. It gives me something to do between weekends.
|
|
|
|
|
Vikas K. wrote: How can we make sure that the GUI doesn't hang during these intensive parsing process?
What about using a worker thread?
- ns ami -
|
|
|
|
|
I've had pretty good performance with memory mapped files (I've only used files up to about 1GB). What you need to do is create a file mapping with CreateFileMapping and then use MapViewOfFile to map segments of the file.
I worked with 16MB windows on the file, which actually meant I mapped 32MB at a time, 16MB before and after the nominal file offset, so that negative random accesses from the file offset didn't fault over into the previous window - something like the diagram below:
+----------------+----------------+
| | |
| | |
| | |
+----------------+----------------+
^ ^ ^
| | |
-16MB Requested +16MB
offset into file
I packaged this up into a MemoryMappedFile class which had an associated randomly-access iterator class (like a vector has an iterator). The memory mapped file could only be accessed using the iterator, which was exposed using 'begin' and 'end' methods of the memory mapped file class.
On a reasonable workstation (2.4GHz Core 2 Duo), I could traverse the file at around 100MB/second when counting data packets whose size was roughly 200 bytes.
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
|
|
|
|
|
Thank you for your suggestions. Willwork on them and update  .
|
|
|
|
|
Hi to All,
I am trying to make a win32 dll.
I just wanted to make sure if we can use the win32 dll in all types of application project or all type of languages. I am sure about Vb. What about other languages? Will it work?
Thanks in advance
HimangshuS
I am a learner Always
|
|
|
|
|
himangshuS wrote: I am trying to make a win32 dll.
Well, have fun with it!
himangshuS wrote: I just wanted to make sure if we can use the win32 dll in all types of application project or all type of languages. I am sure about Vb. What about other languages? Will it work?
The correct answer is no. The practical, common sense answer, is yes (provided the target OS is Windows and you're ready to put some effort in the task...)
If the Lord God Almighty had consulted me before embarking upon the Creation, I would have recommended something simpler.
-- Alfonso the Wise, 13th Century King of Castile.
This is going on my arrogant assumptions. You may have a superb reason why I'm completely wrong.
-- Iain Clarke
[My articles]
|
|
|
|
|
Hi
thanks for your reply
Could you please put some more light into it...
thanks
I am a learner Always
|
|
|
|
|
Most languages allows using Windows API (on Windows, of course) that comes inside DLLs, hence you will be able to use, as well, your DLLs, in such languages.
If the Lord God Almighty had consulted me before embarking upon the Creation, I would have recommended something simpler.
-- Alfonso the Wise, 13th Century King of Castile.
This is going on my arrogant assumptions. You may have a superb reason why I'm completely wrong.
-- Iain Clarke
[My articles]
|
|
|
|
|
|
But my search bar is powerful than your search bar I could to see more. 
Of one Essence is the human race
thus has Creation put the base
One Limb impacted is sufficient
For all Others to feel the Mace
(Saadi )
|
|
|
|
|
Well, I got all impatient! That cut and pasting gets tiring you know...
Iain.
Codeproject MVP for C++, I can't believe it's for my lounge posts...
|
|
|
|
|
Hello all,
I have been given the work which includes writing a add-in for Outlook Express.
The working of this add-in should be as follows:
Whenever a mail is opened in a new window & a right-click is made on a sender's name,then a context menu should pop-up which will include some menu items apart from
existing ones.
I would also like to provide a separate tool bar which will have some buttons on it,
providing same functionality.
Is there a way of doing this without making use of Third-party libraries.
reply me at : n.chudekar.31@gmail.com
Thanks n Regards,
Nikhil.
|
|
|
|
|
I don't know about Outlook Express but for MS Outlook, you can do the operations that you have given using Microsoft VSTO. Please read more about it and see if it helps you.
|
|
|
|
|
Hello All,
I am also facing same problem.I want to add context menu and tool bar in Inspector Window of Outlook Express  . I have done it for MS outlook.
How to archive above task in Outlook Express
Thanks & Regards,
Jitendra Shirolkar
|
|
|
|
|
hello,i have seen all your forums to set the background color of the dialog.i am able to set the background color.But i want to change the background color of the main dialog during run time on selection of different menus.i want to call repaint ,how i do this?.i am using mfc dialog based application
software enggineer
|
|
|
|
|
Member 1631945 wrote: software enggineer
Sure?
If the Lord God Almighty had consulted me before embarking upon the Creation, I would have recommended something simpler.
-- Alfonso the Wise, 13th Century King of Castile.
This is going on my arrogant assumptions. You may have a superb reason why I'm completely wrong.
-- Iain Clarke
[My articles]
|
|
|
|
|
Yeah but it seems we need to a duel for it!
Of one Essence is the human race
thus has Creation put the base
One Limb impacted is sufficient
For all Others to feel the Mace
(Saadi )
|
|
|
|
|
Calling Invalidate will redraw the dialog.
You can handle WM_CTLCOLORDLG and return the appropriate brush.
«_Superman_»
I love work. It gives me something to do between weekends.
|
|
|
|
|
Another approach than the other reply.
You can map WM_ERASEBKGND and write like...
BOOL CMyDlg::OnEraseBkGnd( CDC* pDC )
{
CRect rect;
GetClientRect( &rect );
pDC->FillSolidRect( &rect, m_clrBK );
}
You can set the m_clrBK where ever you needed and call Inbvalidate( TRUE ); to update the background.
- ns ami -
|
|
|
|
|
You can also use RedrawWindow[^].
> The problem with computers is that they do what you tell them to do and not what you want them to do. <
> Life: great graphics, but the gameplay sux. <
|
|
|
|
|
|
Did you handle WM_CTLCOLOR and you need to declare a COLORREF variable and use it on this event with SetBKColor.
Of one Essence is the human race
thus has Creation put the base
One Limb impacted is sufficient
For all Others to feel the Mace
(Saadi )
|
|
|
|
|
I have a MFC Ribbon application that I developed on 32bit system. I have statically linked to MFC.
My question is whether this application will run fine on 64bit XP/Vista?
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
