|
Hi,
I'm looking for an algorithm or a freeware/shareware utility that can handle a large file - 12G with over one billion records. Everything I've been able to find chokes with that and I'm not sure I can write one that will be efficient enough.
Thanks,
David
|
|
|
|
|
i would break it up into managable pieces, run qsort on each, then do a sorted merge on all the pieces.
p.s. sounds like you need a database 
-c
A | B - it's not a choice.
|
|
|
|
|
I don't want to use a database - too much overhead for what I want to do. All I am really trying to do is determine if I have any duplicate data.
I do have an app that will work with about 500M records, so a merge wouldn't be so bad. Biggest problem right now is time, I'm busy. So if someone had something already...
Thanks for the reply,
D
|
|
|
|
|
Why not use a simple hash table. Possibly just add up the ASCII values of the data and when the hash table points to more than 1 record check if they are duplicates, this won't trap all duplicates, i.e. the pointers to the records may be different and not duplicates so you need to check these individually, but depending on the hash method should get most of them
If I have seen further it is by standing on the shoulders of Giants. - Isaac Newton 1676
|
|
|
|
|
If its sorting of a large file you needing then this sounds like a job for a distruibuted solution, or if you don't have mulitiple prcessors avaliable, break the file up into managable chunks of say 100MB each then sort those, and take it from there.
Alternatively, make use of a database to manage the sorting for you. You will need to tell it which fields to index - those your going to need to sort on. In terms of free tools MySQL or MSDE.
|
|
|
|
|
You can give my CDataFile[^] class a try if your file is ascii CSV. It uses std::vector to store the data. You will need to modify the datatype from double to CString or std::string if you have more than just numeric data. Of course, make sure you have enough memory and swap space to handle it. I've handled data files over 350MB with it no prob. But 12GB  ... I agree as above, you may need to break it down.
- Nitron
"Those that say a task is impossible shouldn't interrupt the ones who are doing it." - Chinese Proverb
|
|
|
|
|
Hi!
I'm looking for creating an MFC-based application capable of creating a document which can be displayed in several ways on the screen. For example, a graph or a set of values (like Excel).
I found in the MFC documentation that I must create as many CMultiDocTemplate objects in InitInstance () as I have kinds of documents.
But when the program starts, it asks me what kind of document I would like to create.
How can I prevent this screen from being displayed? And how to manage the several views via a menu?
Thks in advance!
Appstmd
http://www.appstmd.com
|
|
|
|
|
hello,
i have a heirarchical set of data which is created from the following simplified class defintion:
class Extent
{
vector<surface*>m_pSurfaces;
vector<extent*>m_pExtents;
};
CMyAppDoc.cpp:
Extent *globalExtent=new Extent();
...
as can be seen: extents can have surfaces, and other extents.
the best way to represent this data is to use a CTreeCtrl class. this appears in a dialog with add surface, add extent, remove item buttons. i have never programmed such a complex data structure with a tree-view before and am having difficulty acquiring pointers to my data objects when they are selected in the tree view. every item in the tree view corresponds to an item in the globalExtent data heirarchy.
one way to get the pointer i thought would be to associate a HTREEITEM to each surface and extent - that way, when a CTreeCtrl item was selected i could use GetSelectedItem to get the selected HTREEITEM and search my globalExtent heirarchy for a match. This would not be a good idea though due to the search required - a large globalExtent heirarchy will require a time-consuming search.
I wondered if anyone had come across a similar situation and if so how they solved the problem, or if anyone had any ideas as to how to go about the probelm.
Thanks,
Paul.
|
|
|
|
|
Keep this association completely out of you data structure.
One way is to use the treeCtrl's SetItemData() / GetItemData()
(cast the pointer to/from LPARAM, that's perfectly ok)
The other way is to use a map <htreeitem, pointer-to-item="">, or even a two-way map
Both have advantages and disadvantages. If the data doesn't change without the tree control knowing, the first is easier.
It's a royal pain to watch a sex drugs and rock'n'roll design decay into an aids crack and techno implementation [sighist] [Agile Programming] [doxygen]
|
|
|
|
|
What about using the the user defined data attribue (lParam ?) in the tree item to point back to the associated item in your globalExtent class.
Neville Franks, Author of ED for Windows. www.getsoft.com
Make money with our new Affilate program
|
|
|
|
|
hello,
Is there any function is vc++ for getting processor ID , motherboard serial number and BIOS information ?
Thanks in advance
Mahesh
|
|
|
|
|
|
hello Riazi,
Thanks..i downloaded the source. But when i tried to compile it is showing error. Error is " cant open OXVerInfo.h header file "
In the downloaded source there is no file called OXVerInfo.h.
what to do ????? 
Mahesh
|
|
|
|
|
Post you question in the article message board.
Perhaps your question is already answered in there?
Rickard Andersson@Suza Computing
C# and C++ programmer from SWEDEN!
UIN: 50302279
E-Mail: nikado@pc.nu
Speciality: I love C#, ASP.NET and C++!
|
|
|
|
|
As I mentioned in message board of article, this file is part of Dundas Ultimate Toolbox Library. Remove this line and complie project with no errors!
A. Riazi
|
|
|
|
|
Friends,
I do not want title bar or a caption in a window. But the problem is that whenever i remove title bar from a window(or dialog), the taskbar button of the application becomes empty. I want to add some text and icon on the task bar button..can anyone tell me how ??
Thanks in advance for your reply.
|
|
|
|
|
|
 Unfortunately, the solution of my problem is not present in the faq. I actually want the taskbar button showing both icon and some text, but at the same time i don't want the title bar (caption) of a window. Is there any API with the help of which we can deal with taskbar button ???
|
|
|
|
|
Ooops!
I think I read your message a little to careless!
Sorry, but then I don't know how to do
Rickard Andersson@Suza Computing
C# and C++ programmer from SWEDEN!
UIN: 50302279
E-Mail: nikado@pc.nu
Speciality: I love C#, ASP.NET and C++!
|
|
|
|
|
I would look into WM_NCCALCSIZE and WM_NCPAINT. You could effectively remove the visuals of the caption of your window, while it still contains a caption internally.
Then the taskbar will still display the icon and title.
Good Luck
Build a man a fire, and he will be warm for a day
Light a man on fire, and he will be warm for the rest of his life!
|
|
|
|
|
I have problem with a so simple thing as adding a damn string to a ListBox in a non-MFC app.
BUT!
When I use the WindowsX.h macro ListBox_AddString(hwndList, "Hello") the item is added but no text is visible!!
I have sure missed something very simple, can you tell me what I've missed!?
Rickard Andersson@Suza Computing
C# and C++ programmer from SWEDEN!
UIN: 50302279
E-Mail: nikado@pc.nu
Speciality: I love C#, ASP.NET and C++!
|
|
|
|
|
If it has the LBS_OWNERDRAWFIXED style, then be sure to add the LBS_HASSTRINGS style as well.
|
|
|
|
|
Arrgh!!
It was not meant to be an ownerdraw ListBox, but in some way I did hit the "Ownerdraw" selection property in VC++ by chance!!!
Rickard Andersson@Suza Computing
C# and C++ programmer from SWEDEN!
UIN: 50302279
E-Mail: nikado@pc.nu
Speciality: I love C#, ASP.NET and C++!
|
|
|
|
|
Hi,
I just made a program to download a file from the web (using URLDownloadToFile). I'm trying to download a html file with the latest news in a particular site, but it always comes unrefreshed... I always get the same file!
How can I do to refresh it before download ?
Or download the refreshed one ?
thanks.
|
|
|
|
|
It sounds like you're fetching via a proxy... -or- you forced to use a proxy (via your ISP for example). have a look on the HTTP headers, if you find a proxy notice.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
