Processing huge amount of data

Question

5.00/5 (1 vote)

See more:

Hi, I want to ask: What is the best way that can be used to represent a huge matrix that contains digits to be processed? It is too big that it can be processed in the main memory at once.

Posted 26-May-10 22:39pm

Member 7203335

Updated 26-May-10 22:47pm

Moak

v2

Add a Solution

Comments

Moak 27-May-10 4:50am

Updated question, perhaps narrow down which programming language you want to use.

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Aescleal · Answer 1 · 2010-05-26T23:36:00

It all depends on what you want to do with your matrix and what sort of data it holds.

If it's a sparse matrix then you can represent it fairly efficiently as a std::map<std::pair<unsigned, unsigned>, double>.

If you want sequential access then store your data in a file and use an fstream or an istream_iterator if you're feeling flash.

If you're doing clustered access have a look at memory mapped files. This enables you to treat a chunk of a file as memory and map it into the address space of your process.

If you want random access with a working set bigger than your current system memory then whatever data format you use you're going to be stuffed, it's going to be slow. You might be able to use some form of database style hashing or indexing to alleviate this but it really depends on your application.

Cheers,

Ash