vector max_size calculation, how?

Question

1.00/5 (2 votes)

See more:

C++

#include "stdafx.h"
#include <iostream>
#include <vector>
#include "conio.h"
int main ()
{
	typedef struct
	{
		int iArr[10];
	}BIGSTRUCT;

	BIGSTRUCT st;
  std::vector<BIGSTRUCT> myvector;
  int ii = sizeof(BIGSTRUCT);
  std::cout << "max_size: " << (int) myvector.max_size() << '\n';
  _getch();
  return 0;
}

I am using following code to get maximum vector I can allocate; If the size of iArr is 1, then I am getting it 1,073,741,823; But if I increase the size to 10, it is 1,073,741,82;
http://stackoverflow.com/questions/4321040/c-vectors-of-objects-and-pointers?rq=1[^]
http://stackoverflow.com/questions/3813124/c-vector-max-size[^]
I have too many confusions here:
1) how this number is coming? Ho wit is calculated?

Now, if I am on a 32bit machine, with 2GB RAM, and if I have a vector of size say 170,000,000 members, what is the maximum size of each member do I have to have so I will not go out of memory?

Thanks

Posted 21-Apr-13 8:33am

iDebD

Add a Solution

3 solutions

Solution 1

The calculation is based on the formula (for your case):

C++

max_size = (size_t)(-1) / sizeof(BIGSTRUCT);

For an array with 1 int (4 bytes) this yields 1,073,741,823.
For an array with 10 int (40 bytes) this yields 107,374,182.

Unfortunately it is not based on your actual machine configuration.

Posted 21-Apr-13 18:03pm

Steve44

Updated 23-Apr-13 10:19am

v3

Solution 3

When using the stl, you can never pre-calculate the exact maximum size of a collection of items. The best you can do is approximate. What follows is an estimate I make here and now, with a few unstated assumptions. If you ask 2 other people, you might get 5 or more different answers. If you ask me again tomorrow I might have a different estimate.

For std::vector, memory consumed for each element will be a pointer to the struct, plus an instance of the struct, which I assume is on the heap, which means that there will be memory management overhead.

On recent Windows machines:
For debug builds, this will be a minimum of sizeof(void*) + (sizeof(BIGSTRUCT) + 16) where the last item is rounded up to whatever the memory management "chunk size" is.
For release builds, this will be a minimum of sizeof(void*) + (sizeof(BIGSTRUCT) + 8) where the last item is rounded up to whatever the memory management "chunk size" is.

Both of the rounded values might be different (haven't checked lately).

The st::vector array of pointers will likely be contiguous (or if not, then close to it). The structs in the heap will not likely be contiguous (and will depend on how you create them).

Posted 22-Apr-13 9:08am

H.Brydon

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Stefan_Lang · Accepted Answer · 2013-04-22T02:14:00

Now, if I am on a 32bit machine, with 2GB RAM, and if I have a vector of size say 170,000,000 members, what is the maximum size of each member do I have to have so I will not go out of memory?

First, the physical amount of RAM has nothing to do with it: a modern OS will shuffle your memory contents around as needed. E. g. on Windows, that's what the pagefile is for!

That said, the maximum amount does depend on both your application, and your OS: in theory, you can address a total of 4 GB on a 32 bit machine. However, that address space covers every object of your entire application! So if you've got a lot of other spacious objects lying around, this will limit the amount of space available for your array accordingly! Second, your OS may reserve part of your address space: e. g. Windows likes to use the address space above ~3 GB for system functions (this may in fact vary depending on the version of Windows you are using). Therefore your address space may be considerably smaller than you think.

A word of advice: you should never strive to use up the maximum amount of memory available! If nothing else, you're severely hampering every other application that is running on the same machine because they're forced to constantly move memory contents around!

If you need to store that much data in memory, then you should really try to redesign your algorithm and make it use less memory! The usual way for working with big data is to use a database! They let you load all the data necessary to work on one record individually. Or you can just read the data record by record.