C / C++ / MFC

27-Feb-09 22:58

I use

vector<string>

to save a huge file
Each item maybe a word or a sentence
but

vector<string>

cost too many memory

eg:
20M file cost 200M+ memory

How can I reduce memory using??? Poke tongue | ;-P

Thanks
Techie York

Stuart Dootson27-Feb-09 23:30

27-Feb-09 23:30

How about holding the file in a buffer then storing pointers to word beginning & end in the vector:

char* fileBuffer = new char[fileSize];
typedef std::pair<const const=""> Word;

std::vector<word> words;

// Now find each item in the fileBuffer and store the pointers defining its extent in a Word which you can add to words</word></const>

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

bcoide28-Feb-09 0:14

bcoide

28-Feb-09 0:14

Maybe you can just use vector to hold character pointers.

Working example

york52828-Feb-09 3:11

28-Feb-09 3:11

bcoide

Thank you for your share

but I deal with different language

Full Text Search

The vector saves the result of segment

so use string is convinient

but vector's memory grows so fast and huge

a 20M file's result can cost 200M memory

that a waste!!

thank you

maybe I need another idea!

york52828-Feb-09 2:59

28-Feb-09 2:59

It's like this

vec[0] = "you";
vec[1] = "friend";
vec[2] = "你好";
vec[3] = "朋友";

I use

vector<string>

to hold word and build index to search

If I save pointer

The speed of building index maybe not so fast

because I deal with Chinese and English together

I have to consider the speed

but the cost of memory is so big

I wanna find a way on balance

Thank you so much

Stuart Dootson28-Feb-09 3:13

28-Feb-09 3:13

If you had

wchar_t fileBuffer[] = "youfriend你好朋友";

You could have:

typedef std::pair<wchar_t*,> Word;
std::vector<word> words;

words.push_back(std::make_pair(fileBuffer, fileBuffer+3); // Specifies "you"
words.push_back(std::make_pair(fileBuffer+3, fileBuffer+9); // Specifies "friend"
words.push_back(std::make_pair(fileBuffer+9, fileBuffer+11); // Specifies "你好"
words.push_back(std::make_pair(fileBuffer+11, fileBuffer+13); // Specifies "朋友"</word>

?

That could be less overhead than std::string, possibly.

Alternatively, you could use something like a trie[^] to hold all the words?

What's the intent, anyway - why are you building this list of words?

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

york52828-Feb-09 3:26

28-Feb-09 3:26

I am doing a program about Full Text Search

English is easy to segment

but Chinese is complicated

First I should separate text that to make Chinese and English text to two vector

Second do segment to each text and building index for search

I have to consider about speed ,the size of index file and Search speed

some difficulty I've already solved

I wanna do it well

Thank you so much

Rolf Kristensen1-Mar-09 0:49

Rolf Kristensen

1-Mar-09 0:49

Have a feeling that it is caused by the automatic growth of strings / vector.

You can use swap to trim the vector to minimum size. Guess the same thing should be possible with strings.

vector<string>(trim).swap(trim);

Herb Sutter: The Incredible Shrinking vector

It is even better to resize the strings / vectors to their expected size from the start.

york5281-Mar-09 5:51

The sequence of menu command in doc/view architecture?

1-Mar-09 5:51

Yes

That's the problem

I'll try later

Thank you so much for your advise

Poke tongue | ;-P

Joseph Marzbani27-Feb-09 22:56

Joseph Marzbani

27-Feb-09 22:56

Suppose that we've defined (for example) OnAppAbout() function in CMyAppView to be called when the user clicks the About item from Help menu. Now if we define it in both CMainFrame and CMyAppDoc as well, which of them is called when the user clicks the menu item?

Re: The sequence of menu command in doc/view architecture?

Stuart Dootson27-Feb-09 23:33

How to change the size and position of a CView Derived window in a SDI app?

27-Feb-09 23:33

Why don't you try it & see? It's not like you can break anything...it's just software.

This article[^] by Paul DiLascia (RIP) could also help.

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

Joseph Marzbani27-Feb-09 22:13

Joseph Marzbani

27-Feb-09 22:13

I wonder if is there a way to change the size and position of a CView derived window within a SDI application? If not why?

Re: How to change the size and position of a CView Derived window in a SDI app?

Stuart Dootson27-Feb-09 23:34

Re: How to change the size and position of a CView Derived window in a SDI app?

27-Feb-09 23:34

Isn't the view size going to be tied to the main frame size in an SDI app?

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

Joseph Marzbani28-Feb-09 2:30

Joseph Marzbani

28-Feb-09 2:30

no !!!

Re: How to change the size and position of a CView Derived window in a SDI app?

Stuart Dootson28-Feb-09 3:17

Convert CString to wchar_t* - For very very large CString

28-Feb-09 3:17

OK - first hit for "SDI resize view" on Google is this CodeProject article[^] - does that help?

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

Md Saleem Navalur27-Feb-09 21:12

Md Saleem Navalur

27-Feb-09 21:12

Hi,,
I need this info.

I am interacting with MSMQ using VC++.

I am able to send messages. But i am not able to send really really large messages, really close to 4 MB but not more than that.

I guess the issue here is not with MSMQ, but to convert CString to wchar_t*.

I am using this code,
int length = data.GetLength(); // data is CString
USES_CONVERSION;
wchar_t *pwcharMessageBody = T2W(data.GetBuffer()); //stack overflow,This line fails for large string
data.ReleaseBuffer();

This is working for most cases. But not working for very large string size.

Any help will be great.

Thanks,
Saleem

Re: Convert CString to wchar_t* - For very very large CString

«_Superman_»27-Feb-09 21:27

«_Superman_»

27-Feb-09 21:27

Use MultiByteToWideChar with a large buffer.

«_Superman_»
I love work. It gives me something to do between weekends.

Re: Convert CString to wchar_t* - For very very large CString

Stuart Dootson27-Feb-09 21:50

Re: Convert CString to wchar_t* - For very very large CString

27-Feb-09 21:50

If you want to stick with the handy ATL conversion mechanisms, and you're using Visual Studio 2003, 2005 or 2008, check out this page[^] and use a CT2W object - you should just be able to add a C to your current T2W usage.

The Cx2y objects default to using an internal buffer for 64 byte strings, but allocate a buffer with malloc for larger strings, which is probably just what you want.

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

Md Saleem Navalur27-Feb-09 23:10

Md Saleem Navalur

27-Feb-09 23:10

Thanks,,,but there seems to be a problem.
Code compiles fine.
But the wchar_t* is blank.

this is my code....
wchar_t *pwcharMessageBody = CT2W(data);

tried this too..
wchar_t *pwcharMessageBody = CT2W(data.GetBuffer());

No luck.
Any help will be great.

Saleem

Re: Convert CString to wchar_t* - For very very large CString

Stuart Dootson27-Feb-09 23:27