Implementing custom iterator/enumerator for batch data

Question

5.00/5 (2 votes)

See more:

I have a C++ class which models an object. This object has a list of data, which can be quite large. And retrieving this data can be expensive. So when i need to access the items in the list, my object will return one batch at a time (say, 2Kb of data).

Is there a way to write a custom iterator or enumerator so i can loop over all items and have the iterator/enumerator handle getting each batch? I'm not sure if iterators are used for this purpose, i don't have much experience with the standard C++ iterators.

Basically, i want to be able to do something like this:

Model::custom_iterator i;
for (i = model.items.begin(); i != model.items.end(); i++) {
    process item *i;
}

And in the Model class (or the iterator class), there would be some code that would get a batch of data and cache it, then when more data is required it gets the next batch. The iterator would then be able to access the items in the list, and request the next batch when it needs it.

I would also like to use a similar technique for iterating over a list that is expensive to generate (ancestors of a model with a parent/child relationship (read: tree)), so that i don't have to create the list before i iterate over it.

Posted 18-Jan-11 11:01am

XTAL256

Add a Solution

Comments

XTAL256 18-Jan-11 21:14pm

Since some of you mentioned C# and Boost, i should mention that i am using the Qt framework.

To expand on my question, if i had a Model class which gets the batches, would i create an iterator class in a similar manner to Qt's (e.g. class iterator defined inside QList) and override the ++ operator to handle paging?

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Espen Harlinn · Accepted Answer · 2011-01-18T11:18:00

You may find this helpful The Boost.Iterator Library[^] - it will help to implement an iterator correctly.

If your data is comming from a database you can wrap the "result/cursor" handle in an iterator. Like OTL, the Oracle, Odbc and DB2-CLI Template Library[^]

One approach could be to use a vector for the "batches", wrap the iterator for the vector in a new custom iterator and get next batch each time the inner itertor reaches the end of the vector.

Regards
Espen Harlinn

Manfred Rudolf Bihy · Accepted Answer · 2011-01-18T11:09:00

A simple answer to a complex question: Yes that is what iterators are here for after all. You can build an enumerator that would effectively return all of N or the fibonacci series or the faculty function. How much of that series is in fact consumed depends on the code the user runs (and the data type involved of course as you might hit a limit there).

The following site deals with an enumerator in C# but applies also to C++:http://thoughtfulcode.wordpress.com/2011/01/14/ienumerable-is-lazy-and-thats-cool/[^]

Modification:

XTAL256 wrote:
there would be some code that would get a batch of data and cache it, then when more data is required it gets the next batch

I actually missed this part in your question and want to elaborate a bit on the possibilities enumerators give you: Since you have total control over what the enumerators do internally you can also implement some kind of paging mechanism that will fetch batches of data as you put it. As long as you understand and implement all of the IEmumerable interface all of this can be transparent to the user of your code. Why did I say "can be transparent to the user"?
It means you can hide all the "behind the curtain" details from the user or else give him some control by defining some constructors for your IEnumerable implementing class that let him specify fetched batch size etc.

End Modification

Best Regards,
Manfred

Implementing custom iterator/enumerator for batch data

2 solutions

Solution 2

Solution 1

Add your solution here

Preview 0