|
Hi,
I'm developing a database application using CDatabase & CRecordSet classes. My SQL Server database
is sitting on other machine and it's effecting application performance when I tried to insert large number of records
and retrieving the records for generating some reports(i.e. to fetch 1000000 records at a time for creating a report). Application is getting too slow.
Need some suggestions in this regard.
Thanks,
|
|
|
|
|
Hi,
probably instead of using mysql_store_result you should using mysql_use_result. It will give you a boost, but also a lot of drawbacks. See the C-API documentation for that.
GSte
|
|
|
|
|
Hi,
I recommend to segment this query or install more physics memory.
navisoft
www.latindevelopers.com
|
|
|
|
|
How are you using CRecordset? Are you predefining the query and the columns it returns with CRecordset, or are you simply creating a CRecordset from a query at run-time? If you are doing the former (predefining the query and the columns), there are some operations in which CRecordset performs very inefficient methods on the data, and it could be that the slowness you are seeing is because CRecordset is clogging the system memory.
Check which way you are using CRecordset and also pull up Task Manager while your long query is running to see how much system memory it is sucking up.
Sincerely,
Alexander Wiseman
Est melior esse quam videri
It is better to be than to seem
|
|
|
|
|
Hi,
I'm using the former one. I have to read a text file which has more than million records and insert each record at a time. Also, for genearting some reports, i've to fetch huge number of records at a time from the database and process all the records.
Thanx,
|
|
|
|
|
So, just to clarify exactly what you are trying to do: you are opening and reading a text file from which data you want to insert records into a database. Because you want to insert so many records, I would suggest using a SQL command to insert the data (INSERT INTO...) instead of CRecordset::AddNew() and CRecordsSet::Update(). The CRecordset functions for adding records tend to be slower than simply executing a SQL statement against the database. You can execute a SQL statement by using the function in CDatabase (I forget the exact name, but it is something like "Execute").
For fetching large numbers of records, I would also suggest that you move away from the default RFX (Recordset Field Exchange - where you predefine the columns returned from the query) implemented with CRecordset. Again, this tends to be slower than simple qeurying the database with SQL statement and not using RFX. For some more information on this topic, take a look at the ODBC section on CodeProject: http://www.codeproject.com/database/#ODBC[^].
If you need more specific information, or some example code, please let me know.
Sincerely,
Alexander Wiseman
Est melior esse quam videri
It is better to be than to seem
|
|
|
|
|
Hi Alexander,
Thanks for your input. Right now, I'm not using RFX for fetching the records nor
AddNew() and Update() functions of CRecordset classes. As you said, i'm reading each record from the text file and create INSERT INTO.. query and execute it using CDatabase
ExecuteSQL() function. However, for every call to this function, database has to parse , prepare an execution plan and compile it. Instead i'm thinking of using stored procedures. But my application has to support multipe databases like MSAccess, MySQl, Oracle & MS SQL Server. I don't think MSAccess supports Stored procedures and I'm not sure of MySQL either.
Do you have any generic database module? If so, can you send it to me?
I would appreciate your help.
Thanx,
|
|
|
|
|
Hi Sridhar,
Unfortunately, there won't be something which you can code which will take the place of a stored procedure for those DBs that don't support them. MySQL should have something you can use for optimization, although I thought it supported stored procedures. Maybe your best bet is to implement the insertion code inside a worker thread, so the main application UI doesn't seem like it isn't responding when you are doing the insert. Long operations, when necessary, are OK as long as you give the user some indication of what is actually happening and maybe a "Stop" button.
Although I can't think of anything else right now, I will keep my eyes open for something that can help you.
Sincerely,
Alexander Wiseman
Est melior esse quam videri
It is better to be than to seem
|
|
|
|
|
Hi Alexander,
I thought of implementing intensive database operations in the worker thread.
Think this strategy will work for inserting records. But, what about fetching if you implement it in the worker thread?
I mean, if my query is taking X number of sec or minutes to fetch Million plus records(Select * from table where condition). HOw do I cancel the worker thread if it's still in the process of retrieving?
Some insight would be greatly appreciated.
Thanks,
|
|
|
|
|
Hi Alexander,
I thought of implementing intensive database operations in the worker thread.
Think this strategy will work for inserting records. But, what about fetching if you implement it in the worker thread?
I mean, if my query is taking X number of sec or minutes to fetch Million plus records(Select * from table where condition). HOw do I cancel the worker thread if it's still in the process of retrieving?
Some insight would be greatly appreciated.
Thanks,
Sridhar
|
|
|
|
|
|
Hello everyone,
I am searching the way to find for in archives and folders recursively.
You can say me the optimal way to do it?
Thanks!
navisoft
www.latindevelopers.com
|
|
|
|
|
Take a look at this article: http://www.codeproject.com/file/fileglob.asp[^], it implements exactly what you want and more.
Hope that helps!
Sincerely,
Alexander Wiseman
Est melior esse quam videri
It is better to be than to seem
|
|
|
|
|
you can do a function like this
int findfiles()
{
...
findfirstfile(*.*)
if file=a directory
findfile(directory)
}
gabby
|
|
|
|
|
Hai,
I'm working on win32 vc++.I want to get the user logon ID from the system and display it in my dialog(probably in a textbox).I dont know how to do that.
Can anyone help pls.
Thanks a lot in advance.
|
|
|
|
|
Hi,
You can obtain the current username writing follow code:
DWORD userNameCharCount = 255;
TCHAR userName[255];
if( !GetUserName( userName, &userNameCharCount) ){
// error!
}
CString str = userName;
GetDlgItem(IDC_YOUREDIT_CTRL)->SetWindowText(str);
navisoft
www.latindevelopers.com
|
|
|
|
|
I am trying to create an OpenGL MDI application using Doc/View architecture.
When the ID_FILE_NEW is handled, a child window with the correct openGL rendered scene appears. however when the ID_FILE_OPEN is handled by CWinApp:: OnFileOpen in the application, a 'VERIFY' in the following bit of code called causes an error. i am unable to determine why it occurs.
BOOL CDocManager::DoPromptFileName(CString& fileName, UINT nIDSTitle, DWORD lFlags, BOOL bOpenFileDialog, CDocTemplate* pTemplate)
{
CFileDialog dlgFile(bOpenFileDialog);
CString title;
VERIFY(title.LoadString(nIDSTitle));
any help would be much appreciated.
thanks.
|
|
|
|
|
LoadString() has failed to load the string resource with the ID contained in nIDSTitle. Chances are you do not have that string in your string table.
"You're obviously a superstar." - Christian Graus about me - 12 Feb '03
"Obviously ??? You're definitely a superstar!!!" mYkel - 21 Jun '04
Within you lies the power for good - Use it!
|
|
|
|
|
Run your code in debug mode, and step through all function calls. Stepping allows you to see the parameters passed to the function, and simultaneously allows you to see the string table entries. If you encounter a situation where an invalid ID is passed to the function, try tracing back to see why it gets passed.
-Antti Keskinen
----------------------------------------------
The definition of impossible is strictly dependant
on what we think is possible.
|
|
|
|
|
Thanks.
i hadn't included "afxres.rc" in the resource file - that was causing the problem. i had included only "afxres.h" earlier.
the trouble i am having now is that the child windows don't have a title. the resource string is fine - i used it in another program, it works ok.
any comments would be most appreciated.
thanks.
|
|
|
|
|
If a file has been mapped with MapViewOfFile, will the offsets in memory correspond to the ones in the disk. For example if I try to access the contents at file offset 0x100 in the disk, after I map the file would I still be using offset 0x100 or will the memory alignment make the offset change.
// Afterall, I realized that even my comment lines have bugs
If the sun were to blow up, it would take us 7-8 minutes to realize it.
|
|
|
|
|
Toni78 wrote:
will the offsets in memory correspond to the ones in the disk
Yes, they will. However, ensure that either the whole file has been mapped, or the relevant portion that you want to access, has been mapped.
Note that the "offset" must be added to the address returned by MapViewOfFile(). Optionally, MapViewOfFileEx() allows you to specify your own base address where the file will be "loaded".
Memory alignment has to be taken into consideration when
the file was opened with FILE_FLAG_NO_BUFFERING in the CreateFile() call.
Bikram Singh
|
|
|
|
|
Thank you for your help.
After I posted the question and I got no reply, I realized that I could test this by myself by loading the file and doing a byte compare with the file in the disk . At the end the files were identical so apparently there were no memory alignment issues. But I wasn't aware about FILE_FLAG_NO_BUFFERING . Either way, thank you again.
// Afterall, I realized that even my comment lines have bugs
If the sun were to blow up, it would take us 7-8 minutes to realize it.
|
|
|
|
|
Hi All, was hoping I mightbe able to get some help. I have been looking at a few of the clipboard examples on this site and made this code based on some of it:
if(capEditCopy(captureWindow)) //Capture frame from video stream to clipboard
{
if ( OpenClipboard(NULL) )
{
if (!IsClipboardFormatAvailable(CF_BITMAP))
return;
//Get the clipboard data
HBITMAP handle = (HBITMAP)GetClipboardData(CF_BITMAP);
CBitmap * bm = CBitmap::FromHandle(handle);
CSize sz = bm->GetBitmapDimension();
long foo = sz.cx;
CloseClipboard();
}
The long foo was merely created to check whether the bitmap was defined properly, the problem is, foo always comes back as zero. Why is this? Was the bitmap ever created? My aim here is to merely get a bitmap so that I can manipulate some of the pixel values.
Any help would be greatly appreciated. Thanks in advance.
Micheal
|
|
|
|
|
RTFM[^]
[quote]
CBitmap::GetBitmapDimension
CSize GetBitmapDimension( ) const;
Return Value
The width and height of the bitmap, measured in 0.1-millimeter units. The height is in the cy member of the CSize object, and the width is in the cx member. If the bitmap width and height have not been set by using SetBitmapDimension, the return value is 0.
Remarks
Returns the width and height of the bitmap. The height and width are assumed to have been set previously by using the SetBitmapDimension member function.
CBitmap Overview | Class Members | Hierarchy Chart
See Also CBitmap::SetBitmapDimension
[/quote]
Use a BITMAP structure filled with CBitmap::GetBitmap(BITMAP *) to get the bitmaps dimensions.
"You're obviously a superstar." - Christian Graus about me - 12 Feb '03
"Obviously ??? You're definitely a superstar!!!" mYkel - 21 Jun '04
Within you lies the power for good - Use it!
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
