OleDbConnection multiple inserts (to Access DB) - memory issue

Question

0.00/5 (No votes)

See more:

Running the following method - passing in a List<OleDbCommand> about 65K in list

public static int bulkInsertToDataBase(string fileNamePath, bool JetOLEDB,
    List<OleDbCommand> insertStatements, int commitEveryNo) {

    int messagesSent;
    using (OleDbConnection conn = new OleDbConnection(
        JetOLEDB == true ?
        ConnBuilders.getJetConnStrOLEDB(fileNamePath, false, false) :
        ConnBuilders.getAceConnStrOLEDB(fileNamePath, false, false))) {
        conn.Open();
        OleDbTransaction tran = conn.BeginTransaction();
        for (messagesSent = 0; messagesSent < insertStatements.Count; messagesSent++) {
            insertStatements[messagesSent].Connection = conn;
            insertStatements[messagesSent].Transaction = tran;
            insertStatements[messagesSent].ExecuteNonQuery();
            if ((messagesSent % commitEveryNo == 0) && (messagesSent != 0)) {
                tran.Commit();
                tran.Dispose();
                tran = null;
                tran = conn.BeginTransaction();
            }
        }
        tran.Commit();
        tran.Dispose();
        conn.Close();
        conn.Dispose();
    }
    return messagesSent;
}

While running memory usage gets to about 1.2GB and after execution sits at about 400MB?

commitEveryNo = 1000
Each OleDbCommand contains 16 parameters (4 optional) ie

cmd.Parameters.AddWithValue("[End Date]", obj.endDate ?? (object)DBNull.Value);

*** Can anyone explain why the memory usage is so high? ***

Additional code

XML

public static string getJetConnStrOLEDB(string fileName,
    bool excel = false, bool csv = false) {

    Dictionary<string, string> props = new Dictionary<string, string>();
    props["Provider"] = "Microsoft.Jet.OLEDB.4.0";
    props["Data Source"] = fileName;

    if (excel) props["Extended Properties"] = "\"Excel 8.0;IMEX=1\"";
    if (csv) props["Extended Properties"] = "\"text;HDR=No;FMT=Delimited()\"";

    StringBuilder sb = new StringBuilder();
    foreach (KeyValuePair<string, string> prop in props) {
        sb.Append(prop.Key);
        sb.Append('=');
        sb.Append(prop.Value);
        sb.Append(';');
    }
    return sb.ToString();
}

Posted 20-Mar-13 19:53pm

strOngHand

Updated 21-Mar-13 14:05pm

v3

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Prasad Khandekar · Accepted Answer · 2013-03-22T00:52:00

Hello,

There is no need to have a list of OleDBCommand. IDbCommand infact is used when your SQL is static and only data values are changing e.g. Insert 1000 emplouees. This way database does not have to recompile the SQL each time and there by getting improved performance in such operations.

Basically to your function you will pass a List<t> of object whose property values are to be inserted in the database. Then inside function create a oleDBCommand object and add parameters (in your case 16). Write a for loop around List<t> and inside loop assign value to each parameter and execute the query. Your code should look something like shown below.

C#

for (Employee obj in lstData) {
    cmd.parameters[0].value = obj.endDate ?? (object)DBNull.Value;
    cmd.parameters[1].value = obj.startDate ?? (object)DBNull.Value;
    ...
    cmd.parameters[15].value = obj.desc ?? (object) DBNull.Value;

    cmd.ExecuteNonQuery();
    if ((messagesSent % commitEveryNo == 0) && (messagesSent != 0)) {
        tran.Commit();
        tran.Dispose();
        tran = null;
        tran = conn.BeginTransaction();
    }
    messagesSent++;
}

let me know if it helpes.
Regards,