Click here to Skip to main content
15,885,244 members
Please Sign up or sign in to vote.
5.00/5 (2 votes)
See more:
Hi,

I'm a entry level developer working with C#.NET 4.0...I have a table that's inserting approx 20 rows per second. I also wrote a C# app that queries this data and displays the results to the user.

The C# winForm keeps freezing and isn't performing. I'd like to know if this is the correct way to do the inserts on the database side considering that it inserts 20 rows per second and I'm also trying to query the results.


Server side:
try
{
 SqlConnection con = new SqlConnection(conString);
 SqlCommand    cmd = new SqlCommand();
 con.Open()
 cmd.Connection = con;
 cmd.CommandText = sqlInsertString;
 cmd.ExecuteNonQuery();
}
catch(SqlExecption ex)
{
 throw ex.Message;
}
finally
{
 con.Close();
 cmd.Dispose();
}


the query logic is ("SELECT * From tbl WHERE col1 > col2 AND col3 = last 5 mins") and attempts to display the results in a listview control.

Can someone please tell/show me the correct way to go about this?

Thank you!

-Donald
Posted
Updated 16-Jan-11 3:09am
v4
Comments
Espen Harlinn 17-Jan-11 7:55am    
Answer now includes a "working" example - inserting more than the required 20 rows per second, so I hope you'll find some use for the code :)

Well, your insert code should probably be in a thread. You can save time by putting the thread into a class, and creating the SQL connection and command just one time. The body of the thread should be concerned with repeatedly stuffing the data into the database. It would go something like this (this isn't everything, it's just an approximation):

C#
public class DBInsert
{
    SqlConnection m_conn   = null;
    SqlCommand    m_sqlCmd = null;
    Thread        m_thread = null;

    public DBInsert()
    {
        try
        {
            m_conn = new SqlConnection(conString);
            m_conn.Open();
            m_sqlCmd = new SqlCommand(m_conn);
            m_thread = new Thread(new THreadStart(this.InsertThread));
        }
        catch (Exception ex)
        {
            throw new Exception("DBInsert could not establish database connection", ex);
        }
    }

    ~DBInsert()
    {
        if (m_thread != null)
        {
            m_thread.Abort();
        }
        if (m_conn != null)
        {
            m_conn.Close();
        }
        if (m_sqlCmd != null)
        {
            m_sqlCmd.Dispose();
        }
    }

    public void InsertThread()
    {
        try
        {
            while (true)
            {
                m_sqlCmd.ExecuteNonQuery(sqlInsertString);
            }
        }
        catch (ThreadAbortException tae)
        {
        }
        catch (Exception ex)
        {
            throw new Exception("Exception encountered during DBInsert thread execution", ex);
        }
    }
}


EDIT ============================

@Donald Allen - nope, don't close them until the thread is killed. If you're adding 20 records every second, you're spending as more time opening and closing the connection as you are actually adding records. You should see at least a 300% increase in performance, in just your inserts. You didn't say how often you're retrieving the data, but I'm assuming that you wait until the user clicks a button or something.
 
Share this answer
 
v2
Comments
thatraja 16-Jan-11 9:38am    
Good answer john.
d.allen101 16-Jan-11 10:21am    
hey thanks john! i seen something along these lines...but anyway I have a question: so after I create the connection, command objects and establish a connection I "DON'T" open and close the connection after each insert that the thread is running?
Espen Harlinn 16-Jan-11 16:10pm    
5+ - I took your idea for a spin, hope it's ok ...
#realJSOP 16-Jan-11 16:17pm    
I don't mind. BTW, did it work? :)
Espen Harlinn 16-Jan-11 17:07pm    
It will probably end up as an article, but I'll make a number of improvements - something more robust using "real" synchronization classes instead of "lock(synchObj)". It works though, but I can't say I've tested it thoroughly. If I factor out the "connection pool" and convert it to a generic ( where T : IDbConnection ) it may actually become something usefull ... currently looking at System.Threading.Tasks.TaskScheduler and company. I usually do this kind of stuff in C++. On .Net stuff like this usually ends up in the mid-tier, and then I'm usually better off leveraging existing functionality in the framework :)
ListView hasn't exactly stellar performance characteristics, you will probably be better off using the DataGridView control.

Here is a nice example on populating the DataGridView:
How to populate DataGridView, GridView with SQL statement in C#[^]

Possible easy solution:
Try altering the isolation level http://msdn.microsoft.com/en-us/library/system.data.isolationlevel.aspx[^] for the transactions. See: SqlConnection.BeginTransaction[^]

It has severe impact on performance, but check the implications carefully.

If you like John's solution better - check up on the active object pattern, here is a nice explanation:
Applied Long-Running Active Object Pattern[^], I haven't tried the code though - it is a thorough and good explanation.

If you don't mind using C++/CLI you can create a high quality solution using ACE[^], and here is a nice tutorial[^] by Umar Syyid showing how to create Active Objects and more using ACE. This is what I would recommend for a production quality solution.


Here is a little component, a sort of minimalistic approach/hack at John’s idea. SqlCommand objects can be enqued using the Enqueue method.

Update
The answer now includes a little bit of test functionality - showing how SqlCommand objects can be enqueued.



The component maintains a simple pool of connections and executes the ExecuteNonQuery method on the queued commands.

Remember to use the InvokeRequired/Invoke pattern for the ExceptionEvent event.

ConnectionPool.cs:
C#
using System;
using System.ComponentModel;
using System.Collections.Generic;
using System.Data.SqlClient;
using System.Configuration;
using System.Threading;
namespace QueuedDatabaseCommands.Harlinn.Com
{
    public class ExceptionEventArgs : EventArgs
    {
        Exception exception;
        public ExceptionEventArgs(Exception exc)
        {
            exception = exc;
        }
        public Exception Exception
        {
            get
            {
                return exception;
            }
        }
    }
    public delegate void ExceptionEventHandler(object sender,
                     ExceptionEventArgs eventArgs);
    public class SqlCommandQueue : Component,INotifyPropertyChanged
    {
        private string connectionStringName = string.Empty;
        private string connectionString = string.Empty;
        private ushort maxFreeConnections = 10;
        private LinkedList<SqlConnection> freeConnections = new
                 LinkedList<SqlConnection>();
        private LinkedList<SqlConnection> busyConnections = new
                 LinkedList<SqlConnection>();
        private object synchObject = new object();
        public event PropertyChangedEventHandler PropertyChanged;
        public event ExceptionEventHandler ExceptionEvent;
        public SqlCommandQueue()
        {
            
        }
        private void DisposeConnections(LinkedList<SqlConnection>
                                    connections)
        {
            foreach (SqlConnection sqlConnection in connections)
            {
                try
                {
                    sqlConnection.Dispose();
                }
                catch (Exception exc)
                {
                    OnException(exc);
                }
            }
        }
        protected override void Dispose(bool disposing)
        {
            lock (synchObject)
            {
                if (freeConnections != null)
                {
                    LinkedList<SqlConnection> freeConnections_ =
                                   freeConnections;
                    freeConnections = null;
                    DisposeConnections(freeConnections_);
                }
                if (busyConnections != null)
                {
                    LinkedList<SqlConnection> busyConnections_ =
                                    busyConnections;
                    busyConnections = null;
                    DisposeConnections(busyConnections_);
                }
            }
            base.Dispose(disposing);
        }
        protected virtual void OnException(Exception exc)
        {
            if ((ExceptionEvent != null)&&(exc != null))
            {
                ExceptionEventArgs eventArgs = 
                             new ExceptionEventArgs(exc);
                ExceptionEvent(this, eventArgs);
            }
        }

        protected void OnPropertyChanged(string propertyName)
        {
            if (PropertyChanged != null)
            {
                PropertyChangedEventArgs eventArgs = 
                        new PropertyChangedEventArgs(propertyName);
                PropertyChanged(this, eventArgs);
            }
        }
        [DefaultValue("")]
        public string ConnectionStringName
        {
            get
            {
                string result;
                lock (synchObject)
                {
                    if (connectionStringName != null)
                    {
                        result = (string)connectionStringName.Clone();
                    }
                    else
                    {
                        result = string.Empty;
                    }
                }
                return result;
            }
            set
            {
                bool connectionStringChanged = false;
                lock (synchObject)
                {
                    if (connectionStringName == value)
                        return;
                    connectionStringName = 
                              string.IsNullOrWhiteSpace(value) ?
                               string.Empty : (string)value.Clone();
                    if (string.IsNullOrWhiteSpace(connectionString) ==
                        false)
                    {
                        connectionString = string.Empty;
                        connectionStringChanged = true;
                    }
                }
                OnPropertyChanged("ConnectionStringName");
                if (connectionStringChanged)
                {
                    OnPropertyChanged("ConnectionString");
                }
            }
        }
        [DefaultValue("")]
        public string ConnectionString
        {
            get
            {
                bool connectionStringChanged = false;
                string result;
                lock (synchObject)
                {
                    if (DesignMode == false)
                    {
                        if (string.IsNullOrWhiteSpace(connectionString)
                                 == false)
                        {
                            if (string.IsNullOrWhiteSpace
                                (connectionStringName))
                            {
                                ConnectionStringSettings
                                 connectionStringSettings = ConfigurationManager.ConnectionStrings[connectionStringName];
                                if (connectionStringSettings != null)
                                {
                                    string cs = connectionStringSettings.ConnectionString;
                                    if (cs != connectionString)
                                    {
                                        connectionString = cs;
                                        connectionStringChanged = true;
                                    }
                                }
                            }
                        }
                    }
                    if (string.IsNullOrWhiteSpace(connectionString) == false)
                    {
                        result = (string)connectionString.Clone();
                    }
                    else
                    {
                        result = string.Empty;
                    }
                }
                if (connectionStringChanged)
                {
                    OnPropertyChanged("ConnectionString");
                }
                return result;
            }
            set
            {
                lock (synchObject)
                {
                    if (connectionString == value)
                        return;
                    connectionString = value;
                }
                OnPropertyChanged("ConnectionString");
            }
        }

        public ushort MaxFreeConnections
        {
            get
            {
                lock (synchObject)
                {
                    return maxFreeConnections;
                }
            }
            set
            {
                try
                {
                    if (value == 0)
                    {
                        throw new ArgumentException(
   "0 is not allowed for MaxFreeConnections", "value");
                    }
                    lock (synchObject)
                    {
                        if (maxFreeConnections == value)
                            return;
                        maxFreeConnections = value;
                    }
                    OnPropertyChanged("MaxFreeConnections");
                }
                catch (Exception exc)
                {
                    OnException(exc);
                    throw;
                }
            }
        }
        private SqlConnection GetConnection()
        {
            try
            {
                SqlConnection result = null;
                lock (synchObject)
                {
                    if (freeConnections.Count > 0)
                    {
                        result = freeConnections.Last.Value;
                        freeConnections.RemoveLast();
                        busyConnections.AddLast(result);
                    }
                    else
                    {
                        string conStr = ConnectionString;
                        result = new SqlConnection(conStr);
                        result.Open();
                        busyConnections.AddLast(result);
                    }
                    return result;
                }
            }
            catch (Exception exc)
            {
                OnException(exc);
                throw;
            }
        }
        private void ReleaseConnection(SqlConnection sqlConnection)
        {
            try
            {
                lock (synchObject)
                {
                    busyConnections.Remove(sqlConnection);
                    if (freeConnections.Count >= maxFreeConnections)
                    {
                        sqlConnection.Dispose();
                    }
                    else
                    {
                        freeConnections.AddLast(sqlConnection);
                    }
                }
            }
            catch (Exception exc)
            {
                OnException(exc);
                throw;
            }
        }
        class QueuedExecuteInfo
        {
            WeakReference sqlCommandQueueReference; 
            SqlCommand sqlCommand;
            public QueuedExecuteInfo(SqlCommandQueue sqlCommandQueue, SqlCommand sqlCommand)
            {
                sqlCommandQueueReference = new WeakReference( sqlCommandQueue );
                this.sqlCommand = sqlCommand;
            }
            public SqlCommandQueue SqlCommandQueue
            {
              get
              {
                return sqlCommandQueueReference.Target as SqlCommandQueue;
              }
            }
            public SqlCommand SqlCommand
            {
              get
              {
                return sqlCommand;
              }
            }
        }
        private static void Execute(object param)
        {
            QueuedExecuteInfo queuedExecuteInfo = (QueuedExecuteInfo)param;
            SqlCommand sqlCommand = queuedExecuteInfo.SqlCommand;
            if (sqlCommand != null)
            {
                using (sqlCommand)
                {
                    SqlCommandQueue sqlCommandQueue = queuedExecuteInfo.SqlCommandQueue;
                    if (sqlCommandQueue != null)
                    {
                        sqlCommandQueue.ExecuteCommand(sqlCommand);
                    }
                }
            }
        }
        private void ExecuteCommand(SqlCommand sqlCommand)
        {
            try
            {
                SqlConnection sqlConnection = GetConnection();
                try
                {
                    sqlCommand.Connection = sqlConnection;
                    sqlCommand.ExecuteNonQuery();
                }
                finally
                {
                    ReleaseConnection(sqlConnection);
                }
            }
            catch (Exception exc)
            {
                OnException(exc);
            }
        }

        public bool Enqueue(SqlCommand sqlCommand)
        {
            try
            {
                QueuedExecuteInfo queuedExecuteInfo = new QueuedExecuteInfo(this, sqlCommand);
                bool result = ThreadPool.QueueUserWorkItem(new WaitCallback(Execute), queuedExecuteInfo);
                return result;
            }
            catch (Exception exc)
            {
                OnException(exc);
                throw;
            }
        }
        
    }
}


Small test:

Table:
CREATE TABLE QueuedData
(
  ID bigint identity NOT NULL PRIMARY KEY,
  CreatedTime datetime2 NOT NULL default SYSDATETIME(),
  MessageText nvarchar(2048) 
)
go


MainForm.Designer.cs:
namespace QueuedDatabaseCommands
{
    partial class MainForm
    {
        /// <summary>
        /// Required designer variable.
        /// </summary>
        private System.ComponentModel.IContainer components = null;
        /// <summary>
        /// Clean up any resources being used.
        /// </summary>
        /// <param name="disposing">true if managed resources should be disposed; otherwise, false.</param>
        protected override void Dispose(bool disposing)
        {
            if (disposing && (components != null))
            {
                components.Dispose();
            }
            base.Dispose(disposing);
        }
        #region Windows Form Designer generated code
        /// <summary>
        /// Required method for Designer support - do not modify
        /// the contents of this method with the code editor.
        /// </summary>
        private void InitializeComponent()
        {
            this.components = new System.ComponentModel.Container();
            this.enqueueTimer = new System.Windows.Forms.Timer(this.components);
            this.stopButton = new System.Windows.Forms.Button();
            this.startButton = new System.Windows.Forms.Button();
            this.sqlCommandQueue = new QueuedDatabaseCommands.Harlinn.Com.SqlCommandQueue();
            this.enqueueButton = new System.Windows.Forms.Button();
            this.SuspendLayout();
            // 
            // enqueueTimer
            // 
            this.enqueueTimer.Interval = 50;
            this.enqueueTimer.Tick += new System.EventHandler(this.enqueueTimer_Tick);
            // 
            // stopButton
            // 
            this.stopButton.Anchor = ((System.Windows.Forms.AnchorStyles)((System.Windows.Forms.AnchorStyles.Bottom | System.Windows.Forms.AnchorStyles.Right)));
            this.stopButton.Enabled = false;
            this.stopButton.Location = new System.Drawing.Point(720, 408);
            this.stopButton.Name = "stopButton";
            this.stopButton.Size = new System.Drawing.Size(75, 23);
            this.stopButton.TabIndex = 0;
            this.stopButton.Text = "Stop";
            this.stopButton.UseVisualStyleBackColor = true;
            this.stopButton.Click += new System.EventHandler(this.stopButton_Click);
            // 
            // startButton
            // 
            this.startButton.Anchor = ((System.Windows.Forms.AnchorStyles)((System.Windows.Forms.AnchorStyles.Bottom | System.Windows.Forms.AnchorStyles.Right)));
            this.startButton.Location = new System.Drawing.Point(639, 408);
            this.startButton.Name = "startButton";
            this.startButton.Size = new System.Drawing.Size(75, 23);
            this.startButton.TabIndex = 1;
            this.startButton.Text = "Start";
            this.startButton.UseVisualStyleBackColor = true;
            this.startButton.Click += new System.EventHandler(this.startButton_Click);
            // 
            // sqlCommandQueue
            // 
            this.sqlCommandQueue.ConnectionStringName = "QueuedDatabaseName";
            this.sqlCommandQueue.MaxFreeConnections = ((ushort)(10));
            // 
            // enqueueButton
            // 
            this.enqueueButton.Location = new System.Drawing.Point(558, 408);
            this.enqueueButton.Name = "enqueueButton";
            this.enqueueButton.Size = new System.Drawing.Size(75, 23);
            this.enqueueButton.TabIndex = 2;
            this.enqueueButton.Text = "Manual";
            this.enqueueButton.UseVisualStyleBackColor = true;
            this.enqueueButton.Click += new System.EventHandler(this.enqueueButton_Click);
            // 
            // MainForm
            // 
            this.AutoScaleDimensions = new System.Drawing.SizeF(6F, 13F);
            this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;
            this.ClientSize = new System.Drawing.Size(807, 434);
            this.Controls.Add(this.enqueueButton);
            this.Controls.Add(this.startButton);
            this.Controls.Add(this.stopButton);
            this.Name = "MainForm";
            this.Text = "Queued Database Commands by Espen Harlinn";
            this.ResumeLayout(false);
        }
        #endregion
        private Harlinn.Com.SqlCommandQueue sqlCommandQueue;
        private System.Windows.Forms.Timer enqueueTimer;
        private System.Windows.Forms.Button stopButton;
        private System.Windows.Forms.Button startButton;
        private System.Windows.Forms.Button enqueueButton;
    }
}


MainForm.cs:
C#
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Data.SqlClient;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;

namespace QueuedDatabaseCommands
{
    public partial class MainForm : Form
    {
        public MainForm()
        {
            InitializeComponent();
        }

        private void Enqueue()
        {
            SqlCommand sqlCommand = new SqlCommand();
            DateTime dt = DateTime.Now;
            string sql = 
              "INSERT INTO QueuedData(MessageText) VALUES(@msg)";
            string message = "Message created:" + dt.ToString("o");
            sqlCommand.CommandText = sql;
            sqlCommand.Parameters.Add("@msg", 
                  SqlDbType.NVarChar).Value = message;
            sqlCommandQueue.Enqueue(sqlCommand);
        }

        private void enqueueTimer_Tick(object sender, EventArgs e)
        {
            Enqueue();
            Enqueue();
            Enqueue();
        }

        private void startButton_Click(object sender, EventArgs e)
        {
            startButton.Enabled = false;
            stopButton.Enabled = true;
            enqueueTimer.Start();
        }

        private void stopButton_Click(object sender, EventArgs e)
        {
            startButton.Enabled = true;
            stopButton.Enabled = false;
            enqueueTimer.Stop();
        }

        protected override void OnFormClosing(FormClosingEventArgs e)
        {
            if (enqueueTimer.Enabled)
            {
                enqueueTimer.Stop();
            }
            base.OnFormClosing(e);
        }

        private void enqueueButton_Click(object sender, EventArgs e)
        {
            enqueueTimer_Tick(sender, e);
        }

    }
}


App.config:
XML
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <connectionStrings>
    <add name="QueuedDatabaseName" connectionString="Data Source=(local);Initial Catalog=QueuedDatabase;Persist Security Info=False;Integrated Security=SSPI;" providerName="System.Data.SqlClient" />
  </connectionStrings>
</configuration>


Program.cs:
C#
using System;
using System.Collections.Generic;
using System.Linq;
using System.Windows.Forms;

namespace QueuedDatabaseCommands
{
    static class Program
    {
        /// <summary>
        /// The main entry point for the application.
        /// </summary>
        [STAThread]
        static void Main()
        {
            Application.EnableVisualStyles();
            Application.SetCompatibleTextRenderingDefault(false);
            Application.Run(new MainForm());
        }
    }
}




Regards
Espen Harlinn
 
Share this answer
 
v8
Comments
JF2015 17-Jan-11 7:35am    
Very detailed answer. 5+
Espen Harlinn 17-Jan-11 7:49am    
Doing more than 20 inserts per second in a somewhat flexible manner required some details :)
d.allen101 20-Jan-11 13:22pm    
THANKS Espen Harlinn!!!! This is exactly what I was looking for! I didn't take the time to read it when you initially posted it because it was so much code but that was a bad decision on my part! But anyway, I just wanted to thank you!

-Donald
Espen Harlinn 20-Jan-11 13:42pm    
Nice that you found it useful, happy coding :)
fjdiewornncalwe 20-Jan-11 20:03pm    
Wow! This may be the most detailed answer of the year thus far. +5

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900