Click here to Skip to main content
15,886,873 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
I was trying to write one code that needs to fetch the old values stored in some proprietary data base.
To fetch the data, some API and associated functions are provided.The code is as follows.

Now if I run this code, it takes almost 355 seconds. I have measured the time of execution for one point and it took almost 10 seconds(in an avg) to do that.
The test was done with approximately 37 such 'pnt' and it took almost 355 seconds to fetch the required values.The I need to export the same to excel
also.But exporting to excel is not time consuming compared to fetching of data from a time series database through predefined APIs.
I do not have any control in the APIs and I can not do anything on the same.
Considering all this, can I implement parallel processing.I have machine with 24 cores CPU where the program needs to be run.
If somebody can tell me how to implement the same in the code it will be helpful for me.
I was reading some article in this regard, but it seems difficult for me to implement in the code.
If some one cane help me out implementing parallel processing in this code, it will be a great help in reducing time taken by the program.

What I have tried:

int i = 0, j = 0;
                string pnt = "";
                TimeSpan span;
                if (periodtime.SelectedIndex == 0)
                {
                    span = TimeSpan.FromSeconds(Convert.ToInt16(periodval.Text));
                }
                else if (periodtime.SelectedIndex == 1)
                {
                    span = TimeSpan.FromMinutes(Convert.ToInt16(periodval.Text));
                }
                else
                {
                    span = TimeSpan.FromHours(Convert.ToInt16(periodval.Text));
                }	
          	uint s = 0;
                double dval = 0;
                DateTime ts = DateTime.Now;
                string st = "";
                int nRet = 0;
foreach (string slpnt in pntid)
                {

//==========================================================================================
/*The point consist of a collection of strings separated by .,$ etc.It is defined inside a text file.The code reads the whole line and split a particular portion (required one) that we need to pass to API's function.*/
		  //That portion is stored in 'pnt' variable. 
                   pnt = slpnt.Split('-')[0].Split('$')[0];
                    i = 0;
                   span = TimeSpan.FromHours(24);
                   //GetHistMin is a function(Part of an application API) to fetch historical minimum values stored historixcal information database.Hence if .nRet =0 , it enters the loop.
//Here the time range is from Convert.ToDateTime(Fdt.Text + " 00:00:00" =00:00:00 hrs to 24:00:00 hrs (Convert.ToDateTime(Fdt.Text + " 00:00:00").AddHours(24)
/*Span is user given  and here it is 30 seconds.Please note that it is a large data base and for 24 hrs we approximately have to search 4000 samples for a pnt variable and get the minimum out of it.We need to fetch such 50 pnt in minimum to get the desired result.*/
nRet = History.GetHistMin(pnt, Convert.ToDateTime(Fdt.Text + " 00:00:00"), Convert.ToDateTime(Fdt.Text + " 00:00:00").AddHours(24), span, out s);
                   if (nRet == 0)
                   {
                       nRet = History.GetNextHist(s, out dval, out  ts, out st);
					   //GetNextHist is used to collect the resulting value the returned by GetHistMin(). 
                       if (j == 0)
                       {
//Doing the required calculation and adding it to new row of data table.
                           dr = dt.NewRow();
                           dr[0] = "Minimum(of the day)";
                           if (slpnt.Contains("$"))
                           {
                           dr["C" + j.ToString()] = Math.Round(dval, 2);

                           }
                           else
                           {
                           dr["C" + j.ToString()] = Math.Round(dval, 1);
                           }
                           dt.Rows.Add(dr);
                         }
                       else
                 {
                           if (slpnt.Contains("$"))
                 {
                dt.Rows[dt.Rows.Count - 1]["C" + j.ToString()] = Math.Round(dval, 2);
                 }
                           else
                 {
                dt.Rows[dt.Rows.Count - 1]["C" + j.ToString()] = Math.Round(dval, 1);
                 }
             }
         } 


//==========================================================
//Like Minimum, it is calculating the Maximum of the day value.
                  nRet = History.GetHistMax(pnt, Convert.ToDateTime(Fdt.Text + " 00:00:00"), Convert.ToDateTime(Fdt.Text + " 00:00:00").AddHours(24), span, out s);
                   if (nRet == 0)
                   {

                       nRet = History.GetNextHist(s, out dval, out  ts, out st);
                       if (j == 0)
                       {
                           dr = dt.NewRow();
                           dr[0] = "Maximum(of the day)";
                           if (slpnt.Contains("$"))
                           {
                               dr["C" + j.ToString()] = Math.Round(dval, 2);
                           }
                           else
                           {
                               dr["C" + j.ToString()] = Math.Round(dval, 1);
                           }
                           dt.Rows.Add(dr);
                           }
                       else
                       {
                           if (slpnt.Contains("$"))
                           {
         dt.Rows[dt.Rows.Count - 1]["C" + j.ToString()] = Math.Round(dval, 2);
                           }
                           else
                           {
         dt.Rows[dt.Rows.Count - 1]["C" + j.ToString()] = Math.Round(dval, 1);
                           }
                        }			   
                    }  

//=========================================================================================================================
//Like Minimum, it is calculating the Average of the day value.
                    nRet = History.GetHistAvg(pnt, Convert.ToDateTime(Fdt.Text + " 00:00:00"), Convert.ToDateTime(Fdt.Text + " 00:00:00").AddHours(24), span, out s);
                    
                    if (nRet == 0)
                    {
                        nRet = History.GetNextHist(s, out dval, out  ts, out st);
                        if (j == 0)
                        {
                            dr = dt.NewRow();
                            dr[0] = "Average(of the day)";
                            if (slpnt.Contains("$"))
                            {
                                dr["C" + j.ToString()] = Math.Round(dval, 2);
                            }
                            else
                            {
                                dr["C" + j.ToString()] = Math.Round(dval, 1);
                            }
                            dt.Rows.Add(dr);
                        }
                        else
                        {
                            if (slpnt.Contains("$"))
                            {
              dt.Rows[dt.Rows.Count - 1]["C" + j.ToString()] = Math.Round(dval, 2);
                            }
                            else
                            {
              dt.Rows[dt.Rows.Count - 1]["C" + j.ToString()] = Math.Round(dval, 1);
                            }
                        }
                    }
	j++;
}
Posted
Updated 13-Nov-17 22:26pm
v5
Comments
Bernhard Hiller 15-Nov-17 4:28am    
Are you sure that it is correct to call History.GetNextHist regardless if you call GetHistMin, GetHistMax or GetHistAvg just before?

It could be useful to see the History class as well. There are some optimizations that can be done on already shown code:

- Second block: span variable is affected a standard value which overwrites the one it has been affected in the first block (before the for each loop). Is it on purpose?

- Second block: I would suggest to use DateTime.TryParse instead of Convert.ToDateTime (Convert class generally has terrible performance). And cache the result before the GetHistMin call so that it is not computed twice at each iteration. Something like:
C#
DateTime fdtTime;
if (DateTime.TryParse(Fdt.Text, out fdtTime)) {
   nRet = History.GetHistMin(pnt, fdtTime, fdtTime.AddHours(24), span, out s);
  // ...
}

As a general matter, when you see some code which repeats the same computation several times in the same logical block, then it may be useful to cache the result in a variable.
TryParse method has several overloads which allows you to have a fine control over the process (culture-related, mainly).
No need to append the 00:00:00 time, it saves you from a lot of string copy operations.

- Third and fourth blocks: same remark for DateTime variable handling in GetHistMax and GetHistAvg methods.

There may be possible optimizations elsewhere, you should try and see where the culprit is when you execute your code (i.e., where the process is spending most of its time); it will give you an idea of what has really to be optimized.
 
Share this answer
 
Comments
Member 12708425 14-Nov-17 2:55am    
I have done the modification and used Tryparse but it is taking the same time.Also I do not have the code for GetHistMIn/Max/Avg methods
phil.o 14-Nov-17 4:05am    
Then you need to find where the processing is taking much of its time:
Beginners Guide to Performance Profiling
Member 12708425 14-Nov-17 4:31am    
I found the methods GetHistMin,GetHistAvg and GetHistMax are taking time.Average time to process to get 24 hrs is almost 3 seconds for each of this method.e.g. GetHistMin check the samples through out 24 hrs( almost 4000 values for each point through out 24 hrs.)
and get the minimum double value out of 4000 samples. This is for one point and for Minimum Method.Similarly for Max and Avg method and it is for such 37 points. That is why I am thinking of Parallel Programming.Can this be implemented here?
I doubt that you can speed up the procedure with parallelization: it is the slow processing on the other side which consumes the time. I think that parallel requests will make individual requests slower because that other side is at its limits anyway.
 
Share this answer
 
Comments
Member 12708425 14-Nov-17 4:33am    
Can you please help me out to test the code implementing parallel processing.Let me try once.I do not have any other option with me than this.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900