Click here to Skip to main content
15,881,746 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more: , +
We are calling a Rest API URL via the HttpWebRequest / HttpWebResponse method using IDs that are already stored in a database table.
We am currently using a for each loop to iterate through each of the IDs fetched in a DataTable and then making a call to the Rest Api URL and finally dumping the results into multiple tables.
The code works fine however, the number of IDs is likely to increase in the future therefore is there any way we could use multithreading / parallel processing to make the code more efficient?

Any help would be much appreciated as we are newbies in multi threading / parallel processing. The pseudo code is as pasted below.

What I have tried:

foreach (DataRow drow in ds.Tables[0].Rows)
                {
                    string uri2 = "http://myrestURL/Transaction/" + drow.ItemArray[0].ToString();
                    HttpWebRequest req2 = HttpWebRequest.CreateHttp(uri2);



                    req2.CookieContainer = new CookieContainer();
                    req2.Method = "GET";
                    req2.UseDefaultCredentials = true;
                    req2.ContentLength = 0;
                    req2.Accept = "application/xml,*/*";
                    req2.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;


                    

                    using (HttpWebResponse resp2 = (HttpWebResponse)req2.GetResponse())
                    {
                        StringReader theReader;
                        DataSet theDataSet;
                        using (StreamReader reader2 = new StreamReader(resp2.GetResponseStream()))
                        {
                            string message2 = reader2.ReadToEnd();
                            theReader = new StringReader(message2.ToString());
                            theDataSet = new DataSet();
                            theDataSet.ReadXml(theReader);

                            DataTable dt = theDataSet.Tables["table1"];
                            DataTable dt1 = theDataSet.Tables["table2"];
                            DataTable dt2 = theDataSet.Tables["table3"];
                            DataTable dt3 = theDataSet.Tables["table4"];

                            dt.Columns.Add("ID", typeof(string));
                            dt.Columns["ID"].SetOrdinal(0);

                            foreach (DataRow row in dt.Rows)
                            {
                                row["ID"] = drow.ItemArray[0].ToString();
                            }

                            dt1.Columns.Add("ID", typeof(string));
                            dt1.Columns["ID"].SetOrdinal(0);

                            foreach (DataRow row in dt1.Rows)
                            {
                                row["ID"] = drow.ItemArray[0].ToString();
                            }

                            dt2.Columns.Add("ID", typeof(string));
                            dt2.Columns["ID"].SetOrdinal(0);

                            foreach (DataRow row in dt2.Rows)
                            {
                                row["ID"] = drow.ItemArray[0].ToString();
                            }

                            dt3.Columns.Add("ID", typeof(string));
                            dt3.Columns["ID"].SetOrdinal(0);

                            foreach (DataRow row in dt3.Rows)
                            {
                                row["ID"] = drow.ItemArray[0].ToString();
                            }

                            SqlConnection insertConn = new SqlConnection(strConn);
                                insertConn.Open();

                            
                            using (SqlBulkCopy bulkCopy1 = new SqlBulkCopy(insertConn))
                            {
                                
                                bulkCopy1.DestinationTableName = dt1.TableName;
                                try
                                {
                                    bulkCopy1.WriteToServer(dt1);
                                }
                                catch (Exception ex)
                                {
                                    Console.WriteLine(ex.Message);
                                }
                            }
Posted
Comments
PIEBALDconsult 11-Jul-21 0:02am    
All the web services I have encountered are single-threaded, so throwing tons of requests at them via multiple threads does not directly lead to a decrease in time required for those requests to complete. The requests get queued up and processed one at a time.
At worst, the requests at the back of the queue time out.
At best, your requests will be ahead of the requests of others.
However, I find it better to "play nice" and not put a large number of requests in the queue. The Golden Rule applies here -- would you want to find that your requests are waiting because another application had filled the queue with requests?
aakar 16-Jul-21 5:10am    
Hi, my code which was working fine has run into issues. worse part is that it runs most of the times but at times I am getting a 401 "Unauthorized" error at the response step. Can you pls help?
Greg Utas 16-Jul-21 8:38am    
Single-threaded? I'm curious. What happens when the thread blocks on disk I/O, for example? Or is there a thread pool for that, which you're excluding from the main thread?
aakar 16-Jul-21 10:43am    
No, presently the code is single threaded, any leads would be much appreciated!
Greg Utas 16-Jul-21 11:01am    
Sorry, can't help you. I write C++ and am not familiar with what you're using.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900