Click here to Skip to main content
15,894,896 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
The issue I am having is in the parallel.for I have a function to show another form at the end once the iterations complete. I noticed that this function keeps getting called every iteration. Is there something that would allow for all tasks to complete then do this function at the end?

 Public Sub Match_Merge(I As Integer)



        Dim lock As New Object

        '  temptable.Reset()

        'if you are comparing 2 tables then both tables need to be sorted
        'order by last name, then first name






        'this will be table 1



        countI = countI + 1
        'this will be table 2

        SyncLock dt.Rows
            For I2 = I3 To dt.Rows.Count - 1

                counti2 = counti2 + 1

                Call match_specifications(I, I2, I3, dt, count, Duplicatestart, nxrecord, matchedmiddlename, newindividual.Middle_name)

                If nxrecord = True Then
                    nxrecord = False
                    GoTo nextrecord
                ElseIf nxI = True Then
                    nxI = False
                    GoTo nxI

                Else

                End If

nxI2:

            Next I2

nextrecord:


            'This will be to collect the duplicate information in second table
            lastduplicaterow = count - 1

            If dt.Rows.Count = 0 Then
                GoTo nxI
            End If

            If newindividual.Individual_ID = Nothing Then
                newindividual.Individual_ID = dt.Rows(Duplicatestart)("Individual_ID")
            End If

            If newindividual.Death_ID = Nothing And dt.Columns.Contains("Death_ID") Then
                If Not IsDBNull(dt.Rows(Duplicatestart)("Death_ID")) Then
                    newindividual.Death_ID = dt.Rows(Duplicatestart)("Death_ID")
                End If
            End If

            If newindividual.Cemetery_ID = Nothing And dt.Columns.Contains("Cemetery_ID") Then
                If Not IsDBNull(dt.Rows(Duplicatestart)("Cemetery_ID")) Then
                    newindividual.Cemetery_ID = dt.Rows(Duplicatestart)("Cemetery_ID")
                End If
            End If

            If newindividual.Burial_ID = Nothing And dt.Columns.Contains("Burial_ID") Then
                If Not IsDBNull(dt.Rows(Duplicatestart)("Burial_ID")) Then
                    newindividual.Burial_ID = dt.Rows(Duplicatestart)("Burial_ID")
                End If
            End If

            'sources and references
            If newindividual.Source_ID = Nothing And dt.Columns.Contains("Source_ID") Then
                If Not IsDBNull(dt.Rows(Duplicatestart)("Source_ID")) Then
                    newindividual.Source_ID = dt.Rows(Duplicatestart)("Source_ID")
                End If
            End If

            If newindividual.Reference_ID = Nothing And dt.Columns.Contains("Reference_ID") Then
                If Not IsDBNull(dt.Rows(Duplicatestart)("Reference_ID")) Then
                    newindividual.Reference_ID = dt.Rows(Duplicatestart)("Reference_ID")
                End If
            End If

            If newindividual.SSN_ID = Nothing And dt.Columns.Contains("SSN_ID") Then
                If Not IsDBNull(dt.Rows(Duplicatestart)("SSN_ID")) Then
                    newindividual.SSN_ID = dt.Rows(Duplicatestart)("SSN_ID")
                End If
            End If

            ' For duprows = Duplicatestart To lastduplicaterow


            For firsttable = 0 To lastduplicaterow

                'collect variables from table 
                Call collect_all_varaibles_from_table(I, I2, firsttable, deathcount, burialcount, cemeterycount, Referencecount, SSNcount, dt, dtcolumns)

                If firsttable = lastduplicaterow Then
                    I = I + lastduplicaterow

                    If updatecount > 0 Then

                        If Not newindividual.Individual_ID = 0 Then

                            If Not newindividual.Individual_ID = Nothing Then
                                UpdateModule.profile(newindividual.Individual_ID, newindividual.First_name, newindividual.Middle_name, newindividual.Last_name, newindividual.Sex, newindividual.Race, newindividual.Place_of_birth, newindividual.State_of_birth, newindividual.County_of_birth, newindividual.Date_of_birth)
                                updatevalue = "yes"
                            End If

                            If Not newindividual.Death_ID = Nothing And deathcount >= 1 Then
                                UpdateModule.death(newindividual.Death_ID, newindividual.Date_of_death, newindividual.Cause_of_death, newindividual.Place_of_death, newindividual.State_of_death, newindividual.County_of_death, newindividual.Age, newindividual.Age_units)
                                updatevalue = "yes"
                            ElseIf newindividual.Death_ID = Nothing And deathcount >= 1 Then
                                Insertmodule.death(newindividual.Individual_ID, newindividual.Date_of_death, newindividual.Cause_of_death, newindividual.Place_of_death, newindividual.State_of_death, newindividual.County_of_death, newindividual.Age, newindividual.Age_units)
                                insertvalue = "Yes"
                            End If

                            If Not newindividual.Cemetery_ID = Nothing And cemeterycount >= 1 And dt.Rows(lastduplicaterow)("Cemetery_name").ToString = newindividual.Cemetery_name Then
                                UpdateModule.Cemetery_reference(newindividual.Cemetery_ID, newindividual.Cemetery_name, newindividual.Cemetery_state, newindividual.Cemetery_county, newindividual.Cemetery_address)
                                updatevalue = "yes"

                            ElseIf newindividual.Cemetery_ID = Nothing And cemeterycount >= 1 Then
                                Insertmodule.Cemetery_reference(newindividual.Cemetery_name, newindividual.Cemetery_state, newindividual.Cemetery_county, newindividual.Cemetery_address)
                                insertvalue = "Yes"

                            End If

                            If Not newindividual.Burial_ID = Nothing And burialcount >= 1 Then
                                UpdateModule.burials(newindividual.Burial_ID, newindividual.Cemetery_Section, newindividual.Cemetery_Row, newindividual.Cemetery_Lot, newindividual.Grave, newindividual.Burial_date)
                                updatevalue = "yes"
                            ElseIf newindividual.Burial_ID = Nothing And burialcount >= 1 Then
                                Insertmodule.burials(newindividual.Individual_ID, newindividual.Cemetery_ID, newindividual.Cemetery_Section, newindividual.Cemetery_Row, newindividual.Cemetery_Lot, newindividual.Grave, newindividual.Burial_date)
                                insertvalue = "Yes"
                            End If

                            If Not newindividual.Source_ID = Nothing Then
                                Insertmodule.Sources(newindividual.Source_ID, newindividual.Source, newindividual.Source_location)
                                'ElseIf newindividual.Source_ID = 0 And Sourcecount >= 1 Then
                                ' Insertmodule.Sources(newindividual.Source_ID, newindividual.Source, newindividual.Source_location)
                                insertvalue = "Yes"
                            End If

                            'If Not newindividual.Reference_ID = 0 And Referencecount >= 1 Then
                            'UpdateModule.References(newindividual.Reference_ID, newindividual.Source_ID, newindividual.Reference_type, newindividual.Reference, newindividual.Individual_ID, newindividual.Source, newindividual.Source_location)
                            'ElseIf newindividual.Reference_ID = 0 And Referencecount >= 1 Then
                            'Insertmodule.References(newindividual.Reference_ID, newindividual.Source_ID, newindividual.Reference_type, newindividual.Reference, newindividual.Individual_ID, newindividual.Source, newindividual.Source_location)
                            'End If

                            If Not newindividual.SSN_ID = 0 And SSNcount >= 1 Then
                                UpdateModule.Social_security("", newindividual.Social_security_number)
                            ElseIf newindividual.SSN_ID = 0 And SSNcount >= 1 Then
                                Insertmodule.SSN(newindividual.SSN_ID, newindividual.Social_security_number, newindividual.Individual_ID)
                                insertvalue = "Yes"
                            End If
                        End If
                    End If
                End If

            Next firsttable

nxI:

            If count < 1 Then

                'non duplicate information
                'before moving to the next record in the temp table copy nonduplicates to another table
                'you can use this table later on to compare to manually match or bulk insert into.
                'get count for all matches

                'remove all duplicates to make a master list
                If dt.Rows.Count > 0 And Not I2 > dt.Rows.Count - 1 Then
                    Call remove_duplicates_from_temptable(I, I2, count, Duplicatestart, dt)
                End If


                removeduptable.ImportRow(temptable.Rows(I))


            Else


                'make a duplicates table to show you which duplicates were found and how many
                duplicaterow = Duplicatetable.NewRow()
                duplicaterow.Item("First_name") = dt.Rows(I2).Item("First_name")
                duplicaterow.Item("Middle_name") = dt.Rows(I2).Item("Middle_name")
                duplicaterow.Item("Last_name") = dt.Rows(I2).Item("Last_name")
                duplicaterow.Item("Duplicatecount") = count

                'add rows to table
                Duplicatetable.Rows.Add(duplicaterow)

            End If



            'always remove all string values before proceeding to next match
            'if this step is not completed you will bring over data from other matches                 
            newindividual.emptyvalues()

            'only for matches
            If count >= 1 Then
                I3 = Duplicatestart + count
            Else
                I3 = I2 + 1
            End If

            count = 0
            deathcount = 0
            Referencecount = 0
            updatecount = 0
            Duplicatestart = Nothing

nxIvalue:
        End SyncLock

movetoend:


        'bring up table that shows duplicates and non duplicates
        'insert into tables if no match is found




        '  MessageBox.Show("table 1: " & countI & "  table 2: " & counti2)

    End Sub


Public Sub ParallelForprocess(ByVal startOfIteration As Integer, ByVal endOfIteration As Integer, ByVal subFunctionName As String)

       startOfIteration = 0
       endOfIteration = temptable.Rows.Count
       Const flags As BindingFlags = BindingFlags.Public Or BindingFlags.NonPublic Or BindingFlags.Static Or BindingFlags.Instance

       Dim modType As Type = GetType(allmodules)
       ' Find the method with the specified name which accepts a single Integer parameter:
       Dim method As MethodInfo = modType.GetMethod(subFunctionName, flags, Nothing, New Type() {GetType(Integer)}, Nothing)

       If method Is Nothing Then
           Throw New ArgumentException(String.Format("Method '{0}' was not found.", subFunctionName), "subFunctionName")
       End If

       ' Create a delegate of the correct type pointing to the method:
       Dim del As System.Delegate

       del = System.Delegate.CreateDelegate(GetType(Action(Of Integer)), method)

       ' Cast the delegate to the correct type:
       Dim action As Action(Of Integer) = DirectCast(del, Action(Of Integer))

       ' NB: Second parameter to Range is the number of items to return, not the last number to return:
       Dim numbers As IEnumerable(Of Integer) = ParallelEnumerable.Range(startOfIteration, endOfIteration).AsParallel.AsOrdered


       dtcolumns.Add("Cemetery_name")
       dtcolumns.Add("Cemetery_county")
       dtcolumns.Add("Cemetery_state")
       dtcolumns.Add("Cemetery_address")
       dtcolumns.Add("Cemetery_Section")
       dtcolumns.Add("Cemetery_Row")
       dtcolumns.Add("Cemetery_Lot")
       dtcolumns.Add("Grave")
       dtcolumns.Add("Burial_date")

       'connect to
       Dim sqlconn1 As New SqlConnection("my sql statement")
       Dim sqladaptor = New SqlDataAdapter

       sqlconn1.Open()

       sqladaptor.SelectCommand = New SqlCommand("Select Profile.Individual_ID, First_name, Middle_name, Last_name, Sex, Race, Place_of_birth, County_of_birth, State_of_birth, Date_of_birth, death.Death_ID, Place_of_death, State_of_death, County_of_death, Date_of_death, Cause_of_death, Cemetery_reference.Cemetery_ID, Cemetery_name, Cemetery_state, Cemetery_county, Cemetery_address, Cemetery_Section, Cemetery_Row, Cemetery_Lot, Grave, Burial_date from Profile full join Death On profile.Individual_ID = death.individual_ID full join Burial On Profile.Individual_ID = Burial.Individual_ID full join Cemetery_reference On Burial.Cemetery_ID = Cemetery_reference.Cemetery_ID ORDER BY Last_name ASC, First_name ASC, Date_of_birth ASC", sqlconn1)

       sqladaptor.SelectCommand.ExecuteNonQuery()
       sqladaptor.Fill(dt)

       sqlconn1.Close()
       sqlconn1.Dispose()
       sqladaptor.Dispose()

       Dim view As New DataView(temptable)
       view.Sort = "Last_name ASC, First_name ASC, Date_of_birth ASC"
       Dim temptab As DataTable = view.ToTable
       temptable.Clear()
       temptable = temptab

       start_diagnostic("match")
       System.Threading.Tasks.Parallel.ForEach(numbers.AsParallel.AsOrdered, processorcount, action)

       If updatevalue = "yes" Then
           UpdateModule.update_all_tables()
       End If

       If insertvalue = "Yes" Then
           Insertmodule.insert_all()
       End If
       end_diagnostic("match")

       Manualimport.Show()

   End Sub


What I have tried:

tried synclocks and tried to use if I = temptable.rows.count -1 then do my function. However, it never made it to the function.
Posted
Updated 17-Aug-19 6:48am
v4
Comments
Dave Kreskowiak 17-Aug-19 0:03am    
Show the code you're using.
Member 11856456 17-Aug-19 0:13am    
I have updated the question, all I care about is how to skip over this section:
SyncLock lock
If updatevalue = "yes" Then
UpdateModule.update_all_tables()
End If
End SyncLock

SyncLock lock
If insertvalue = "Yes" Then
Insertmodule.insert_all()
End If
End SyncLock
' end_diagnostic("match")

' MessageBox.Show("table 1: " & countI & " table 2: " & counti2)
SyncLock lock
Manualimport.Show()
End SyncLock

until the last parallel for iteration
Member 11856456 17-Aug-19 12:44pm    
Dave I have rearranged some of the code outside the parallel.for. the stuff above the for statement is now above the parallel.for and the stuff after the I2 iteration is under the parallel for. after trying this out the code looked like it was running correctly. However, because it is out of sequence, 1,2,3,4,5 the match statement I have does not work. I was hoping to keep that specific order and just send to individual threads and then write to the table. is there a way to accomplish this? Please keep in mind this is my first attempt at doing anything parallel.
Dave Kreskowiak 17-Aug-19 14:40pm    
You don't get to dictate which order the threads run in. If you're trying to maintain and order, you cannot break up the execution into multiple threads.

 
Share this answer
 
If you're doing this:
VB.NET
System.Windows.Forms.Control.CheckForIllegalCrossThreadCalls = False

You're doing something very wrong and just creating problems for yourself that you're not going to be able to duplicate. This property only does anything during debugging. Compile your code in Release and this line doesn't even make it into your .EXE.

Forms and controls should NEVER be touched from anything other than the UI (startup) thread.

This entire code snippet looks wrong. Why would you be calling Match_Merge from multiple threads? Why would each of those threads be making a database SELECT when they all would be using the exact same data?

You've got a loop inside of a loop, each with their own SyncLock? This makes it so only a single thread can go through a non-performant loop at a time. The inner SyncLock isn't even needed and is redundant.

This entire thing needs to be scrapped, thoroughly rethought and redesigned, and rewritten. This monolithic method needs to be broken into multiple methods that are responsible for much smaller problems to solve.

For example, that SQL query you run only needs to be done ONCE. The data needs to be retrieved by its own method and then that data can be passed to a method that will use it to make comparisons with some other data.
 
Share this answer
 
v2

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900