Count of duplicate lines in text file

Question

1.00/5 (1 vote)

See more:

Hi, how can I add to existing code count of lines duplicate?
Thank you.

What I have tried:

Private Sub remove_duplicite(sender As Object, e As EventArgs)
        Dim sFiles() As String
        sFiles = Directory.GetFiles(filesPath1, remove_dupl)

        Dim path As String = String.Join("", sFiles)
        'MessageBox.Show(path)
        <pre> Dim i As Integer = 0


        Using sr As StreamReader = New StreamReader(path)
            Do While sr.Peek() >= 0
                lines.Add(sr.ReadLine())
                Dim sLine = sr.ReadLine

                If lines.Contains(sLine) Then
                    i += 1
                    pocet_duplicit = i
                    'lines.Add(i & "," & sLine)
                Else
                    lines.Add(sLine)
                End If
            Loop
        End Using

        'Write to file
        Using sw As StreamWriter = New StreamWriter(path)
            For Each line As String In lines
                sw.WriteLine(line)
            Next
        End Using
        Close()
    End Sub

Posted 20-Sep-22 3:25am

Member 15251555

Updated 25-Sep-22 1:26am

v2

Add a Solution

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

OriginalGriff · Answer 1 · 2022-09-20T03:43:00

Solution 1

Sort them: then identical lines are next to each other and simple to find and count.

Posted 20-Sep-22 3:43am

OriginalGriff

Comments

Member 15251555 21-Sep-22 8:57am

thi is a file with 140000 rows, I dont known how fast compare lines

OriginalGriff 21-Sep-22 9:59am

"Sort them: then identical lines are next to each other and simple to find and count."

Member 15251555 22-Sep-22 2:59am

Grif, do you have some examples ? I dont known how start.

OriginalGriff 22-Sep-22 3:17am

Oh, come on!

You know how to read a file into an array of lines, yes? It's one line of code!

Member 15251555 22-Sep-22 5:10am

Griff, I edited it what do you think, is it correct?

Richard Deeming 22-Sep-22 5:41am

Definitely not! What's wrong with:

For Each filePath As String In Directory.GetFiles(filesPath1)
    Dim lines() As String = File.ReadAllLines(filePath)
    Dim uniqueLines() As String = lines.Distinct().ToArray()
    Dim duplicateLineCount = uniqueLines.Length - lines.Length
    If duplicateLineCount <> 0 Then
        File.WriteAllLines(filePath, uniqueLines)
    End If
Next

Member 15251555 22-Sep-22 7:28am

Hi Richard, I tried this code - working well, thank you.

Maciej Los · Answer 2 · 2022-09-25T01:26:00

I'd use Linq[^]. See:

VB.NET

Dim lines As String() = New String(){"APPLE", "APPLE", "MANGO", "BANANA", "BANANA", "APPLE", "MANGO"}

Dim dups = lines. _
	Select(Function(l, i) New With{i, l}). _
	GroupBy(Function(x) x.l). _
	Select(Function(grp)  New With _
	{ _
		.Value=grp.Key,  _
		.Lines = String.Join(";", grp.Select(Function(x) x.i)) _
	}). _
	ToList()

C#

string[] lines = {"APPLE", "APPLE", "MANGO", "BANANA", "BANANA", "APPLE", "MANGO"};

var dups = lines
	.Select((l, i) => new {i, l})
	.GroupBy(x => x.l)
	.Select(grp => new
	{
		Value=grp.Key, 
		Lines = string.Join(";", grp.Select(x=> x.i))
	})
	.ToList();

dups variable holds:
Value Lines
APPLE 0;1;5
MANGO 2;6
BANANA 3;4