Duplicate file finder

Question

1.00/5 (1 vote)

See more:

Dear Guys.......

i want to find and process all duplicate files in my system.

i supposed to use byte by byte comparison of files. but here the problem is i want to search each files with other files in a directory,folder and sub folders. how to find and list all files in a folder and its sub folder hierarchy.

please give me an idea to find duplicate files in my directory tree.
if possible give me the idea and code if you guys have...

With regards

Irfad C

Posted 8-Dec-11 22:47pm

irfad

Add a Solution

2 solutions

Solution 1

This will retrieve the paths of all files in a directory including those in sub directories.

C#

string[] files = System.IO.Directory.GetFiles(
    "PathToDirectory", "*.*", System.IO.SearchOption.AllDirectories);

Posted 8-Dec-11 23:04pm

LanFanNinja

Updated 8-Dec-11 23:05pm

v2

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Manfred Rudolf Bihy · Accepted Answer · 2011-12-08T23:06:00

Solution 2

Here is the idea of what you should do:

Use the class DirectoryInfo[^] to traverse the file system structure.
Add all files found with their complete path to a list making sure not to add the same file more than once.
Calculate the MD5 hash[^] for the content of each and every file in your list and store that value along with the files path. (maybe SHA-2[^])
Partition all files into sets so that any two files will be in one set if and only if their MD5 or SHA-2 hashes are identical.
For each set do a binary compare of all files in that set against each other.

Regards,

—MRB

Posted 8-Dec-11 23:06pm

Manfred Rudolf Bihy

Updated 18-Dec-11 10:14am

v2

Comments

Mehdi Gholam 9-Dec-11 5:12am

5'ed

thatraja 9-Dec-11 5:15am

5!

LanFanNinja 9-Dec-11 5:18am

+5 I agree!

Michel [mjbohn] 9-Dec-11 5:19am

and my 5

_Tushar Patil 18-Dec-11 22:48pm

+5

Espen Harlinn 30-Dec-11 9:42am

5'ed!

Manfred Rudolf Bihy 30-Dec-11 10:38am

Thank you, Espen!