share
Super UserWhich duplicate files and folders finders exist for Windows?
[+59] [15] Andrija
[2009-07-18 20:50:03]
[ windows file-management deduplication ]
[ https://superuser.com/questions/8223/which-duplicate-files-and-folders-finders-exist-for-windows ]

I need a free duplicate file finder/remover app, with ability to find duplicate files/folders by name and/or by size and to remove one of duplicates.

It's possible to do this using a BASH shell script: superuser.com/questions/386199/… - Anderson Green
[+28] [2010-04-08 03:59:54] Synetech

I have tried literally dozens of duplicate file finders (I still have the installers/ZIP files for about 20 of them sitting around). I used CloneMaster 2.19 for a time because it was the best one I could find, though even that wans’t perfect (I wanted one that could also detect duplicate MP3s by audio content, ignoring the tags). All of them had problems that frustrated me enough that I decided to write my own and laid out a list of the features I demand of a DFF.

And then I found AllDup [1]. It is the first and only one that actually made me abandon a project (technically mine isn’t completely abandoned, it’s just no longer being worked on since I don’t need to write it anymore because AllDup does everything I want of it). Anyway, AllDup searches for duplicate files, but unlike the others, it uses a lot of the tricks and techniques that I was going to use in my own DFF. As such, it is very fast: it can for example scan >250,000 of sizes from 1B-5GB in ~30mins (I have done it on my system several times).

Another great (and for some reason rare) feature of AllDup is as I mentioned that it can scan for duplicate MP3s by their actually audio data while ignoring tags, so two MP3s that are the same but have differing tags (very common when downloading) will be detected as duplicates. (It could even detect when I ripped two identical songs from two different CDs—with different tags of course.) Of course it can detect dupes based on other factors besides byte-for-byte content, like filenames, dates, etc.

Other great features of AllDup include the ability to filter (include or exclude) based on filename and/or folder name and/or filesize. The results screen is also very versatile with options to select files based on date, path, name, this, that, the other… It also has variety in what to do with detected duplicates

Michael Thummerer is also very receptive. I have reported bugs and suggested features to him several times which he addressed extremely quickly (to the point that he told me to download and try out a beta with the updates in the very next emails the same day).

Oh and it’s free.

HTH

[1] http://www.alldup.de/

The link doesn't work. - Joseph
Thanks for the notice. I guess I must have typed the URL in manually and perseverated the GTLD. - Synetech
No, what happened was Chrome’s auto-correct reversed the ‘e’ and ‘d’ (like it does when I try to type dr.) Grrr. (It’s fixed, for the record.) - Synetech
I have also used many, and this is what I've settled on. The interface is a little clunky, but it has lots of useful options. - endolith
Have you tried version 3? It is pretty sold. - Synetech
Can you use AllDup to find duplicate folders instead of duplicate folders? - Anderson Green
@AndersonGreen, huh? o.O - Synetech
I meant "duplicate folders instead of duplicate files". Can AllDup find duplicate folders, as well as duplicate files? - Anderson Green
Hmm, there’s no specific duplicate-folder-detection function and I can’t say that I’ve ever seen a dupe-file-finder that does. (You can certainly request; I find that Michael is very responsive to feature-requests and bug-reports.) However, it is fairly simple to at least clean up duplicate folders: run a scan using the two folders as sources, then delete all dupes from one of them, finally compare the results (and merge the two if desired). You could also use a compare/diff program like WinDiff/WinMerge to compare two folders and their contents. - Synetech
Very good... took me a few seconds to understand how to setup the search params correctly, and then it did the trick! Thanks. - TacB0sS
(1) I like AllDup. The only thing I found that it doesn't handle well is rotated JPG files. - JcMaco
@AndersonGreen tou can use ashisoft Duplicate File Finder to search duplicate folders. - Alex78191
Great app so far... - Brethlosze
1
[+15] [2009-10-07 08:45:16] Dimitri C.

Duplicate Cleaner [1] is very fast and it has extensive result set filtering possibilities.

[1] http://www.digitalvolcano.co.uk/content/duplicate-cleaner

This looks nice. It's freeware, works in Windows 7, has a variety of filtering methods, and supports creation of NTFS hardlinks instead of deletion. - endolith
(1) Can't handle Unicode filenames, though? - endolith
(1) Does it handle duplicate folders? - Anderson Green
(2) Actually, it does handle duplicate folders. This is the first program that I've found (so far) that can identify duplicate folders as well as duplicate files. - Anderson Green
The Pro version handles rotated image files. Great! I can finally sort my iOS photo from my Dropbox auto-backups. - JcMaco
@AndersonGreen 11 years later, it still remains the only program that I've found that identifies duplicate folder trees. - Milind R
Okay, I have to correct myself - ashisoft.com/# also does find duplicate folders. But one crucial lacuna is that they are identified by name, size, date - not by content. So if they are named differently, such duplicate folders with same content don't show up here. The other crucial flaw is that only leaf folders are identified - this doesn't find the topmost-level folder which is duplicated. In my case at least, it meant 5x more entries to examine manually. The one in the answer does not have either of these two flaws. - Milind R
2
[+9] [2009-07-18 20:54:11] nik [ACCEPTED]

There was this Duplicate File Finder available for some time but now its only on secondary sites, like this Softpedia reference [1].

The Wareseeker site shows the correct reference http://dff.nazrashid.com/ which is no longer around. I would be careful fetching executables from such sites though.

It can do search for duplicates within multiple directory trees based on,

  • contents (i think it does a md5 match)
  • size
  • name
  • name and size
  • contents and name

And, it lets you filter your search by a minimum and maximum files size (speeds up things when you know the bounds).

A very thoughtful piece of software. Don't know if there is something as fast and featured for free around these days.


The comments refer to a similar Sourceforge tool called Doubles [2]

[1] http://www.softpedia.com/get/System/File-Management/Duplicate-File-Finder.shtml
[2] http://sourceforge.net/projects/doubles/

(2) Actually it is available on SourceForge here: doubles.sourceforge.net - BinaryMisfit
(3) Can I suggest removing the Wareseaker link. I think we need to be careful when referring to Warez sites in general. - BinaryMisfit
I agree with you Diago. I tried locating the DupeFinder tool and could never find it. The doubles tool looks similar but its not the same thing. - nik
(1) For what it's worth, I found the website (home page, at least) in the Wayback Machine: wayback.archive.org/web/20070115135708/http://dff.nazrashid.‌​com with another download link: wayback.archive.org/web/20061207062803/http://dff.nazrashid.‌​com/… - seanf
3
[+8] [2009-07-19 04:51:41] pcapademic

Clonespy http://www.clonespy.com

I find this very helpful and more useful than doublekiller. A particularly nice feature is the "Pools" feature, where you can compare one group of directories with another group of directories while ignoring duplicates within the groups.

For example, my partner likes to keep duplicate image files while working with images. I want to see if I have any copies of those files in my directories.

Pool 1 - Partner Home Directory Pool 2 - My Home Directory.

The only duplicates found are if a file is present both in my directories and my partner's directories. The files only duplicated in my partner's directories are ignored.


Can you use CloneSpy to find all of the duplicate files (or duplicate folders) in a specific directory? - Anderson Green
@AndersonGreen Yes for duplicate files - FelipeAls
This is the only dupe finder that I have come across that allows you to do cross compare of a single file (provided you put it in an adequate folder location) or a folder tree structure, against another folder tree structure (or a set of folders or tree structures). All the other tools I have tried throughout the years have failed at exactly this point! This is the only one that does this right. - Samir
I have often found myself in a situation where I know I am looking at a file that has a duplicate copy somewhere else, but I can't remember where that other copy is stored. Sounds familiar? Or I know that the file is a copy of another file, and I even know the location of that other copy, but I have to compare it against it to see which one is the one I want to keep. I might even have two or three other copies or versions of it on that second location. This is the breaking point where the distinction between a diff tool and a dupe tool becomes foggy. - Samir
The pool to pool comparison is simply a killer! This is a major advantage over other tools. As far as diff and dupe tools go I have yet to find one that forges both into one software. I think Beyond Compare comes close to that, but it's still a mile off. It definitely has the potential, and it's the best diff tools I've used. - Samir
4
[+7] [2009-07-18 21:52:57] Joey

I have the following batch file lying around for some time:

@ECHO OFF
REM TODO: Help when run with /? and switch for recursion
REM Furthermore check whether we might have enough files to hit the envvar length limit
REM and switch strategies accordingly (slower but finds all dupes then)
SETLOCAL ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
REM This method speeds up comparison but suffers from long file lists
REM as environment variables have a length limit
SET FILELIST=
FOR %1 %%i IN (*) DO (
    FOR %%j IN (!FILELIST!) DO (
        IF %%~zi EQU %%~zj (
            fc /b "%%~i" "%%~j">NUL && echo "%%~i" = "%%~j"
        )
    )
    SET FILELIST=!FILELIST! "%%~i"
)
ENDLOCAL
GOTO :EOF

You can run it with /r as argument to run recursively through the directory tree.


(1) Awesome geek solution! - JBRWilkinson
Downvoter: It would help if you said why this wasn't helpful in your opinion. - Joey
A small word of warning is that nested for loops on Windows sometimes result in the outer loop silently being ended prematurely on large directory structures. I've had better luck with having the outer loop call a separate batch file containing the inner loop. - mikewse
5
[+4] [2009-07-18 21:06:25] BinaryMisfit

Total Commander [1] has a brilliant duplicate file finder utility. Unfortunately it is not free.

As nik mentioned there is the Duplicate File Finder which is open source and cross platform. It's available on SourceForge here [2]

[1] http://www.ghisler.com/
[2] http://doubles.sourceforge.net/

6
[+4] [2009-07-18 23:03:59] Chris Driver

DoubleKiller is an excellent free (they also have a pro version) duplicate file finder/remover. I've been using it for years and would thoroughly recommend it.

http://www.bigbangenterprises.de/en/doublekiller


(1) Thanks: still works fine in 2022. Download from web.archive.org/web/20220516125012/http://… is a simple ZIP file which you can extract and just run the executable. No need for complicated installers or admin rights. Matches on (combinations of) name/size/date/CRC32, then intuitive GUI to select the files you don't want any more, then either delete or move those selected files. - Jeroen Wiert Pluimers
7
[+4] [2009-07-19 05:04:14] pcapademic

Winmerge http://winmerge.org/

Compare very similar directories, perhaps between your "Documents" folder and your impromptu "backup", "Copy of Documents" folder, you created several weeks ago "just in case".

You can figure out what files are still identical, which files are the newest (or oldest, if you have encountered a problem), and which files exist only in one place. Then you can "merge" the directories and delete the duplicates.


Can you use WinMerge to obtain a list of duplicate files and folders within one specific directory? - Anderson Green
I think winmerge can be used as a very powerful ally in the hunt for duplicate folders. First digitalvolcano.co.uk/dcdownloads.html can find 100% identical folders, and then the next step is to find "close enough" matches based on duplicate file finding - if there are a lot of files that seem to be duplicated from the same pair of folders, then the two directories can be compared in WinMerge to see how they are different (perhaps they were identical at some point, but a few additional files found their way in later). - Milind R
8
[+2] [2009-07-18 21:57:16] mutewinter

Duplocator [1]. A cross-platform duplicate-finding python script.

[1] http://brainerror.net/scripts/python/duplocator/

(1) Link is broken. - Nifle
9
[+2] [2010-09-05 19:45:42] James Mertz

Advanced system care free [1] comes with an Clone Files Finder with the program. I use it, and it works fine.

[1] http://www.iobit.com/advancedwindowscareper.html

10
[+1] [2009-11-16 02:17:25] endolith

I've always used an old program Find Duplicates [1], but now that Geocities has been killed off, I guess it doesn't exist anymore, so here's a copy of the file [2].

It displays the files in easy-to-see groups isolated by color. It warns you if you've marked all the copies of a file for deletion, can automark them, can selectively automark all files in a directory (I used this a lot), can delete directories that become empty after the deletions, etc. Be careful of links if you're using it in Wine. It will delete all copies of a file without knowing that they are links.

[1] http://www.geocities.com/hirak_99/goodies/finddups.html
[2] http://www.endolith.com/wordpress/2010/01/15/find-duplicates/

I've switched to AllDup. - endolith
11
[+1] [2010-01-15 15:46:43] therobyouknow

Fast Duplicate File Finder:

  • Completely Free
  • Can identify duplicates regardless of where they are in the directory/folder structure
  • Easy to use Windows UI
  • Can produce reports in XML form
  • Can Move the duplicates to the recycle bin or another root folder retaining the original folder structure, e.g. if a duplicate file was within a folder within another folder, this path would be retained: the file would still exist within the folder within the folder, and this who structure would be moved to the folder that the user chooses - whether that be the recycling bin or a user-specified folder
  • Fast binary comparison algorith as well as checking filename and size.

http://www.mindgems.com/products/Fast-Duplicate-File-Finder/Fast-Duplicate-File-Finder-About.htm


can you please explain points 2 and 5? - Milind R
updated @MilindR please let me know if it helps - therobyouknow
Thanks.. I still don't get point 2 though.. Duplicates files should be found anywhere right? - Milind R
Yes correct. For example: consider 2 files with exactly the same content. File 1 is in path /folder1/folder2/file1 and file 2 is in in path /folder3/folder4/folder5/file2 - Fast Duplicate File Finder will find both of these and report them as the same and the user can choose which one they want removed. - therobyouknow
Ah that, yeah that clarifies it. Though I think that is usually taken for granted when considering a program looking for duplicate files. - Milind R
12
[+1] [2010-04-08 03:03:39] Joseph

I have found Yet Another Duplicate File Remover [1] to be awesome. It is pretty new and very easy to use. I used it to compare over 50,000 files from a hard drive recovery project, so it can handle a large amount of files.

[1] http://sourceforge.net/projects/yadfr/

13
[+1] [2011-12-01 10:29:29] Kyle

The best solution I tested on windows was the already mentioned Alldup [1]. Props to Synetech inc. for sharing it. Brilliant and very professional freeware tool.

In the post I provide tested bash solutions that will work under Cygwin, Linux and OS X (or anything that uses bash-like shell).

Looking for duplicate files using freeware and open source tools [2]

[1] http://www.alldup.de/en_alldup.htm
[2] https://htks.digiflakes.com/2011/11/looking-for-duplicate-files-using.html

(2) Can AllDup generate a list of duplicate folders instead of duplicate files? - Anderson Green
not that i'm aware of - Kyle
@AndersonGreen No - Milind R
14
[+1] [2013-01-11 12:26:17] user87856

Another utility:

http://yurichev.com/ddff.html

* Comparison to other duplicate finding utilities:

+ Very fast
+ Comparing directories too.
+ Often, two directories contain, let's say, 4 equal files and 5th file is different.
  We handle it too and output these as "common files in directories"
+ Absence of unnecessary switches.

- Win32 only
- Command-line only

15