Awesome Open Source
Awesome Open Source
Combined Topics
dedupe x
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194

The Top 50 Dedupe Open Source Projects on Github

Restic ⭐ 13,671
Fast, secure, efficient backup program
Borg ⭐ 7,583
Deduplicating archiver with compression and authenticated encryption.
Dedupe ⭐ 3,137
🆔 A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.
Yarn Deduplicate ⭐ 981
Deduplication tool for yarn.lock files
Jdupes ⭐ 839
A powerful duplicate file finder and an enhanced fork of 'fdupes'.
Free Style ⭐ 688
Make CSS easier and more maintainable by using JavaScript
Recordlinkage ⭐ 490
A toolkit for record linkage and duplicate detection in Python
Duplicut ⭐ 474
Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)
Csvdedupe ⭐ 317
🆔 Command line tool for deduplicating CSV files
Dedupe Examples ⭐ 270
🆔 Examples for using the dedupe library
Bees ⭐ 230
Best-Effort Extent-Same, a btrfs dedupe agent
Mail Deduplicate ⭐ 117
📧 CLI to deduplicate mails from mail boxes.
Dupe Krill ⭐ 115
A fast file deduplicator
Zingg ⭐ 82
Scalable fuzzy matching for data mastering, deduplication and entity resolution.
Imgdupes ⭐ 78
Finding and deleting near-duplicate images based on perceptual hash.
Dduper ⭐ 62
Fast block-level out-of-band BTRFS deduplication tool.
Pgdedupe ⭐ 32
A simple command line interface to the datamade/dedupe library.
Watsondedupe ⭐ 26
Self-contained C# library for data deduplication using Sqlite
Django Super Deduper ⭐ 23
Utilities for de-duping Django model instances
Duplex ⭐ 19
Duplicate code finder for Elixir
Yadf ⭐ 19
Yet Another Dupes Finder
Apollo Dedup Batch Http Link ⭐ 7
👓@tipe/apollo-dedup-batch-http-link: batches multiple operations into a single HTTP dedup request. Instead of sending a single operation, it sends an array of operations to the server.
Dedupe Variable Name ⭐ 5
name variable type for dedupe
Dedupe Variable Address ⭐ 5
Address Variable Type for dedupe
Dedupe Variable Employer ⭐ 3
Dedup ⭐ 3
Deduplicator
Dedupe Variable Fuzzycategory ⭐ 2
Dedupe Variable for Fuzzy Categories
Watsondedupeui ⭐ 2
UI for WatsonDedupe library
De Dupe ⭐ 2
Deduplicate strings from javascript files
Android Dedupefs ⭐ 2
A filesystem for reading Android dedupe backup
Iter Intersection By Hash ⭐ 1
Create an iterator which returns the intersection of two or more iterators according to a hash function.
Iter Intersection ⭐ 1
Create an iterator which returns the intersection of two or more iterators.
Iter Unique By Hash ⭐ 1
Create an iterator which returns unique values according to a hash function.
Iter Unique By ⭐ 1
Create an iterator which returns unique values according to a predicate function.
Parseratorvariable ⭐ 1
Base class for dedupe variables for parsed fields
Godedupe ⭐ 1
Improved and modern fdupes alternative
Swuniq ⭐ 1
A command-line tool for deduplicating entries in a file or stream with constant memory usage
Sort Columns ⭐ 1
Sort, uniq, reverse, and randomize data
Csvdedupe Docker ⭐ 1
Run csvdedupe on Docker container.
Pocket Deduper ⭐ 1
Remove duplicates from your Pocket list
Twicejs ⭐ 1
Manage duplicates, count item occurences, dedupe an Array.
Simplededup ⭐ 1
Yet another block-level btrfs deduplication tool. Other filesystems with FIEMAP and FIDEDUPERANGE may also compatible.
Zipfile Dedup ⭐ 1
Project to take two similar zipfiles, and to dedupe files that have the same tiemstamp in the older file.
Btrfslime ⭐ 1
A few useful BTRFS utilities
Borg ⭐ 1
Borg backup in a container.
Rayyan Dedup Training ⭐ 1
Command line tool for training and deduplicating review articles in rayyan
Iter Unique ⭐ 1
Create an iterator which returns unique values.
Iter Dedupe ⭐ 1
Create an iterator which removes consecutive duplicated values.
Iter Dedupe By ⭐ 1
Create an iterator which removes consecutive values that resolve to the same value according to a provided function.
Iter Union ⭐ 1
Create an iterator which returns the union of two or more iterators.
1-50 of 50 projects

Related Projects

Deduplication Dedupe Projects (17)
Python Dedupe Projects (16)
Javascript Dedupe Projects (14)
Dedupe Deduplicate Projects (11)
Dedupe Uniq Projects (11)
Dedupe Uniq Deduplicate Projects (9)
Javascript Dedupe Uniq Projects (9)
Python Deduplication Dedupe Projects (6)
C Dedupe Projects (6)
Python Dedupe Variable Projects (5)
Record Linkage Dedupe Projects (5)
Duplicates Dedupe Projects (5)
Cli Dedupe Projects (5)
Btrfs Dedupe Projects (5)
Go Dedupe Projects (4)
Backup Dedupe Projects (4)
Python Record Linkage Dedupe Projects (4)
Entity Resolution Record Linkage Dedupe Projects (4)
Entity Resolution Dedupe Projects (4)
Javascript Duplicates Dedupe Projects (4)
Compression Deduplication Dedupe Projects (4)
Compression Dedupe Projects (4)
Deduplication Dedupe Deduplicate Projects (4)
Go Deduplication Dedupe Projects (3)
Python Entity Resolution Record Linkage Dedupe Projects (3)
Python Entity Resolution Dedupe Projects (3)
Deduplication Btrfs Dedupe Projects (3)
Duplicate Detection Dedupe Projects (3)
Javascript Deduplication Dedupe Projects (3)
C Deduplication Dedupe Projects (3)
Deduplication Dedupe Uniq Projects (3)
Deduplication Dedupe Uniq Deduplicate Projects (3)
Python3 Dedupe Projects (3)
Backup Deduplication Dedupe Projects (2)
Python Library Entity Resolution Record Linkage Dedupe Projects (2)
Python Library Entity Resolution Dedupe Projects (2)
Python Library Record Linkage Dedupe Projects (2)
Python Library Dedupe Projects (2)
C Plus Plus Btrfs Dedupe Projects (2)
C Plus Plus Dedupe Projects (2)
Dedupe Dedup Projects (2)
Rust Dedupe Projects (2)
Rust Dedupe File Deduplication Projects (2)
Macos Dedupe Projects (2)
Macos Dedupe Hardlink Projects (2)
Dedupe Hardlink Projects (2)
Dedupe File Deduplication Projects (2)
Python Dedupe Deduplicate Projects (2)
C Sharp Compression Deduplication Dedupe Projects (2)
C Sharp Compression Dedupe Projects (2)
C Sharp Deduplication Dedupe Projects (2)
C Sharp Dedupe Projects (2)
Deduplication Dedupe Fdupes Projects (2)
Dedupe Fdupes Projects (2)
Typescript Dedupe Projects (2)
C Backup Dedupe Projects (2)
C Dedupe Uniq Projects (2)
Cli Deduplication Dedupe Projects (2)
Dedupe Uniq Remove Duplicates Projects (2)
Dedupe Remove Duplicates Projects (2)
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open Source 2018-2021