Awesome Open Source
Awesome Open Source
Combined Topics
deduplication x
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194

The Top 157 Deduplication Open Source Projects on Github

Categories > Data Processing > Deduplication
Restic ⭐ 13,790
Fast, secure, efficient backup program
Borg ⭐ 7,583
Deduplicating archiver with compression and authenticated encryption.
Alertmanager ⭐ 4,396
Prometheus Alertmanager
Libpostal ⭐ 3,158
A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
Dupeguru ⭐ 2,240
Find duplicate files
Rmlint ⭐ 1,109
Extremely fast tool to remove duplicates and other lint from your filesystem
Kopia ⭐ 1,107
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
Borgmatic ⭐ 1,003
Simple, configuration-driven backup software for servers and workstations
Jdupes ⭐ 839
A powerful duplicate file finder and an enhanced fork of 'fdupes'.
Rdedup ⭐ 712
Data deduplication engine, supporting optional compression and public key encryption.
Talisman ⭐ 590
Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
Recordlinkage ⭐ 490
A toolkit for record linkage and duplicate detection in Python
Autorestic ⭐ 273
High level CLI utility for restic
Zingg ⭐ 183
Scalable fuzzy matching for data mastering, deduplication and entity resolution.
Kvdo ⭐ 172
A pair of kernel modules which provide pools of deduplicated and/or compressed block storage.
Data Matching Software ⭐ 166
A list of free data matching and record linkage software.
Dejavu ⭐ 147
Quickly detect already witnessed data.
Lsh ⭐ 144
Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents
Vdo ⭐ 144
Userspace tools for managing VDO volumes.
Spark Lucenerdd ⭐ 121
Spark RDD with Lucene's query and entity linkage capabilities
Mail Deduplicate ⭐ 117
📧 CLI to deduplicate mails from mail boxes.
Benji ⭐ 92
Benji Backup: A block based deduplicating backup software for Ceph RBD images, iSCSI targets, image files and block devices
Fingerprints ⭐ 91
Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.
Splink ⭐ 87
Implementation in Apache Spark of the EM algorithm to estimate parameters of Fellegi-Sunter's canonical model of record linkage.
Blobstash ⭐ 83
You personal database. Mirror of https://git.sr.ht/~tsileo/blobstash
Dupd ⭐ 83
CLI utility to find duplicate files
Rltk ⭐ 78
Record Linkage ToolKit (Find and link entities)
Rocketmqdeduplistener ⭐ 73
RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用
Entity Embed ⭐ 70
PyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.
Intraarchivededuplicator ⭐ 70
Tool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.
Lieu ⭐ 64
Dedupe/batch geocode addresses and venues around the world with libpostal
Dduper ⭐ 62
Fast block-level out-of-band BTRFS deduplication tool.
Horcrux ⭐ 61
The Dropbox for IPFS (without the icky stuff)
Py Image Dedup ⭐ 54
CLI utility to find near duplicate images and remove all but the best copy.
Record Linkage Resources ⭐ 49
Resources for tackling record linkage / deduplication / data matching problems
Cargo Limit ⭐ 48
Cargo with less noise: warnings are skipped until errors are fixed, Neovim integration, etc.
Deduplication ⭐ 38
Fast multi-threaded content-dependent chunking deduplication for Buffers in C++ with a reference implementation in Javascript. Ships with extensive tests, a fuzz test and a benchmark.
Fs Curator ⭐ 36
Automation for the serious data hoarder that wants to have their data and use it
Pgdedupe ⭐ 32
A simple command line interface to the datamade/dedupe library.
Imgdedup ⭐ 29
CLI tool for image duplicate detection
Acid Store ⭐ 28
A library for secure, deduplicated, transactional, and verifiable data storage
Watsondedupe ⭐ 26
Self-contained C# library for data deduplication using Sqlite
Fastcdc Rs ⭐ 25
FastCDC implementation in Rust
Umicollapse ⭐ 24
Accelerating the deduplication and collapsing process for reads with Unique Molecular Identifiers (UMI). Heavily optimized for scalability and orders of magnitude faster than a previous tool.
Sauvegarde ⭐ 21
Continuous data protection for GNU/Linux (cdpfgl).
Frost ⭐ 21
A backup program that does deduplication, compression, encryption
Sparkclean ⭐ 20
A Scalable Data Cleaning Library for PySpark.
Spark Search ⭐ 20
Spark Search - high performance advanced search features based on Apache Lucene
Yadf ⭐ 19
Yet Another Dupes Finder
Dedupsqlfs ⭐ 18
Deduplicating filesystem via Python3, FUSE and SQLite
Borg Qt ⭐ 15
A Qt frontend for the command line software BorgBackup.
Dupandas ⭐ 14
📊 python package for performing deduplication using flexible text matching and cleaning in pandas dataframe
Fastcdc Py ⭐ 14
FastCDC implementation in Python https://pypi.org/project/fastcdc/
Container Bootfs ⭐ 13
Container image converter aiming to minimize image size and speed up boot time dramatically with block-level de-dupliction and lazy-pull technology.
Febrl Fork V0.4.2 ⭐ 12
Fork of the Freely Extensible Biomedical Record Linkage program
Zpaqfranz ⭐ 12
Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
Recordlinkage Annotator ⭐ 11
A browser user interface for manual labeling of record pairs.
Dedup ⭐ 11
A command-line tool for deduplicating entries in a file or stream.
Marty ⭐ 11
An efficient backup tool inspired by Git, saving your bandwidth and providing global deduplication at file level.
Blobfs ⭐ 11
New project: https://git.sr.ht/~tsileo/blobfs
Nominally ⭐ 10
A maximum-strength name parser for record linkage.
Backend ⭐ 9
Backend (Docker & API) for matchID project
Dedup Resolve Webpack Plugin ⭐ 9
Webpack plugin that resolves copies of the same module in different locations to a single path.
Phash ⭐ 9
Perceptual hashing command-line tool
Lafs Backup Tool ⭐ 9
Tool to securely push incremental (think "rsync --link-dest") backups to tahoe-lafs
Dugu ⭐ 8
Find, remove and avoid duplicates with dugu: The Duplicates Guru
Soph ⭐ 8
Efficiently import pictures while handling duplicates gracefully
Bucketsync ⭐ 8
S3 backed FUSE Filesystem written in Go with dedup and encryption.
Borg_wrapper ⭐ 8
another borg wrapper with some cool features for borg backup
Dedup ⭐ 8
Deduplication for cfDNA sequencing data
Sparklink ⭐ 8
WORK IN PROGRESS: Implementation in Apache Spark of the EM algorithm to estimate parameters of Fellegi-Sunter's canonical model of record linkage
Dedupe ⭐ 8
Java DSL for (online) deduplication
Ingestion ⭐ 6
Ingestion Pipeline
Venti Streamchunk ⭐ 6
Variable-sized block deduplication archival backed by Plan9's venti
Sparklyclean ⭐ 6
Optimal distributed data deduplication and supervised learning pipeline using Apache Spark
Dataloader Php ⭐ 6
A speed layer that enables query batching, de-duplication, and caching for efficient data fetching over any storage backend.
Qdda ⭐ 5
Quick and Dirty Deduplication Analyzer
Cafs ⭐ 5
Content-Addressable File System (used by BitWrk)
Oc_graphenricher ⭐ 5
A tool to enrich any OCDM compliant Knowledge Graph, finding new identifiers and deduplicating entities.
Deduper ⭐ 5
General deduping engine for JDBC sources with output to JDBC/csv targets
Celery Unique ⭐ 5
Python factory for limiting Celery tasks by configuration
Deduper ⭐ 5
A general purpose deduplication framework
Throttle Queue ⭐ 5
A promise based priority queue with task deduplication, concurrency control, serial resolution and aging
Qlink ⭐ 5
Entity Resolution and Record Linkage library
Mfdedup ⭐ 5
A Management Friendly Deduplication Prototype System for Backup
Gob ⭐ 4
This is a mirror of https://gitlab.com/gob-backup/gob
Face_amnesia ⭐ 4
Face detection and retrieval in image and video files.
Smered ⭐ 4
Mirror of https://bitbucket.org/resteorts/smered
Arius ⭐ 4
Arius is a lightweight archival solution, specifically built to leverage the Azure Blob Archive tier.
Text Dedup ⭐ 4
All-in-one text de-duplication
Eplouribousse ⭐ 3
Application web pour le dédoublonnement des périodiques [maintenu par : unistra-scd]
Docker_srcdocks ⭐ 3
Deduplicated hosting of Source Engine based gameservers - With Docker!
Dedup_signature ⭐ 3
A collection of algorithms to generate a signature/fingerprint/hash in order to be used for detecting duplicate/near duplicate documents.
Deduplication ⭐ 3
Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.
Dedup ⭐ 3
Find duplicate text files.
Dedup ⭐ 3
Deduplicator
Cd4py ⭐ 3
CD4Py: Code De-Duplication for Python
Cpdd ⭐ 2
cpdd - a simple copy and deduplication tool written in Rust
De Dupe ⭐ 2
Deduplicate strings from javascript files
Umi Atac Seq ⭐ 2
1-100 of 157 projects

Related Projects

Python Deduplication Projects (46)
Backup Deduplication Projects (28)
Deduplication Dedupe Projects (19)
C Deduplication Projects (19)
Deduplication Record Linkage Projects (19)
Go Deduplication Projects (16)
Compression Deduplication Projects (16)
Javascript Deduplication Projects (15)
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open Source 2018-2021