Awesome Open Source
Awesome Open Source
Combined Topics
deduplication
x
Advertising
📦 10
All Projects
Application Programming Interfaces
📦 124
Applications
📦 192
Artificial Intelligence
📦 78
Blockchain
📦 73
Build Tools
📦 113
Cloud Computing
📦 80
Code Quality
📦 28
Collaboration
📦 32
Command Line Interface
📦 49
Community
📦 83
Companies
📦 60
Compilers
📦 63
Computer Science
📦 80
Configuration Management
📦 42
Content Management
📦 175
Control Flow
📦 213
Data Formats
📦 78
Data Processing
📦 276
Data Storage
📦 135
Economics
📦 64
Frameworks
📦 215
Games
📦 129
Graphics
📦 110
Hardware
📦 152
Integrated Development Environments
📦 49
Learning Resources
📦 166
Legal
📦 29
Libraries
📦 129
Lists Of Projects
📦 22
Machine Learning
📦 347
Mapping
📦 64
Marketing
📦 15
Mathematics
📦 55
Media
📦 239
Messaging
📦 98
Networking
📦 315
Operating Systems
📦 89
Operations
📦 121
Package Managers
📦 55
Programming Languages
📦 245
Runtime Environments
📦 100
Science
📦 42
Security
📦 396
Social Media
📦 27
Software Architecture
📦 72
Software Development
📦 72
Software Performance
📦 58
Software Quality
📦 133
Text Editors
📦 49
Text Processing
📦 136
User Interface
📦 330
User Interface Components
📦 514
Version Control
📦 30
Virtualization
📦 71
Web Browsers
📦 42
Web Servers
📦 26
Web User Interface
📦 210
The Top 23 Deduplication Open Source Projects
Categories
>
Data Processing
>
Deduplication
Restic
⭐
12,689
Fast, secure, efficient backup program
Borg
⭐
7,265
Deduplicating archiver with compression and authenticated encryption.
Alertmanager
⭐
3,985
Prometheus Alertmanager
Libpostal
⭐
3,110
A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
Dupeguru
⭐
2,047
Find duplicate files
Rmlint
⭐
1,027
Extremely fast tool to remove duplicates and other lint from your filesystem
Borgmatic
⭐
949
Simple, configuration-driven backup software for servers and workstations
Jdupes
⭐
830
A powerful duplicate file finder and an enhanced fork of 'fdupes'.
Rdedup
⭐
698
Data deduplication engine, supporting optional compression and public key encryption.
Kopia
⭐
625
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
Talisman
⭐
590
Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
Recordlinkage
⭐
551
A toolkit for record linkage and duplicate detection in Python
Data Matching Software
⭐
220
A list of free data matching and record linkage software.
Autorestic
⭐
207
High level CLI utility for restic
Lsh
⭐
187
Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents
Kvdo
⭐
169
A pair of kernel modules which provide pools of deduplicated and/or compressed block storage.
Dejavu
⭐
152
Quickly detect already witnessed data.
Vdo
⭐
142
Userspace tools for managing VDO volumes.
Spark Lucenerdd
⭐
115
Spark RDD with Lucene's query and entity linkage capabilities
Fingerprints
⭐
91
Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.
Rltk
⭐
74
Record Linkage ToolKit (Find and link entities)
Fastcdc Rs
⭐
31
FastCDC implementation in Rust
Dupandas
⭐
20
📊 python package for performing deduplication using flexible text matching and cleaning in pandas dataframe
1-23 of 23 projects
Advertising
📦 10
All Projects
Application Programming Interfaces
📦 124
Applications
📦 192
Artificial Intelligence
📦 78
Blockchain
📦 73
Build Tools
📦 113
Cloud Computing
📦 80
Code Quality
📦 28
Collaboration
📦 32
Command Line Interface
📦 49
Community
📦 83
Companies
📦 60
Compilers
📦 63
Computer Science
📦 80
Configuration Management
📦 42
Content Management
📦 175
Control Flow
📦 213
Data Formats
📦 78
Data Processing
📦 276
Data Storage
📦 135
Economics
📦 64
Frameworks
📦 215
Games
📦 129
Graphics
📦 110
Hardware
📦 152
Integrated Development Environments
📦 49
Learning Resources
📦 166
Legal
📦 29
Libraries
📦 129
Lists Of Projects
📦 22
Machine Learning
📦 347
Mapping
📦 64
Marketing
📦 15
Mathematics
📦 55
Media
📦 239
Messaging
📦 98
Networking
📦 315
Operating Systems
📦 89
Operations
📦 121
Package Managers
📦 55
Programming Languages
📦 245
Runtime Environments
📦 100
Science
📦 42
Security
📦 396
Social Media
📦 27
Software Architecture
📦 72
Software Development
📦 72
Software Performance
📦 58
Software Quality
📦 133
Text Editors
📦 49
Text Processing
📦 136
User Interface
📦 330
User Interface Components
📦 514
Version Control
📦 30
Virtualization
📦 71
Web Browsers
📦 42
Web Servers
📦 26
Web User Interface
📦 210
