Awesome Open Source
Awesome Open Source
Combined Topics
deduplication x

The Top 623 Deduplication Open Source Projects

Categories > Data Processing > Deduplication
Restic ⭐ 16,820
Fast, secure, efficient backup program
total releases 76most recent commit 2 days ago
Borg ⭐ 8,306
Deduplicating archiver with compression and authenticated encryption.
most recent commit 7 days ago
Alertmanager ⭐ 4,921
Prometheus Alertmanager
dependent packages 229total releases 149most recent commit 2 days ago
Duplicacy ⭐ 4,083
A new generation cloud backup tool
total releases 51most recent commit 2 months ago
Imagededup ⭐ 3,901
😎 Finding duplicate images made easy!
most recent commit 5 months ago
Libpostal ⭐ 3,429
A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
most recent commit a month ago
Dedupe ⭐ 3,381
🆔 A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.
dependent packages 9total releases 164most recent commit a day agopypi dedupe} Downloads
Dupeguru ⭐ 2,732
Find duplicate files
most recent commit 14 days ago
Node Dev ⭐ 2,111
Zero-conf Node.js reloading
dependent packages 333total releases 75most recent commit a month agonpm node-dev} Downloads
Kopia ⭐ 1,798
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
dependent packages 1total releases 220most recent commit 10 hours ago
Zbox ⭐ 1,233
Zero-details, privacy-focused in-app file system.
dependent packages 2total releases 27most recent commit 8 months agocargo zbox} Downloads
Rmlint ⭐ 1,186
Extremely fast tool to remove duplicates and other lint from your filesystem
most recent commit 3 months ago
Yarn Deduplicate ⭐ 1,159
Deduplication tool for yarn.lock files
dependent packages 113total releases 16most recent commit a month agonpm yarn-deduplicate} Downloads
Borgmatic ⭐ 1,138
Simple, configuration-driven backup software for servers and workstations
most recent commit 15 hours ago
Attic ⭐ 1,083
Deduplicating backup program
most recent commit 3 years ago
Jdupes ⭐ 1,042
A powerful duplicate file finder and an enhanced fork of 'fdupes'.
most recent commit 2 months ago
Zbackup ⭐ 769
ZBackup, a versatile deduplicating backup tool
most recent commit 2 years ago
Rdedup ⭐ 741
Data deduplication engine, supporting optional compression and public key encryption.
dependent packages 2total releases 19most recent commit 3 months agocargo rdedup-lib} Downloads
Gifcap ⭐ 725
Capture your screen to a GIF in your browser
most recent commit a month ago
Free Style ⭐ 698
Make CSS easier and more maintainable by using JavaScript
dependent packages 24total releases 54most recent commit 7 months agonpm free-style} Downloads
Dropship ⭐ 687
Instantly transfer files between Dropbox accounts using only their hashes.
most recent commit 7 years ago
Recordlinkage ⭐ 652
A powerful and modular toolkit for record linkage and duplicate detection in Python
dependent packages 1total releases 22most recent commit a month agopypi recordlinkage} Downloads
Duke ⭐ 602
Duke is a fast and flexible deduplication engine written in Java
dependent packages 2total releases 4most recent commit 4 months ago
Talisman ⭐ 590
Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
dependent packages 43total releases 30most recent commit a year agonpm talisman} Downloads
Dwarfs ⭐ 572
A fast high compression read-only file system
most recent commit a month ago
Zingg ⭐ 508
Scalable entity resolution, data mastering and deduplication using ML
most recent commit 4 hours ago
Duplicut ⭐ 506
Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)
most recent commit 5 months ago
Autorestic ⭐ 457
Config driven, easy backup cli for restic.
total releases 38most recent commit a day ago
Duperemove ⭐ 406
Tools for deduping file systems
most recent commit 8 months ago
Cc_net ⭐ 406
Tools to download and cleanup Common Crawl data
total releases 2most recent commit 5 months ago
React Request ⭐ 346
Declarative HTTP requests for React
dependent packages 3total releases 24most recent commit a year agonpm react-request} Downloads
Csvdedupe ⭐ 317
🆔 Command line tool for deduplicating CSV files
most recent commit 2 years ago
Sdfs ⭐ 315
Deduplication Based Filesystem
most recent commit 2 months ago
Dedupe Examples ⭐ 306
🆔 Examples for using the dedupe library
most recent commit 4 months ago
Bees ⭐ 294
Best-Effort Extent-Same, a btrfs dedupe agent
most recent commit a month ago
Warcprox ⭐ 289
WARC writing MITM HTTP/S proxy
total releases 24most recent commit a month ago
Bedup ⭐ 287
Btrfs deduplication
total releases 11most recent commit 2 years ago
Data Matching Software ⭐ 257
A list of free data matching and record linkage software.
most recent commit a month ago
Fast Sass Loader ⭐ 239
High performance sass loader for webpack
dependent packages 5,617total releases 27most recent commit 6 months agonpm fast-sass-loader} Downloads
Tagsistant ⭐ 231
Semantic filesystem for Linux, with relation reasoner, autotagging plugins and a deduplication service
most recent commit 4 years ago
Imgdupes ⭐ 220
Finding and deleting near-duplicate images based on perceptual hash.
total releases 28most recent commit 2 months ago
Pcompress ⭐ 219
A Parallelized Data Deduplication and Compression utility
most recent commit 3 years ago
Flim Springfield ⭐ 210
Analysis of The Simpsons
most recent commit 2 years ago
Elasticsearch Entity Resolution ⭐ 202
Elasticsearch entity resolution plugin based on Duke
most recent commit 5 years ago
Splink ⭐ 193
Implementation of Fellegi-Sunter's canonical model of record linkage in Apache Spark, including EM algorithm to estimate parameters
dependent packages 1total releases 56most recent commit 5 days agopypi splink} Downloads
Kvdo ⭐ 186
A pair of kernel modules which provide pools of deduplicated and/or compressed block storage.
most recent commit 3 months ago
Rabbitmq Message Deduplication ⭐ 178
RabbitMQ Plugin for filtering message duplicates
most recent commit 4 months ago
Dedup ⭐ 168
Streaming Deduplication Package for Go
most recent commit 4 years ago
Postgresql Patterns Library ⭐ 168
Коллекция готовых SQL запросов для PostgreSQL по часто возникающим задачам (получение и модификация данных, обслуживание БД)
most recent commit 6 hours ago
Dejavu ⭐ 153
Quickly detect already witnessed data.
most recent commit 5 years ago
Vdo ⭐ 148
Userspace tools for managing VDO volumes.
most recent commit 3 months ago
Lsh ⭐ 144
Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents
most recent commit 3 years ago
Nomenklatura ⭐ 141
Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources
dependent packages 1total releases 32most recent commit 20 hours agopypi nomenklatura} Downloads
Haxlsharp ⭐ 134
Automatically concurrent data fetching and request deduplication in C#.
dependent packages 1total releases 2most recent commit 6 years agonuget HaxlSharp.Core} Downloads
Spark Lucenerdd ⭐ 126
Spark RDD with Lucene's query and entity linkage capabilities
total releases 39most recent commit 9 hours ago
Mail Deduplicate ⭐ 123
📧 CLI to deduplicate mails from mail boxes.
total releases 13most recent commit 3 months ago
Dupe Krill ⭐ 123
A fast file deduplicator
total releases 10most recent commit 4 months ago
Encbup ⭐ 118
Encrypted backups (without the backups)
most recent commit 7 years ago
Benji ⭐ 114
Benji Backup: A block based deduplicating backup software for Ceph RBD images, iSCSI targets, image files and block devices
most recent commit a month ago
Go Rsync ⭐ 99
Best GTK+ frontend (backup application) for RSYNC utility.
total releases 7most recent commit 7 months ago
Entity Embed ⭐ 98
PyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.
total releases 6most recent commit a month ago
Pandas Dedupe ⭐ 93
Simplifies use of the Dedupe library via Pandas
total releases 12most recent commit 3 months ago
Fingerprints ⭐ 91
Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.
dependent packages 2total releases 22most recent commit a year agopypi fingerprints} Downloads
Destor ⭐ 91
An experimental platform for chunk-level data deduplication. Key words: DDFS, Sparse Index, Extreme Binning, SiLo, Sample Index, BLC; CBR, CFL, CAP, HAR; ASM, OPT; GC, Cumulus
most recent commit 6 years ago
Dedupfs ⭐ 90
A Python FUSE file system that features transparent deduplication and compression which make it ideal for archiving backups.
most recent commit 12 years ago
Dupd ⭐ 87
CLI utility to find duplicate files
most recent commit 8 months ago
Blobstash ⭐ 83
You personal database. Mirror of https://git.sr.ht/~tsileo/blobstash
most recent commit 2 years ago
Gencore ⭐ 82
Generate duplex/single consensus reads to reduce sequencing noises and remove duplications
most recent commit 7 months ago
Pinnacleapi Documentation ⭐ 81
Pinnacle API Documentation
most recent commit a month ago
Rltk ⭐ 81
Record Linkage ToolKit (Find and link entities)
dependent packages 3total releases 20most recent commit 8 months agopypi rltk} Downloads
Cargo Limit ⭐ 80
Cargo with less noise: warnings are skipped until errors are fixed, Neovim integration, etc.
total releases 9most recent commit 5 months ago
Laravel Console Logger ⭐ 80
Logging and Notifications for Laravel Console Commands.
total releases 110most recent commit 7 months ago
Rocketmqdeduplistener ⭐ 73
RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用
most recent commit 9 months ago
Lieu ⭐ 72
Dedupe/batch geocode addresses and venues around the world with libpostal
most recent commit 6 months ago
Py Image Dedup ⭐ 71
CLI utility to find near duplicate images and remove all but the best copy.
total releases 2most recent commit 20 days ago
Intraarchivededuplicator ⭐ 70
Tool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.
most recent commit 3 years ago
Moss ⭐ 67
The default package manager of Serpent OS
most recent commit a year ago
Dduper ⭐ 62
Fast block-level out-of-band BTRFS deduplication tool.
most recent commit a year ago
Horcrux ⭐ 61
The Dropbox for IPFS (without the icky stuff)
most recent commit 5 years ago
Sfmc Example Jb Custom Activity ⭐ 59
Custom activity examples for Journey Builder.
most recent commit 7 months ago
Webpack Deduplication Plugin ⭐ 59
Plugin for webpack that de-duplicates transitive dependencies in yarn and webpack-based projects.
total releases 8most recent commit a year ago
Vite Plugin Svelte ⭐ 58
Svelte integration for Vite, a fast web dev tool.
dependent packages 2total releases 28most recent commit a year agonpm vite-plugin-svelte} Downloads
Minihttp ⭐ 57
A small HTTP server.
most recent commit 2 years ago
Zpaqfranz ⭐ 55
Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
most recent commit 5 days ago
Kuromojin ⭐ 52
Provide a high-level wrapper for kuromoji.js. Cache/Promise API
dependent packages 40total releases 16most recent commit a year agonpm kuromojin} Downloads
Fastcdc Rs ⭐ 52
FastCDC implementation in Rust
dependent packages 5total releases 8most recent commit 4 months agocargo fastcdc} Downloads
Record Linkage Resources ⭐ 49
Resources for tackling record linkage / deduplication / data matching problems
most recent commit a year ago
Autoaction ⭐ 49
Declarative data loading and action calling within react-redux
most recent commit 2 years ago
Wretch Middlewares ⭐ 48
Collection of middlewares for the Wretch library. 🎁
most recent commit a month ago
Wget Lua ⭐ 47
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
most recent commit a month ago
Cyphertite ⭐ 45
A high-security scalable solution for online backups.
most recent commit 8 years ago
Es Dedupe ⭐ 43
Tool for removing duplicate documents from Elasticsearch
most recent commit 8 months ago
Quipucords ⭐ 42
A tool for discovery, inspection, collection/deduplication, and reporting on an IT environment
most recent commit 16 days ago
Edge Dev ⭐ 42
NexentaEdge DevOps Edition, scale-out S3, SWIFT, NFS and iSCSI/NBD unified storage solution
most recent commit 3 years ago
Zvault ⭐ 42
Deduplicating backup solution
most recent commit 4 years ago
Clonefile Dedup ⭐ 41
Use clonefile to deduplicate files on APFS.
most recent commit 2 years ago
Savefile ⭐ 40
An easy to use library to save arbitrary rust data-structures to disk (or serialize to any other stream)
most recent commit 3 months ago
Deduplication ⭐ 38
Fast multi-threaded content-dependent chunking deduplication for Buffers in C++ with a reference implementation in Javascript. Ships with extensive tests, a fuzz test and a benchmark.
most recent commit 2 years ago
Cnbuilder ⭐ 38
Yet another classname string builder (the fastest one)
most recent commit 15 hours ago
Deduplication Slides ⭐ 37
"1 + 1 = 1 or Record Deduplication with Python" Jupyter Notebook
most recent commit 2 years ago
1-100 of 623 projects
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,048,735)
Python  (796,103)
Java  (385,413)
Php  (284,589)
Typescript  (239,722)
C Plus Plus  (236,213)
Ruby  (221,696)
C  (180,396)
Shell  (169,594)
C Sharp  (162,015)
Golang  (156,089)
Swift  (64,698)
R  (57,248)
Objective C  (57,169)
Rust  (56,117)
Dart  (51,466)
Kotlin  (45,511)
Lua  (33,293)
Matlab  (30,891)
Perl  (30,483)
Language  (30,085)
Scala  (28,601)
Haskell  (20,260)
Clojure  (19,072)
Powershell  (18,797)
Bash  (18,606)
Coffeescript  (17,331)
Elixir  (16,565)
Assembly  (14,526)
Processing  (13,298)
Julia  (12,536)
Basic  (10,765)
Flow  (10,528)
Emacs Lisp  (10,368)
Erlang  (9,058)
Scheme  (8,437)
Groovy  (8,431)
Solidity  (7,611)
Ocaml  (6,668)
Pascal  (6,378)
Hack  (6,365)
Lisp  (6,340)
Fortran  (5,483)
Elm  (5,477)
Common Lisp  (5,277)
Dlang  (4,747)
Shell Script  (4,633)
Python Library  (4,446)
Crystal  (4,374)
Scripting  (4,203)
Top Projects
Freecodecamp ⭐ 346,432
996.icu ⭐ 262,320
Free Programming Books ⭐ 233,678
Coding Interview University ⭐ 220,499
Awesome ⭐ 202,578
Vue ⭐ 196,186
Developer Roadmap ⭐ 195,398
Public Apis ⭐ 193,925
React ⭐ 188,417
System Design Primer ⭐ 180,789
Tensorflow ⭐ 165,144
Bootstrap ⭐ 157,483
You Dont Know Js ⭐ 153,649
Cs Notes ⭐ 150,780
Ohmyzsh ⭐ 145,459
Build Your Own X ⭐ 142,246
Flutter ⭐ 140,926
Javascript Algorithms ⭐ 140,055
Python ⭐ 136,303
Gitignore ⭐ 133,450
Linux ⭐ 132,199
Vscode ⭐ 132,056
Awesome Python ⭐ 127,528
Javascript ⭐ 123,347
Javaguide ⭐ 121,667
Python 100 Days ⭐ 117,585
Computer Science ⭐ 113,544
Youtube Dl ⭐ 110,239
Fucking Algorithm ⭐ 106,048
The Art Of Command Line ⭐ 105,874
React Native ⭐ 102,862
Electron ⭐ 101,813
D3 ⭐ 100,901
Go ⭐ 99,493
30 Seconds Of Code ⭐ 95,528
Create React App ⭐ 95,174
Axios ⭐ 93,562
Awesome Selfhosted ⭐ 89,230
Kubernetes ⭐ 88,689
Free Programming Books Zh_cn ⭐ 88,386
Node ⭐ 87,823
Next.js ⭐ 87,114
Terminal ⭐ 83,251
Deno ⭐ 82,597
Three.js ⭐ 82,214
Angular ⭐ 81,516
Awesome Go ⭐ 81,089
Typescript ⭐ 80,876
Ant Design ⭐ 80,416
Material Ui ⭐ 78,638
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.