Deduplication Open Source Projects

Fast, secure, efficient backup program

Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

Deduplicating archiver with compression and authenticated encryption.

Prometheus Alertmanager

Find duplicate files

A new generation cloud backup tool

😎 Finding duplicate images made easy!

A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.

:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.

rustic - fast, encrypted, and deduplicated backups powered by Rust

A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows

Extremely fast tool to remove duplicates and other lint from your filesystem

Simple, configuration-driven backup software for servers and workstations

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

Config driven, easy backup cli for restic.

A powerful duplicate file finder and an enhanced fork of 'fdupes'.

Capture your screen to a GIF in your browser

Zero-details, privacy-focused in-app file system.

Scalable master data management, identity resolution, entity resolution, and deduplication using ML

Deduplicating backup program

A powerful and modular toolkit for record linkage and duplicate detection in Python

Tools for deduping file systems

Best-Effort Extent-Same, a btrfs dedupe agent

Data deduplication engine, supporting optional compression and public key encryption.

ZBackup, a versatile deduplicating backup tool

Simple bash script to rebalance pool data between all mirrors when adding vdevs to a pool.

Коллекция готовых SQL запросов для PostgreSQL по часто возникающим задачам (получение и модификация данных, ускорение запросов, обслуживание БД)

Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.

Instantly transfer files between Dropbox accounts using only their hashes.

Duke is a fast and flexible deduplication engine written in Java

Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix

WARC writing MITM HTTP/S proxy

A list of free data matching and record linkage software.

Deduplication Based Filesystem

Filter, Sort & Delete Duplicate Files Recursively

A secure and efficient file backup solution that fits both system administrators (CLI) and end users (GUI)

RabbitMQ Plugin for filtering message duplicates

Btrfs deduplication

Semantic filesystem for Linux, with relation reasoner, autotagging plugins and a deduplication service

Enable deduplication with non-Synology SSDs and unsupported NAS models

A high-performance tool for video upscaling, interpolation, depth estimation, and more. Available as a CLI and Adobe Extension.

Cargo with less noise: warnings are skipped until errors are fixed, LSP-independent Neovim integration, etc.

Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents

A Parallelized Data Deduplication and Compression utility

Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources

A kernel module which provide a pool of deduplicated and/or compressed block storage.

RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用

A batch manager that will deduplicate and batch requests for a certain data type made within a window. Useful to batch requests made from multiple react components that uses react-query

Elasticsearch entity resolution plugin based on Duke

A fast file deduplicator

FastCDC implementation in Rust

Streaming Deduplication Package for Go

Userspace tools for managing VDO volumes.

Fast block-level out-of-band BTRFS deduplication tool.

📧 CLI to deduplicate mails from mail boxes

Duplicates Detector is a cross-platform GUI utility for finding duplicate files, allowing you to delete or link them to save space. Duplicate files are displayed and processed on two synchronized panels for efficient and convenient operation.

CLI utility to find near duplicate images and remove all but the best copy.

An experimental platform for chunk-level data deduplication. Key words: DDFS, Sparse Index, Extreme Binning, SiLo, Sample Index, BLC; CBR, CFL, CAP, HAR; ASM, OPT; GC, Cumulus

Quick and dirty backup tool benchmark with reproducible results

Pinnacle API Documentation