|Project Name||Stars||Downloads||Repos Using This||Packages Using This||Most Recent Commit||Total Releases||Latest Release||Open Issues||License||Language|
|Awesome Postgres||8,644||a month ago||64||cc0-1.0|
|A curated list of awesome PostgreSQL software, libraries, tools and resources, inspired by awesome-mysql|
|Wal E||3,327||14||a year ago||33||February 04, 2020||91||bsd-3-clause||Python|
|Continuous Archiving for Postgres|
|Pghoard||1,213||1||a month ago||17||May 27, 2019||42||apache-2.0||Python|
|PostgreSQL® backup and restore service|
|Gitlab Mirrors||743||2 years ago||19||mit||Shell|
|A set of scripts adding the ability of managing remote mirrors to GitLab.|
|Zrepl||602||12 days ago||23||February 15, 2022||175||mit||Go|
|One-stop ZFS backup & replication solution|
|Replication Manager||584||2 days ago||24||March 09, 2020||100||gpl-3.0||Go|
|Signal 18 repman - Replication Manager for MySQL / MariaDB / Percona Server|
|Awesome Storage||423||2 months ago||6||mit|
|A curated list of storage open source tools. Backups, redundancy, sharing, distribution, encryption, etc.|
|Zfs_autobackup||391||2 months ago||37||July 04, 2022||17||gpl-3.0||Python|
|ZFS autobackup is used to periodicly backup ZFS filesystems to other locations. Easy to use and very reliable.|
|Storagetapper||269||a year ago||4||November 19, 2021||21||mit||Go|
|StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service|
|Pglogrepl||172||a month ago||11||mit||Go|
|PostgreSQL logical replication library for Go.|
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service.
Storagetapper is deployed in production at Uber and used to produce snapshot and realtime changed data of thousands of MySQL tables across multiple datacenters.
It is also used as a backup service to snapshot hundreds of terrabytes of Schemaless data to HDFS and S3 with optional asymmetric encryption and compression.
It reads data from source transforms according to the specified event format and produces data to destination.
Supported event sources:
Supported event destinations:
Supported event formats:
Storagetapper keeps its jobs state in MySQL database and automatically distributes jobs between configured number of workers.
It is also aware of node roles and takes snapshot from the slave nodes in order to reduce load on master nodes. StorageTapper can also optionally further throttle its reads. Binlogs are streamed from master nodes for better SLAs.
Service is dynamically configurable through RESTful API or built-in UI.
cd storagetapper make deb && dpkg -i ../storagetapper_1.0_amd64.deb
cd storagetapper make && make install
/bin/bash scripts/install_deps.sh # install all dependencies: MySQL, Kafka, HDFS, S3, ... make test # run all tests GO111MODULE=on TEST_PARAM="-test.run=TestLocalBasic" /bin/bash scripts/run_tests.sh ./pipe # individual test
make test-env $ make test
Storagetapper loads configuration from the following files and location in the given order:
/etc/storagetapper/base.yaml /etc/storagetapper/production.yaml $(HOME)/base.yaml $(HOME)/production.yaml $(STORAGETAPPER_CONFIG_DIR)/base.yaml $(STORAGETAPPER_CONFIG_DIR)/production.yaml
Available options described in Configuration section
This software is licensed under the MIT License.