Project Name	Stars	Repos Using This	Most Recent Commit	Total Releases	Latest Release	Open Issues	License	Language
Scrapy S3pipeline	66	1	2 years ago	8	January 31, 2021	1	mit	Python
Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.
Scrapy History Middleware	9		7 years ago			1	mit	Python
Scrapy middleware for building historical S3 cache for crawled web resources.
Scrapy Dotpersistence	8	9	5 years ago	4	August 04, 2016	1	bsd-3-clause	Python
A scrapy extension to sync `.scrapy` folder to an S3 bucket

Alternatives To Scrapy Dotpersistence

Select To Compare

Scrapy S3pipeline ⭐ 66

Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.

total releases 8most recent commit 2 years ago

Scrapy History Middleware ⭐ 9

Scrapy middleware for building historical S3 cache for crawled web resources.

most recent commit 7 years ago

Scrapy Dotpersistence ⭐ 8

A scrapy extension to sync `.scrapy` folder to an S3 bucket

total releases 4most recent commit 5 years ago

Suggest An Alternative To scrapy-dotpersistence

Alternative Project Comparisons

Scrapy Dotpersistence vs Scrapy S3pipeline

Scrapy Dotpersistence vs Scrapy History Middleware

Popular S3 Projects

Rclone ⭐ 43,997

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files

dependent packages 43total releases 268latest release November 26, 2023most recent commit 25 days ago

Minio ⭐ 42,715

The Object Store for AI Data Infrastructure

dependent packages 129total releases 377latest release December 01, 2023most recent commit 4 months ago

Seaweedfs ⭐ 21,236

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

dependent packages 2total releases 296latest release April 24, 2021most recent commit 15 days ago

Transfer.sh ⭐ 14,640

Easy and fast file sharing from the command-line.

dependent packages 2total releases 27latest release December 04, 2023most recent commit 5 months ago

Siyuan ⭐ 14,236

A privacy-first, self-hosted, fully open source personal knowledge management software, written in typescript and golang.

total releases 1latest release July 07, 2022most recent commit 4 months ago

Popular Scrapy Projects

Scrapy ⭐ 49,918

Scrapy, a fast high-level web crawling & scraping framework for Python.

dependent packages 445total releases 96latest release September 18, 2023most recent commit 4 months ago

Learn_python3_spider ⭐ 14,425

python爬虫教程系列、从0到1学习python爬虫，包括浏览器抓包，手机APP抓包，如 fiddler、mitmproxy，各种爬虫涉及的模块的使用，如：requests、beautifu 爬虫加密逆向破解，JS爬虫逆向，分布式爬虫，爬虫项目实战实例等

total releases 2latest release August 07, 2019most recent commit 5 months ago

Cs Book ⭐ 11,024

计算机类常用电子书整理，并且附带下载链接，包括Java，Python，Linux，Go，C，C++，

most recent commit 10 months ago

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

total releases 1latest release March 03, 2019most recent commit 5 months ago

Portia ⭐ 8,982

Visual scraping for Scrapy

dependent packages 2total releases 26latest release May 25, 2015most recent commit 9 months ago

Popular Cloud Computing Categories

Google Cloud Platform

Aws Lambda

Aws Ecs

Get A Weekly Email With Trending Projects For These Categories

No Spam. Unsubscribe easily at any time.

Python

Scrapy

S3 Bucket

Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.