Scrapy Cluster
modified by http://github.com/istresearch/scrapy-cluster.git to make it appropriate for project.
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Scrapy Cluster | 1,137 |  | 18 | 2 | 6 months ago | 15 | December 23, 2020 | 17 | mit | Python |
| This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster. | ||||||||||
| Dataengineeringproject | 644 | a year ago | 4 | mit | Python | |||||
| Example end to end data engineering project. | ||||||||||
| Jikan Rest | 391 | 3 months ago | 33 | mit | PHP | |||||
| The REST API for Jikan | ||||||||||
| Letterboxd_recommendations | 190 | 3 months ago | 7 | gpl-3.0 | Python | |||||
| Scraping publicly-accessible Letterboxd data and creating a movie recommendation model with it that can generate recommendations when provided with a Letterboxd username | ||||||||||
| Awesome Python Primer | 78 | 2 years ago | mit | Python | ||||||
| 自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向 | ||||||||||
| Collyzar | 65 | 3 years ago | 2 | January 31, 2021 | Go | |||||
| Distributed redis-based web crawler framework for colly | ||||||||||
| Scraper Boilerplate | 54 | 3 years ago | Python | |||||||
| Scrapy Distributed | 40 | a year ago | 8 | February 20, 2021 | 10 | Python | ||||
| A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy. | ||||||||||
| Scrapy Kafka Redis | 35 | 3 years ago | 5 | July 24, 2018 | apache-2.0 | Python | ||||
| Distributed crawling/scraping, Kafka And Redis based components for Scrapy | ||||||||||
| Crawler | 32 | 4 years ago | JavaScript | |||||||
| Chromium / Puppeteer site crawler | ||||||||||
Alternatives To Scrapy ClusterSelect To Compare
Alternative Project Comparisons
Popular Redis Projects
Popular Scraper Projects
Popular Data Storage Categories
Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Redis
Scraper
Kafka
Crawler
Spider
Scrapy