Scrapy Redis

Redis-based components for Scrapy.
Alternatives To Scrapy Redis
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Crawlab10,521
6 months ago1March 03, 201958bsd-3-clauseGo
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Scrapy Redis5,4881762112 days ago18July 26, 202229mitPython
Redis-based components for Scrapy.
Haipproxy5,384
12 years ago7June 18, 201844mitPython
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
Distribute_crawler3,176
7 years ago26Python
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
Scrapy Cluster1,1371828 months ago15December 23, 202017mitPython
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
Funpyspidersearchengine862
2 years ago3mitPython
Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索
Python Spider680
2 years agoapache-2.0Python
豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章
Spiderman498
a year ago3mitPython
基于 scrapy-redis 的通用分布式爬虫框架
Awesome_crawl206
4 years agoPython
腾讯新闻、知乎话题、微博粉丝,Tumblr爬虫、斗鱼弹幕、妹子图爬虫、分布式设计等
Zi5book183
5 years agoPython
book.zi5.me全站kindle电子书籍爬取,按照作者书籍名分类,每本书有mobi和equb两种格式,采用分布式进行全站爬取
Alternatives To Scrapy Redis
Select To Compare


Alternative Project Comparisons
Popular Scrapy Projects
Popular Redis Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Redis
Crawler
Scrapy