Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Redis Scrapy Open Source Projects

Open source projects categorized as Redis Scrapy
Crawlab ⭐ 10,521
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
total releases 1most recent commit 4 months ago
Scrapy Redis ⭐ 5,438
Redis-based components for Scrapy.
dependent packages 21total releases 18most recent commit 5 months agopypi scrapy-redis} Downloads
Haipproxy ⭐ 5,384
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
total releases 7most recent commit a year ago
Distribute_crawler ⭐ 3,176
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re
most recent commit 7 years ago
Scrapy Cluster ⭐ 1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15most recent commit 6 months agopypi scutils} Downloads
Funpyspidersearchengine ⭐ 862
Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索
most recent commit 2 years ago
Python Spider ⭐ 680
豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红
most recent commit 2 years ago
Spiderman ⭐ 498
基于 scrapy-redis 的通用分布式爬虫框架
most recent commit a year ago
Awesome_crawl ⭐ 206
腾讯新闻、知乎话题、微博粉丝，Tumblr爬虫、斗鱼弹幕、妹子图爬虫、分布式设计等
most recent commit 4 years ago
Zi5book ⭐ 183
book.zi5.me全站kindle电子书籍爬取，按照作者书籍名分类，每本书有mobi和equb两
most recent commit 5 years ago
Zhihuspider ⭐ 149
知乎分布式爬虫（Scrapy、Redis）
most recent commit 6 years ago
Crawlerproject ⭐ 147
爬虫项目：链家网（普通/scrapy）、虎扑、维基百科、百度地图api、房天下（分布式爬虫）、微信公
most recent commit 2 years ago
Du App Sign ⭐ 145
毒app(得物app)逆向js破解sign参数，获取毒app品牌列表页数据，毒app商品详情页数据， 毒app关键词搜索数据，鞋穿不炒😝
most recent commit 4 years ago
Scrapyredisbloomfilter ⭐ 144
Scrapy Redis Bloom Filter
total releases 5most recent commit 3 years ago
Weibosearch ⭐ 144
A distributed Sina Weibo Search spider base on Scrapy and Redis.
most recent commit 11 years ago
Scrapy Cve Cnvd ⭐ 127
漏洞监控，基于scrapy，scrapy-redis，获取每日最新的CVE和CNVD漏洞，邮件通知
most recent commit 2 years ago
Python3 Scrapy Spider Phantomjs Selenium ⭐ 125
基于Python3的动态网站爬虫，使用selenium+phantomjs实现爬取动态网站, 本项目以爬取今日头条为例
most recent commit 3 years ago
Datamining And Social Sentiment Analysis Based On Weibo ⭐ 121
基于微博的数据挖掘与社交舆情分析
most recent commit 6 years ago
Docs ⭐ 102
《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志
most recent commit 5 years ago
Hive ⭐ 101
lots of spider (很多爬虫）
most recent commit 5 years ago
Scrapy Monitor ⭐ 99
scrapy-monitor，实现爬虫可视化，监控实时状态
most recent commit 7 years ago
Scrapyd Cluster On Heroku ⭐ 90
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉
most recent commit 4 years ago
Nscrapy ⭐ 82
NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider
most recent commit 2 years ago
The 2 Most Depended On Redis Scrapy Open Source Projects
Scrapy Redis ⭐ 5,438
Redis-based components for Scrapy.
dependent packages 21total releases 18latest release July 26, 2022most recent commit 5 months agopypi scrapy-redis} Downloads
Scrapy Cluster ⭐ 1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15latest release December 23, 2020most recent commit 6 months agopypi scutils} Downloads
The 10 Latest Releases In Redis Scrapy Open Source Projects
Scrapy Redis ⭐ 5,438
Redis-based components for Scrapy.
dependent packages 21total releases 18latest release July 26, 2022most recent commit 5 months agopypi scrapy-redis} Downloads
Scrapyredisbloomfilter ⭐ 144
Scrapy Redis Bloom Filter
total releases 5latest release January 03, 2021most recent commit 3 years ago
Scrapy Cluster ⭐ 1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15latest release December 23, 2020most recent commit 6 months agopypi scutils} Downloads
Scrapy_plus ⭐ 17
scrapy 常用爬网必备工具包
total releases 6latest release December 17, 2020most recent commit 2 years ago
Scrapyredisbloomfilterblockcluster ⭐ 18
Scrapy Redis with Bloom Filter，support redis sentinel and cluster
total releases 12latest release December 26, 2019most recent commit 2 years ago
Scrapy Kafka Redis ⭐ 35
Distributed crawling/scraping, Kafka And Redis based components for Scrapy
total releases 5latest release July 24, 2018most recent commit 3 years ago
Haipproxy ⭐ 5,384
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
total releases 7latest release June 18, 2018most recent commit a year ago
Popular Data Storage Categories
Database
Mysql
Redis
Mongodb
Postgresql
Firebase
Elasticsearch
Backup
Sqlite
Archive
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.