Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Redis Crawler Open Source Projects

Open source projects categorized as Redis Crawler
Categories > Data Processing > Crawler
Categories > Data Storage > Redis
Proxy_pool ⭐ 19,442
Python ProxyPool for web spider
most recent commit 4 months ago
Pyspider ⭐ 15,943
A Powerful Spider(Web Crawler) System in Python.
dependent packages 2total releases 17most recent commit 10 months agopypi pyspider} Downloads
Crawlab ⭐ 10,521
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
total releases 1most recent commit 3 months ago
Scrapy Redis ⭐ 5,438
Redis-based components for Scrapy.
dependent packages 21total releases 18most recent commit 5 months agopypi scrapy-redis} Downloads
Haipproxy ⭐ 5,384
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
total releases 7most recent commit a year ago
Proxypool ⭐ 5,154
An Efficient ProxyPool with Getter, Tester and Server
most recent commit 4 months ago
Distribute_crawler ⭐ 3,176
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re
most recent commit 7 years ago
Anemone ⭐ 1,615
Anemone web-spider framework
dependent packages 34total releases 23most recent commit 4 years agorubygems anemone} Downloads
Scrapy Cluster ⭐ 1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15most recent commit 6 months agopypi scutils} Downloads
Listed Company News Crawl And Text Analysis ⭐ 689
从新浪财经、每经网、金融界、中国证券网、证券时报网上，爬取上市公司（个股）的历史新闻文本数据进行文本
most recent commit a year ago
Magnet Dht ⭐ 591
✌️ Python3 BitTorrent DHT crawler
most recent commit 5 years ago
Netdiscovery ⭐ 557
NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。
most recent commit 3 years ago
Go_jobs ⭐ 536
带你了解一下Golang的市场行情
total releases 1most recent commit 4 years ago
Go Movies ⭐ 232
golang spider Crawler 爬虫 电影
most recent commit 5 months ago
Awesome_crawl ⭐ 206
腾讯新闻、知乎话题、微博粉丝，Tumblr爬虫、斗鱼弹幕、妹子图爬虫、分布式设计等
most recent commit 4 years ago
Zhihu Crawler People ⭐ 179
A simple distributed crawler for zhihu && data analysis
most recent commit 4 years ago
Spoon ⭐ 177
🥄 A package for building specific Proxy Pool for different Sites.
total releases 23most recent commit 2 years ago
Weibosearch ⭐ 144
A distributed Sina Weibo Search spider base on Scrapy and Redis.
most recent commit 11 years ago
Poopak ⭐ 110
POOPAK - TOR Hidden Service Crawler
most recent commit a year ago
Docs ⭐ 102
《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志
most recent commit 5 years ago
Zhihu_crawler ⭐ 100
a crawler for zhihu
most recent commit 7 years ago
Polipus ⭐ 95
Polipus: distributed and scalable web-crawler framework
dependent packages 3total releases 20most recent commit 9 years agorubygems polipus} Downloads
Scrapyd Cluster On Heroku ⭐ 90
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉
most recent commit 4 years ago
The 8 Most Depended On Redis Crawler Open Source Projects
Anemone ⭐ 1,615
Anemone web-spider framework
dependent packages 34total releases 23latest release May 30, 2012most recent commit 4 years agorubygems anemone} Downloads
Scrapy Redis ⭐ 5,438
Redis-based components for Scrapy.
dependent packages 21total releases 18latest release July 26, 2022most recent commit 5 months agopypi scrapy-redis} Downloads
Seenreq ⭐ 32
Generate an object for testing if a request is sent, request is Mikeal's request.
dependent packages 11total releases 26latest release April 15, 2019most recent commit 5 years agonpm seenreq} Downloads
Polipus ⭐ 95
Polipus: distributed and scalable web-crawler framework
dependent packages 3total releases 20latest release July 17, 2015most recent commit 9 years agorubygems polipus} Downloads
Pyspider ⭐ 15,943
A Powerful Spider(Web Crawler) System in Python.
dependent packages 2total releases 17latest release April 18, 2018most recent commit 10 months agopypi pyspider} Downloads
Ciao Ssr ⭐ 12
A server side render service based on puppeteer
dependent packages 2total releases 5latest release August 15, 2018most recent commit 2 years agonpm puppeteer-server-side-render} Downloads
Scrapy Cluster ⭐ 1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15latest release December 23, 2020most recent commit 6 months agopypi scutils} Downloads
The 10 Latest Releases In Redis Crawler Open Source Projects
Laravel Block Bots ⭐ 34
Block crawlers and high traffic users on your site by IP using Redis
total releases 28latest release September 21, 2023most recent commit 7 months ago
Deva ⭐ 9
函数流，数据流，同步异步，事件驱动，多种模式channel
total releases 38latest release September 21, 2023most recent commit 7 months ago
Scrapy Redis ⭐ 5,438
Redis-based components for Scrapy.
dependent packages 21total releases 18latest release July 26, 2022most recent commit 5 months agopypi scrapy-redis} Downloads
Soundclouder ⭐ 11
:musical_keyboard: Crawling SoundCloud at scale (Go + Redis)
latest release April 08, 2022most recent commit 8 years ago
Collyzar ⭐ 65
Distributed redis-based web crawler framework for colly
total releases 2latest release January 31, 2021most recent commit 3 years ago
Scrapy Cluster ⭐ 1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15latest release December 23, 2020most recent commit 6 months agopypi scutils} Downloads
Airspider ⭐ 8
A Fast and Light Python Spider Framework 🕷️
total releases 6latest release September 02, 2020most recent commit 4 years ago
Go_jobs ⭐ 536
带你了解一下Golang的市场行情
total releases 1latest release May 17, 2020most recent commit 4 years ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.