Awesome Open Source

Programming Languages

Search results for redis web crawler

11 search results found

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Requests Cache ⭐ 1,244

Persistent HTTP cache for python requests

Letterboxd_recommendations ⭐ 190

Scraping publicly-accessible Letterboxd data and creating a movie recommendation model with it that can generate recommendations when provided with a Letterboxd username

Zhihu Crawler People ⭐ 179

A simple distributed crawler for zhihu && data analysis

Zhihu_crawler ⭐ 100

a crawler for zhihu

Polipus: distributed and scalable web-crawler framework

Scrapyd Cluster On Heroku ⭐ 90

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉

Yet Another Web Spider

Distributed task redisqueue(最简单python分布式函数调度框架)

Rust Web Crawler saving pages on Redis

Scrapybook ⭐ 21

精通Scrapy网络爬虫

Pubcrawl ⭐ 18

*Deprecated* A short and sweet Python web crawler using Redis as the process queue, seen set and Memcache style rate limiter for robots.txt

Robots.txt ⭐ 13

🤖 robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

Opennews ⭐ 11

OpenNews is a REST API made in Python to extract news from Portuguese journals. It is intended for academic use.

Scrapy_book_code ⭐ 8

《精通 Scrapy 网络爬虫》刘硕书中源代码

Xiaohongshu Spider Visualizer ⭐ 7

A distributed web crawler for xiaohongshu.com and visualization for the crawled content.

Anemone_lite ⭐ 7

Distributed web crawler using mongodb

Open Source Crawler ⭐ 5

Web crawler finding open source GitHub repositories, parsing README files, and scanning for typo/security issues.

Related Searches

Javascript Redis (4,842)

Java Redis (3,565)

Python Redis (3,473)

Docker Redis (2,405)

Mysql Redis (2,199)

Golang Redis (2,049)

Php Redis (1,915)

Redis Spring (1,500)

Mongodb Redis (1,419)

Scraper Web Crawler (1,388)

1-11 of 11 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.