Open Source Crawler
Web crawler finding open source GitHub repositories, parsing README files, and scanning for typo/security issues.
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Crawlab | 10,521 | 4 months ago | 1 | March 03, 2019 | 58 | bsd-3-clause | Go | |||
| Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架 | ||||||||||
| Requests Cache | 1,244 |  | 701 | 466 | 2 months ago | 90 | November 19, 2023 | 26 | bsd-2-clause | Python |
| Persistent HTTP cache for python requests | ||||||||||
| Letterboxd_recommendations | 190 | 4 months ago | 7 | gpl-3.0 | Python | |||||
| Scraping publicly-accessible Letterboxd data and creating a movie recommendation model with it that can generate recommendations when provided with a Letterboxd username | ||||||||||
| Zhihu Crawler People | 179 | 4 years ago | 2 | gpl-2.0 | Python | |||||
| A simple distributed crawler for zhihu && data analysis | ||||||||||
| Zhihu_crawler | 100 | 7 years ago | 2 | apache-2.0 | Python | |||||
| a crawler for zhihu | ||||||||||
| Polipus | 95 |  | 3 | 3 | 9 years ago | 20 | July 17, 2015 | 13 | mit | Ruby |
| Polipus: distributed and scalable web-crawler framework | ||||||||||
| Scrapyd Cluster On Heroku | 90 | 4 years ago | 1 | gpl-3.0 | Python | |||||
| Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO :point_right: | ||||||||||
| Tspider | 71 | 7 years ago | 1 | Python | ||||||
| Yet Another Web Spider | ||||||||||
| Leek | 61 | 9 months ago | 64 | August 02, 2022 | 1 | mit | Python | |||
| Distributed task redisqueue(最简单python分布式函数调度框架) | ||||||||||
| Maman | 40 | 3 years ago | 16 | December 01, 2018 | mit | Rust | ||||
| Rust Web Crawler saving pages on Redis | ||||||||||
Alternatives To Open Source CrawlerSelect To Compare
Alternative Project Comparisons
Popular Web Crawler Projects
Popular Redis Projects
Popular Data Processing Categories
Related Searches
Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Javascript
Redis
Aws Lambda
Web Crawler