Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Scrapy | 49,918 | 4,185 | 445 | 5 months ago | 96 | September 18, 2023 | 692 | bsd-3-clause | Python | |
Scrapy, a fast high-level web crawling & scraping framework for Python. | ||||||||||
Crawlab | 10,521 | 6 months ago | 1 | March 03, 2019 | 58 | bsd-3-clause | Go | |||
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架 | ||||||||||
Awesome Crawler | 5,859 | 7 months ago | 27 | mit | ||||||
A collection of awesome web crawler,spider in different languages | ||||||||||
Wechatsogou | 5,822 | 2 | 7 months ago | 25 | April 10, 2019 | 81 | apache-2.0 | Python | ||
基于搜狗微信搜索的微信公众号爬虫接口 | ||||||||||
Scrapy Redis | 5,468 | 176 | 21 | a month ago | 18 | July 26, 2022 | 29 | mit | Python | |
Redis-based components for Scrapy. | ||||||||||
Haipproxy | 5,384 | 1 | 2 years ago | 7 | June 18, 2018 | 44 | mit | Python | ||
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis | ||||||||||
Ecommercecrawlers | 3,724 | a year ago | 43 | mit | Python | |||||
实战🐍多种网站、电商数据爬虫🕷。包含🕸:淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目: | ||||||||||
Distribute_crawler | 3,176 | 7 years ago | 26 | Python | ||||||
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现 | ||||||||||
Gerapy | 3,144 | 8 | 6 months ago | 49 | July 19, 2023 | 60 | mit | Python | ||
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js | ||||||||||
Python3 Spider | 2,582 | 8 months ago | 6 | Python | ||||||
Python爬虫实战 - 模拟登陆各大网站 包含但不限于:滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝,如果喜欢请start ❤️ |