Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Learn_python3_spider | 14,425 | 5 months ago | 2 | August 07, 2019 | 29 | mit | Python | |||
python爬虫教程系列、从0到1学习python爬虫,包括浏览器抓包,手机APP抓包,如 fiddler、mitmproxy,各种爬虫涉及的模块的使用,如:requests、beautifulSoup、selenium、appium、scrapy等,以及IP代理,验证码识别,Mysql,MongoDB数据库的python使用,多线程多进程爬虫的使用,css 爬虫加密逆向破解,JS爬虫逆向,分布式爬虫,爬虫项目实战实例等 | ||||||||||
Crawlab | 10,521 | 5 months ago | 1 | March 03, 2019 | 58 | bsd-3-clause | Go | |||
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架 | ||||||||||
Awesome Crawler | 5,859 | 6 months ago | 27 | mit | ||||||
A collection of awesome web crawler,spider in different languages | ||||||||||
Haipproxy | 5,384 | 1 | a year ago | 7 | June 18, 2018 | 44 | mit | Python | ||
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis | ||||||||||
Ecommercecrawlers | 3,724 | a year ago | 43 | mit | Python | |||||
实战🐍多种网站、电商数据爬虫🕷。包含🕸:淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目: | ||||||||||
Weibospider | 3,294 | 5 months ago | 7 | mit | Python | |||||
持续维护的新浪微博采集工具🚀🚀🚀 | ||||||||||
Distribute_crawler | 3,176 | 7 years ago | 26 | Python | ||||||
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现 | ||||||||||
Gerapy | 3,144 | 8 | 5 months ago | 49 | July 19, 2023 | 60 | mit | Python | ||
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js | ||||||||||
Scrapydweb | 2,839 | 3 | 8 months ago | 18 | August 31, 2023 | 56 | gpl-3.0 | Python | ||
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO :point_right: | ||||||||||
Scrapyd | 2,766 | 187 | 15 | 5 months ago | 11 | September 25, 2023 | 31 | bsd-3-clause | Python | |
A service daemon to run Scrapy spiders |