Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Proxy_pool | 19,442 | 3 months ago | 273 | mit | Python | |||||
Python ProxyPool for web spider | ||||||||||
Scrapy Examples | 2,550 | 6 years ago | 6 | Python | ||||||
Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc. | ||||||||||
Gain | 1,972 | 1 | 5 years ago | 5 | June 19, 2017 | 8 | gpl-3.0 | Python | ||
Web crawling framework based on asyncio. | ||||||||||
Pspider | 1,675 | 2 years ago | 1 | bsd-2-clause | Python | |||||
简单易用的Python爬虫框架,QQ交流群:597510560 | ||||||||||
Scrapy Rotating Proxies | 474 | 8 | 1 | 3 years ago | 13 | May 25, 2019 | 40 | mit | Python | |
use multiple proxies with Scrapy | ||||||||||
Free_proxy_website | 333 | 4 years ago | 2 | Python | ||||||
获取免费socks/https/http代理的网站集合 | ||||||||||
Httpproxymiddleware | 318 | 6 years ago | mit | Python | ||||||
A middleware for scrapy. Used to change HTTP proxy from time to time. | ||||||||||
Ppspider | 278 | 1 | 2 | 2 years ago | 85 | December 07, 2020 | 5 | mit | TypeScript | |
web spider built by puppeteer, support task-queue and task-scheduling by decorators,support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架,提供灵活的任务队列管理调度方案,提供便捷的数据保存方案(nedb/mongodb),提供数据可视化和用户交互的实现方案 | ||||||||||
Sasila | 264 | 4 years ago | 17 | November 02, 2017 | 1 | apache-2.0 | Python | |||
一个灵活、友好的爬虫框架 | ||||||||||
Lagoujob | 250 | 5 years ago | apache-2.0 | Python | ||||||
Job data mining repo for lagou.com |