Crawlerflow
Web Crawlers orchestration framework that lets you create datasets from multiple web sources using yaml configurations.
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Crawlab | 10,521 | 4 months ago | 1 | March 03, 2019 | 58 | bsd-3-clause | Go | |||
| Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架 | ||||||||||
| Distribute_crawler | 3,176 | 7 years ago | 26 | Python | ||||||
| 使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现 | ||||||||||
| Lianjia Beike Spider | 2,464 | 10 months ago | 13 | Python | ||||||
| 链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 ，点星支持，仅供学习参考，请勿用于商业用途，后果自负。 | ||||||||||
| Anemone | 1,615 |  | 385 | 34 | 4 years ago | 23 | May 30, 2012 | 55 | mit | Ruby |
| Anemone web-spider framework | ||||||||||
| Weixin Game Helper | 1,352 | 10 months ago | 24 | gpl-3.0 | JavaScript | |||||
| 微信小游戏辅助合集（加减大师、包你懂我、大家来找茬腾讯版、头脑王者、好友画我、悦动音符、我最在行、星途WeGoing、猜画小歌、知乎答题王、腾讯中国象棋、跳一跳、题多多黄金版） | ||||||||||
| Zhihu Crawler | 843 | 5 years ago | 2 | other | Java | |||||
| zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目 | ||||||||||
| Ppspider | 278 |  | 1 | 2 | 2 years ago | 85 | December 07, 2020 | 5 | mit | TypeScript |
| web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（nedb/mongodb），提供数据可视化和用户交互的实现方案 | ||||||||||
| Taobao_bra_crawler | 189 | 5 years ago | December 26, 2023 | mit | Python | |||||
| a taobao web crawler just for fun. | ||||||||||
| Zhihu Crawler People | 179 | 4 years ago | 2 | gpl-2.0 | Python | |||||
| A simple distributed crawler for zhihu && data analysis | ||||||||||
| Github_commit_crawler | 167 | 8 years ago | 7 | Python | ||||||
| Tool used to continuously monitor a Github org for mistaken public commits | ||||||||||
Alternatives To CrawlerflowSelect To Compare
Alternative Project Comparisons
Popular Crawler Projects
Popular Mongodb Projects
Popular Data Processing Categories
Related Searches
Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Mongodb
Elasticsearch
Crawler
Scrapy