Roach
A very adaptable web crawler framework. Impossible to kill.
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Spring Boot Quick | 2,282 | 8 months ago | 13 | Java | ||||||
| :herb: 基于springboot的快速学习示例,整合自己遇到的开源框架,如：rabbitmq(延迟队列)、Kafka、jpa、redies、oauth2、swagger、jsp、docker、k3s、k3d、k8s、mybatis加解密插件、异常处理、日志输出、多模块开发、多环境打包、缓存cache、爬虫、jwt、GraphQL、dubbo、zookeeper和Async等等:pushpin: | ||||||||||
| Ipfs Search | 779 | 1 | a year ago | 19 | April 20, 2021 | 42 | agpl-3.0 | Go | ||
| Search engine for the Interplanetary Filesystem. | ||||||||||
| Jlitespider | 151 | 7 years ago | 3 | February 12, 2017 | 2 | apache-2.0 | Java | |||
| A lite distributed Java spider framework :-) | ||||||||||
| Gocrawler | 60 | 4 years ago | mit | Go | ||||||
| A distributed web crawler implemented using Go, Postgres, RabbitMQ and Docker | ||||||||||
| Umbra | 54 | 4 years ago | 7 | apache-2.0 | Python | |||||
| A queue-controlled browser automation tool for improving web crawl quality | ||||||||||
| Findopendata | 49 | a year ago | 6 | December 26, 2019 | 3 | apache-2.0 | Python | |||
| A search engine for Open Data | ||||||||||
| Docker Django Celery Tutorial | 45 | 2 years ago | 2 | mit | Python | |||||
| docker-django-celery-tutorial 基本教學 📝 | ||||||||||
| Scrapy Distributed | 40 | a year ago | 8 | February 20, 2021 | 10 | Python | ||||
| A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy. | ||||||||||
| Go Crawler Distributed | 39 | 8 months ago | mit | Go | ||||||
| 分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微服务架构，通过消息队列实现消息的异步发送，使用到的框架包括：redigo, gorm, goquery, easyjson, viper, amqp, zap, go-micro，并通过Docker实现容器化部署，中间爬虫节点支持水平拓展。 | ||||||||||
| Crawler | 23 | 12 years ago | gpl-3.0 | Python | ||||||
| Toy web crawler | ||||||||||
Alternatives To RoachSelect To Compare
Alternative Project Comparisons
Popular Crawler Projects
Popular Rabbitmq Projects
Popular Data Processing Categories
Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Javascript
Crawler
Rabbitmq
Transport
Moment