Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About
Search results for kafka crawler
crawler
x
kafka
x
13 search results found
Frontera
⭐
1,244
A scalable frontier for web crawlers
Scrapy Cluster
⭐
1,137
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
Netdiscovery
⭐
557
NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。
Scrapy_demo
⭐
150
all kinds of scrapy demo
Dodder
⭐
71
A distributed DHT crawler that sniffs torrents from BitTorrent network
Dig Etl Engine
⭐
65
Download DIG to run on your laptop or server.
Scrapy Kafka
⭐
63
Kafka-based components for Scrapy
Scrapy Distributed
⭐
40
A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.
Scrapy Kafka Redis
⭐
35
Distributed crawling/scraping, Kafka And Redis based components for Scrapy
Pomp Craigslist Example
⭐
33
Extract data from Craigslist.org by python3 and pomp framework
Engineeringteam
⭐
32
와이빅타 엔지니어링팀의 자료를 정리해두는 곳입니다.
Crawler Framework
⭐
12
分布式爬虫框架,基于webdrvier模拟用户请求,kafka消息传递,分布式网页存储使用hbase ip服务和号码验证服务等, proxy page使用H5和we版进行接入
Webcrawler
⭐
11
A focused web crawler based on Playwright, RMQ, Kafka and Flink.
Weather Mrs
⭐
9
天气爬虫采集,kafka实时分发,flume 收集数据导入到 Hbase, 再由 Hive 与 Hbase 建立映射,Superset 分析和展示数据。
Php Securityservicesscraper
⭐
8
Ukwa Heritrix
⭐
8
The UKWA Heritrix3 custom modules and Docker builder.
Visee
⭐
6
Just a typical search engine in this universe 🔥🔥🔥
Crawler
⭐
6
基于django和scrapy的采集系统
Skeleton X
⭐
6
🎉基于Springboot的SSM脚手架,目前已整合spring-scurity,websocke
Apiestas
⭐
5
Backend, modern REST API for obtaining match and odds data crawled from multiple sites. Using FastAPI, MongoDB as database, Motor as async MongoDB client, Scrapy as crawler and Docker.
Scrapy Cluster
⭐
5
modified by http://github.com/istresearch/scrapy-cluster.git to make it appropriate for project.
Hunter Parent
⭐
5
分布式爬虫系统-hunter
Related Searches
Python Crawler (4,545)
Java Kafka (3,237)
Kafka Zookeeper (1,229)
Docker Kafka (1,191)
Javascript Crawler (1,142)
Python Kafka (1,133)
Crawler Scrapy (988)
Spark Kafka (979)
Scala Kafka (969)
Golang Kafka (919)
1-13 of 13 search results
Privacy
|
About
|
Terms
|
Follow Us On Twitter
Copyright 2018-2024 Awesome Open Source. All rights reserved.