Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Hawk | 2,638 | 4 years ago | 65 | apache-2.0 | C# | |||||
visualized crawler & ETL IDE written with C#/WPF | ||||||||||
Etlpy | 393 | 5 years ago | 8 | apache-2.0 | Python | |||||
a smart stream-like crawler & etl python library | ||||||||||
Sentinel Crawler | 122 | a year ago | 33 | mit | JavaScript | |||||
Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure :dizzy: 多语言执行器,分布式爬虫 | ||||||||||
Sqloogle | 21 | 5 years ago | apache-2.0 | C# | ||||||
Crawl, Index, and Search Your SQL. | ||||||||||
Amazon S3 Step Functions Ingestion Orchestration | 19 | 5 years ago | apache-2.0 | Python | ||||||
Design pattern for orchestrating an incremental data ingestion pipeline using AWS Step Functions from an on premise location into an Amazon S3 datalake bucket | ||||||||||
Oeh Search Etl | 7 | 7 months ago | 10 | Python | ||||||
The Backend includes all data for the ETL process (Scrapy, Postgres, Elasticsearch) | ||||||||||
Nutchpighive | 5 | 7 years ago | Java | |||||||
crawl GooglePlay data with Nutch, ETL with Pig, analyze with Hive |