Awesome Open Source

Programming Languages

The Top 23 Crawler Webspider Open Source Projects

Open source projects categorized as Crawler Webspider

Categories > Data Processing > Crawler

Categories > Webspider

Crawlee ⭐ 11,957

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747most recent commit 7 days ago

npm crawlee} Downloads

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

total releases 1most recent commit 3 months ago

Spider Flow ⭐ 8,075

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

most recent commit 10 months ago

Katana ⭐ 7,995

A next-generation crawling and spidering framework.

dependent packages 1total releases 8most recent commit 3 months ago

Awesome Crawler ⭐ 5,859

A collection of awesome web crawler,spider in different languages

most recent commit 4 months ago

Proxypool ⭐ 5,154

An Efficient ProxyPool with Getter, Tester and Server

most recent commit 4 months ago

Hakrawler ⭐ 4,120

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

total releases 11most recent commit 3 months ago

Gerapy ⭐ 3,144

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

total releases 49most recent commit 3 months ago

Webcollector ⭐ 2,974

WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.

dependent packages 1total releases 22most recent commit a year ago

Nutch ⭐ 2,742

Apache Nutch is an extensible and scalable web crawler

dependent packages 1total releases 26most recent commit 3 months ago

Gecco ⭐ 2,403

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

dependent packages 4total releases 28most recent commit a year ago

Gospider ⭐ 2,190

Gospider - Fast web spider written in Go

dependent packages 1total releases 17most recent commit 5 months ago

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

dependent packages 8total releases 153most recent commit 2 years ago

nuget Abot} Downloads

Pspider ⭐ 1,675

简单易用的Python爬虫框架，QQ交流群：597510560

most recent commit 2 years ago

Storm Crawler ⭐ 834

A scalable, mature and versatile web crawler based on Apache Storm

dependent packages 10total releases 36most recent commit 3 months ago

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

dependent packages 16total releases 26most recent commit 3 months ago

rubygems spidr} Downloads

Fetchbot ⭐ 758

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

dependent packages 3total releases 7most recent commit 3 years ago

Browsertrix Crawler ⭐ 470

Run a high-fidelity browser-based crawler in a single Docker container

most recent commit 3 months ago

Spidersuite ⭐ 447

Advance web spider/crawler for cyber security professionals

most recent commit 7 months ago

Pulsarrpa ⭐ 413

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.

dependent packages 3total releases 58most recent commit 3 months ago

Simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names.

most recent commit 5 years ago

Sparkler ⭐ 401

Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

most recent commit a year ago

Python3webcrawler ⭐ 347

🌈Python3网络爬虫实战：QQ音乐歌曲、京东商品信息、房天下、破解有道翻译、构建代理池、豆瓣读

most recent commit a year ago

The 10 Most Depended On Crawler Webspider Open Source Projects

Crawlee ⭐ 11,957

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747latest release December 10, 2023most recent commit 7 days ago

npm crawlee} Downloads

X Ray Crawler ⭐ 39

Friendly web crawler for x-ray

dependent packages 19total releases 7latest release August 26, 2020most recent commit 3 years ago

npm x-ray-crawler} Downloads

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

dependent packages 16total releases 26latest release October 25, 2019most recent commit 3 months ago

rubygems spidr} Downloads

Storm Crawler ⭐ 834

A scalable, mature and versatile web crawler based on Apache Storm

dependent packages 10total releases 36latest release October 25, 2023most recent commit 3 months ago

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

dependent packages 8total releases 153latest release August 09, 2021most recent commit 2 years ago

nuget Abot} Downloads

Crawler Commons ⭐ 217

A set of reusable Java components that implement functionality common to any web crawler

dependent packages 7total releases 10latest release July 13, 2023most recent commit 5 months ago

Node Krawler ⭐ 13

Fast and lightweight web crawler with built-in cheerio, xml and json parser.

dependent packages 7total releases 10latest release June 23, 2014most recent commit 10 years ago

npm krawler} Downloads

Device Detective ⭐ 8

A Node.js module which determines whether a user agent is a phone, tablet, desktop, text browser, or search crawler.

dependent packages 4total releases 6latest release December 19, 2015most recent commit 8 years ago

npm device-detective} Downloads

Blazingly fast web crawler for mapping and updating data

dependent packages 4total releases 2latest release April 07, 2016most recent commit 7 years ago

npm hawk-webcrawler} Downloads

Painlesscrawler ⭐ 10

(WIP) A painless Node.js web crawler that simply works

dependent packages 4total releases 3latest release August 05, 2015most recent commit 8 years ago

npm painless-crawler} Downloads

The 10 Latest Releases In Crawler Webspider Open Source Projects

Crawlee ⭐ 11,957

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747latest release December 10, 2023most recent commit 7 days ago

npm crawlee} Downloads

Pagemunch Php ⭐ 6

A PHP library for the PageMunch web crawler API

latest release December 10, 2023most recent commit 7 years ago

Crawley ⭐ 208

The unix-way web crawler

total releases 60latest release December 07, 2023most recent commit 4 months ago

Google News Scraper ⭐ 144

Lightweight scraper for Google News

dependent packages 2total releases 12latest release December 06, 2023most recent commit 4 months ago

npm google-news-scraper} Downloads

Crawler ⭐ 285

Library for Rapid (Web) Crawler and Scraper Development

total releases 25latest release December 01, 2023most recent commit 3 months ago

Web crawler based on Puppeteer

dependent packages 4total releases 23latest release November 13, 2023most recent commit 5 months ago

npm @opd/crawler} Downloads

Pulsarrpa ⭐ 413

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.

dependent packages 3total releases 58latest release November 11, 2023most recent commit 3 months ago

Storm Crawler ⭐ 834

A scalable, mature and versatile web crawler based on Apache Storm

dependent packages 10total releases 36latest release October 25, 2023most recent commit 3 months ago

Katana ⭐ 7,995

A next-generation crawling and spidering framework.

dependent packages 1total releases 8latest release September 14, 2023most recent commit 3 months ago

Gerapy ⭐ 3,144

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

total releases 49latest release July 19, 2023most recent commit 3 months ago

Popular Data Processing Categories

Jupyter Notebook

Categories

Application Programming Interfaces

Artificial Intelligence

Cloud Computing

Command Line Interface

Computer Science

Configuration Management

Content Management

Data Processing

Integrated Development Environments

Learning Resources

Lists Of Projects

Machine Learning

Operating Systems

Package Managers

Programming Languages

Runtime Environments

Software Architecture

Software Development

Software Performance

Software Quality

Text Processing

User Interface Components

Version Control

Web User Interface

Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source. All rights reserved.