Awesome Open Source

Programming Languages

The Top 23 Javascript Webspider Open Source Projects

Open source projects categorized as Javascript Webspider

Categories > Programming Languages > Javascript

Categories > Webspider

Crawlee ⭐ 12,106

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747most recent commit 2 days ago

npm crawlee} Downloads

Browsertrix Crawler ⭐ 470

Run a high-fidelity browser-based crawler in a single Docker container

most recent commit 3 months ago

Archivebot ⭐ 328

ArchiveBot, an IRC bot for archiving websites

most recent commit 5 months ago

Supercrawler ⭐ 324

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

dependent packages 3total releases 45most recent commit 2 years ago

npm supercrawler} Downloads

Spider Less ⭐ 186

Web spider as a service, spider on serverless

most recent commit a year ago

Google News Scraper ⭐ 144

Lightweight scraper for Google News

dependent packages 2total releases 12most recent commit 4 months ago

npm google-news-scraper} Downloads

Node Web Crawler ⭐ 104

A web scraper with a web user interface which shows scraping stats in realtime. Uses Node.JS, jQuery, socket.io and Express.

most recent commit 10 years ago

A platform displaying the latest software engineer job information to entry-level new graduates

most recent commit 2 years ago

Node Search Engine ⭐ 79

Sample search engine with web crawler, built on Node.js + CouchDB + Limestone

most recent commit 13 years ago

Amazon_scraper ⭐ 64

Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt

dependent packages 2total releases 13most recent commit 4 months ago

npm @scrapingant/amazon-proxy-scraper} Downloads

Simplestorm ⭐ 62

Simple Storm-like distributed application implementation

dependent packages 1total releases 6most recent commit 10 years ago

npm simplestorm} Downloads

Siteshooter ⭐ 58

📷 Automate full website screenshots and PDF generation with multiple viewport support.

dependent packages 1total releases 73most recent commit 5 years ago

npm siteshooter} Downloads

Blazingly fast web crawler for mapping and updating data

dependent packages 4total releases 2most recent commit 7 years ago

npm hawk-webcrawler} Downloads

X Ray Crawler ⭐ 39

Friendly web crawler for x-ray

dependent packages 19total releases 7most recent commit 3 years ago

npm x-ray-crawler} Downloads

Puppeteer Service ⭐ 27

🎠 Run headless Chrome (aka Puppeteer) as a service.

dependent packages 3total releases 23most recent commit 6 years ago

npm puppeteer-service} Downloads

Mimo Crawler ⭐ 24

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

most recent commit 8 months ago

Nian Crawler ⭐ 23

A web crawler for api.nian.so

most recent commit 6 years ago

Node Web Crawler ⭐ 21

Web Crawler in Node.js

most recent commit 10 years ago

Pipe2time.ir ⭐ 18

Web Crawler for Time.ir to Retrive JSON File, jalali, qamari, miladi JSON Calendar API.

most recent commit a year ago

Ptt Crawler ⭐ 18

ptt-crawler is a web crawler module designed to scarpe data from Ptt.

total releases 12most recent commit a year ago

Json Web Crawler ⭐ 17

Use JSON to list all elements (with css 3 and jquery selector) that you want to crawl.

total releases 19most recent commit 2 years ago

Web Crawler created with Node.js and Puppeteer

most recent commit 6 years ago

Js Crawler ⭐ 16

A short and simple python crawler, that uses Webkit and executes Javascript

most recent commit 11 years ago

The 10 Most Depended On Javascript Webspider Open Source Projects

Crawlee ⭐ 12,106

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747latest release December 10, 2023most recent commit 2 days ago

npm crawlee} Downloads

X Ray Crawler ⭐ 39

Friendly web crawler for x-ray

dependent packages 19total releases 7latest release August 26, 2020most recent commit 3 years ago

npm x-ray-crawler} Downloads

Node Krawler ⭐ 13

Fast and lightweight web crawler with built-in cheerio, xml and json parser.

dependent packages 7total releases 10latest release June 23, 2014most recent commit 10 years ago

npm krawler} Downloads

Device Detective ⭐ 8

A Node.js module which determines whether a user agent is a phone, tablet, desktop, text browser, or search crawler.

dependent packages 4total releases 6latest release December 19, 2015most recent commit 8 years ago

npm device-detective} Downloads

Blazingly fast web crawler for mapping and updating data

dependent packages 4total releases 2latest release April 07, 2016most recent commit 7 years ago

npm hawk-webcrawler} Downloads

Painlesscrawler ⭐ 10

(WIP) A painless Node.js web crawler that simply works

dependent packages 4total releases 3latest release August 05, 2015most recent commit 8 years ago

npm painless-crawler} Downloads

Floodesh ⭐ 11

Floodesh is a distributed web spider written with Nodejs.

dependent packages 3total releases 78latest release April 27, 2020most recent commit 4 years ago

npm floodesh} Downloads

Supercrawler ⭐ 324

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

dependent packages 3total releases 45latest release November 24, 2019most recent commit 2 years ago

npm supercrawler} Downloads

Puppeteer Service ⭐ 27

🎠 Run headless Chrome (aka Puppeteer) as a service.

dependent packages 3total releases 23latest release March 14, 2018most recent commit 6 years ago

npm puppeteer-service} Downloads

Ajfabriqnode ⭐ 6

A Distributed Application Framework for NodeJs

dependent packages 3total releases 3latest release April 29, 2013most recent commit 10 years ago

npm ajfabriq} Downloads

The 10 Latest Releases In Javascript Webspider Open Source Projects

Crawlee ⭐ 12,106

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747latest release December 10, 2023most recent commit 2 days ago

npm crawlee} Downloads

Google News Scraper ⭐ 144

Lightweight scraper for Google News

dependent packages 2total releases 12latest release December 06, 2023most recent commit 4 months ago

npm google-news-scraper} Downloads

Amazon_scraper ⭐ 64

Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt

dependent packages 2total releases 13latest release June 10, 2022most recent commit 4 months ago

npm @scrapingant/amazon-proxy-scraper} Downloads

Ptt Crawler ⭐ 18

ptt-crawler is a web crawler module designed to scarpe data from Ptt.

total releases 12latest release January 29, 2022most recent commit a year ago

Json Web Crawler ⭐ 17

Use JSON to list all elements (with css 3 and jquery selector) that you want to crawl.

total releases 19latest release December 22, 2021most recent commit 2 years ago

X Ray Crawler ⭐ 39

Friendly web crawler for x-ray

dependent packages 19total releases 7latest release August 26, 2020most recent commit 3 years ago

npm x-ray-crawler} Downloads

Spiderman ⭐ 5

Minimalistic web crawler for Node.js

dependent packages 2total releases 32latest release July 19, 2020most recent commit 2 years ago

npm @albert-team/spiderman} Downloads

Floodesh ⭐ 11

Floodesh is a distributed web spider written with Nodejs.

dependent packages 3total releases 78latest release April 27, 2020most recent commit 4 years ago

npm floodesh} Downloads

Supercrawler ⭐ 324

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

dependent packages 3total releases 45latest release November 24, 2019most recent commit 2 years ago

npm supercrawler} Downloads

Siteshooter ⭐ 58

:camera: Automate full website screenshots and PDF generation with multiple viewport support.

dependent packages 1total releases 73latest release May 15, 2019most recent commit 5 years ago

npm siteshooter} Downloads

Popular Programming Languages Categories

Categories

Application Programming Interfaces

Artificial Intelligence

Cloud Computing

Command Line Interface

Computer Science

Configuration Management

Content Management

Data Processing

Integrated Development Environments

Learning Resources

Lists Of Projects

Machine Learning

Operating Systems

Package Managers

Programming Languages

Runtime Environments

Software Architecture

Software Development

Software Performance

Software Quality

Text Processing

User Interface Components

Version Control

Web User Interface

Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source. All rights reserved.