The Top 23 Nodejs Web Crawler Open Source Projects

Open source projects categorized as Nodejs Web Crawler

Categories > Runtime Environments > Nodejs

Categories > Data Processing > Web Crawler

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747most recent commit 4 hours ago

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

total releases 1most recent commit 3 months ago

Node Osmosis ⭐ 4,083

Web scraper for NodeJS

dependent packages 58total releases 27most recent commit 9 months ago

Thal ⭐ 2,268

Getting started with Puppeteer and Chrome Headless for Web Scraping

most recent commit 3 years ago

Gogoanime Api ⭐ 575

Anime Streaming, Discovery API made with Cheerio and Express. Uses data from Gogoanime

most recent commit 5 months ago

Netflix Clone ⭐ 245

Netflix like full-stack application with SPA client and backend implemented in service oriented architecture

most recent commit a year ago

Html Metadata ⭐ 115

MetaData html scraper and parser for Node.js (supports Promises and callback style)

dependent packages 26total releases 20most recent commit 4 years ago

Node Web Crawler ⭐ 104

A web scraper with a web user interface which shows scraping stats in realtime. Uses Node.JS, jQuery, socket.io and Express.

most recent commit 10 years ago

Torrents Api ⭐ 90

Torrent Api ✨

most recent commit 2 years ago

Animeez ⭐ 88

AnimeEZ - An Anime Streaming website without any ads for free (Demo - https://animeez.live) BTW ITS MADE IN HTML

most recent commit a year ago

Raise ⭐ 83

A simple (and unofficial) GitHub Trending client that lives in your menubar.

most recent commit 2 years ago

Node Search Engine ⭐ 79

Sample search engine with web crawler, built on Node.js + CouchDB + Limestone

most recent commit 13 years ago

Tatooine ⭐ 78

A powerful scraper for JavaScript Developers.

dependent packages 1total releases 84most recent commit a year ago

Amazon_scraper ⭐ 64

Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt

dependent packages 2total releases 13most recent commit 3 months ago

npm @scrapingant/amazon-proxy-scraper} Downloads

Newspaperjs ⭐ 63

News extraction and scraping. Article Parsing

dependent packages 2total releases 7most recent commit a year ago

Gocrawler ⭐ 60

A distributed web crawler implemented using Go, Postgres, RabbitMQ and Docker

most recent commit 3 years ago

Node Js Functionalities ⭐ 58

This repository contains very useful restful API's and functionalities in node-js containing many important tutorial code for mastering node-js, all tutorials have been published on medium.com, tutorials link is given below

most recent commit 3 years ago

Google Search Results Nodejs ⭐ 53

SerpApi client library for Node.js. Previously: Google Search Results Node.js.

dependent packages 3total releases 7most recent commit 10 months ago

npm google-search-results-nodejs} Downloads

Nightmare Heroku ⭐ 48

😱 a setup for nightmarejs on heroku

most recent commit 5 years ago

Maps To Lead ⭐ 48

Esse projeto tem como objetivo obter leads em formato JSON e enviar para um webhook

most recent commit a year ago

Keeper Core Api ⭐ 44

Nunux Keeper core API

most recent commit 4 years ago

React Node Web Scraper ⭐ 28

Final Year project, scraping data of e-commerce stores and display in ReactJS app.

most recent commit 6 months ago

Professional Javascript ⭐ 28

Fast-track your web development career using the powerful features of advanced JavaScript

dependent packages 2total releases 3most recent commit 2 years ago

The 10 Most Depended On Nodejs Web Crawler Open Source Projects

Node Osmosis ⭐ 4,083

Web scraper for NodeJS

dependent packages 58total releases 27latest release March 01, 2019most recent commit 9 months ago

Crawlee ⭐ 12,101

dependent packages 42total releases 747latest release December 10, 2023most recent commit 4 hours ago

Webmiddle ⭐ 13

Node.js framework for modular web scraping and data extraction

dependent packages 27total releases 9latest release November 06, 2018most recent commit 4 years ago

Html Metadata ⭐ 115

MetaData html scraper and parser for Node.js (supports Promises and callback style)

dependent packages 26total releases 20latest release November 20, 2019most recent commit 4 years ago

Google Search Results Nodejs ⭐ 53

SerpApi client library for Node.js. Previously: Google Search Results Node.js.

dependent packages 3total releases 7latest release December 26, 2021most recent commit 10 months ago

Kirinuki Core ⭐ 12

Kirinuki is a library that convert any html to JSON using CSS selectors.

dependent packages 3total releases 14latest release March 06, 2019most recent commit a year ago

Ajfabriqnode ⭐ 6

A Distributed Application Framework for NodeJs

dependent packages 3total releases 3latest release April 29, 2013most recent commit 10 years ago

Professional Javascript ⭐ 28

Fast-track your web development career using the powerful features of advanced JavaScript

dependent packages 2total releases 3latest release September 11, 2019most recent commit 2 years ago

Spiderman ⭐ 5

Minimalistic web crawler for Node.js

dependent packages 2total releases 32latest release July 19, 2020most recent commit 2 years ago

Scraper ⭐ 15

All In One API to easily scrape data from any website, without worrying about captchas and bot detection mecanisms.

dependent packages 2total releases 12latest release September 02, 2021most recent commit 8 months ago

The 10 Latest Releases In Nodejs Web Crawler Open Source Projects

Crawlee ⭐ 12,101

dependent packages 42total releases 747latest release December 10, 2023most recent commit 4 hours ago

Zenrows Node Sdk ⭐ 6

SDK to access ZenRows API directly from Node.js. We handle proxies rotation, headless browsers and CAPTCHAs for you.

dependent packages 1total releases 15latest release October 21, 2023most recent commit 6 months ago

Tatooine ⭐ 78

A powerful scraper for JavaScript Developers.

dependent packages 1total releases 84latest release March 27, 2023most recent commit a year ago

Node Fetch Dom ⭐ 13

Magic utility that extract javascript global variables from a remote html page.

dependent packages 2total releases 6latest release September 08, 2022most recent commit a year ago

Amazon_scraper ⭐ 64

Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt

dependent packages 2total releases 13latest release June 10, 2022most recent commit 3 months ago

Ptt Crawler ⭐ 18

ptt-crawler is a web crawler module designed to scarpe data from Ptt.

total releases 12latest release January 29, 2022most recent commit a year ago

Google Search Results Nodejs ⭐ 53

SerpApi client library for Node.js. Previously: Google Search Results Node.js.

dependent packages 3total releases 7latest release December 26, 2021most recent commit 10 months ago

Scraper ⭐ 15

All In One API to easily scrape data from any website, without worrying about captchas and bot detection mecanisms.

dependent packages 2total releases 12latest release September 02, 2021most recent commit 8 months ago

Botmation ⭐ 8

A simple TypeScript framework for declaratively composing bots with Puppeteer

dependent packages 2total releases 10latest release April 28, 2021most recent commit 3 years ago

Spiderman ⭐ 5

Minimalistic web crawler for Node.js

dependent packages 2total releases 32latest release July 19, 2020most recent commit 2 years ago

Popular Runtime Environments Categories