The Top 23 Javascript Web Crawler Open Source Projects

Open source projects categorized as Javascript Web Crawler

Categories > Programming Languages > Javascript

Categories > Data Processing > Web Crawler

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747most recent commit 6 days ago

Node Osmosis ⭐ 4,083

Web scraper for NodeJS

dependent packages 58total releases 27most recent commit 9 months ago

Browser Fingerprinting ⭐ 3,353

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

most recent commit a year ago

Thal ⭐ 2,268

Getting started with Puppeteer and Chrome Headless for Web Scraping

most recent commit 3 years ago

100projectsofcode ⭐ 1,293

A list of practical knowledge-building projects.

most recent commit 3 years ago

Stealth ⭐ 923

🚀 Stealth - Secure, Peer-to-Peer, Private and Automateable Web Browser/Scraper/Proxy

most recent commit 7 months ago

User Agents ⭐ 871

A JavaScript library for generating random user agents with data that's updated daily.

dependent packages 104total releases 1,550most recent commit 3 months ago

Rpa ⭐ 858

UI.Vision: Open-Source RPA Software (formerly Kantu) - Modern Robotic Process Automation with Selenium IDE++

most recent commit 10 months ago

Coolqlcool ⭐ 626

Nextjs server to query websites with GraphQL

most recent commit 3 years ago

Browsertrix Crawler ⭐ 470

Run a high-fidelity browser-based crawler in a single Docker container

most recent commit 3 months ago

Archivebot ⭐ 328

ArchiveBot, an IRC bot for archiving websites

most recent commit 5 months ago

Supercrawler ⭐ 324

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

dependent packages 3total releases 45most recent commit 2 years ago

Youtube Projects ⭐ 272

This repository contains all the code I use in my YouTube tutorials.

most recent commit 3 years ago

Netflix Clone ⭐ 245

Netflix like full-stack application with SPA client and backend implemented in service oriented architecture

most recent commit a year ago

Nofasel ⭐ 239

A streaming app with no ADs.

most recent commit 9 months ago

Imdb Api ⭐ 193

Serverless IMDB API powered by Cloudflare Worker

most recent commit 4 months ago

Awesomewebscraping ⭐ 189

List of libraries, tools and APIs for web scraping and data processing.

most recent commit 7 months ago

Spider Less ⭐ 186

Web spider as a service, spider on serverless

most recent commit a year ago

Helena ⭐ 165

A Chrome extension for writing custom web scraping programs and web automation programs. Just demonstrate how to collect the first row of data, then let the extension write the program for collecting all rows.

most recent commit 3 years ago

Animos ⭐ 154

Animos - Clean and minimal Anime-streaming desktop application without any ads.

most recent commit 10 months ago

Imghash ⭐ 152

Perceptual image hashing for Node.js

dependent packages 21total releases 9most recent commit a year ago

Web Scraper Chrome Extension ⭐ 149

Web data extraction tool implemented as chrome extension

most recent commit 9 months ago

Google News Scraper ⭐ 144

Lightweight scraper for Google News

dependent packages 2total releases 12most recent commit 4 months ago

The 10 Most Depended On Javascript Web Crawler Open Source Projects

User Agents ⭐ 871

A JavaScript library for generating random user agents with data that's updated daily.

dependent packages 104total releases 1,550latest release December 10, 2023most recent commit 3 months ago

Node Osmosis ⭐ 4,083

Web scraper for NodeJS

dependent packages 58total releases 27latest release March 01, 2019most recent commit 9 months ago

Crawlee ⭐ 11,957

dependent packages 42total releases 747latest release December 10, 2023most recent commit 6 days ago

Webmiddle ⭐ 13

Node.js framework for modular web scraping and data extraction

dependent packages 27total releases 9latest release November 06, 2018most recent commit 4 years ago

Html Metadata ⭐ 115

MetaData html scraper and parser for Node.js (supports Promises and callback style)

dependent packages 26total releases 20latest release November 20, 2019most recent commit 4 years ago

Imghash ⭐ 152

Perceptual image hashing for Node.js

dependent packages 21total releases 9latest release May 07, 2021most recent commit a year ago

X Ray Crawler ⭐ 39

Friendly web crawler for x-ray

dependent packages 19total releases 7latest release August 26, 2020most recent commit 3 years ago

Node Krawler ⭐ 13

Fast and lightweight web crawler with built-in cheerio, xml and json parser.

dependent packages 7total releases 10latest release June 23, 2014most recent commit 10 years ago

Nodejs Web Scraper ⭐ 54

dependent packages 5total releases 81latest release January 07, 2023most recent commit a year ago

Hawk ⭐ 43

Blazingly fast web crawler for mapping and updating data

dependent packages 4total releases 2latest release April 07, 2016most recent commit 7 years ago

The 10 Latest Releases In Javascript Web Crawler Open Source Projects

User Agents ⭐ 871

A JavaScript library for generating random user agents with data that's updated daily.

dependent packages 104total releases 1,550latest release December 10, 2023most recent commit 3 months ago

Crawlee ⭐ 11,957

dependent packages 42total releases 747latest release December 10, 2023most recent commit 6 days ago

Google News Scraper ⭐ 144

Lightweight scraper for Google News

dependent packages 2total releases 12latest release December 06, 2023most recent commit 4 months ago

Amazon Scraper ⭐ 140

Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.

total releases 1latest release November 13, 2023most recent commit 5 months ago

Zenrows Node Sdk ⭐ 6

SDK to access ZenRows API directly from Node.js. We handle proxies rotation, headless browsers and CAPTCHAs for you.

dependent packages 1total releases 15latest release October 21, 2023most recent commit 6 months ago

Tt Mobile ⭐ 6

PWA and web scraper for swiss table tennis results

dependent packages 1total releases 11latest release August 03, 2023most recent commit 4 months ago

Tatooine ⭐ 78

A powerful scraper for JavaScript Developers.

dependent packages 1total releases 84latest release March 27, 2023most recent commit a year ago

Nodejs Web Scraper ⭐ 54

dependent packages 5total releases 81latest release January 07, 2023most recent commit a year ago

Amazon_scraper ⭐ 64

Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt

dependent packages 2total releases 13latest release June 10, 2022most recent commit 3 months ago

npm @scrapingant/amazon-proxy-scraper} Downloads

Ptt Crawler ⭐ 18

ptt-crawler is a web crawler module designed to scarpe data from Ptt.

total releases 12latest release January 29, 2022most recent commit a year ago

Popular Programming Languages Categories