The Top 23 Javascript Crawler Open Source Projects

Open source projects categorized as Javascript Crawler

Categories > Data Processing > Crawler

Categories > Programming Languages > Javascript

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747most recent commit 6 days ago

Node Crawler ⭐ 6,610

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

dependent packages 146total releases 31most recent commit 3 months ago

Headless Chrome Crawler ⭐ 5,051

Distributed crawler powered by Headless Chrome

dependent packages 12total releases 21most recent commit 3 years ago

Arachni ⭐ 3,632

Web Application Security Scanner Framework

most recent commit a year ago

Browser Fingerprinting ⭐ 3,353

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

most recent commit a year ago

Work_crawler ⭐ 2,671

Download comics novels 小说漫画下载工具小説漫画のダウンローダ小說漫畫下載:腾讯漫画大角虫漫画有妖气咪咕 SF漫画哦漫画看漫画漫画柜汗汗酷漫動漫伊甸園快看漫画微博动漫 733动漫网大古漫画网漫画DB 無限動漫動漫狂卡推漫画动漫之家动漫屋古风漫画网 36漫画网亲亲漫画网乙女漫画 webtoons 咚漫ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミックサイコミ;アルファポリスカクヨムハーメルン小説家になろう起点中文网八一中文网顶点小说落霞小说网努努书坊笔趣阁→epub.

most recent commit 3 months ago

Crawler ⭐ 2,417

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

dependent packages 28total releases 101most recent commit 4 months ago

Google Play Scraper ⭐ 2,108

Node.js scraper to get data from Google Play

dependent packages 29total releases 86most recent commit 5 months ago

Rendora ⭐ 1,950

dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites

total releases 1most recent commit a year ago

Geziyor ⭐ 1,892

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

dependent packages 11total releases 6most recent commit a year ago

Ambar ⭐ 1,797

🔍 Ambar: Document Search Engine

most recent commit 3 years ago

React Snapshot ⭐ 1,619

A zero-configuration static pre-renderer for React apps

dependent packages 19total releases 11most recent commit 3 years ago

Static Site Generator Webpack Plugin ⭐ 1,538

Minimal, unopinionated static site generator powered by webpack

dependent packages 214total releases 21most recent commit 5 years ago

npm static-site-generator-webpack-plugin} Downloads

Lightcrawler ⭐ 1,354

Crawl a website and run it through Google lighthouse

most recent commit 3 years ago

Article Extractor ⭐ 1,297

To extract main article from given URL with Node.js

dependent packages 10total releases 156most recent commit 3 months ago

Fdir ⭐ 1,191

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

dependent packages 98total releases 35most recent commit 5 months ago

Prerender Node ⭐ 916

Express middleware for prerendering javascript-rendered pages on the fly for SEO

dependent packages 38total releases 71most recent commit a year ago

Scrapy Selenium ⭐ 842

Scrapy middleware to handle javascript pages using selenium

dependent packages 2total releases 6most recent commit 6 months ago

X Crawl ⭐ 718

x-crawl is a flexible Node.js multifunctional crawler library. Flexible usage and numerous functions can help you quickly, safely, and stably crawl pages, interfaces, and files. ---------------- x-crawl 是一个灵活的 Node.js 多功能爬虫库。灵活的使用方式和众多的功能可以帮助您快速、安全、稳定地爬取页面、接口以及文件。

dependent packages 1total releases 59most recent commit 3 months ago

Xxl Crawler ⭐ 650

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

dependent packages 1total releases 6most recent commit a year ago

Tailwindui Crawler ⭐ 642

tailwindui-crawler downloads the component HTML files locally

most recent commit a year ago

Newcrawler ⭐ 583

Free Web Scraping Tool with Java

most recent commit 5 months ago

Thea11ymachine ⭐ 553

The A11y Machine is an automated accessibility testing tool which crawls and tests pages of any web application to produce detailed reports.

dependent packages 2total releases 26most recent commit 4 years ago

The 10 Most Depended On Javascript Crawler Open Source Projects

Static Site Generator Webpack Plugin ⭐ 1,538

Minimal, unopinionated static site generator powered by webpack

dependent packages 214total releases 21latest release November 19, 2018most recent commit 5 years ago

Node Crawler ⭐ 6,610

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

dependent packages 146total releases 31latest release December 30, 2022most recent commit 3 months ago

Zhihu Api ⭐ 267

Unofficial API for zhihu.

dependent packages 103total releases 39latest release July 16, 2017most recent commit 7 years ago

Fdir ⭐ 1,191

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

dependent packages 98total releases 35latest release November 05, 2023most recent commit 5 months ago

Readdir Enhanced ⭐ 66

fs.readdir() with filter, recursion, absolute paths, promises, streams, and more!

dependent packages 54total releases 39latest release July 18, 2020most recent commit 4 years ago

Npm License Crawler ⭐ 141

Analyzes license information for multiple node.js modules (package.json files) as part of your software project.

dependent packages 43total releases 16latest release March 28, 2019most recent commit 4 years ago

Crawlee ⭐ 11,957

dependent packages 42total releases 747latest release December 10, 2023most recent commit 6 days ago

Jssoup ⭐ 240

JavaScript + BeautifulSoup = JSSoup

dependent packages 41total releases 15latest release October 03, 2021most recent commit 2 years ago

Prerender Node ⭐ 916

Express middleware for prerendering javascript-rendered pages on the fly for SEO

dependent packages 38total releases 71latest release May 22, 2023most recent commit a year ago

Google Play Scraper ⭐ 2,108

Node.js scraper to get data from Google Play

dependent packages 29total releases 86latest release August 09, 2023most recent commit 5 months ago

The 10 Latest Releases In Javascript Crawler Open Source Projects

Crawlee ⭐ 11,957

dependent packages 42total releases 747latest release December 10, 2023most recent commit 6 days ago

Google News Scraper ⭐ 144

Lightweight scraper for Google News

dependent packages 2total releases 12latest release December 06, 2023most recent commit 4 months ago

I_love_indexes ⭐ 7

latest release November 26, 2023most recent commit 9 years ago

Reffy ⭐ 61

Reffy is a Web spec crawler and analyzer tool. It is notably used to update Webref

dependent packages 2total releases 138latest release November 24, 2023most recent commit 3 months ago

Crawler ⭐ 2,417

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

dependent packages 28total releases 101latest release November 22, 2023most recent commit 4 months ago

Js Stellar Node Crawler ⭐ 12

Crawl the Stellar Network. Identify the nodes and determine their validating status.

dependent packages 2total releases 83latest release November 20, 2023most recent commit 4 months ago

npm @stellarbeat/js-stellar-node-crawler} Downloads

Htcrawl ⭐ 46

Htcrawl is nodejs module for the recursive crawling of single page applications (SPA) using javascript

dependent packages 2total releases 8latest release November 16, 2023most recent commit 4 months ago

X Crawl ⭐ 718

dependent packages 1total releases 59latest release November 09, 2023most recent commit 3 months ago

Webster ⭐ 465

a reliable high-level web crawling & scraping framework for Node.js.

dependent packages 2total releases 42latest release November 09, 2023most recent commit 5 months ago

Fdir ⭐ 1,191

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

dependent packages 98total releases 35latest release November 05, 2023most recent commit 5 months ago

Popular Data Processing Categories