The Top 23 Nodejs Crawler Open Source Projects

Open source projects categorized as Nodejs Crawler

Categories > Data Processing > Crawler

Categories > Runtime Environments > Nodejs

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747most recent commit a day ago

Crawlab ⭐ 10,521

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

total releases 1most recent commit 3 months ago

Node Crawler ⭐ 6,610

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

dependent packages 146total releases 31most recent commit 4 months ago

Google Play Scraper ⭐ 2,108

Node.js scraper to get data from Google Play

dependent packages 29total releases 86most recent commit 5 months ago

Weixin Game Helper ⭐ 1,352

微信小游戏辅助合集（加减大师、包你懂我、大家来找茬腾讯版、头脑王者、好友画我、悦动音符、我最在行、星

most recent commit 10 months ago

Article Extractor ⭐ 1,297

To extract main article from given URL with Node.js

dependent packages 10total releases 156most recent commit 3 months ago

Fdir ⭐ 1,191

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

dependent packages 98total releases 35most recent commit 5 months ago

Creeper ⭐ 762

🐾 Creeper - The Next Generation Crawler Framework (Go)

total releases 1most recent commit 7 years ago

X Crawl ⭐ 718

x-crawl is a flexible Node.js multifunctional crawler library. Flexible usage and numerous functions can help you quickly, safely, and stably crawl pages, interfaces, and files. ---------------- x-crawl 是一个灵活的 Node.js 多功能爬虫库。灵活的使用方式和众多的功能可以帮助您快速、安全、稳定地爬取页面、接口以及文件。

dependent packages 1total releases 59most recent commit 3 months ago

Nodejs Stuff ⭐ 484

Node.js libs I want to keep in mind.

most recent commit 4 years ago

Webster ⭐ 465

a reliable high-level web crawling & scraping framework for Node.js.

dependent packages 2total releases 42most recent commit 6 months ago

Linkedin Profile Scraper Api ⭐ 404

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

most recent commit 10 months ago

Bitcoin Seeder ⭐ 369

most recent commit 8 months ago

Rental ⭐ 344

🏠 Rental app base on Node.js

most recent commit 7 years ago

Harvestman ⭐ 278

Quick and dirty web crawling.

most recent commit 8 years ago

Ppspider ⭐ 278

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（ne

dependent packages 2total releases 85most recent commit 2 years ago

Jssoup ⭐ 240

JavaScript + BeautifulSoup = JSSoup

dependent packages 41total releases 15most recent commit 3 years ago

Goose Parser ⭐ 222

Universal scraping tool, which allows you to extract data using multiple environments

dependent packages 6total releases 87most recent commit 5 years ago

Leetcode Spider ⭐ 166

用 node.js 爬你自己的 leetcode 解题源码

dependent packages 2total releases 32most recent commit 4 years ago

Tor_crawler ⭐ 148

Web crawling with IP rotation via Tor

most recent commit 4 years ago

Xvideos ⭐ 147

xvideos API library

dependent packages 2total releases 9most recent commit 2 years ago

Spyder ⭐ 138

A Python web crawler using Tornado and ZeroMQ

most recent commit 12 years ago

Acm Statistics ⭐ 137

An online tool (crawler) to analyze users performance in online judges (coding competition websites). Supported OJ: POJ, HDU, HYSBZ, CodeForces, UVA, ICPC Live Archive, FZU, SPOJ, Timus (URAL), LeetCode_CN, CSU, LibreOJ, 洛谷, 牛客OJ, Lutece (UESTC), AtCoder, AIZU, CodeChef, El Judge, BNUOJ, Codewars, UOJ, NBUT, 51Nod, DMOJ, VJudge

most recent commit 4 months ago

The 10 Most Depended On Nodejs Crawler Open Source Projects

Node Crawler ⭐ 6,610

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

dependent packages 146total releases 31latest release December 30, 2022most recent commit 4 months ago

Fdir ⭐ 1,191

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

dependent packages 98total releases 35latest release November 05, 2023most recent commit 5 months ago

Readdir Enhanced ⭐ 66

fs.readdir() with filter, recursion, absolute paths, promises, streams, and more!

dependent packages 54total releases 39latest release July 18, 2020most recent commit 4 years ago

Crawlee ⭐ 12,106

dependent packages 42total releases 747latest release December 10, 2023most recent commit a day ago

Jssoup ⭐ 240

JavaScript + BeautifulSoup = JSSoup

dependent packages 41total releases 15latest release October 03, 2021most recent commit 3 years ago

Google Play Scraper ⭐ 2,108

Node.js scraper to get data from Google Play

dependent packages 29total releases 86latest release August 09, 2023most recent commit 5 months ago

Spidex ⭐ 26

A web requester for node.js.

dependent packages 23total releases 21latest release March 31, 2017most recent commit 5 years ago

Xml Parser ⭐ 13

A Node.js XML DOM, Parser & Stringifier.

dependent packages 21total releases 6latest release November 06, 2019most recent commit 2 years ago

Article Extractor ⭐ 1,297

To extract main article from given URL with Node.js

dependent packages 10total releases 156latest release December 01, 2022most recent commit 3 months ago

Goose Parser ⭐ 222

Universal scraping tool, which allows you to extract data using multiple environments

dependent packages 6total releases 87latest release April 17, 2019most recent commit 5 years ago

The 10 Latest Releases In Nodejs Crawler Open Source Projects

Crawlee ⭐ 12,106

dependent packages 42total releases 747latest release December 10, 2023most recent commit a day ago

Js Stellar Node Crawler ⭐ 12

Crawl the Stellar Network. Identify the nodes and determine their validating status.

dependent packages 2total releases 83latest release November 20, 2023most recent commit 4 months ago

npm @stellarbeat/js-stellar-node-crawler} Downloads

Htcrawl ⭐ 46

Htcrawl is nodejs module for the recursive crawling of single page applications (SPA) using javascript

dependent packages 2total releases 8latest release November 16, 2023most recent commit 4 months ago

X Crawl ⭐ 718

dependent packages 1total releases 59latest release November 09, 2023most recent commit 3 months ago

Webster ⭐ 465

a reliable high-level web crawling & scraping framework for Node.js.

dependent packages 2total releases 42latest release November 09, 2023most recent commit 6 months ago

Fdir ⭐ 1,191

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

dependent packages 98total releases 35latest release November 05, 2023most recent commit 5 months ago

Js Stellar Node Connector ⭐ 15

Connect and interact with nodes in the Stellar Network over the tcp protocol.

dependent packages 3total releases 70latest release October 30, 2023most recent commit 4 months ago

npm @stellarbeat/js-stellar-node-connector} Downloads

Noscrape ⭐ 32

obfuscate text via node to make scraping your content really difficult

dependent packages 2total releases 20latest release September 28, 2023most recent commit 4 months ago

Google Play Scraper ⭐ 2,108

Node.js scraper to get data from Google Play

dependent packages 29total releases 86latest release August 09, 2023most recent commit 5 months ago

Nekopoi Scraper ⭐ 11

API scraper for nekopoi.care

dependent packages 2total releases 23latest release February 23, 2023most recent commit 9 months ago

Popular Data Processing Categories