The Top 23 Javascript Nodejs Crawler Open Source Projects

Open source projects categorized as Javascript Nodejs Crawler

Categories > Data Processing > Crawler

Categories > Programming Languages > Javascript

Categories > Runtime Environments > Nodejs

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747most recent commit a day ago

Node Crawler ⭐ 6,610

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

dependent packages 146total releases 31most recent commit 3 months ago

Google Play Scraper ⭐ 2,108

Node.js scraper to get data from Google Play

dependent packages 29total releases 86most recent commit 5 months ago

Article Extractor ⭐ 1,297

To extract main article from given URL with Node.js

dependent packages 10total releases 156most recent commit 3 months ago

Fdir ⭐ 1,191

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

dependent packages 98total releases 35most recent commit 5 months ago

X Crawl ⭐ 718

x-crawl is a flexible Node.js multifunctional crawler library. Flexible usage and numerous functions can help you quickly, safely, and stably crawl pages, interfaces, and files. ---------------- x-crawl 是一个灵活的 Node.js 多功能爬虫库。灵活的使用方式和众多的功能可以帮助您快速、安全、稳定地爬取页面、接口以及文件。

dependent packages 1total releases 59most recent commit 3 months ago

Nodejs Stuff ⭐ 484

Node.js libs I want to keep in mind.

most recent commit 4 years ago

Webster ⭐ 465

a reliable high-level web crawling & scraping framework for Node.js.

dependent packages 2total releases 42most recent commit 5 months ago

Rental ⭐ 344

🏠 Rental app base on Node.js

most recent commit 7 years ago

Jssoup ⭐ 240

JavaScript + BeautifulSoup = JSSoup

dependent packages 41total releases 15most recent commit 3 years ago

Leetcode Spider ⭐ 166

用 node.js 爬你自己的 leetcode 解题源码

dependent packages 2total releases 32most recent commit 4 years ago

Xvideos ⭐ 147

xvideos API library

dependent packages 2total releases 9most recent commit 2 years ago

Acm Statistics ⭐ 137

An online tool (crawler) to analyze users performance in online judges (coding competition websites). Supported OJ: POJ, HDU, HYSBZ, CodeForces, UVA, ICPC Live Archive, FZU, SPOJ, Timus (URAL), LeetCode_CN, CSU, LibreOJ, 洛谷, 牛客OJ, Lutece (UESTC), AtCoder, AIZU, CodeChef, El Judge, BNUOJ, Codewars, UOJ, NBUT, 51Nod, DMOJ, VJudge

most recent commit 4 months ago

Islandbeauty ⭐ 131

A spider/crawler edit by Node.js to download torrents of Adult videos.

most recent commit 5 years ago

Reqman ⭐ 124

Reqman is a tool that can quickly help back-end engineers with api testing, as well as a nodejs-based crawler tool.

dependent packages 1total releases 6most recent commit 5 years ago

Crawl ⭐ 110

Utility to crawl and diff websites for node.js

dependent packages 4total releases 4most recent commit 7 years ago

Holoapi ⭐ 106

Platform to serve Hololive content to the community tools.

most recent commit 3 years ago

Instagram Profilecrawl ⭐ 105

💻 Quickly crawl the information (e.g. followers, tags, etc...) of an instagram profile. No login required!

dependent packages 2total releases 44most recent commit 6 years ago

Node Web Crawler ⭐ 104

A web scraper with a web user interface which shows scraping stats in realtime. Uses Node.JS, jQuery, socket.io and Express.

most recent commit 10 years ago

Skill Share Crawler Dl ⭐ 104

Download Videos Skill Share per ID or per Class

most recent commit 5 years ago

Is Google ⭐ 81

Verify that a request is from Google crawlers using Google's DNS verification steps

dependent packages 2total releases 3most recent commit 5 years ago

Node Search Engine ⭐ 79

Sample search engine with web crawler, built on Node.js + CouchDB + Limestone

most recent commit 13 years ago

Strumentalia Seealsology ⭐ 76

most recent commit 5 months ago

The 10 Most Depended On Javascript Nodejs Crawler Open Source Projects

Node Crawler ⭐ 6,610

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

dependent packages 146total releases 31latest release December 30, 2022most recent commit 3 months ago

Fdir ⭐ 1,191

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

dependent packages 98total releases 35latest release November 05, 2023most recent commit 5 months ago

Readdir Enhanced ⭐ 66

fs.readdir() with filter, recursion, absolute paths, promises, streams, and more!

dependent packages 54total releases 39latest release July 18, 2020most recent commit 4 years ago

Crawlee ⭐ 12,059

dependent packages 42total releases 747latest release December 10, 2023most recent commit a day ago

Jssoup ⭐ 240

JavaScript + BeautifulSoup = JSSoup

dependent packages 41total releases 15latest release October 03, 2021most recent commit 3 years ago

Google Play Scraper ⭐ 2,108

Node.js scraper to get data from Google Play

dependent packages 29total releases 86latest release August 09, 2023most recent commit 5 months ago

Spidex ⭐ 26

A web requester for node.js.

dependent packages 23total releases 21latest release March 31, 2017most recent commit 5 years ago

Xml Parser ⭐ 13

A Node.js XML DOM, Parser & Stringifier.

dependent packages 21total releases 6latest release November 06, 2019most recent commit 2 years ago

Article Extractor ⭐ 1,297

To extract main article from given URL with Node.js

dependent packages 10total releases 156latest release December 01, 2022most recent commit 3 months ago

Node Tarantula ⭐ 23

web crawler/spider for nodejs

dependent packages 5total releases 9latest release April 18, 2014most recent commit 9 years ago

The 10 Latest Releases In Javascript Nodejs Crawler Open Source Projects

Crawlee ⭐ 12,059

dependent packages 42total releases 747latest release December 10, 2023most recent commit a day ago

Js Stellar Node Crawler ⭐ 12

Crawl the Stellar Network. Identify the nodes and determine their validating status.

dependent packages 2total releases 83latest release November 20, 2023most recent commit 4 months ago

npm @stellarbeat/js-stellar-node-crawler} Downloads

Htcrawl ⭐ 46

Htcrawl is nodejs module for the recursive crawling of single page applications (SPA) using javascript

dependent packages 2total releases 8latest release November 16, 2023most recent commit 4 months ago

X Crawl ⭐ 718

dependent packages 1total releases 59latest release November 09, 2023most recent commit 3 months ago

Webster ⭐ 465

a reliable high-level web crawling & scraping framework for Node.js.

dependent packages 2total releases 42latest release November 09, 2023most recent commit 5 months ago

Fdir ⭐ 1,191

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

dependent packages 98total releases 35latest release November 05, 2023most recent commit 5 months ago

Js Stellar Node Connector ⭐ 15

Connect and interact with nodes in the Stellar Network over the tcp protocol.

dependent packages 3total releases 70latest release October 30, 2023most recent commit 4 months ago

npm @stellarbeat/js-stellar-node-connector} Downloads

Google Play Scraper ⭐ 2,108

Node.js scraper to get data from Google Play

dependent packages 29total releases 86latest release August 09, 2023most recent commit 5 months ago

Node Crawler ⭐ 6,610

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

dependent packages 146total releases 31latest release December 30, 2022most recent commit 3 months ago

Article Extractor ⭐ 1,297

To extract main article from given URL with Node.js

dependent packages 10total releases 156latest release December 01, 2022most recent commit 3 months ago

Popular Data Processing Categories