The Top 23 Typescript Crawler Open Source Projects

Open source projects categorized as Typescript Crawler

Categories > Data Processing > Crawler

Categories > Programming Languages > Typescript

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747most recent commit an hour ago

Node Rate Limiter ⭐ 1,444

A generic rate limiter for node.js. Useful for API clients, web crawling, or other tasks that need to be throttled

dependent packages 329total releases 16most recent commit 8 months ago

Fdir ⭐ 1,191

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

dependent packages 98total releases 35most recent commit 5 months ago

Linkinator ⭐ 955

🐿 Scurry around your site and find all those broken links.

dependent packages 313total releases 106most recent commit 4 months ago

X Crawl ⭐ 718

x-crawl is a flexible Node.js multifunctional crawler library. Flexible usage and numerous functions can help you quickly, safely, and stably crawl pages, interfaces, and files. ---------------- x-crawl 是一个灵活的 Node.js 多功能爬虫库。灵活的使用方式和众多的功能可以帮助您快速、安全、稳定地爬取页面、接口以及文件。

dependent packages 1total releases 59most recent commit 3 months ago

Nintendo Switch Eshop ⭐ 513

Crawler for Nintendo Switch eShop

most recent commit 3 years ago

Lighthouse Parade ⭐ 355

A Node.js command line tool that crawls a domain and gathers lighthouse performance data for every page.

dependent packages 2total releases 14most recent commit 6 months ago

Crawling Infrastructure ⭐ 321

Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues.

most recent commit 2 years ago

Clipper.js ⭐ 311

HTML to Markdown converter and crawler.

most recent commit 4 months ago

Ppspider ⭐ 278

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（ne

dependent packages 2total releases 85most recent commit 2 years ago

Algoliasearch Netlify ⭐ 258

Official Algolia Plugin for Netlify. Index your website to Algolia when deploying your project to Netlify with the Algolia Crawler

dependent packages 2total releases 34most recent commit 7 months ago

npm @algolia/algoliasearch-netlify-frontend} Downloads

Authcov ⭐ 199

Web app authorisation coverage scanning

dependent packages 2total releases 10most recent commit 2 years ago

Estela ⭐ 142

estela, an elastic web scraping cluster 🕸

most recent commit 3 months ago

Douyubarrage Pro ⭐ 138

most recent commit a year ago

Javbus Api ⭐ 136

一个自我托管的 JavBus API 服务

most recent commit 5 months ago

Taki ⭐ 135

Take a snapshot of any website.

dependent packages 4total releases 40most recent commit 2 years ago

Npm Search ⭐ 122

🗿 npm ↔️ Algolia replication tool ⛷️ 🐌 🛰️

most recent commit 5 months ago

Double Agent ⭐ 120

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

most recent commit 2 years ago

Dungeondash ⭐ 76

Phaser 3 Open Source Dungeon Crawler Experiment

most recent commit 4 years ago

Disfactory ⭐ 63

違章工廠舉報系統

most recent commit 5 months ago

Ncovr ⭐ 56

most recent commit 4 years ago

Local Api Client Typescript ⭐ 54

Official JavaScript/TypeScript library for interacting with Kameleo Client

dependent packages 2total releases 11most recent commit 10 months ago

npm @kameleo/local-api-client} Downloads

Ui Router Metatags ⭐ 52

Metatags support for the AngularUI Router

dependent packages 2total releases 8most recent commit 6 years ago

The 10 Most Depended On Typescript Crawler Open Source Projects

Node Rate Limiter ⭐ 1,444

A generic rate limiter for node.js. Useful for API clients, web crawling, or other tasks that need to be throttled

dependent packages 329total releases 16latest release May 19, 2021most recent commit 8 months ago

Linkinator ⭐ 955

🐿 Scurry around your site and find all those broken links.

dependent packages 313total releases 106latest release November 22, 2023most recent commit 4 months ago

Fdir ⭐ 1,191

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

dependent packages 98total releases 35latest release November 05, 2023most recent commit 5 months ago

Crawlee ⭐ 12,250

dependent packages 42total releases 747latest release December 10, 2023most recent commit an hour ago

Extension ⭐ 32

web scraping extension

dependent packages 13total releases 7latest release September 26, 2020most recent commit a year ago

npm get-set-fetch-extension-commons} Downloads

Node Webcheck ⭐ 14

A node.js module to analyse websites for SEO, validation and code-quality

dependent packages 11total releases 10latest release January 19, 2019most recent commit 5 years ago

Crawler ⭐ 13

Web crawler based on Puppeteer

dependent packages 4total releases 23latest release November 13, 2023most recent commit 6 months ago

Web Master ⭐ 5

Web mastering tools for my personal services

dependent packages 4total releases 17latest release November 01, 2019most recent commit 5 years ago

npm @web-master/node-web-scraper} Downloads

Usetube ⭐ 33

search & get datas from youtube no google account needed

dependent packages 4total releases 35latest release April 03, 2023most recent commit a year ago

A11y Sitechecker ⭐ 29

Automatic accessibility checker with website crawling + screenshots for easy use

dependent packages 4total releases 86latest release February 21, 2022most recent commit a year ago

The 10 Latest Releases In Typescript Crawler Open Source Projects

Crawlee ⭐ 12,250

dependent packages 42total releases 747latest release December 10, 2023most recent commit an hour ago

Linkinator ⭐ 955

🐿 Scurry around your site and find all those broken links.

dependent packages 313total releases 106latest release November 22, 2023most recent commit 4 months ago

Js Stellar Node Crawler ⭐ 12

Crawl the Stellar Network. Identify the nodes and determine their validating status.

dependent packages 2total releases 83latest release November 20, 2023most recent commit 5 months ago

npm @stellarbeat/js-stellar-node-crawler} Downloads

Crawler ⭐ 13

Web crawler based on Puppeteer

dependent packages 4total releases 23latest release November 13, 2023most recent commit 6 months ago

X Crawl ⭐ 718

dependent packages 1total releases 59latest release November 09, 2023most recent commit 3 months ago

Fdir ⭐ 1,191

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

dependent packages 98total releases 35latest release November 05, 2023most recent commit 5 months ago

Js Stellar Node Connector ⭐ 15

Connect and interact with nodes in the Stellar Network over the tcp protocol.

dependent packages 3total releases 70latest release October 30, 2023most recent commit 5 months ago

npm @stellarbeat/js-stellar-node-connector} Downloads

Noscrape ⭐ 32

obfuscate text via node to make scraping your content really difficult

dependent packages 2total releases 20latest release September 28, 2023most recent commit 4 months ago

Local Api Client Typescript ⭐ 54

Official JavaScript/TypeScript library for interacting with Kameleo Client

dependent packages 2total releases 11latest release July 20, 2023most recent commit 10 months ago

Lighthouse Parade ⭐ 355

A Node.js command line tool that crawls a domain and gathers lighthouse performance data for every page.

dependent packages 2total releases 14latest release July 17, 2023most recent commit 6 months ago

Popular Data Processing Categories