Awesome Open Source

Programming Languages

Search results for typescript crawler

71 search results found

Crawlee ⭐ 12,106

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Node Rate Limiter ⭐ 1,444

A generic rate limiter for node.js. Useful for API clients, web crawling, or other tasks that need to be throttled

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

Linkinator ⭐ 955

🐿 Scurry around your site and find all those broken links.

X Crawl ⭐ 718

x-crawl is a flexible Node.js multifunctional crawler library. Flexible usage and numerous functions can help you quickly, safely, and stably crawl pages, interfaces, and files. ---------------- x-crawl 是一个灵活的 Node.js 多功能爬虫库。灵活的使用方式和众多的功能可以帮助您快速、安全、稳定地爬取页面、接口以及文件。

Nintendo Switch Eshop ⭐ 513

Crawler for Nintendo Switch eShop

Lighthouse Parade ⭐ 355

A Node.js command line tool that crawls a domain and gathers lighthouse performance data for every page.

Crawling Infrastructure ⭐ 321

Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues.

Clipper.js ⭐ 311

HTML to Markdown converter and crawler.

Ppspider ⭐ 278

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（ne

Algoliasearch Netlify ⭐ 258

Official Algolia Plugin for Netlify. Index your website to Algolia when deploying your project to Netlify with the Algolia Crawler

Authcov ⭐ 199

Web app authorisation coverage scanning

estela, an elastic web scraping cluster 🕸

Douyubarrage Pro ⭐ 138

(2020年最新)斗鱼弹幕抓取及可视化管理平台第二版，提供弹幕抓取、弹幕实时发送速度可视化、抓取记录

Javbus Api ⭐ 136

一个自我托管的 JavBus API 服务

Take a snapshot of any website.

Npm Search ⭐ 122

🗿 npm ↔️ Algolia replication tool ⛷️ 🐌 🛰️

Double Agent ⭐ 120

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

Dungeondash ⭐ 76

Phaser 3 Open Source Dungeon Crawler Experiment

Disfactory ⭐ 63

違章工廠舉報系統

Local Api Client Typescript ⭐ 54

Official JavaScript/TypeScript library for interacting with Kameleo Client

Ui Router Metatags ⭐ 52

Metatags support for the AngularUI Router

Lara Dotng ⭐ 49

Twitter Bot for the awesome Public Transit Directions Assistant - Lara.ng

nodejs 爬虫框架. crawler framework for nodejs

Openartbrowser ⭐ 33

Exploring the world of arts using open data

search & get datas from youtube no google account needed

Medium Stat Box ⭐ 33

Practical pinned gist which show your latest medium status 📌

Noscrape ⭐ 32

obfuscate text via node to make scraping your content really difficult

Extension ⭐ 32

web scraping extension

Nodespider ⭐ 32

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

Serverless Instagram Crawler ⭐ 29

serverless, instagram hashtag crawler with lambda, dynamoDB

A11y Sitechecker ⭐ 29

Automatic accessibility checker with website crawling + screenshots for easy use

Billboard Json ⭐ 27

🎧 Get json type billboard hot 100 chart

Webmagician Ui ⭐ 26

An admin UI project for a configurable web crawler platform

Nest Crawler ⭐ 26

An easiest crawling and scraping module for NestJS

Retro Env Can Weather Chan ⭐ 24

Retro Environment Canada Weather Channel for your browser

Puppet Master ⭐ 22

Puppeteer as a service hosted on Saasify.

Anime Tracker ⭐ 20

🕸️ All in one place to track your favorite animes

Recursive ⭐ 20

A chrome extension web crawler written in Typescript / Javascript

Movierater ⭐ 20

A useful website for finding movie's rating in Chinese and English. By crawling Yahoo, Ptt, IMDB.

Sentinel Cendertron ⭐ 19

Cendertron = Crawler + cendertron, Crawl AJAX-heavy client-side Single Page Applications (SPAs), deploying with docker, focusing on scraping requests(page urls, apis, etc.), followed by pentest tools(Sqlmap, etc.). Cendertron can be used for extracting requests(page urls, apis, etc.) from your Web 2.0 page.

🇰🇷 Realtime integrated information analysis service

Websight ⭐ 18

🕷A simple but *really* fast crawler built with Node.js & TypeScript

Flixhq Core ⭐ 18

Nodejs library that provides an Api for obtaining the movies information from FlixHQ website.

Logo Scrape ⭐ 17

🕷🚀 Scrapes/Crawls the logo from a provided url(s)/website for your Node.js applications.

Images Grabber ⭐ 17

🖼️ Get all images from pixiv/twitter/deviantart

Bright Cli ⭐ 16

Command Line Interface (CLI) tool for NeuraLegion's solutions.

Js Stellar Node Connector ⭐ 15

Connect and interact with nodes in the Stellar Network over the tcp protocol.

Papercut ⭐ 15

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

All In One API to easily scrape data from any website, without worrying about captchas and bot detection mecanisms.

Ppspider_example ⭐ 14

ppspider爬虫例子，B站视频信息及评论爬取，qq音乐信息及评论爬取，推特主题评论和用户信息爬取

Translators ⭐ 14

🇻🇨 구글, 파파고, 카카오 번역기 비교 서비스

Node Webcheck ⭐ 14

A node.js module to analyse websites for SEO, validation and code-quality

Book Spider ⭐ 14

🎉 开箱即用的高性能可自定义的笔趣阁小说爬虫快速下载无广告小说

Web crawler based on Puppeteer

Test Crawler ⭐ 13

⚒ Web crawler that analyzes and dissects subtitles into database entries

Search for videos, playlists, channels, movies. live and musics on youtube without api key.

Flatcrawl Processors ⭐ 12

A set of processors that will instantly inform users via a set of channels (ie. Telegram) of new flats that are found on different rental websites.

Serverless Prerender ⭐ 12

Serverless implementation of Prerender service

Puppeteer Typescript Boilerplate ⭐ 12

A boilerplate for Puppeteer + TypeScript.

Spiderwebai ⭐ 12

The fastest way to scrape data to feed your AI models and LLMs

Js Stellar Node Crawler ⭐ 12

Crawl the Stellar Network. Identify the nodes and determine their validating status.

Crawl_freess ⭐ 11

用于爬取ssr地址，仅供学习🤖

Microdocs ⭐ 11

Documentation that scales with your Microservices

Arachnid Seo Js ⭐ 11

Web crawler for extracting internal site links info for SEO auditing & optimization purposes

A fairly intuitive & powerful framework that enables you to collect & save articles and news from all over the web.

Billboard Player ⭐ 10

🎹 Free billboard hot 100 M/V streaming service

Batterypatcher ⭐ 9

Playwright Task Server ⭐ 9

A headless browser manager with multi tasking RESTful API, crawling oriented

Distributed web scraping framework

It Events ⭐ 9

中国 IT 活动日历

Scrapyteer ⭐ 9

Web crawling & scraping framework for Node.js on top of headless Chrome browser

Bundesdatenkrake ⭐ 8

Extraction, versioning and machine-readable provisioning of public data.

Metafy Svg ⭐ 8

Easily crawl a website's metadata and generate SVG as a service.

Thai Gold Api ⭐ 8

API สำหรับแสดงราคาทองคำ

Crawler Br News ⭐ 8

📰 A node crawler to get brazilian news in main websites.

(ˈhɑr və stər) A Page-Object-Generator for Protractor-Tests in TypeScript.

Scrape Them All ⭐ 7

🚀 An easy-to-handle Node.js scraper that allow you to scrape them all in a record time.

Spider Nodejs ⭐ 6

Spider ported to Node.js

Angular7 Universal ⭐ 6

A sample project including basic angular concepts with server side rendering

Vscode Hydrate ⭐ 6

VS Code extension for Hydrate, which crawls a kubernetes cluster and generates a high level description of your deployments.

Zalando Scraper ⭐ 5

Apify actor extracting data from Zalando

Express Bot ⭐ 5

Crawler(robots) decision middleware for Express

Crawldiff ⭐ 5

Crawler to check two different website for differences, great for website migrations.

Casper Metrics ⭐ 5

Casper Metrics - fast blockchain crawler, API and analysis tool

Spiderman ⭐ 5

Minimalistic web crawler for Node.js

Web Master ⭐ 5

Web mastering tools for my personal services

Automator Sdk ⭐ 5

Automator CI Scripts and SDK project

Yasse Crawlers ⭐ 5

A crawler manager for scraping series (manga, anime, webtoons and novels) info from a lot of websites and APIs.

Makes visual reviews of web app releases easy.

Node Hoyowiki Api ⭐ 5

A unofficial Node.js library for HoYoWiki API, can get the details of Genshin Impact items!

Related Searches

Typescript Angular (35,567)

Typescript Reactjs (31,235)

Typescript Command Line (21,993)

Typescript Testing (21,336)

Typescript Vue (5,971)

Typescript Nextjs (5,496)

Typescript Html (5,128)

Typescript Redux (5,035)

Typescript Docker (4,994)

Typescript Nest (4,761)

1-71 of 71 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.