Awesome Open Source

Programming Languages

Search results for crawler puppeteer

56 search results found

Crawlee ⭐ 12,059

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Headless Chrome Crawler ⭐ 5,051

Distributed crawler powered by Headless Chrome

Browser Fingerprinting ⭐ 3,353

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

Puppeteer Sharp ⭐ 3,135

Headless Chrome .NET API

Awesome Puppeteer ⭐ 2,245

A curated list of awesome puppeteer resources.

Rendora ⭐ 1,950

dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites

X Crawl ⭐ 718

x-crawl is a flexible Node.js multifunctional crawler library. Flexible usage and numerous functions can help you quickly, safely, and stably crawl pages, interfaces, and files. ---------------- x-crawl 是一个灵活的 Node.js 多功能爬虫库。灵活的使用方式和众多的功能可以帮助您快速、安全、稳定地爬取页面、接口以及文件。

Jvppeteer ⭐ 549

Headless Chrome For Java （Java 爬虫）

Browsertrix Crawler ⭐ 470

Run a high-fidelity browser-based crawler in a single Docker container

Webster ⭐ 465

a reliable high-level web crawling & scraping framework for Node.js.

Linkedin Profile Scraper Api ⭐ 404

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

Crawling Infrastructure ⭐ 321

Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues.

Fakebrowser ⭐ 290

🤖 Fake fingerprints to bypass anti-bot systems. Simulate mouse and keyboard operations to make behavior like a real person.

Ppspider ⭐ 278

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（ne

Make a ZIM file from any Web site and surf offline!

Chromium_for_spider ⭐ 182

dynamic crawler for web vulnerability scanner

Squidwarc ⭐ 163

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

Site Audit Seo ⭐ 151

Web service and CLI tool for SEO site audit: crawl site, lighthouse all pages, view public reports in browser. Also output to console, json, csv, xlsx, Google Drive.

Double Agent ⭐ 120

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

Csharpcrawler ⭐ 118

C#爬虫示例程序，想学习爬虫入门知识的可以看过来。后续会慢慢加入更多爬虫相关的知识。

Tracker Radar Collector ⭐ 109

🕸 Modular, multithreaded, puppeteer-based crawler

Actor Scraper ⭐ 93

House of Apify Scrapers. Generic scraping actors with a simple UI to handle complex web crawling and scraping use cases.

Bots Zoo ⭐ 90

A command-line tool to crawl websites using puppeteer.

Puppeteer Walker ⭐ 72

a puppeteer walker 🕷 🕸

Local Api Examples ⭐ 64

Useful and easy to understand examples written in Node.js and .NET Core about web scraping and automated browsing with Kameleo Client

Crawler Cases Demo ⭐ 56

Local Api Client Typescript ⭐ 54

Official JavaScript/TypeScript library for interacting with Kameleo Client

Local Api Client Python ⭐ 53

Official Python library for interacting with Kameleo Client

Meirim is an open-source smart city application that facilitates transparency in urban planning.

Lara Dotng ⭐ 49

Twitter Bot for the awesome Public Transit Directions Assistant - Lara.ng

Browser As A Service ⭐ 43

A web browser 🌎 hosted as a service, to render your JavaScript web pages as HTML

Pdf Crawler ⭐ 39

SimFin's open source PDF crawler

Local Api Client Csharp ⭐ 39

This .NET Standard package provides convenient access to the Local API REST interface of the Kameleo Client.

Crawlersamples ⭐ 34

This is a Puppeteer+AngleSharp crawler console app samples, used C# 7.1 coding and dotnet core build.

Tw Stock Telegram Bot ⭐ 33

台股機器人，提供即時個股及大盤報價、走勢、新聞、盤後資料等 Telegram bot to query real-time TW stock quotes, charts, news, and other related information

Chromium / Puppeteer site crawler

A11y Sitechecker ⭐ 29

Automatic accessibility checker with website crawling + screenshots for easy use

Micro Website Api ⭐ 29

An API microservice that crawls dynamic website powered by puppeteer.

A web spider framework

Puppet Master ⭐ 22

Puppeteer as a service hosted on Saasify.

Selector Finder ⭐ 19

Find a CSS selector on a public site

Actor Templates ⭐ 19

This project is the 🏠 home of Apify actor template projects to help users quickly get started.

Slackwebhooksgithubcrawler ⭐ 19

Search for Slack Webhooks token publicly exposed on Github

pppr is a prerender service

Web Crawler created with Node.js and Puppeteer

Spider Video ⭐ 16

Node 爬取头条视频并保存

Soccer Scrape ⭐ 16

📃 Scrape football data from Bet365

Spiderpuppeteer ⭐ 15

Use Puppeteer crawl a SPA (Single-Page Application)/generate pre-rendered content and etc...

Puppeteer Pdf ⭐ 15

使用Node.js爬取网页内容并且生成本地PDF文件

Querylist Puppeteer ⭐ 15

QueryList Plugin: Use Puppeteer to crawl Javascript dynamically rendered pages.(Headless Chrome ) 使用Puppeteer采集JavaScript动态渲染的页面

Ppspider_example ⭐ 14

ppspider爬虫例子，B站视频信息及评论爬取，qq音乐信息及评论爬取，推特主题评论和用户信息爬取

Just enough framework to make puppeteer your best friend.

Web crawler based on Puppeteer

Ciao Ssr ⭐ 12

A server side render service based on puppeteer

A lightweight web crawler.

Headless Crawler ⭐ 12

A crawler implemented using a headless browser (Chrome).

Puppeteer Typescript Boilerplate ⭐ 12

A boilerplate for Puppeteer + TypeScript.

Arachnid Seo Js ⭐ 11

Web crawler for extracting internal site links info for SEO auditing & optimization purposes

Awesome Puppeteer Zh ⭐ 10

🇨🇳翻译: <awesome-puppeteer> Puppeteer 资源的精选列表 ❤️ 校对 ✅

Storybook A11y Report ⭐ 10

CLI tool for storybook-addon-a11y.

Gitbook Printer ⭐ 8

Exports New Gitbooks to PDF using Puppeteer

Node Crawler On Mongodb ⭐ 8

🕷 NodeJS + Puppeteer crawler on MongoDB

Puppeteer Scraper ⭐ 6

Implement crawlers using a sane api on top of codeceptjs and puppeteer

Puppeteer Page Pool ⭐ 6

A Page resource pool for Puppeteer.

Makes visual reviews of web app releases easy.

Parentaljobs ⭐ 5

Parents friendly jobs portal

Coinmarketcap Historical Data Crawler ⭐ 5

Command-line interface to fetch historical from coinmarketcap.com with puppeteer.

Wuzzuf Web Scrapper ⭐ 5

A web scrapper to fetch jobs with specific salaries or with queries given to the code, it was made with puppeteer and i made it for fun.

Google Flights Crawler ⭐ 5

Related Searches

Python Crawler (4,545)

Javascript Puppeteer (1,642)

Javascript Crawler (1,142)

Crawler Spider (1,072)

Crawler Scrapy (988)

Scraper Crawler (896)

Java Crawler (807)

1-56 of 56 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.