Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About
Search results for crawler headless browsers
crawler
x
headless-browsers
x
1 search results found
Crawlee
⭐
12,871
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Katana
⭐
7,995
A next-generation crawling and spidering framework.
Headless Chrome Crawler
⭐
5,051
Distributed crawler powered by Headless Chrome
Rod
⭐
4,505
A Devtools driver for web automation and scraping
Crawlergo
⭐
2,642
A powerful browser crawler for web vulnerability scanners
Awesome Puppeteer
⭐
2,245
A curated list of awesome puppeteer resources.
Rendora
⭐
1,950
dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites
Kimuraframework
⭐
874
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
Jvppeteer
⭐
549
Headless Chrome For Java (Java 爬虫)
Nodejs Stuff
⭐
484
Node.js libs I want to keep in mind.
Wscan
⭐
415
Wscan is a web security scanner that focuses on web security, dedicated to making web security accessible to everyone.
Dataflowkit
⭐
394
Extract structured data from web sites. Web sites scraping.
Fakebrowser
⭐
290
🤖 Fake fingerprints to bypass anti-bot systems. Simulate mouse and keyboard operations to make behavior like a real person.
Ppspider
⭐
278
web spider built by puppeteer, support task-queue and task-scheduling by decorators,support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架,提供灵活的任务队列管理调度方案,提供便捷的数据保存方案(ne
Intoli Article Materials
⭐
255
All of the supporting materials for articles from Intoli's blog.
Arachnid
⭐
246
Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites
Selenium Crawler
⭐
119
Sometimes sites make crawling hard. Selenium-crawler uses selenium automation to fix that.
Abotx
⭐
106
Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.
Scrapper
⭐
83
Web scraper with a simple REST API living in Docker and using a Headless browser and Readability.js for parsing.
Puppeteer Walker
⭐
72
a puppeteer walker 🕷 🕸
Querylist Phantomjs
⭐
45
QueryList Plugin: Use PhantomJS to crawl Javascript dynamically rendered pages.(headless WebKit ) 使用PhantomJS采集JavaScript动态渲染的页面
Koa Seo
⭐
35
koa SEO middleware
Crawlersamples
⭐
34
This is a Puppeteer+AngleSharp crawler console app samples, used C# 7.1 coding and dotnet core build.
Yurun Crawler
⭐
28
宇润爬虫框架(Yurun Crawler) 是一个低代码、高性能、分布式爬虫采集框架,基于 imi 框架开发,运行在 Swoole 常驻内存的协程环境。
Mimo Crawler
⭐
24
A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.
Kimurai
⭐
19
Kimurai is a modern web scraping framework written in Ruby which works out of box with headless chromium/firefox, phantomjs, or simple HTTP requests and allows to scrape and interact with javascript rendered websites
Screamingfrog Docker
⭐
19
Docker image for ScreamingFrog version 16
Querylist Puppeteer
⭐
15
QueryList Plugin: Use Puppeteer to crawl Javascript dynamically rendered pages.(Headless Chrome ) 使用Puppeteer采集JavaScript动态渲染的页面
Doffy
⭐
13
a web auto run lib base on chrome headless
Headless Crawler
⭐
12
A crawler implemented using a headless browser (Chrome).
Axegrinder
⭐
12
Crawl websites for accessibility issues from the command line.
Spiking
⭐
12
A lightweight web crawler.
Awesome Puppeteer Zh
⭐
10
🇨🇳翻译: <awesome-puppeteer> Puppeteer 资源的精选列表 ❤️ 校对 ✅
Playwright Task Server
⭐
9
A headless browser manager with multi tasking RESTful API, crawling oriented
Scrapyteer
⭐
9
Web crawling & scraping framework for Node.js on top of headless Chrome browser
Express Middleware Seo
⭐
8
Webpage pre-rendering middleware, base on headless chrome⚡️
Botasaurus Starter
⭐
7
🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖
Puppeteer Scraper
⭐
6
Implement crawlers using a sane api on top of codeceptjs and puppeteer
Puppeteer Page Pool
⭐
6
A Page resource pool for Puppeteer.
Hkexnews_scrapy
⭐
5
使用 Scrapy 拿滬港通及深港通持股紀錄
Full Page Cache Warmer
⭐
5
🔥=> A website crawler that fully loads pages using headless Chrome AND mimics browser HTTP headers to NGINX or Varnish
Arachnida
⭐
5
App to scrap the web, for people without coding skills. Fully integrates WebCrawlers (Headless Chrome) and the interface to deal with it.
Related Searches
Python Crawler (4,545)
Javascript Headless Browsers (1,178)
Javascript Crawler (1,142)
Crawler Scrapy (988)
Scraper Crawler (896)
Java Crawler (807)
Crawler Spider (709)
1-1 of 1 search results
Privacy
|
About
|
Terms
|
Follow Us On Twitter
Copyright 2018-2024 Awesome Open Source. All rights reserved.