Awesome Open Source
Awesome Open Source
Combined Topics
crawler
x
Advertising
📦 10
All Projects
Application Programming Interfaces
📦 124
Applications
📦 192
Artificial Intelligence
📦 78
Blockchain
📦 73
Build Tools
📦 113
Cloud Computing
📦 80
Code Quality
📦 28
Collaboration
📦 32
Command Line Interface
📦 49
Community
📦 83
Companies
📦 60
Compilers
📦 63
Computer Science
📦 80
Configuration Management
📦 42
Content Management
📦 175
Control Flow
📦 213
Data Formats
📦 78
Data Processing
📦 276
Data Storage
📦 135
Economics
📦 64
Frameworks
📦 215
Games
📦 129
Graphics
📦 110
Hardware
📦 152
Integrated Development Environments
📦 49
Learning Resources
📦 166
Legal
📦 29
Libraries
📦 129
Lists Of Projects
📦 22
Machine Learning
📦 347
Mapping
📦 64
Marketing
📦 15
Mathematics
📦 55
Media
📦 239
Messaging
📦 98
Networking
📦 315
Operating Systems
📦 89
Operations
📦 121
Package Managers
📦 55
Programming Languages
📦 245
Runtime Environments
📦 100
Science
📦 42
Security
📦 396
Social Media
📦 27
Software Architecture
📦 72
Software Development
📦 72
Software Performance
📦 58
Software Quality
📦 133
Text Editors
📦 49
Text Processing
📦 136
User Interface
📦 330
User Interface Components
📦 514
Version Control
📦 30
Virtualization
📦 71
Web Browsers
📦 42
Web Servers
📦 26
Web User Interface
📦 210
The Top 372 Crawler Open Source Projects
Categories
>
Data Processing
>
Crawler
Scrapy
⭐
40,796
Scrapy, a fast high-level web crawling & scraping framework for Python.
Pyspider
⭐
15,010
A Powerful Spider(Web Crawler) System in Python.
Annie
⭐
14,896
👾 Fast, simple and clean video downloader
Colly
⭐
14,015
Elegant Scraper and Crawler Framework for Golang
Proxy_pool
⭐
12,581
Python爬虫代理IP池(proxy pool)
Newspaper
⭐
11,075
News, full-text, and article metadata extraction in Python 3. Advanced docs:
Examples Of Web Crawlers
⭐
10,212
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
Webmagic
⭐
9,832
A scalable web crawler framework for Java.
Photon
⭐
7,906
Incredibly fast crawler designed for OSINT.
Crawlab
⭐
7,887
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
Avbook
⭐
7,844
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
Python
⭐
6,838
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
Node Crawler
⭐
5,900
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Wechatsogou
⭐
5,055
基于搜狗微信搜索的微信公众号爬虫接口
Headless Chrome Crawler
⭐
4,991
Distributed crawler powered by Headless Chrome
Scrapy Redis
⭐
4,899
Redis-based components for Scrapy.
Haipproxy
⭐
4,831
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
Awesome Crawler
⭐
4,601
A collection of awesome web crawler,spider in different languages
Ferret
⭐
4,517
Declarative web scraping
Autoscraper
⭐
3,570
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Dom Crawler
⭐
3,356
The DomCrawler component eases DOM navigation for HTML and XML documents.
Scylla
⭐
3,241
Intelligent proxy pool for Humans™ (Maintainer needed)
Toapi
⭐
3,144
Every web site provides APIs.
Dotnetspider
⭐
3,003
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
Arachni
⭐
2,873
Web Application Security Scanner Framework
Ecommercecrawlers
⭐
2,745
实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:
Proxybroker
⭐
2,518
Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭
Querylist
⭐
2,301
🕷 The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Googlescraper
⭐
2,292
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
Gecco
⭐
2,224
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
Crawler_illegal_cases_in_china
⭐
2,207
Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。 [AD]中文知识图谱门户
Lianjia Beike Spider
⭐
2,180
链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 ，点星支持，仅供学习参考，请勿用于商业用途，后果自负。
Gain
⭐
1,992
Web crawling framework based on asyncio.
Instagram Scraper
⭐
1,986
scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot
Python3 Spider
⭐
1,951
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
Gocrawl
⭐
1,939
Polite, slim and concurrent web crawler.
Crawler
⭐
1,904
An easy to use, powerful crawler implemented in PHP. Can execute Javascript.
Abot
⭐
1,893
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Dxy Covid 19 Crawler
⭐
1,826
2019新型冠状病毒疫情实时爬虫及API | COVID-19/2019-nCoV Realtime Infection Crawler and API
Rendora
⭐
1,807
dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites
Red_hawk
⭐
1,712
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
Go_spider
⭐
1,711
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
Decryptlogin
⭐
1,624
APIs for loginning some websites by using requests.
Pspider
⭐
1,575
简单易用的Python爬虫框架，QQ交流群：597510560
Google Play Scraper
⭐
1,502
Node.js scraper to get data from Google Play
Crawler Detect
⭐
1,459
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Ruia
⭐
1,424
Async Python 3.6+ web scraping micro-framework based on asyncio
Lightcrawler
⭐
1,360
Crawl a website and run it through Google lighthouse
Scrapoxy
⭐
1,354
Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!
Work_crawler
⭐
1,349
Download comics novels 小说漫画下载工具 小説漫画のダウンローダ 小說漫畫下載:腾讯漫画 大角虫漫画 有妖气 知音漫客 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫伊甸園 快看漫画 微博动漫 733动漫网 大古漫画网 漫画DB 無限動漫 動漫狂 卡推漫画 动漫之家 动漫屋 古风漫画网 36漫画网 亲亲漫画网 乙女漫画 comico webtoons 咚漫 ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミック サイコミ;アルファポリス カクヨム ハーメルン 小説家になろう 起点中文网 八一中文网 顶点小说 落霞小说网 努努书坊 笔趣阁→epub.
Geziyor
⭐
1,295
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Vulnx
⭐
1,244
vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform a quick CMS security detection, information collection (including sub-domain name, ip address, country information, organizational information and time zone, etc.) and vulnerability scanning.
Swiftlinkpreview
⭐
1,235
It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.
Wombat
⭐
1,225
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Jd Autobuy
⭐
1,192
Python爬虫，京东自动登录，在线抢购商品
Crawlergo
⭐
1,175
A powerful browser crawler for web vulnerability scanners
Weibo Crawler
⭐
1,152
新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频
Openwpm
⭐
1,133
A web privacy measurement framework
Tumblr Crawler
⭐
1,124
Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片，视频
Pixeval
⭐
1,113
A Strong, Fast and Flexible Pixiv Client based on .NET Core and WPF
Beanbun
⭐
1,106
Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。
News Please
⭐
1,068
news-please - an integrated web crawler and information extractor for news that just works.
Dirhunt
⭐
1,021
Find web directories without bruteforce
Diskover
⭐
1,002
File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch
Autocrawler
⭐
999
Google, Naver multiprocess image web crawler (Selenium)
Finalrecon
⭐
990
The Last Web Recon Tool You'll Need
Appcrawler
⭐
953
基于appium的app自动遍历工具
Fscrawler
⭐
949
Elasticsearch File System Crawler (FS Crawler)
Mzitu
⭐
937
👧 美女写真套图爬虫（二）
Tumblthree
⭐
921
A Tumblr Blog Backup Application
Zhihu Crawler
⭐
896
zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目
Torbot
⭐
886
Dark Web OSINT Tool
Gospider
⭐
861
Gospider - Fast web spider written in Go
Sqliv
⭐
859
massive SQL injection vulnerability scanner
Instagram Profilecrawl
⭐
848
📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.
Pic Gather
⭐
836
[ Closed ] 🎨 image collector, which supports custom acquisition source configuration and is compatible with MacOS and Windows operating systems.
Xalpha
⭐
812
基金投资管理回测引擎
Pxer
⭐
802
A tool for pixiv.net. 人人可用的P站爬虫
Crawler
⭐
796
A high performance web crawler in Elixir.
Lulu
⭐
794
[Unmaintained] A simple and clean video/music/image downloader 👾
Kimuraframework
⭐
781
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
Creeper
⭐
763
🐾 Creeper - The Next Generation Crawler Framework (Go)
Fetchbot
⭐
760
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Grab Site
⭐
718
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Magnet Dht
⭐
713
✌️ Python3 BitTorrent DHT crawler
Scrapyrt
⭐
666
HTTP API for Scrapy spiders
Spidr
⭐
662
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Icrawler
⭐
644
A multi-thread crawler framework with many builtin image crawlers provided.
Price Monitor
⭐
634
京东商品价格监控：监控用户设定商品价格，降价邮件/微信提醒。技术：Python爬虫/IP代理池/JS接口爬取/Selenium页面爬取
Course Crawler
⭐
616
🎓 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载。
Baiduimagespider
⭐
605
一个超级轻量的百度图片爬虫
Easy Scraping Tutorial
⭐
598
Simple but useful Python web scraping tutorial code.
Xsrfprobe
⭐
589
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
Newcrawler
⭐
587
Free Web Scraping Tool with Java
Douyin
⭐
584
API of DouYin for Humans used to Crawl Popular Videos and Musics
Filemasta
⭐
582
A search application to explore, discover and share online files
Netdiscovery
⭐
579
NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。
Fess
⭐
577
Fess is very powerful and easily deployable Enterprise Search Server.
Xxl Crawler
⭐
568
A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）
Fbcrawl
⭐
555
A Facebook crawler
1-100 of 372 projects
Next >
Advertising
📦 10
All Projects
Application Programming Interfaces
📦 124
Applications
📦 192
Artificial Intelligence
📦 78
Blockchain
📦 73
Build Tools
📦 113
Cloud Computing
📦 80
Code Quality
📦 28
Collaboration
📦 32
Command Line Interface
📦 49
Community
📦 83
Companies
📦 60
Compilers
📦 63
Computer Science
📦 80
Configuration Management
📦 42
Content Management
📦 175
Control Flow
📦 213
Data Formats
📦 78
Data Processing
📦 276
Data Storage
📦 135
Economics
📦 64
Frameworks
📦 215
Games
📦 129
Graphics
📦 110
Hardware
📦 152
Integrated Development Environments
📦 49
Learning Resources
📦 166
Legal
📦 29
Libraries
📦 129
Lists Of Projects
📦 22
Machine Learning
📦 347
Mapping
📦 64
Marketing
📦 15
Mathematics
📦 55
Media
📦 239
Messaging
📦 98
Networking
📦 315
Operating Systems
📦 89
Operations
📦 121
Package Managers
📦 55
Programming Languages
📦 245
Runtime Environments
📦 100
Science
📦 42
Security
📦 396
Social Media
📦 27
Software Architecture
📦 72
Software Development
📦 72
Software Performance
📦 58
Software Quality
📦 133
Text Editors
📦 49
Text Processing
📦 136
User Interface
📦 330
User Interface Components
📦 514
Version Control
📦 30
Virtualization
📦 71
Web Browsers
📦 42
Web Servers
📦 26
Web User Interface
📦 210
