Awesome Open Source
Awesome Open Source
Combined Topics
crawler x
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194

The Top 2,877 Crawler Open Source Projects on Github

Categories > Data Processing > Crawler
Scrapy ⭐ 42,593
Scrapy, a fast high-level web crawling & scraping framework for Python.
Lux ⭐ 16,806
👾 Fast and simple video download library and CLI tool written in Go
Colly ⭐ 15,763
Elegant Scraper and Crawler Framework for Golang
Pyspider ⭐ 15,282
A Powerful Spider(Web Crawler) System in Python.
Proxy_pool ⭐ 14,044
Python爬虫代理IP池(proxy pool)
Newspaper ⭐ 11,537
News, full-text, and article metadata extraction in Python 3. Advanced docs:
Webmagic ⭐ 10,225
A scalable web crawler framework for Java.
Crawlab ⭐ 8,538
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
Examples Of Web Crawlers ⭐ 8,424
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
Avbook ⭐ 7,936
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
Photon ⭐ 7,822
Incredibly fast crawler designed for OSINT.
Python ⭐ 7,400
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
Node Crawler ⭐ 6,076
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Wechatsogou ⭐ 5,241
基于搜狗微信搜索的微信公众号爬虫接口
Headless Chrome Crawler ⭐ 5,051
Distributed crawler powered by Headless Chrome
Scrapy Redis ⭐ 5,013
Redis-based components for Scrapy.
Ferret ⭐ 4,858
Declarative web scraping
Haipproxy ⭐ 4,835
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
Awesome Crawler ⭐ 4,747
A collection of awesome web crawler,spider in different languages
Spider Flow ⭐ 4,260
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
Dom Crawler ⭐ 3,513
The DomCrawler component eases DOM navigation for HTML and XML documents.
Scylla ⭐ 3,418
Intelligent proxy pool for Humans™ (Maintainer needed)
Autoscraper ⭐ 3,281
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Dotnetspider ⭐ 3,204
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
Toapi ⭐ 3,153
Every web site provides APIs.
Ecommercecrawlers ⭐ 3,124
实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:
Arachni ⭐ 2,954
Web Application Security Scanner Framework
Proxybroker ⭐ 2,664
Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭
Browser Fingerprinting ⭐ 2,594
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?
Crawler_illegal_cases_in_china ⭐ 2,467
Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。 [AD]中文知识图谱门户
Querylist ⭐ 2,333
🕷 The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Googlescraper ⭐ 2,325
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
Gecco ⭐ 2,314
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
Instagram Scraper ⭐ 2,161
scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot
Crawler ⭐ 2,071
An easy to use, powerful crawler implemented in PHP. Can execute Javascript.
Python3 Spider ⭐ 1,995
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
Gain ⭐ 1,972
Web crawling framework based on asyncio.
Gocrawl ⭐ 1,929
Polite, slim and concurrent web crawler.
Abot ⭐ 1,919
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Decryptlogin ⭐ 1,897
DecryptLogin: APIs for loginning some websites by using requests.
Red_hawk ⭐ 1,842
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
Dxy Covid 19 Crawler ⭐ 1,822
2019新型冠状病毒疫情实时爬虫及API | COVID-19/2019-nCoV Realtime Infection Crawler and API
Work_crawler ⭐ 1,756
Download comics novels 小说漫画下载工具 小説漫画のダウンローダ 小說漫畫下載:腾讯漫画 大角虫漫画 有妖气 知音漫客 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫伊甸園 快看漫画 微博动漫 733动漫网 大古漫画网 漫画DB 無限動漫 動漫狂 卡推漫画 动漫之家 动漫屋 古风漫画网 36漫画网 亲亲漫画网 乙女漫画 comico webtoons 咚漫 ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミック サイコミ;アルファポリス カクヨム ハーメルン 小説家になろう 起点中文网 八一中文网 顶点小说 落霞小说网 努努书坊 笔趣阁→epub.
Crawlergo ⭐ 1,714
A powerful browser crawler for web vulnerability scanners
Lianjia Beike Spider ⭐ 1,695
链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 🚁，点星支持
Rendora ⭐ 1,651
dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites
Go_spider ⭐ 1,629
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
Google Play Scraper ⭐ 1,626
Node.js scraper to get data from Google Play
Pspider ⭐ 1,617
简单易用的Python爬虫框架，QQ交流群：597510560
Geziyor ⭐ 1,589
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Weibo Crawler ⭐ 1,541
新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频
Crawler Detect ⭐ 1,537
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Ruia ⭐ 1,507
Async Python 3.6+ web scraping micro-framework based on asyncio
Lightcrawler ⭐ 1,354
Crawl a website and run it through Google lighthouse
Vulnx ⭐ 1,273
vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform a quick CMS security detection, information collection (including sub-domain name, ip address, country information, organizational information and time zone, etc.) and vulnerability scanning.
Swiftlinkpreview ⭐ 1,260
It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.
Scrapoxy ⭐ 1,243
Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!
Finalrecon ⭐ 1,219
The Last Web Recon Tool You'll Need
Wombat ⭐ 1,217
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Jd Autobuy ⭐ 1,208
Python爬虫，京东自动登录，在线抢购商品
Openwpm ⭐ 1,175
A web privacy measurement framework
Gospider ⭐ 1,166
Gospider - Fast web spider written in Go
Torbot ⭐ 1,119
Dark Web OSINT Tool
Xalpha ⭐ 1,117
基金投资管理回测引擎
News Please ⭐ 1,116
news-please - an integrated web crawler and information extractor for news that just works
Tumblr Crawler ⭐ 1,105
Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片，视频
Diskover Community ⭐ 1,086
Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch
Autocrawler ⭐ 1,038
Google, Naver multiprocess image web crawler (Selenium)
Beanbun ⭐ 1,025
Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。
Fscrawler ⭐ 1,025
Elasticsearch File System Crawler (FS Crawler)
Appcrawler ⭐ 1,023
基于appium的app自动遍历工具
Dirhunt ⭐ 1,009
Find web directories without bruteforce
Tumblthree ⭐ 922
A Tumblr Blog Backup Application
Instagram Profilecrawl ⭐ 901
📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.
Sqliv ⭐ 864
massive SQL injection vulnerability scanner
Mzitu ⭐ 853
👧 美女写真套图爬虫（二）
Zhihu Crawler ⭐ 843
zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目
Pxer ⭐ 824
A tool for pixiv.net. 人人可用的P站爬虫
Crawler ⭐ 819
A high performance web crawler / scraper in Elixir.
Kimuraframework ⭐ 808
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
Grab Site ⭐ 795
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Till ⭐ 770
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
Creeper ⭐ 762
🐾 Creeper - The Next Generation Crawler Framework (Go)
Fetchbot ⭐ 758
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Lulu ⭐ 752
[Unmaintained] A simple and clean video/music/image downloader 👾
Feapder ⭐ 707
feapder是一款支持分布式、批次采集、任务防丢、报警丰富的python爬虫框架
Scrapyrt ⭐ 701
HTTP API for Scrapy spiders
Bilili ⭐ 677
🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器
Spidr ⭐ 663
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Icrawler ⭐ 653
A multi-thread crawler framework with many builtin image crawlers provided.
Course Crawler ⭐ 633
🎓 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载。
Fess ⭐ 623
Fess is very powerful and easily deployable Enterprise Search Server.
Easy Scraping Tutorial ⭐ 618
Simple but useful Python web scraping tutorial code.
Xsrfprobe ⭐ 614
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
Filemasta ⭐ 605
A search application to explore, discover and share online files
Pyptt ⭐ 599
支援 PTT 還有 PTT2 的 PTT API
Baiduimagespider ⭐ 591
一个超级轻量的百度图片爬虫
Magnet Dht ⭐ 591
✌️ Python3 BitTorrent DHT crawler
Xxl Crawler ⭐ 576
A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）
Newcrawler ⭐ 576
Free Web Scraping Tool with Java
1-100 of 2,877 projects

Related Projects

Python Crawler Projects (1,455)
Web Crawler Projects (595)
Crawler Spider Projects (477)
Javascript Crawler Projects (476)
Crawler Scraper Projects (360)
Python3 Crawler Projects (294)
Python Python3 Crawler Projects (253)
Crawler Scrapy Projects (246)
Nodejs Crawler Projects (229)
Go Crawler Projects (228)
Python Crawler Spider Projects (219)
Python Crawler Scrapy Projects (198)
Java Crawler Projects (193)
Javascript Nodejs Crawler Projects (178)
Python Crawler Scraper Projects (158)
Php Crawler Projects (158)
Crawler Scraping Projects (151)
Crawler Crawling Projects (150)
Html Crawler Projects (114)
Selenium Crawler Projects (103)
C Sharp Crawler Projects (93)
Python Selenium Crawler Projects (80)
Crawler Puppeteer Projects (77)
Crawler Webscraping Projects (77)
Bot Crawler Projects (76)
Python Crawler Scraping Projects (75)
Crawler Scraper Scraping Projects (75)
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194
Privacy policy
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open Source 2018-2022