Awesome Open Source
Awesome Open Source
Combined Topics
spider x
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194

The Top 1,573 Spider Open Source Projects on Github

Awesome Spider ⭐ 16,075
爬虫集合
Colly ⭐ 15,844
Elegant Scraper and Crawler Framework for Golang
Proxy_pool ⭐ 14,044
Python爬虫代理IP池(proxy pool)
Awesome Python Login Model ⭐ 13,571
😮python模拟登陆一些大型网站，还有一些简单的爬虫，希望对你们有所帮助❤️，如果喜欢记得给个star哦🌟
Crawlab ⭐ 8,554
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
Examples Of Web Crawlers ⭐ 8,424
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
Avbook ⭐ 7,936
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
Photon ⭐ 7,822
Incredibly fast crawler designed for OSINT.
Pholcus ⭐ 6,907
Pholcus is a distributed high-concurrency crawler software written in pure golang
Anti Anti Spider ⭐ 6,824
越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）
Node Crawler ⭐ 6,081
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Infospider ⭐ 6,016
INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。
Haipproxy ⭐ 4,835
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
Awesome Crawler ⭐ 4,747
A collection of awesome web crawler,spider in different languages
Toplist ⭐ 4,362
今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:https://mo.fish
Spider Flow ⭐ 4,260
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
Toapi ⭐ 3,153
Every web site provides APIs.
Fiction_house ⭐ 2,682
小说精品屋是一个多平台（web、安卓app、微信小程序）、功能完善的屏幕自适应小说漫画连载系统，包含精品小说专区、轻小说专区和漫画专区。包括小说/漫画分类、小说/漫画搜索、小说/漫画排行、完本小说/漫画、小说/漫画评分、小说/漫画在线阅读、小说/漫画书架、小说/漫画阅读记录、小说下载、小说弹幕、小说/漫画自动采集/更新/纠错、小说内容自动分享到微博、邮件自动推广、链接自动推送到百度搜索引擎等功能。
Core ⭐ 2,667
🔞 JAVClub - 让你的大姐姐不再走丢
Gerapy ⭐ 2,634
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
Browser Fingerprinting ⭐ 2,594
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?
Spiderkeeper ⭐ 2,539
admin ui for scrapy/open source scrapinghub
Dht ⭐ 2,424
BitTorrent DHT Protocol && DHT Spider.
Scrapydweb ⭐ 2,408
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO 👉
Querylist ⭐ 2,333
🕷 The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Novel Plus ⭐ 2,305
小说精品屋-plus是一个多端（PC、WAP）阅读、功能完善的原创文学CMS系统，由前台门户系统、作家后台管理系统、平台后台管理系统、爬虫管理系统等多个子系统构成，支持多模版、会员充值、订阅模式、新闻发布和实时统计报表等功能，新书自动入库，老书自动更新。
Owllook ⭐ 2,165
owllook-小说搜索引擎
Grab ⭐ 2,054
Web Scraping Framework
Python3 Spider ⭐ 1,995
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
Gain ⭐ 1,972
Web crawling framework based on asyncio.
Abot ⭐ 1,919
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Decryptlogin ⭐ 1,897
DecryptLogin: APIs for loginning some websites by using requests.
Lianjia Beike Spider ⭐ 1,695
链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 🚁，点星支持
Go_spider ⭐ 1,629
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
Pspider ⭐ 1,617
简单易用的Python爬虫框架，QQ交流群：597510560
Geziyor ⭐ 1,589
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Go Demo ⭐ 1,565
Go语言实例教程从入门到进阶，包括基础库使用、设计模式、面试易错点、工具类、对接第三方等
Crawler Detect ⭐ 1,537
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Ruia ⭐ 1,507
Async Python 3.6+ web scraping micro-framework based on asyncio
Bilibili Api ⭐ 1,466
哔哩哔哩的API调用模块
Wechat_articles_spider ⭐ 1,423
微信公众号文章的爬虫
Image Downloader ⭐ 1,254
Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.
Gospider ⭐ 1,166
Gospider - Fast web spider written in Go
Torbot ⭐ 1,119
Dark Web OSINT Tool
Glyphhanger ⭐ 1,099
Your web font utility belt. It can subset web fonts. It can find unicode-ranges for you automatically. It makes julienne fries.
Reptile ⭐ 1,081
🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish
Django Dynamic Scraper ⭐ 1,069
Creating Scrapy scrapers via the Django admin interface
Antcolony ⭐ 1,042
Nodejs实现的一个磁力链接爬虫 http://findit.keenwon.com (原域名http://findit.so )
Wechatsync ⭐ 1,030
一键同步文章到多个内容平台，支持今日头条、WordPress、知乎、简书、掘金、CSDN、typecho各大平台，一次发布，多平台同步发布。解放个人生产力
Beanbun ⭐ 1,025
Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。
Blackwidow ⭐ 986
A Python based web application scanner to gather OSINT and fuzz for OWASP vulnerabilities on a target website.
Pythoncrawler ⭐ 977
💗用python编写的爬虫项目集合
Jspider ⭐ 952
JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816
Spider ⭐ 907
A configurable web spider with a easy-to-use web console
Zhihu Crawler ⭐ 843
zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目
Crawler ⭐ 819
A high performance web crawler / scraper in Elixir.
Funpyspidersearchengine ⭐ 812
Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索
Grab Site ⭐ 795
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Oneblog ⭐ 792
👽 OneBlog，一个简洁美观、功能强大并且自适应的Java博客
V2raycloudspider ⭐ 788
​🚀 采集|免费|优质|的订阅链接；科学上网，从娃娃抓起！
Darknet_chinesetrading ⭐ 788
🚇暗网中文网监控爬虫(DEEPMIX)
Video_spider ⭐ 771
短视频去水印：抖音,皮皮虾,火山,微视,微博,绿洲,最右,轻视频,快手,全民小视频,巴塞电影,陌陌,Before避风,开眼,Vue Vlog 小咖秀,皮皮搞笑,全民K歌,西瓜视频,逗拍,虎牙,6间房,梨视频,新片场,acfun,美拍...
Baiduyunspider ⭐ 769
百度云网盘搜索引擎，包含爬虫 & 网站
Querido Diario ⭐ 765
📰 Brazilian government gazettes, accessible to everyone.
Creeper ⭐ 762
🐾 Creeper - The Next Generation Crawler Framework (Go)
Feapder ⭐ 713
feapder是一款支持分布式、批次采集、任务防丢、报警丰富的python爬虫框架
Qzoneexport ⭐ 694
QQ空间导出助手，用于备份QQ空间的说说、日志、私密日记、相册、视频、留言板、QQ好友、收藏夹、分享、最近访客为文件，便于迁移与保存
Bilili ⭐ 681
🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器
Spidr ⭐ 674
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Istock ⭐ 673
👉一个基于spring boot 实现的java股票爬虫(仅支持A股)，如果你❤️请⭐️ . V2升级版正在开发中！
Icrawler ⭐ 653
A multi-thread crawler framework with many builtin image crawlers provided.
Bdp Platform ⭐ 653
大数据生态解决方案数据平台：基于大数据、数据平台、微服务、机器学习、商城、自动化运维、DevOps、容器部署平台、数据平台采集、数据平台存储、数据平台计算、数据平台开发、数据平台应用搭建的大数据解决方案。
Domain_hunter ⭐ 633
A Burp Suite Extension that try to find all sub-domain, similar-domain and related-domain of an organization automatically! 基于流量自动收集整个企业或组织的子域名、相似域名、相关域名的burp插件
Xsrfprobe ⭐ 614
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
Python Spider ⭐ 594
豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章
Baiduimagespider ⭐ 591
一个超级轻量的百度图片爬虫
Newcrawler ⭐ 576
Free Web Scraping Tool with Java
Xxl Crawler ⭐ 576
A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）
Spider163 ⭐ 571
抓取网易云音乐热门评论
Netdiscovery ⭐ 557
NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。
Douyin ⭐ 550
API of DouYin for Humans used to Crawl Popular Videos and Musics
Crawly ⭐ 543
Crawly, a high-level web crawling & scraping framework for Elixir.
Go_jobs ⭐ 536
带你了解一下Golang的市场行情
Anti Webspider ⭐ 535
Web 端反爬技术方案
91porn_php ⭐ 521
最简单的91porn爬虫php版本
Dumpall ⭐ 483
一款信息泄漏利用工具，适用于.git/.svn源代码泄漏和.DS_Store泄漏
Qqzonemood ⭐ 478
QQZone mood spider and analysis. QQ空间多线程爬虫和数据挖掘。提供线上服务，扫码登陆即可自动爬取和分析数据，还有网易云年度报告风格的数据展示；使用docker-compose打包程序，方便部署；额外提供QQ空间抽奖小程序。
Movieheavens ⭐ 477
🎬 基于Pyqt5的简单电影搜索工具
Sign ⭐ 475
逆向app参数的思路和体验，安卓逆向
Spiders ⭐ 445
Python爬虫，返回一定格式的信息，下载，使用flask提供简易api。抖音无水印、皮皮虾、快手、网易云音乐、qq音乐、咪咕音乐、荔枝FM音频、知乎视频、最右语音、视频、微博......
Learnpython ⭐ 437
Python的基础练习代码与各种爬虫代码
Web_kg ⭐ 435
爬取百度百科中文页面，抽取三元组信息，构建中文知识图谱
Bt Btt ⭐ 432
磁力網站U3C3介紹以及域名更新
Tumblr_spider ⭐ 429
汤不热 python 多线程爬虫
Animesearcher ⭐ 428
整合第三方网站的视频和弹幕资源, 为白嫖党提供最佳看番追剧体验
Html2article ⭐ 425
Html网页正文提取
Baiduspider ⭐ 417
BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。
Fbcrawl ⭐ 415
A Facebook crawler
Gosint ⭐ 411
OSINT Swiss Army Knife
Jdpackage ⭐ 404
跨平台的京东全能工具包 仅供学习使用，技术交流群：108934299
1-100 of 1,573 projects

Related Projects

Python Spider Projects (963)
Crawler Spider Projects (477)
Spider Scrapy Projects (241)
Python3 Spider Projects (241)
Python Crawler Spider Projects (219)
Python Spider Scrapy Projects (202)
Python Python3 Spider Projects (199)
Javascript Spider Projects (194)
Java Spider Projects (137)
Go Spider Projects (111)
Scraper Spider Projects (99)
Html Spider Projects (80)
Nodejs Spider Projects (79)
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194
Privacy policy
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open Source 2018-2022