Awesome Open Source
Awesome Open Source
Combined Topics
spider x

The Top 3,782 Spider Open Source Projects

Colly ⭐ 18,665
Elegant Scraper and Crawler Framework for Golang
dependent packages 201total releases 22most recent commit 11 days ago
Awesome Spider ⭐ 18,426
爬虫集合
most recent commit a month ago
Proxy_pool ⭐ 16,786
Python爬虫代理IP池(proxy pool)
most recent commit 25 days ago
Pyspider ⭐ 15,653
A Powerful Spider(Web Crawler) System in Python.
dependent packages 2total releases 17most recent commit 2 months agopypi pyspider} Downloads
Python Spider ⭐ 15,506
🌈Python3网络爬虫实战：淘宝、京东、网易云、B站、12306、抖音、笔趣阁、漫画小说下载、音
most recent commit 2 months ago
Awesome Python Login Model ⭐ 14,424
😮python模拟登陆一些大型网站，还有一些简单的爬虫，希望对你们有所帮助❤️，如果喜欢记得给个s
most recent commit 6 months ago
Learn_python3_spider ⭐ 11,186
python爬虫教程系列、从0到1学习python爬虫，包括浏览器抓包，手机APP抓包，如 fiddler、mitmproxy，各种爬虫涉及的模块的使用，如：requests、beautifu 爬虫加密逆向破解，JS爬虫逆向，分布式爬虫，爬虫项目实战实例等
total releases 2most recent commit 2 months ago
Examples Of Web Crawlers ⭐ 11,050
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等 interesting examples of python crawlers that are friendly to beginners. )
most recent commit 9 months ago
Crawlab ⭐ 9,480
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
total releases 15most recent commit 12 days ago
Photon ⭐ 9,272
Incredibly fast crawler designed for OSINT.
total releases 18most recent commit a month ago
Avbook ⭐ 8,694
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
most recent commit 23 days ago
Spider Flow ⭐ 7,439
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
most recent commit 2 months ago
Pholcus ⭐ 7,199
Pholcus is a distributed high-concurrency crawler software written in pure golang
most recent commit 3 months ago
Anti Anti Spider ⭐ 6,824
越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库
most recent commit a year ago
Infospider ⭐ 6,415
INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透
most recent commit a month ago
Node Crawler ⭐ 6,340
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
dependent packages 126total releases 30most recent commit 4 days agonpm crawler} Downloads
Weibospider ⭐ 6,278
新浪微博爬虫，用python爬取新浪微博数据
total releases 26most recent commit 2 months ago
Awesome Web Scraping ⭐ 5,576
List of libraries, tools and APIs for web scraping and data processing.
most recent commit a month ago
Awesome Crawler ⭐ 5,308
A collection of awesome web crawler,spider in different languages
most recent commit a month ago
Haipproxy ⭐ 5,222
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
total releases 7most recent commit a month ago
Scrapy Redis ⭐ 5,141
Redis-based components for Scrapy.
dependent packages 21total releases 17most recent commit 6 months agopypi scrapy-redis} Downloads
Font Spider ⭐ 4,770
Smart webfont compression and format conversion tool
dependent packages 16total releases 51most recent commit 2 months agonpm font-spider} Downloads
Toplist ⭐ 4,362
今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:h
most recent commit a year ago
Ecommercecrawlers ⭐ 3,633
实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼
most recent commit 2 months ago
Dotnetspider ⭐ 3,496
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
dependent packages 7total releases 43most recent commit 2 months agonuget DotnetSpider} Downloads
Distribute_crawler ⭐ 3,176
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re
most recent commit 6 years ago
Browser Fingerprinting ⭐ 3,170
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?
most recent commit 3 months ago
Toapi ⭐ 3,153
Every web site provides APIs.
total releases 40most recent commit 2 years ago
Gerapy ⭐ 2,878
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
total releases 47most recent commit a month ago
Novel Plus ⭐ 2,829
novel-plus 是一个多端（PC、WAP）阅读 、功能完善的小说 CMS 系统。包括小说推荐、小说检索、小说排行、小说阅读、小说书架、小说评论、小说爬虫、会员中心、作家专区、
most recent commit 16 days ago
Subbrute ⭐ 2,787
A DNS meta-query spider that enumerates DNS records, and subdomains.
most recent commit a year ago
Weibospider ⭐ 2,752
持续维护的新浪微博采集工具🚀🚀🚀
most recent commit a month ago
Core ⭐ 2,667
🔞 JAVClub - 让你的大姐姐不再走丢
most recent commit 2 years ago
Scrapy Examples ⭐ 2,550
Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc.
most recent commit 4 years ago
Scrapyd ⭐ 2,543
A service daemon to run Scrapy spiders
dependent packages 13total releases 7most recent commit 3 months agopypi scrapyd} Downloads
Spiderkeeper ⭐ 2,539
admin ui for scrapy/open source scrapinghub
total releases 6most recent commit 2 years ago
Dht ⭐ 2,424
BitTorrent DHT Protocol && DHT Spider.
dependent packages 1most recent commit a year ago
Scrapydweb ⭐ 2,408
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO 👉
total releases 17most recent commit a year ago
Java ⭐ 2,380
java项目实战练习
most recent commit a month ago
Decryptlogin ⭐ 2,375
DecryptLogin: APIs for loginning some websites by using requests.
dependent packages 4total releases 49most recent commit 5 months agopypi DecryptLogin} Downloads
Querylist ⭐ 2,333
🕷 The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
dependent packages 71total releases 21most recent commit a year agopackagist jaeger/querylist} Downloads
Owllook ⭐ 2,269
owllook-小说搜索引擎
most recent commit 6 months ago
Grab ⭐ 2,252
Web Scraping Framework
dependent packages 4total releases 120most recent commit a month agopypi grab} Downloads
Python3 Spider ⭐ 1,995
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
most recent commit 3 years ago
Wechatsync ⭐ 1,992
一键同步文章到多个内容平台，支持今日头条、WordPress、知乎、简书、掘金、CSDN、typec
most recent commit 10 days ago
Abot ⭐ 1,991
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
dependent packages 8total releases 153most recent commit a year agonuget Abot} Downloads
Gain ⭐ 1,972
Web crawling framework based on asyncio.
total releases 5most recent commit 4 years ago
Geziyor ⭐ 1,814
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
dependent packages 11total releases 6most recent commit a month ago
Gospider ⭐ 1,790
Gospider - Fast web spider written in Go
total releases 17most recent commit 10 days ago
Torbot ⭐ 1,779
Dark Web OSINT Tool
most recent commit 9 days ago
Go Demo ⭐ 1,774
Go语言实例教程从入门到进阶，包括基础库使用、设计模式、面试易错点、工具类、对接第三方等
most recent commit 9 months ago
Hawkeye ⭐ 1,735
GitHub 泄露监控系统(GitHub Sensitive Information Leakage Monitor Spider)
most recent commit a year ago
Bilibili Api ⭐ 1,734
哔哩哔哩的API调用模块
dependent packages 4total releases 81most recent commit 3 months agopypi bilibili-api} Downloads
Crawler Detect ⭐ 1,726
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
dependent packages 103total releases 151most recent commit 2 days agopackagist jaybizzle/crawler-detect} Downloads
Wechat_articles_spider ⭐ 1,721
微信公众号文章的爬虫
total releases 29most recent commit a month ago
Lianjia Beike Spider ⭐ 1,695
链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新 MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 🚁，点星支持
most recent commit 4 years ago
Pspider ⭐ 1,675
简单易用的Python爬虫框架，QQ交流群：597510560
most recent commit 8 months ago
Go_spider ⭐ 1,629
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
dependent packages 1most recent commit 5 years ago
Anemone ⭐ 1,615
Anemone web-spider framework
dependent packages 32total releases 23most recent commit 3 years agorubygems anemone} Downloads
Ruia ⭐ 1,611
Async Python 3.6+ web scraping micro-framework based on asyncio
dependent packages 6total releases 54most recent commit 5 months agopypi ruia} Downloads
Feapder ⭐ 1,473
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、
dependent packages 3total releases 112most recent commit 17 days agopypi feapder} Downloads
Douyin_tiktok_download_api ⭐ 1,453
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikT
most recent commit 15 days ago
Xsscrapy ⭐ 1,398
XSS spider - 66/66 wavsep XSS detected
most recent commit 9 months ago
Open Source Search Engine ⭐ 1,376
Nov 20 2017 -- A distributed open source search engine and spider/crawler written in C/C++ for Linux on Intel/AMD. From gigablast dot com, which has binaries for download. See the README.md file at the very bottom of this page for instructions.
most recent commit 5 months ago
Spider ⭐ 1,365
Unsurprising JavaScript - No longer active
dependent packages 7total releases 14most recent commit 5 years agonpm spider-script} Downloads
Catvodtvspider ⭐ 1,365
most recent commit 8 months ago
Video_spider ⭐ 1,308
短视频去水印：抖音,皮皮虾,火山,微视,微博,绿洲,最右,轻视频,快手,全民小视频,巴塞电影,陌陌, Vlog 小咖秀,皮皮搞笑,全民K歌,西瓜视频,逗拍,虎牙,6间房,梨视频,新片场,acfun,美拍...
most recent commit a month ago
Php Spider ⭐ 1,282
A configurable and extensible PHP web spider
dependent packages 6total releases 14most recent commit 5 days agopackagist vdb/php-spider} Downloads
Catvodtvspider ⭐ 1,270
most recent commit 8 months ago
Image Downloader ⭐ 1,254
Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.
most recent commit 2 years ago
Blackwidow ⭐ 1,229
A Python based web application scanner to gather OSINT and fuzz for OWASP vulnerabilities on a target website.
most recent commit a month ago
Cdfang Spider ⭐ 1,198
📊 成都房协网数据分析，喜欢请点 star！
most recent commit 2 months ago
Wechat Spider ⭐ 1,188
开源微信爬虫：爬取公众号所有 文章、阅读量、点赞量和评论内容。易部署。持续维护！！！
most recent commit a year ago
Quotesbot ⭐ 1,178
This is a sample Scrapy project for educational purposes
most recent commit 2 months ago
Cewl ⭐ 1,172
CeWL is a Custom Word List Generator
most recent commit 2 months ago
Wechat_spider ⭐ 1,161
微信爬虫，获取文章内容、阅读量、点赞量、评论等，获取公众号所有历史文章链接。
most recent commit a month ago
Checkinpanel ⭐ 1,149
一个主要运行在 𝐞𝐥𝐞𝐜𝐕𝟐𝐏 或 𝐪𝐢𝐧𝐠𝐥𝐨𝐧𝐠 等定时面板，同时支持系统运行环境的签到项目（环境：𝑷𝒚𝒕𝒉𝒐𝒏 3.8+ / 𝑵𝒐𝒅𝒆.𝒋𝒔 10+ / 𝑩𝒂𝒔𝒉 4+ / 𝑶𝒑𝒆𝒏𝑱𝑫𝑲8 / 𝑷𝒆𝒓𝒍5）
most recent commit 18 days ago
Pythoncrawler ⭐ 1,109
💗用python编写的爬虫项目集合
most recent commit 3 months ago
Reptile ⭐ 1,081
🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish
most recent commit 2 years ago
Django Dynamic Scraper ⭐ 1,069
Creating Scrapy scrapers via the Django admin interface
total releases 65most recent commit a year ago
Antcolony ⭐ 1,042
Nodejs实现的一个磁力链接爬虫 http://findit.keenwon.com (原域名http://findit.so )
most recent commit 4 years ago
Beanbun ⭐ 1,025
Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。
dependent packages 1total releases 5most recent commit 3 years agopackagist kiddyu/beanbun} Downloads
Scrapy Cluster ⭐ 1,016
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15most recent commit a year agopypi scutils} Downloads
Jspider ⭐ 1,006
JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816
most recent commit 7 months ago
Qzoneexport ⭐ 976
QQ空间导出助手，用于备份QQ空间的说说、日志、私密日记、相册、视频、留言板、QQ好友、收藏夹、分享
most recent commit 3 months ago
V2raycloudspider ⭐ 942
​🚀 采集|免费|优质|的订阅链接；科学上网，从娃娃抓起！
most recent commit 9 months ago
Grab Site ⭐ 935
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
most recent commit 2 months ago
Darknet_chinesetrading ⭐ 924
🚇暗网中文网监控爬虫(DEEPMIX)
most recent commit 4 months ago
Spider ⭐ 919
Python website crawler.
most recent commit 3 months ago
Spider ⭐ 907
A configurable web spider with a easy-to-use web console
most recent commit 4 years ago
Crawler User Agents ⭐ 902
Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome ⭐️
dependent packages 3total releases 90most recent commit 2 months agonpm crawler-user-agents} Downloads
Vulstudy ⭐ 876
使用docker快速搭建各大漏洞靶场，目前可以一键搭建17个靶场。
most recent commit 3 years ago
Kimuraframework ⭐ 874
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
dependent packages 2total releases 10most recent commit a year agorubygems kimurai} Downloads
Dumpall ⭐ 865
一款信息泄漏利用工具，适用于.git/.svn/.DS_Store泄漏和目录列出
total releases 7most recent commit a month ago
Wechat_spider ⭐ 863
使用“代理”的方式来抓取微信公众账号文章，可以抓取阅读数、点赞数，基于 anyproxy。
most recent commit 3 years ago
Funpyspidersearchengine ⭐ 862
Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索
most recent commit a year ago
Zhihu_spider ⭐ 855
知乎爬虫
most recent commit 6 years ago
Oneblog ⭐ 851
👽 OneBlog，一个简洁美观、功能强大并且自适应的Java博客
most recent commit 8 months ago
Spider Crack Js ⭐ 845
破解各种网站的JS [知乎] [豆瓣][Incapsula-CDN][百度贴吧_BSK][七麦analysis]
most recent commit 2 years ago
Zhihu Crawler ⭐ 843
zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬
most recent commit 4 years ago
1-100 of 3,782 projects
Related Awesome Lists
Python Spider Projects (2,085)
Crawl Spider Projects (1,048)
Spider Scrapy Projects (982)
Crawler Spider Projects (709)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,154,708)
Python  (888,323)
Java  (411,578)
Typescript  (304,714)
Php  (297,724)
C Plus Plus  (257,665)
Ruby  (225,073)
C  (193,605)
Shell  (182,260)
C Sharp  (178,205)
Golang  (171,741)
Swift  (68,947)
Rust  (68,512)
R  (62,080)
Dart  (61,781)
Objective C  (57,059)
Kotlin  (53,231)
Lua  (38,349)
Matlab  (32,815)
Perl  (30,816)
Language  (29,928)
Scala  (29,370)
Haskell  (20,925)
Powershell  (20,379)
Clojure  (19,555)
Bash  (19,201)
Elixir  (17,444)
Coffeescript  (17,147)
Assembly  (15,654)
Julia  (13,994)
Processing  (13,325)
Emacs Lisp  (10,848)
Basic  (10,674)
Flow  (10,509)
Solidity  (10,029)
Erlang  (9,124)
Groovy  (8,612)
Scheme  (8,437)
Ocaml  (6,951)
Pascal  (6,798)
Hack  (6,596)
Lisp  (6,356)
Fortran  (5,792)
Elm  (5,567)
Common Lisp  (5,467)
Shell Script  (4,835)
Dlang  (4,803)
Python Library  (4,516)
Crystal  (4,443)
F Sharp  (4,260)
Privacy | About | Terms | Follow Us On Twitter | Top Projects

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2023 Awesome Open Source.  All rights reserved.