Awesome Open Source
Awesome Open Source
Combined Topics
crawling x

The Top 6,616 Crawling Open Source Projects

Topic > Crawling
Scrapy ⭐ 45,280
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 373total releases 87most recent commit 2 days agopypi Scrapy} Downloads
Lux ⭐ 19,590
👾 Fast and simple video download library and CLI tool written in Go
dependent packages 5total releases 31most recent commit 18 days ago
Colly ⭐ 18,251
Elegant Scraper and Crawler Framework for Golang
dependent packages 201total releases 22most recent commit 18 days ago
Proxy_pool ⭐ 15,864
Python爬虫代理IP池(proxy pool)
most recent commit 4 months ago
Pyspider ⭐ 15,653
A Powerful Spider(Web Crawler) System in Python.
dependent packages 2total releases 17most recent commit 15 days agopypi pyspider} Downloads
Newspaper ⭐ 12,252
News, full-text, and article metadata extraction in Python 3. Advanced docs:
dependent packages 48total releases 18most recent commit 20 days agopypi newspaper3k} Downloads
Tushare ⭐ 11,467
TuShare is a utility for crawling historical data of China stocks
most recent commit 8 months ago
Examples Of Web Crawlers ⭐ 11,050
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等 interesting examples of python crawlers that are friendly to beginners. )
most recent commit 7 months ago
Xsstrike ⭐ 10,736
Most advanced XSS scanner.
most recent commit 2 months ago
Webmagic ⭐ 10,625
A scalable web crawler framework for Java.
dependent packages 15total releases 21most recent commit 9 days ago
Crawlab ⭐ 9,387
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
total releases 15most recent commit 2 days ago
Avbook ⭐ 8,623
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
most recent commit 22 days ago
Photon ⭐ 8,510
Incredibly fast crawler designed for OSINT.
total releases 18most recent commit 9 months ago
Python ⭐ 7,972
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
most recent commit 13 days ago
Spider Flow ⭐ 7,439
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
most recent commit 15 days ago
Crawlee ⭐ 7,199
Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.
dependent packages 16total releases 246most recent commit 3 days agonpm @crawlee/types} Downloads
Infospider ⭐ 6,356
INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透
most recent commit 6 days ago
Node Crawler ⭐ 6,287
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
dependent packages 126total releases 30most recent commit a month agonpm crawler} Downloads
Awesome Web Scraping ⭐ 5,525
List of libraries, tools and APIs for web scraping and data processing.
most recent commit 6 days ago
Wechatsogou ⭐ 5,410
基于搜狗微信搜索的微信公众号爬虫接口
total releases 25most recent commit 13 days ago
Awesome Crawler ⭐ 5,192
A collection of awesome web crawler,spider in different languages
most recent commit 2 months ago
Haipproxy ⭐ 5,157
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
total releases 7most recent commit 4 months ago
Scrapy Redis ⭐ 5,141
Redis-based components for Scrapy.
dependent packages 21total releases 17most recent commit 4 months agopypi scrapy-redis} Downloads
Ferret ⭐ 5,116
Declarative web scraping
dependent packages 3total releases 34most recent commit 21 days ago
Headless Chrome Crawler ⭐ 5,051
Distributed crawler powered by Headless Chrome
dependent packages 9total releases 21most recent commit a year agonpm headless-chrome-crawler} Downloads
Katana ⭐ 4,876
A next-generation crawling and spidering framework.
most recent commit 3 days ago
Weibospider ⭐ 4,653
⚡️ A distributed crawler for weibo, building with celery and requests.
most recent commit 2 years ago
Autoscraper ⭐ 4,498
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
dependent packages 1total releases 16most recent commit 5 months agopypi autoscraper} Downloads
Crawler4j ⭐ 4,192
Open Source Web Crawler for Java
dependent packages 4total releases 10most recent commit a year ago
Proxypool ⭐ 3,967
An Efficient ProxyPool with Getter, Tester and Server
most recent commit 3 months ago
Interesting Python ⭐ 3,748
有趣的Python爬虫和Python数据分析小项目(Some interesting Python crawlers and data analysis projects)
most recent commit a year ago
Wooyun_public ⭐ 3,701
This repo is archived. Thanks for wooyun! 乌云公开漏洞、知识库爬虫和搜索 crawl and search for wooyun.org public bug(vulnerability) and drops
most recent commit 3 years ago
Scylla ⭐ 3,638
Intelligent proxy pool for Humans™
total releases 30most recent commit 2 days ago
Ecommercecrawlers ⭐ 3,611
实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼
most recent commit 13 days ago
Dotnetspider ⭐ 3,488
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
dependent packages 7total releases 43most recent commit 17 days agonuget DotnetSpider} Downloads
Hakrawler ⭐ 3,266
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
total releases 11most recent commit 2 months ago
Rod ⭐ 3,204
A Devtools driver for web automation and scraping
dependent packages 58total releases 385most recent commit 4 days ago
Distribute_crawler ⭐ 3,176
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re
most recent commit 6 years ago
Browser Fingerprinting ⭐ 3,170
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?
most recent commit a month ago
Toapi ⭐ 3,153
Every web site provides APIs.
total releases 40most recent commit a year ago
Proxybroker ⭐ 3,144
Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭
dependent packages 3total releases 12most recent commit a month agopypi proxybroker} Downloads
Arachni ⭐ 3,055
Web Application Security Scanner Framework
most recent commit 6 months ago
Anthelion ⭐ 2,883
Anthelion is a plugin for Apache Nutch to crawl semantic annotations within HTML pages.
most recent commit 7 years ago
Webcollector ⭐ 2,831
WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
dependent packages 1total releases 21most recent commit a year ago
Gerapy ⭐ 2,783
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
total releases 47most recent commit 4 months ago
Novel Plus ⭐ 2,705
novel-plus 是一个多端（PC、WAP）阅读 、功能完善的小说 CMS 系统。包括小说推荐、小说检索、小说排行、小说阅读、小说书架、小说评论、小说爬虫、会员中心、作家专区、
most recent commit 2 months ago
Hawk ⭐ 2,638
visualized crawler & ETL IDE written with C#/WPF
most recent commit 3 years ago
Nutch ⭐ 2,468
Apache Nutch is an extensible and scalable web crawler
dependent packages 1total releases 25most recent commit 6 days ago
Crawler_illegal_cases_in_china ⭐ 2,467
Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的 [AD]中文知识图谱门户
most recent commit a year ago
Dht ⭐ 2,424
BitTorrent DHT Protocol && DHT Spider.
dependent packages 1most recent commit a year ago
Gecco ⭐ 2,403
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
dependent packages 4total releases 28most recent commit a month ago
Gau ⭐ 2,396
Fetch known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl.
dependent packages 1total releases 29most recent commit 19 days ago
Decryptlogin ⭐ 2,375
DecryptLogin: APIs for loginning some websites by using requests.
dependent packages 4total releases 49most recent commit 3 months agopypi DecryptLogin} Downloads
Heritrix3 ⭐ 2,335
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
dependent packages 2total releases 9most recent commit a month ago
Querylist ⭐ 2,333
🕷 The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
dependent packages 71total releases 21most recent commit a year agopackagist jaeger/querylist} Downloads
Googlescraper ⭐ 2,325
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
most recent commit a year ago
Dirmap ⭐ 2,325
An advanced web directory & file scanning tool that will be more powerful than DirBuster, Dirsearch, cansina, and Yu Jian.一个高级web目录、文件扫描工具，功能将会强于DirBuster、Dirsearch、ca
most recent commit 6 months ago
Owllook ⭐ 2,269
owllook-小说搜索引擎
most recent commit 4 months ago
Weibo_terminater ⭐ 2,265
Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
most recent commit 3 years ago
Crawler ⭐ 2,253
An easy to use, powerful crawler implemented in PHP. Can execute Javascript.
dependent packages 27total releases 96most recent commit 2 days agopackagist spatie/crawler} Downloads
Crawlergo ⭐ 2,156
A powerful browser crawler for web vulnerability scanners
most recent commit 2 months ago
Work_crawler ⭐ 2,137
Download comics novels 小说漫画下载工具 小説漫画のダウンローダ 小說漫畫下載:腾讯漫画 大角虫漫画 有妖气 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫伊甸園 快看漫画 微博动漫 733动漫网 大古漫画网 漫画DB 無限動漫 動漫狂 卡推漫画 动漫之家 动漫屋 古风漫画网 36漫画网 亲亲漫画网 乙女漫画 webtoons 咚漫 ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミック サイコミ;アルファポリス カクヨム ハーメルン 小説家になろう 起点中文网 八一中文网 顶点小说 落霞小说网 努努书坊 笔趣阁→epub.
most recent commit 12 days ago
Simplecrawler ⭐ 2,113
Flexible event driven crawler for node.
dependent packages 74total releases 70most recent commit 2 years agonpm simplecrawler} Downloads
Red_hawk ⭐ 2,079
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
most recent commit 6 months ago
Gryffin ⭐ 2,061
Gryffin is a large scale web security scanning platform.
most recent commit 2 years ago
Spring Boot Quick ⭐ 2,020
🌿 基于springboot的快速学习示例,整合自己遇到的开源框架,如：rabbitmq(延迟队列)、K
most recent commit 3 months ago
Python3 Spider ⭐ 1,995
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
most recent commit 2 years ago
Abot ⭐ 1,991
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
dependent packages 8total releases 153most recent commit 9 months agonuget Abot} Downloads
Gain ⭐ 1,972
Web crawling framework based on asyncio.
total releases 5most recent commit 4 years ago
Dxy Covid 19 Crawler ⭐ 1,969
2019新型冠状病毒疫情实时爬虫及API | COVID-19/2019-nCoV Realtime Infection Crawler and API
most recent commit 3 months ago
Weibo Crawler ⭐ 1,947
新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频
most recent commit 3 months ago
Gocrawl ⭐ 1,929
Polite, slim and concurrent web crawler.
total releases 2most recent commit 2 years ago
Awesome Puppeteer ⭐ 1,881
A curated list of awesome puppeteer resources.
most recent commit 4 months ago
Amemv Crawler ⭐ 1,871
🙌Easily download all the videos from TikTok(amemv).下载指定的 抖音（Douyin） 号的视频,抖音爬虫
most recent commit 3 years ago
Crawl ⭐ 1,860
Dungeon Crawl: Stone Soup official repository
most recent commit 2 days ago
Seimicrawler ⭐ 1,857
一个简单、敏捷、分布式的支持SpringBoot的Java爬虫框架;An agile, distributed crawler framework.
total releases 26most recent commit a month ago
Ipproxytool ⭐ 1,851
python ip proxy tool scrapy crawl. 抓取大量免费代理 ip，提取有效 ip 使用
most recent commit 6 months ago
Laravel Sitemap ⭐ 1,832
Create and generate sitemaps with ease
dependent packages 27total releases 57most recent commit 2 days agopackagist spatie/laravel-sitemap} Downloads
Google Play Scraper ⭐ 1,831
Node.js scraper to get data from Google Play
dependent packages 22total releases 84most recent commit 2 months agonpm google-play-scraper} Downloads
Ambar ⭐ 1,797
🔍 Ambar: Document Search Engine
most recent commit a year ago
Domain_analyzer ⭐ 1,747
Analyze the security of any domain by finding all the information possible. Made in python.
most recent commit 16 days ago
Striker ⭐ 1,744
Striker is an offensive information and vulnerability scanner.
most recent commit 2 years ago
Lianjia Beike Spider ⭐ 1,695
链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新 MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 🚁，点星支持
most recent commit 3 years ago
Geziyor ⭐ 1,683
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
dependent packages 11total releases 6most recent commit 7 months ago
Crawler Detect ⭐ 1,677
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
dependent packages 103total releases 151most recent commit 2 months agopackagist jaybizzle/crawler-detect} Downloads
Pspider ⭐ 1,675
简单易用的Python爬虫框架，QQ交流群：597510560
most recent commit 6 months ago
Torbot ⭐ 1,670
Dark Web OSINT Tool
most recent commit 7 days ago
Scrapely ⭐ 1,668
A pure-python HTML screen-scraping library
dependent packages 3total releases 13most recent commit 3 years agopypi scrapely} Downloads
Rendora ⭐ 1,651
dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites
most recent commit 3 years ago
Ucss ⭐ 1,635
Find unused and duplicate CSS
dependent packages 3total releases 22most recent commit 6 years agonpm ucss} Downloads
Go_spider ⭐ 1,629
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
dependent packages 1most recent commit 5 years ago
React Snapshot ⭐ 1,619
A zero-configuration static pre-renderer for React apps
dependent packages 16total releases 11most recent commit 2 years agonpm react-snapshot} Downloads
Anemone ⭐ 1,615
Anemone web-spider framework
dependent packages 32total releases 23most recent commit 3 years agorubygems anemone} Downloads
Ruia ⭐ 1,611
Async Python 3.6+ web scraping micro-framework based on asyncio
dependent packages 6total releases 54most recent commit 3 months agopypi ruia} Downloads
Python Crawler ⭐ 1,576
从头开始 系统化的 学习如何写Python爬虫。 Python版本 3.6
most recent commit a year ago
Gospider ⭐ 1,573
Gospider - Fast web spider written in Go
total releases 17most recent commit 3 months ago
Finalrecon ⭐ 1,541
The Last Web Recon Tool You'll Need
most recent commit 3 months ago
Static Site Generator Webpack Plugin ⭐ 1,538
Minimal, unopinionated static site generator powered by webpack
dependent packages 198total releases 21most recent commit 4 years agonpm static-site-generator-webpack-plugin} Downloads
Hauberk ⭐ 1,505
A web-based roguelike written in Dart.
most recent commit 2 years ago
Vulnx ⭐ 1,450
vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform a quick CMS security detection, information collection (including sub-domain name, ip address, country information, organizational information and time zone, etc.) and vulnerability scanning.
total releases 2most recent commit 6 months ago
1-100 of 6,616 projects
Related Awesome Lists
Crawling Webscraping Projects (889)
Scraper Crawling Projects (866)
Python Crawling Projects (789)
Crawler Crawling Projects (743)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 71
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 343
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 115
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,118,155)
Python  (863,562)
Java  (403,872)
Php  (293,282)
Typescript  (283,207)
C Plus Plus  (251,932)
Ruby  (223,273)
C  (189,870)
Shell  (178,876)
C Sharp  (173,761)
Golang  (167,817)
Swift  (67,809)
Rust  (65,148)
R  (60,980)
Dart  (58,968)
Objective C  (57,046)
Kotlin  (51,248)
Lua  (37,084)
Matlab  (32,369)
Perl  (30,722)
Language  (29,923)
Scala  (29,137)
Haskell  (20,775)
Powershell  (19,976)
Clojure  (19,416)
Bash  (19,028)
Coffeescript  (17,172)
Elixir  (17,172)
Assembly  (15,230)
Julia  (13,658)
Processing  (13,292)
Emacs Lisp  (10,686)
Basic  (10,684)
Flow  (10,513)
Solidity  (9,559)
Erlang  (9,095)
Groovy  (8,516)
Scheme  (8,427)
Ocaml  (6,878)
Pascal  (6,701)
Hack  (6,542)
Lisp  (6,347)
Fortran  (5,738)
Elm  (5,543)
Common Lisp  (5,427)
Dlang  (4,780)
Shell Script  (4,768)
Visual Basic  (4,516)
Python Library  (4,497)
Crystal  (4,421)
Privacy | About | Terms | Follow Us On Twitter | Top Projects

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.