The Top 338 Crawler Open Source Projects

Awesome Open Source

Awesome Open Source

Combined Topics

The Top 338 Crawler Open Source Projects

Categories > Data Processing > Crawler

Scrapy ⭐39,027

Scrapy, a fast high-level web crawling & scraping framework for Python.

Pyspider ⭐14,716

A Powerful Spider(Web Crawler) System in Python.

Annie ⭐13,423

👾 Fast, simple and clean video downloader

Colly ⭐12,452

Elegant Scraper and Crawler Framework for Golang

Proxy_pool ⭐11,172

Python爬虫代理IP池(proxy pool)

Newspaper ⭐10,421

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Webmagic ⭐9,439

A scalable web crawler framework for Java.

Examples Of Web Crawlers ⭐9,365

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Avbook ⭐7,364

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Photon ⭐7,323

Incredibly fast crawler designed for OSINT.

Crawlab ⭐7,222

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Python ⭐6,062

Python脚本。模拟登录知乎，爬虫，操作excel，微信公众号，远程开机

Node Crawler ⭐5,648

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

Headless Chrome Crawler ⭐4,826

Distributed crawler powered by Headless Chrome

Wechatsogou ⭐4,802

基于搜狗微信搜索的微信公众号爬虫接口

Scrapy Redis ⭐4,731

Redis-based components for Scrapy.

Haipproxy ⭐4,645

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Awesome Crawler ⭐4,342

A collection of awesome web crawler,spider in different languages

Ferret ⭐4,331

Declarative web scraping

Dom Crawler ⭐3,132

The DomCrawler component eases DOM navigation for HTML and XML documents.

Scylla ⭐3,070

Intelligent proxy pool for Humans™ (Maintainer needed)

Every web site provides APIs.

Autoscraper ⭐2,855

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Arachni ⭐2,799

Web Application Security Scanner Framework

Dotnetspider ⭐2,791

DotnetSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Proxybroker ⭐2,301

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

Ecommercecrawlers ⭐2,255

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:

Googlescraper ⭐2,220

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Querylist ⭐2,172

🕷 The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

Web crawling framework based on asyncio.

Lianjia Beike Spider ⭐1,965

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 🚁，点星支持

Gocrawl ⭐1,897

Polite, slim and concurrent web crawler.

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Dxy Covid 19 Crawler ⭐1,743

2019新型冠状病毒疫情实时爬虫及API | COVID-19/2019-nCoV Realtime Infection Crawler and API

Rendora ⭐1,741

dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites

Crawler ⭐1,727

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

Instagram Scraper ⭐1,711

scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot

Go_spider ⭐1,684

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

Crawler_illegal_cases_in_china ⭐1,657

Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。 [AD]中文知识图谱门户

Red_hawk ⭐1,516

All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers

Pspider ⭐1,503

简单易用的Python爬虫框架，QQ交流群：597510560

Python3 Spider ⭐1,459

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Google Play Scraper ⭐1,370

Node.js scraper to get data from Google Play

Crawler Detect ⭐1,350

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

Lightcrawler ⭐1,317

Crawl a website and run it through Google lighthouse

Async Python 3.6+ web scraping micro-framework based on asyncio（Python3.6+异步爬虫框架）

Scrapoxy ⭐1,256

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

Wombat ⭐1,212

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

Geziyor ⭐1,200

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Swiftlinkpreview ⭐1,189

It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.

Jd Autobuy ⭐1,116

Python爬虫，京东自动登录，在线抢购商品

Tumblr Crawler ⭐1,098

Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片，视频

Decryptlogin ⭐1,097

APIs for loginning some websites using requests.

Work_crawler ⭐1,093

Download comics novels 小说漫画下载工具小説漫画のダウンローダ小說漫畫下載:腾讯漫画大角虫漫画有妖气知音漫客咪咕 SF漫画哦漫画看漫画漫画柜汗汗酷漫動漫伊甸園快看漫画微博动漫 733动漫网大古漫画网漫画DB 無限動漫動漫狂卡推漫画动漫之家动漫屋古风漫画网 36漫画网亲亲漫画网乙女漫画 comico webtoons 咚漫ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミックサイコミ;アルファポリスカクヨムハーメルン小説家になろう起点中文网八一中文网顶点小说落霞小说网努努书坊笔趣阁→epub.

Beanbun ⭐1,067

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

vulnx 🕷️ is an intelligent bot auto shell injector that detect vulnerabilities in multiple types of cms { `wordpress , joomla , drupal , prestashop .. `}

Crawlergo ⭐958

A powerful dynamic crawler for web vulnerability scanners

Find web directories without bruteforce

Diskover ⭐916

File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch

Tumblthree ⭐912

A Tumblr Blog Backup Application

Autocrawler ⭐903

Google, Naver multiprocess image web crawler (Selenium)

👧 美女写真套图爬虫（二）

A Strong, Fast and Flexible Pixiv Client based on .NET Core and WPF

Zhihu Crawler ⭐880

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

Appcrawler ⭐874

基于appium的app自动遍历工具

News Please ⭐874

news-please - an integrated web crawler and information extractor for news that just works.

Fscrawler ⭐854

Elasticsearch File System Crawler (FS Crawler)

Pic Gather ⭐845

[ Closed ] 🎨 image collector, which supports custom acquisition source configuration and is compatible with MacOS and Windows operating systems.

massive SQL injection vulnerability scanner

Weibo Crawler ⭐810

新浪微博爬虫，用python爬取新浪微博数据

[Unmaintained] A simple and clean video/music/image downloader 👾

A high performance web crawler in Elixir.

Instagram Profilecrawl ⭐764

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

🐾 Creeper - The Next Generation Crawler Framework (Go)

Finalrecon ⭐750

The Last Web Recon Tool You'll Need

Fetchbot ⭐745

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Dark Web OSINT Tool

A tool for pixiv.net. 人人可用的P站爬虫

Magnet Dht ⭐653

✌️ Python3 BitTorrent DHT crawler

Grab Site ⭐641

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

Gospider ⭐626

Gospider - Fast web spider written in Go

Price Monitor ⭐613

京东商品价格监控：监控用户设定商品价格，降价邮件/微信提醒。技术：Python爬虫/IP代理池/JS接口爬取/Selenium页面爬取

Icrawler ⭐601

A multi-thread crawler framework with many builtin image crawlers provided.

Course Crawler ⭐586

🎓 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载。

Newcrawler ⭐584

Free Web Scraping Tool with Java

API of DouYin for Humans used to Crawl Popular Videos and Musics

Filemasta ⭐563

A search application to explore, discover and share online files

Netdiscovery ⭐557

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

Easy Scraping Tutorial ⭐549

Simple but useful Python web scraping tutorial code.

Baiduimagespider ⭐545

一个超级轻量的百度图片爬虫

Fess is very powerful and easily deployable Enterprise Search Server.

Xxl Crawler ⭐530

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

带你了解一下Golang的市场行情

A Facebook crawler

a new crawler based on python with more function including Network fingerprint search

News_feed ⭐491

🐨实时监控1000家中国企业的新闻动态

支援 PTT 還有 PTT2 的 PTT API

Xehentai ⭐463

Doujinshi downloader 绅士漫画下载

Scrapple ⭐460

A framework for creating semi-automatic web content extractors

1-100 of 338 projects