Awesome Open Source
Awesome Open Source
Combined Topics
scraper x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 185 Scraper Open Source Projects

Categories > Data Processing > Scraper
Huginn ⭐ 31,277
Create agents that monitor and act on your behalf. Your agents are standing by!
Cheerio ⭐ 23,590
Fast, flexible, and lean implementation of core jQuery designed specifically for the server.
Annie ⭐ 14,523
👾 Fast, simple and clean video downloader
Colly ⭐ 13,620
Elegant Scraper and Crawler Framework for Golang
Newspaper ⭐ 10,894
News, full-text, and article metadata extraction in Python 3. Advanced docs:
Chinese Xinhua ⭐ 7,797
📙 中华新华字典数据库。包括歇后语，成语，词语，汉字。
Avbook ⭐ 7,714
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
Headless Chrome Crawler ⭐ 4,950
Distributed crawler powered by Headless Chrome
Instagram Scraper ⭐ 4,898
Scrapes an instagram user's photos and videos
Awesome Crawler ⭐ 4,528
A collection of awesome web crawler,spider in different languages
Ferret ⭐ 4,454
Declarative web scraping
Scrape It ⭐ 3,690
🔮 A Node.js scraper for humans.
Autoscraper ⭐ 3,440
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Weibo_terminater ⭐ 2,291
Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
Querylist ⭐ 2,267
🕷 The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Node Ytdl Core ⭐ 2,234
YouTube video downloader in javascript.
Instagram Scraper ⭐ 1,893
scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot
Google Play Scraper ⭐ 1,467
Node.js scraper to get data from Google Play
Jobfunnel ⭐ 1,464
Scrape job websites into a single spreadsheet with no duplicates.
Rod ⭐ 1,462
A Devtools driver for web automation and scraping
Scrapoxy ⭐ 1,336
Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!
Geziyor ⭐ 1,267
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Wombat ⭐ 1,220
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Jd Autobuy ⭐ 1,183
Python爬虫，京东自动登录，在线抢购商品
Django Dynamic Scraper ⭐ 1,027
Creating Scrapy scrapers via the Django admin interface
Emby.plugins.javscraper ⭐ 951
Emby/Jellyfin 的一个日本电影刮削器插件，可以从某些网站抓取影片信息。
Node Website Scraper ⭐ 936
Download website to local directory (including all css, images, js, etc.)
Scanless ⭐ 879
online port scan scraper
Imdbpy ⭐ 803
IMDbPY is a Python package useful to retrieve and manage the data of the IMDb movie database about movies, people, characters and companies
Onlyfans ⭐ 795
Scrape all the media from an OnlyFans account - Updated regularly
Lulu ⭐ 788
[Unmaintained] A simple and clean video/music/image downloader 👾
Crawler ⭐ 783
A high performance web crawler in Elixir.
Informer ⭐ 756
A Telegram Mass Surveillance Bot in Python
Instagram Crawler ⭐ 661
Get Instagram posts/profile/hashtag data without using Instagram API
Spidr ⭐ 659
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Surgeon ⭐ 659
Declarative DOM extraction expression evaluator. 👨‍⚕️
Scrapyrt ⭐ 650
HTTP API for Scrapy spiders
Instagram4j ⭐ 638
📷 Instagram private API in Java
Scala Scraper ⭐ 635
A Scala library for scraping content from HTML pages
Imagescraper ⭐ 632
✂️ High performance, multi-threaded image scraper
Googledictionaryapi ⭐ 547
Google does not provide Google Dictionary API so I created one.
Redditdownloader ⭐ 538
Scrapes Reddit to download media of your choice.
Fbcrawl ⭐ 538
A Facebook crawler
Jikan ⭐ 537
Unofficial MyAnimeList PHP+REST API which provides functions other than the official API
Finviz ⭐ 517
Unofficial API for finviz.com
Operative Framework ⭐ 513
operative framework is a OSINT investigation framework, you can interact with multiple targets, execute multiple modules, create links with target, export rapport to PDF file, add note to target or results, interact with RESTFul API, write your own modules.
Nintendo Switch Eshop ⭐ 469
Crawler for Nintendo Switch eShop
Snscrape ⭐ 465
A social networking service scraper in Python
Scrapedin ⭐ 460
LinkedIn Scraper (currently working 2020)
Dataflowkit ⭐ 458
Extract structured data from web sites. Web sites scraping.
Crawly ⭐ 451
Crawly, a high-level web crawling & scraping framework for Elixir.
Bookcorpus ⭐ 450
Crawl BookCorpus
Linkedin_scraper ⭐ 438
A library that scrapes Linkedin for user data
Osi.ig ⭐ 412
Information Gathering Instagram.
Gosint ⭐ 408
OSINT Swiss Army Knife
Finance Go ⭐ 395
📊 Financial markets data library implemented in go.
Php Goose ⭐ 394
Readability / Html Content / Article Extractor & Web Scrapping library written in PHP
Advanced Web Scraping Tutorial ⭐ 386
The Zipru scraper developed in the Advanced Web Scraping Tutorial.
Katana ⭐ 384
A Python Tool For google Hacking
Scrapers ⭐ 373
A list of scrapers from around the web.
Micro Open Graph ⭐ 371
A tiny Node.js microservice to scrape open graph data with joy.
Freshonions Torscraper ⭐ 352
Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
Xidel ⭐ 346
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Xcrawler ⭐ 344
快速、简洁且强大的PHP爬虫框架
Javgo ⭐ 338
JavGo是一个集合影片管理，影片刮削，视频处理，资源搜索等综合一体的全功能影音软件，支持爬取javbus，jav321，javdb，javlibrary进行刮削，支持db，bus的磁力搜索，支持获取library的影片评论。
Linkedin ⭐ 315
Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy
Cryptocmd ⭐ 310
Cryptocurrency historical price data library in Python. Data from https://coinmarketcap.com.
Hquery.php ⭐ 296
An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.
Socialmanagertools Gui ⭐ 293
🤖 👻 Desktop application for Instagram Bot, Twitter Bot and Facebook Bot
Webinspector ⭐ 289
Ruby gem to inspect completely a web page. It scrapes a given URL, and returns you its meta, links, images more.
Rcrawler ⭐ 277
An R web crawler and scraper
Java Spider ⭐ 277
一个基于webmagic框架二次开发的java爬虫框架实战，已实现能爬取腾讯，搜狐，今日头条（单独集成功能）等资讯内容，配合elasticsearch框架用法，实现了自动爬虫，已投入线上生产使用。
Weibo_terminator_workflow ⭐ 259
Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!
Polite ⭐ 254
Be nice on the web
Heroku_ebooks ⭐ 251
A script to generate Markov chains and to post to an _ebooks account on Twitter using Heroku
Skrape.it ⭐ 248
A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
Scrape Linkedin Selenium ⭐ 247
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
Instagram Proxy Api ⭐ 246
CORS compliant API to access Instagram's public data
Hltv ⭐ 238
The unofficial HLTV Node.js API
Getsy ⭐ 238
A simple browser/client-side web scraper.
Ant ⭐ 236
A web crawler for Go
Scrapysharp ⭐ 229
reborn of https://bitbucket.org/rflechner/scrapysharp
Ruiji.net ⭐ 221
crawler framework, distributed crawler extractor
Goose Parser ⭐ 211
Universal scrapping tool, which allows you to extract data using multiple environments
Media Scraper ⭐ 210
Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok
Tianyancha ⭐ 206
pip安装的天眼查爬虫API，指定的单个/多个企业工商信息一键保存为Excel/JSON格式。A Battery-included Scraper API of Tianyancha, the best Chinese business data and investigation platform.
Jvppeteer ⭐ 204
Headless Chrome For Java （Java 爬虫）
Jsonframe Cheerio ⭐ 196
simple multi-level scraper json input/output for Cheerio
Unfurl ⭐ 194
Scraper for oEmbed, Twitter Cards and Open Graph metadata - fast and Promise-based ⚡️
Anime Dl ⭐ 192
Anime-dl is a command-line program to download anime from CrunchyRoll and Funimation.
Gmdb ⭐ 192
GMDB is the ultra-simple, cross-platform Movie Library with Features (Search, Take Note, Watch Later, Like, Import, Learn, Instantly Torrent Magnet Watch)
Goribot ⭐ 192
[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。
Thepiratebay ⭐ 191
💀 The Pirate Bay node.js client
Docsearch Scraper ⭐ 190
DocSearch - Scraper
Unhtml.rs ⭐ 181
A magic html parser
Instagram Crawler ⭐ 180
Crawl instagram photos, posts and videos for download.
Linkedin Profile Scraper ⭐ 179
🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.
Novel ⭐ 175
基于 Laravel 5.2 的小说网站
Readablewebproxy ⭐ 172
Rewriting web proxy and archival tool. At this point, it just tries to download all the things.
Scrape Twitter ⭐ 166
🐦 Access Twitter data without an API key. [DEPRECATED]
1-100 of 185 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210