Awesome Open Source
Awesome Open Source
Combined Topics
scraper x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 202 Scraper Open Source Projects

Categories > Data Processing > Scraper
Huginn ⭐ 31,951
Create agents that monitor and act on your behalf. Your agents are standing by!
Cheerio ⭐ 24,020
Fast, flexible, and lean implementation of core jQuery designed specifically for the server.
Annie ⭐ 15,159
👾 Fast and simple video download library and CLI tool written in Go
Colly ⭐ 14,454
Elegant Scraper and Crawler Framework for Golang
Newspaper ⭐ 11,202
News, full-text, and article metadata extraction in Python 3. Advanced docs:
Chinese Xinhua ⭐ 7,981
📙 中华新华字典数据库。包括歇后语，成语，词语，汉字。
Avbook ⭐ 7,923
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
Instagram Scraper ⭐ 5,252
Scrapes an instagram user's photos and videos
Headless Chrome Crawler ⭐ 5,023
Distributed crawler powered by Headless Chrome
Awesome Crawler ⭐ 4,664
A collection of awesome web crawler,spider in different languages
Ferret ⭐ 4,567
Declarative web scraping
Autoscraper ⭐ 3,802
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Scrape It ⭐ 3,728
🔮 A Node.js scraper for humans.
Node Ytdl Core ⭐ 2,779
YouTube video downloader in javascript.
Querylist ⭐ 2,328
🕷 The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Weibo_terminater ⭐ 2,289
Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
Instagram Scraper ⭐ 2,049
scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot
Onlyfans ⭐ 1,871
Scrape all the media from an OnlyFans account - Updated regularly
Rod ⭐ 1,687
A Devtools driver for web automation and scraping
Google Play Scraper ⭐ 1,533
Node.js scraper to get data from Google Play
Jobfunnel ⭐ 1,494
Scrape job websites into a single spreadsheet with no duplicates.
Geziyor ⭐ 1,487
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Scrapoxy ⭐ 1,389
Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!
Wombat ⭐ 1,230
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Jd Autobuy ⭐ 1,197
Python爬虫，京东自动登录，在线抢购商品
Emby.plugins.javscraper ⭐ 1,165
Emby/Jellyfin 的一个日本电影刮削器插件，可以从某些网站抓取影片信息。
Django Dynamic Scraper ⭐ 1,045
Creating Scrapy scrapers via the Django admin interface
Bulk Downloader For Reddit ⭐ 1,025
Downloads and archives content from reddit
Node Website Scraper ⭐ 990
Download website to local directory (including all css, images, js, etc.)
Scanless ⭐ 891
online port scan scraper
Imdbpy ⭐ 835
IMDbPY is a Python package useful to retrieve and manage the data of the IMDb movie database about movies, people, characters and companies
Informer ⭐ 835
A Telegram Mass Surveillance Bot in Python
Crawler ⭐ 809
A high performance web crawler in Elixir.
Kimuraframework ⭐ 799
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
Lulu ⭐ 797
[Unmaintained] A simple and clean video/music/image downloader 👾
Instagram Crawler ⭐ 729
Get Instagram posts/profile/hashtag data without using Instagram API
Scrapyrt ⭐ 675
HTTP API for Scrapy spiders
Instagram4j ⭐ 670
📷 Instagram private API in Java
Spidr ⭐ 668
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Freedictionaryapi ⭐ 665
There was no free Dictionary API on the web when I wanted one for my friend, so I created one.
Surgeon ⭐ 664
Declarative DOM extraction expression evaluator. 👨‍⚕️
Imagescraper ⭐ 651
✂️ High performance, multi-threaded image scraper
Snscrape ⭐ 650
A social networking service scraper in Python
Scala Scraper ⭐ 649
A Scala library for scraping content from HTML pages
Finviz ⭐ 623
Unofficial API for finviz.com
Redditdownloader ⭐ 598
Scrapes Reddit to download media of your choice.
Fbcrawl ⭐ 568
A Facebook crawler
Jikan ⭐ 564
Unofficial MyAnimeList PHP+REST API which provides functions other than the official API
Linkedin_scraper ⭐ 560
A library that scrapes Linkedin for user data
Operative Framework ⭐ 531
operative framework is a OSINT investigation framework, you can interact with multiple targets, execute multiple modules, create links with target, export rapport to PDF file, add note to target or results, interact with RESTFul API, write your own modules.
Osi.ig ⭐ 504
Information Gathering Instagram.
Katana ⭐ 494
A Python Tool For google Hacking
Nintendo Switch Eshop ⭐ 490
Crawler for Nintendo Switch eShop
Crawly ⭐ 489
Crawly, a high-level web crawling & scraping framework for Elixir.
Bookcorpus ⭐ 484
Crawl BookCorpus
Scrapedin ⭐ 477
LinkedIn Scraper (currently working 2020)
Dataflowkit ⭐ 469
Extract structured data from web sites. Web sites scraping.
Finance Go ⭐ 459
📊 Financial markets data library implemented in go.
Gosint ⭐ 417
OSINT Swiss Army Knife
Scrapers ⭐ 407
A list of scrapers from around the web.
Php Goose ⭐ 402
Readability / Html Content / Article Extractor & Web Scrapping library written in PHP
Advanced Web Scraping Tutorial ⭐ 392
The Zipru scraper developed in the Advanced Web Scraping Tutorial.
Skrape.it ⭐ 391
A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
Micro Open Graph ⭐ 371
A tiny Node.js microservice to scrape open graph data with joy.
Xidel ⭐ 362
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Freshonions Torscraper ⭐ 362
Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
Xcrawler ⭐ 344
快速、简洁且强大的PHP爬虫框架
Javgo ⭐ 338
JavGo是一个集合影片管理，影片刮削，视频处理，资源搜索等综合一体的全功能影音软件，支持爬取javbus，jav321，javdb，javlibrary进行刮削，支持db，bus的磁力搜索，支持获取library的影片评论。
Cryptocmd ⭐ 333
Cryptocurrency historical price data library in Python. Data from https://coinmarketcap.com.
Linkedin ⭐ 331
Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy
Hquery.php ⭐ 308
An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.
Mlscraper ⭐ 296
🤖 Scrape data from HTML websites automatically with Machine Learning
Socialmanagertools Gui ⭐ 293
🤖 👻 Desktop application for Instagram Bot, Twitter Bot and Facebook Bot
Webinspector ⭐ 290
Ruby gem to inspect completely a web page. It scrapes a given URL, and returns you its meta, links, images more.
Rcrawler ⭐ 285
An R web crawler and scraper
Java Spider ⭐ 281
一个基于webmagic框架二次开发的java爬虫框架实战，已实现能爬取腾讯，搜狐，今日头条（单独集成功能）等资讯内容，配合elasticsearch框架用法，实现了自动爬虫，已投入线上生产使用。
Scrape Linkedin Selenium ⭐ 274
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
Polite ⭐ 265
Be nice on the web
Weibo_terminator_workflow ⭐ 259
Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!
Hltv ⭐ 253
The unofficial HLTV Node.js API
Heroku_ebooks ⭐ 251
A script to generate Markov chains and to post to an _ebooks account on Twitter using Heroku
Instagram Proxy Api ⭐ 249
CORS compliant API to access Instagram's public data
Ant ⭐ 245
A web crawler for Go
Jvppeteer ⭐ 244
Headless Chrome For Java （Java 爬虫）
Youtube Projects ⭐ 243
This repository contains all the code I use in my YouTube tutorials.
Scrapysharp ⭐ 240
reborn of https://bitbucket.org/rflechner/scrapysharp
Getsy ⭐ 238
A simple browser/client-side web scraper.
Media Scraper ⭐ 230
Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok
Ruiji.net ⭐ 224
crawler framework, distributed crawler extractor
Trafilatura ⭐ 217
Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)
Goose Parser ⭐ 211
Universal scrapping tool, which allows you to extract data using multiple environments
Unfurl ⭐ 209
Metadata scraper with support for oEmbed, Twitter Cards and Open Graph Protocol for Node.js ⚡️
Tianyancha ⭐ 206
pip安装的天眼查爬虫API，指定的单个/多个企业工商信息一键保存为Excel/JSON格式。A Battery-included Scraper API of Tianyancha, the best Chinese business data and investigation platform.
Docsearch Scraper ⭐ 201
DocSearch - Scraper
Linkedin Profile Scraper ⭐ 198
🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.
Google Play Scraper ⭐ 198
Google play scraper for Python inspired by <facundoolano/google-play-scraper>
Jsonframe Cheerio ⭐ 196
simple multi-level scraper json input/output for Cheerio
Goribot ⭐ 196
[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。
Anime Dl ⭐ 195
Anime-dl is a command-line program to download anime from CrunchyRoll and Funimation.
Gmdb ⭐ 195
GMDB is the ultra-simple, cross-platform Movie Library with Features (Search, Take Note, Watch Later, Like, Import, Learn, Instantly Torrent Magnet Watch)
1-100 of 202 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210