Awesome Open Source
Awesome Open Source
Combined Topics
scraper x
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194

The Top 2,583 Scraper Open Source Projects on Github

Categories > Data Processing > Scraper
Huginn ⭐ 33,244
Create agents that monitor and act on your behalf. Your agents are standing by!
Cheerio ⭐ 24,733
Fast, flexible, and lean implementation of core jQuery designed specifically for the server.
Lux ⭐ 16,869
👾 Fast and simple video download library and CLI tool written in Go
Colly ⭐ 15,844
Elegant Scraper and Crawler Framework for Golang
Newspaper ⭐ 11,537
News, full-text, and article metadata extraction in Python 3. Advanced docs:
Avbook ⭐ 7,936
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
Chinese Xinhua ⭐ 7,465
📙 中华新华字典数据库。包括歇后语，成语，词语，汉字。
Instagram Scraper ⭐ 5,751
Scrapes an instagram user's photos and videos
Headless Chrome Crawler ⭐ 5,051
Distributed crawler powered by Headless Chrome
Ferret ⭐ 4,858
Declarative web scraping
Awesome Crawler ⭐ 4,747
A collection of awesome web crawler,spider in different languages
Scrape It ⭐ 3,780
🔮 A Node.js scraper for humans.
Autoscraper ⭐ 3,281
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Node Ytdl Core ⭐ 3,044
YouTube video downloader in javascript.
Onlyfans ⭐ 2,778
Scrape all the media from an OnlyFans account - Updated regularly
Browser Fingerprinting ⭐ 2,594
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?
Querylist ⭐ 2,333
🕷 The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Weibo_terminater ⭐ 2,265
Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
Instagram Scraper ⭐ 2,161
scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot
Rod ⭐ 2,072
A Devtools driver for web automation and scraping
Google Play Scraper ⭐ 1,626
Node.js scraper to get data from Google Play
Geziyor ⭐ 1,589
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Jobfunnel ⭐ 1,533
Scrape job websites into a single spreadsheet with no duplicates.
Scrapoxy ⭐ 1,243
Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!
Wombat ⭐ 1,217
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Jd Autobuy ⭐ 1,208
Python爬虫，京东自动登录，在线抢购商品
Bulk Downloader For Reddit ⭐ 1,195
Downloads and archives content from reddit
Emby.plugins.javscraper ⭐ 1,070
Emby/Jellyfin 的一个日本电影刮削器插件，可以从某些网站抓取影片信息。
Django Dynamic Scraper ⭐ 1,069
Creating Scrapy scrapers via the Django admin interface
Node Website Scraper ⭐ 1,066
Download website to local directory (including all css, images, js, etc.)
Snscrape ⭐ 973
A social networking service scraper in Python
Cinemagoer ⭐ 929
Cinemagoer is a Python package useful to retrieve and manage the data of the IMDb (to which we are not affiliated in any way) movie database about movies, people, characters and companies
Informer ⭐ 902
A Telegram Mass Surveillance Bot in Python
Freedictionaryapi ⭐ 892
There was no free Dictionary API on the web when I wanted one for my friend, so I created one.
Scanless ⭐ 848
online port scan scraper
Crawler ⭐ 819
A high performance web crawler / scraper in Elixir.
Kimuraframework ⭐ 808
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
Till ⭐ 770
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
Lulu ⭐ 752
[Unmaintained] A simple and clean video/music/image downloader 👾
Instagram4j ⭐ 727
📷 Instagram private API in Java
Openstates Scrapers ⭐ 722
source for Open States scrapers
Redditdownloader ⭐ 709
Scrapes Reddit to download media of your choice.
Scrapyrt ⭐ 701
HTTP API for Scrapy spiders
Finviz ⭐ 699
Unofficial API for finviz.com
Spidr ⭐ 674
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Instagram Crawler ⭐ 671
Get Instagram posts/profile/hashtag data without using Instagram API
Scala Scraper ⭐ 669
A Scala library for scraping content from HTML pages
Linkedin_scraper ⭐ 640
A library that scrapes Linkedin for user data
Jikan ⭐ 611
Unofficial MyAnimeList PHP+REST API which provides functions other than the official API
Surgeon ⭐ 593
Declarative DOM extraction expression evaluator. 👨‍⚕️
Katana ⭐ 576
python script for Google Dorking
Imagescraper ⭐ 572
✂️ High performance, multi-threaded image scraper
Osi.ig ⭐ 558
Information Gathering Instagram.
Crawly ⭐ 543
Crawly, a high-level web crawling & scraping framework for Elixir.
Nintendo Switch Eshop ⭐ 513
Crawler for Nintendo Switch eShop
Scrapedin ⭐ 504
LinkedIn Scraper (currently working 2020)
Bookcorpus ⭐ 473
Crawl BookCorpus
Skrape.it ⭐ 469
A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
Scrapple ⭐ 452
A framework for creating semi-automatic web content extractors
Operative Framework ⭐ 451
operative framework is a OSINT investigation framework, you can interact with multiple targets, execute multiple modules, create links with target, export rapport to PDF file, add note to target or results, interact with RESTFul API, write your own modules.
Scrapers ⭐ 428
A list of scrapers from around the web.
Xidel ⭐ 427
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Fbcrawl ⭐ 415
A Facebook crawler
Gosint ⭐ 411
OSINT Swiss Army Knife
Finance Go ⭐ 403
📊 Financial markets data library implemented in go.
Dataflowkit ⭐ 394
Extract structured data from web sites. Web sites scraping.
Php Goose ⭐ 380
Readability / Html Content / Article Extractor & Web Scrapping library written in PHP
Advanced Web Scraping Tutorial ⭐ 370
The Zipru scraper developed in the Advanced Web Scraping Tutorial.
Trafilatura ⭐ 369
Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)
Micro Open Graph ⭐ 362
A tiny Node.js microservice to scrape open graph data with joy.
Jvppeteer ⭐ 348
Headless Chrome For Java （Java 爬虫）
Xcrawler ⭐ 346
快速、简洁且强大的PHP爬虫框架
Linkedin ⭐ 344
Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy
Freshonions Torscraper ⭐ 313
Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
Hquery.php ⭐ 309
An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.
Alltheplaces ⭐ 303
A set of spiders and scrapers to extract location information from places that post their location on the internet.
Bot ⭐ 293
Completely free and open-source human-like Instagram bot. Powered by UIAutomator2 and compatible with basically any Android device 5.0+ that can run Instagram - real or emulated.
Scrape Linkedin Selenium ⭐ 288
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
Webinspector ⭐ 286
Ruby gem to inspect completely a web page. It scrapes a given URL, and returns you its meta, links, images more.
Sniff Paste ⭐ 280
Pastebin OSINT Harvester
Google Play Scraper ⭐ 277
Google play scraper for Python inspired by <facundoolano/google-play-scraper>
Java Spider ⭐ 276
一个基于webmagic框架二次开发的java爬虫框架实战，已实现能爬取腾讯，搜狐，今日头条（单独集成功能）等资讯内容，配合elasticsearch框架用法，实现了自动爬虫，已投入线上生产使用。
Scweet ⭐ 276
A simple and unlimited twitter scraper : scape tweets, likes, retweets, following, followers, user info, images...
Hltv ⭐ 275
The unofficial HLTV Node.js API
Youtube Projects ⭐ 272
This repository contains all the code I use in my YouTube tutorials.
Cryptocmd ⭐ 269
Cryptocurrency historical price data library in Python. Data from https://coinmarketcap.com.
Weibo_terminator_workflow ⭐ 259
Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!
Ant ⭐ 252
A web crawler for Go
Heroku_ebooks ⭐ 245
A script to generate Markov chains and to post to an _ebooks account on Twitter using Heroku
Scrapysharp ⭐ 245
reborn of https://bitbucket.org/rflechner/scrapysharp
Rcrawler ⭐ 240
An R web crawler and scraper
Instagram Proxy Api ⭐ 239
CORS compliant API to access Instagram's public data
Polite ⭐ 234
Be nice on the web
Getsy ⭐ 234
A simple browser/client-side web scraper.
Unfurl ⭐ 232
Metadata scraper with support for oEmbed, Twitter Cards and Open Graph Protocol for Node.js ⚡️
Opensanctions ⭐ 231
An open database of international sanctions data, persons of interest and politically exposed persons
Covid19_mobility ⭐ 218
COVID-19 Mobility Data Aggregator. Scraper of Google, Apple, Waze and TomTom COVID-19 Mobility Reports🚶🚘🚉
Docsearch Scraper ⭐ 215
DocSearch - Scraper
Ruiji.net ⭐ 213
crawler framework, distributed crawler extractor
Goose Parser ⭐ 208
Universal scrapping tool, which allows you to extract data using multiple environments
1-100 of 2,583 projects

Related Projects

Python Scraper Projects (1,334)
Javascript Scraper Projects (509)
Crawler Scraper Projects (360)
Scraper Scraping Projects (301)
Python3 Scraper Projects (295)
Python Python3 Scraper Projects (257)
Nodejs Scraper Projects (225)
Scraper Webscraping Projects (209)
Javascript Nodejs Scraper Projects (178)
Go Scraper Projects (171)
Api Scraper Projects (168)
Python Crawler Scraper Projects (158)
Selenium Scraper Projects (157)
Scraper Scraping Websites Projects (147)
Python Scraper Scraping Projects (142)
Python Selenium Scraper Projects (132)
Typescript Scraper Projects (131)
Php Scraper Projects (125)
Scraper Webscraper Projects (122)
Python Scraper Webscraping Projects (118)
Scraper Scrapy Projects (110)
Html Scraper Projects (108)
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194
Privacy policy
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open Source 2018-2022