Awesome Open Source
Awesome Open Source
Combined Topics
scraping x
Advertising πŸ“¦Β 9
All Projects
Application Programming Interfaces πŸ“¦Β 120
Applications πŸ“¦Β 181
Artificial Intelligence πŸ“¦Β 72
Blockchain πŸ“¦Β 70
Build Tools πŸ“¦Β 111
Cloud Computing πŸ“¦Β 79
Code Quality πŸ“¦Β 28
Collaboration πŸ“¦Β 30
Command Line Interface πŸ“¦Β 48
Community πŸ“¦Β 81
Companies πŸ“¦Β 60
Compilers πŸ“¦Β 60
Computer Science πŸ“¦Β 74
Configuration Management πŸ“¦Β 39
Content Management πŸ“¦Β 167
Control Flow πŸ“¦Β 197
Data Formats πŸ“¦Β 77
Data Processing πŸ“¦Β 266
Data Storage πŸ“¦Β 132
Economics πŸ“¦Β 60
Frameworks πŸ“¦Β 198
Games πŸ“¦Β 122
Graphics πŸ“¦Β 103
Hardware πŸ“¦Β 148
Integrated Development Environments πŸ“¦Β 47
Learning Resources πŸ“¦Β 147
Legal πŸ“¦Β 28
Libraries πŸ“¦Β 119
Lists Of Projects πŸ“¦Β 21
Machine Learning πŸ“¦Β 336
Mapping πŸ“¦Β 61
Marketing πŸ“¦Β 15
Mathematics πŸ“¦Β 55
Media πŸ“¦Β 228
Messaging πŸ“¦Β 97
Networking πŸ“¦Β 304
Operating Systems πŸ“¦Β 84
Operations πŸ“¦Β 120
Package Managers πŸ“¦Β 52
Programming Languages πŸ“¦Β 229
Runtime Environments πŸ“¦Β 96
Science πŸ“¦Β 42
Security πŸ“¦Β 375
Social Media πŸ“¦Β 26
Software Architecture πŸ“¦Β 70
Software Development πŸ“¦Β 68
Software Performance πŸ“¦Β 57
Software Quality πŸ“¦Β 127
Text Editors πŸ“¦Β 45
Text Processing πŸ“¦Β 131
User Interface πŸ“¦Β 310
User Interface Components πŸ“¦Β 465
Version Control πŸ“¦Β 29
Virtualization πŸ“¦Β 68
Web Browsers πŸ“¦Β 38
Web Servers πŸ“¦Β 25
Web User Interface πŸ“¦Β 194

The Top 1,290 Scraping Open Source Projects on Github

CategoriesΒ >Β Data ProcessingΒ >Β Scraping
Scrapy ⭐ 41,588
Scrapy, a fast high-level web crawling & scraping framework for Python.
Colly ⭐ 14,759
Elegant Scraper and Crawler Framework for Golang
Requests Html ⭐ 12,044
Pythonic HTML Parsing for Humansβ„’
Webmagic ⭐ 9,951
A scalable web crawler framework for Java.
Tabula ⭐ 5,141
Tabula is a tool for liberating data tables trapped inside PDF files
Headless Chrome Crawler ⭐ 5,051
Distributed crawler powered by Headless Chrome
Ferret ⭐ 4,630
Declarative web scraping
Mechanize ⭐ 4,121
Mechanize is a ruby library that makes automated web interaction easy.
Autoscraper ⭐ 3,281
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Apify Js ⭐ 3,008
Apify SDK β€” The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
Panther ⭐ 2,411
A browser testing and web crawling library for PHP and Symfony
Googlescraper ⭐ 2,325
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
Thal ⭐ 2,268
Getting started with Puppeteer and Chrome Headless for Web Scraping
Embed ⭐ 1,761
Get info from any web service or page
Awesome Puppeteer ⭐ 1,552
A curated list of awesome puppeteer resources.
Geziyor ⭐ 1,512
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Snoop ⭐ 1,090
Snoop β€” инструмСнт Ρ€Π°Π·Π²Π΅Π΄ΠΊΠΈ Π½Π° основС ΠΎΡ‚ΠΊΡ€Ρ‹Ρ‚Ρ‹Ρ… Π΄Π°Π½Π½Ρ‹Ρ… (OSINT world)
Data Science ⭐ 1,065
Collection of useful data science topics along with code and articles
Django Dynamic Scraper ⭐ 1,037
Creating Scrapy scrapers via the Django admin interface
Artoo ⭐ 1,024
artoo.js - the client-side scraping companion.
Scrapy Cluster ⭐ 934
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
Instagram Scraper ⭐ 895
Scrape the Instagram frontend. Inspired from twitter-scraper by @kennethreitz.
Undetected Chromedriver ⭐ 849
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
Facebook Scraper ⭐ 806
Scrape Facebook public pages without an API key
Lulu ⭐ 752
[Unmaintained] A simple and clean video/music/image downloader πŸ‘Ύ
Till ⭐ 750
DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.
Parsel ⭐ 682
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
Easy Scraping Tutorial ⭐ 618
Simple but useful Python web scraping tutorial code.
Oj ⭐ 607
Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.
Newcrawler ⭐ 576
Free Web Scraping Tool with Java
Imagescraper ⭐ 572
βœ‚οΈ High performance, multi-threaded image scraper
Gazpacho ⭐ 543
πŸ₯« The simple, fast, and modern web scraping library
Iiab ⭐ 538
Internet-in-a-Box - Build your own LIBRARY OF ALEXANDRIA with a Raspberry Pi !
Facebook_data_analyzer ⭐ 507
Analyze facebook copy of your data with ruby language. Download zip file from facebook and get info about friends ranking by message, vocabulary, contacts, friends added statistics and more
Crawly ⭐ 499
Crawly, a high-level web crawling & scraping framework for Elixir.
Nickjs ⭐ 488
Web scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)
Katana ⭐ 485
A Python Tool For google Hacking
Scrapple ⭐ 452
A framework for creating semi-automatic web content extractors
Lookyloo ⭐ 448
Lookyloo is a web interface that allows users to capture a website page and then display a tree of domains that call each other.
Edu Mail Generator ⭐ 441
Generate Free Edu Mail(s) within minutes
Clean Text ⭐ 434
🧹 Python package for text cleaning
Jekyll ⭐ 419
Jekyll-based static site for The Programming Historian
Dataflowkit ⭐ 394
Extract structured data from web sites. Web sites scraping.
Comic Dl ⭐ 391
Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.
Coronadatascraper ⭐ 372
COVID-19 Coronavirus data scraped from government and curated data sources.
Social Media Profiles Regexs ⭐ 365
πŸ“‡ Extract social media profiles and more with regular expressions
Loconotion ⭐ 363
πŸ“„ Python tool to turn Notion.so pages into lightweight, customizable static websites
Linkedin ⭐ 344
Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy
Geeksforgeeks.pdf ⭐ 340
Topic wise PDFs of Geeks for Geeks articles. (Last updated in October 2018)
Spidermon ⭐ 339
Scrapy Extension for monitoring spiders execution.
Tinking ⭐ 337
🧢 Extract data from any website without code, just clicks.
Elixir Scrape ⭐ 300
Scrape any website, article or RSS/Atom Feed with ease!
Scrapy Zyte Smartproxy ⭐ 292
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
Lambdasoup ⭐ 290
Functional HTML scraping and rewriting with CSS in OCaml
Scrape Linkedin Selenium ⭐ 288
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
Gopa ⭐ 281
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Post Tuto Deployment ⭐ 269
Build and deploy a machine learning app from scratch πŸš€
Sasila ⭐ 264
δΈ€δΈͺ灡活、友ε₯½ηš„ηˆ¬θ™«ζ‘†ζžΆ
Reaper ⭐ 263
Social media scraping / data collection tool for the Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs
Secret Agent ⭐ 260
The web browser that's built for scraping.
Search Engine Parser ⭐ 258
Lightweight package to query popular search engines and scrape for result titles, links and descriptions
Memorious ⭐ 254
Lightweight web scraping toolkit for documents and structured data.
Musoq ⭐ 250
Use SQL on various data sources
Trafilatura ⭐ 249
Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)
Jsoup Annotations ⭐ 244
Jsoup Annotations POJO
Jikan Rest ⭐ 244
The REST API for Jikan
Socialreaper ⭐ 233
Social media scraping / data collection library for Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs
Arachnid ⭐ 230
Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites
List Of User Agents ⭐ 219
List of major web + mobile browser user agent strings. +1 Bonus script to scrape :)
Dataengineeringproject ⭐ 217
Example end to end data engineering project.
Juriscraper ⭐ 209
An API to scrape American court websites for metadata.
Transistor ⭐ 208
Transistor, a Python web scraping framework for intelligent use cases.
Goose Parser ⭐ 208
Universal scrapping tool, which allows you to extract data using multiple environments
Idt ⭐ 206
Image Dataset Tool (idt) is a cli tool designed to make the otherwise repetitive and slow task of creating image datasets into a fast and intuitive process.
Scweet ⭐ 198
A simple and unlimited twitter scraper : scape tweets, likes, retweets, following, followers, user info, images...
Scrapysharp ⭐ 195
reborn of https://bitbucket.org/rflechner/scrapysharp
Jsonframe Cheerio ⭐ 191
simple multi-level scraper json input/output for Cheerio
Linkedin Profile Scraper ⭐ 191
πŸ•΅οΈβ€β™‚οΈ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.
Spatula ⭐ 179
A modern Python library for writing maintainable web scrapers.
Antch ⭐ 177
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
Phpscraper ⭐ 174
PHP Scraper - an highly opinionated web-interface for PHP
Fantasy Basketball ⭐ 156
Scraping statistics, predicting NBA player performance with neural networks and boosting algorithms, and optimising lineups for Draft Kings with genetic algorithm. Capstone Project for Machine Learning Engineer Nanodegree by Udacity.
Xquery ⭐ 155
Extract data or evaluate value from HTML/XML documents using XPath
Sqrape ⭐ 145
Simple Query Scraping with CSS and Go Reflection (MOVED to Gitlab)
Scan For Webcams ⭐ 137
scan for webcams on the internet
Souqscraper ⭐ 132
Simple scriptes for Level UP your scraping Skills, and source code for Level UP playlist on Youtube
Anime Dl ⭐ 131
Anime-dl is a command-line program to download anime from CrunchyRoll and Funimation.
Torchbear ⭐ 129
πŸ”₯🐻 The Speakeasy Scripting Engine Which Combines Speed, Safety, and Simplicity
Linkedin Learning Downloader ⭐ 125
Linkedin Learning videos downloader
Serpscrap ⭐ 125
SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type from searchresults for given keywords. Detect Ads or make automated screenshots. You can also fetch text content of urls provided in searchresults or by your own. It's usefull for SEO and business related research tasks.
Udemycoursegrabber ⭐ 125
Your will to enroll in Udemy course is here, but the money isn't? Search no more! This python program searches for your desired course in more than [insert big number here] websites, compares the last updated date, and gives you the download link of the latest one back, but you also have the choice to see the other ones as well!
Shadow Useragent ⭐ 122
Pick the most common user-agents on the Internet πŸ‘»
Educative.io Downloader ⭐ 121
πŸ“– This tool is to download course from educative.io for offline usage. It uses your login credentials and download the course.
Search Engine Google ⭐ 121
πŸ•· Google client for SERPS
Htmlsql ⭐ 121
htmlSQL is a experimental PHP library which allows you to access HTML values by an SQL like syntax.
Od Database ⭐ 113
Distributed crawler, database and web frontend for public directories indexing
Languagepod101 Scraper ⭐ 106
Python scraper for Language Pods such as Japanesepod101.com πŸ‘Ή πŸ—Ύ 🍣 Compatible with Japanese, Chinese, French, German, Italian, Korean, Portuguese, Russian, Spanish and many more! ✨
D4n155 ⭐ 106
OWASP D4N155 - Intelligent and dynamic wordlist using OSINT
Humanoid ⭐ 105
Node.js package to bypass CloudFlare's anti-bot JavaScript challenges
Seleniumcrawler ⭐ 105
An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site
1-100 of 1,290 projects

Related Projects

Python Scraping Projects (679)
Scraper Scraping Projects (268)
Javascript Scraping Projects (207)
Python3 Scraping Projects (162)
Python Python3 Scraping Projects (146)
Crawler Scraping Projects (133)
Python Scraper Scraping Projects (125)
Webscraping Scraping Projects (124)
Advertising πŸ“¦Β 9
All Projects
Application Programming Interfaces πŸ“¦Β 120
Applications πŸ“¦Β 181
Artificial Intelligence πŸ“¦Β 72
Blockchain πŸ“¦Β 70
Build Tools πŸ“¦Β 111
Cloud Computing πŸ“¦Β 79
Code Quality πŸ“¦Β 28
Collaboration πŸ“¦Β 30
Command Line Interface πŸ“¦Β 48
Community πŸ“¦Β 81
Companies πŸ“¦Β 60
Compilers πŸ“¦Β 60
Computer Science πŸ“¦Β 74
Configuration Management πŸ“¦Β 39
Content Management πŸ“¦Β 167
Control Flow πŸ“¦Β 197
Data Formats πŸ“¦Β 77
Data Processing πŸ“¦Β 266
Data Storage πŸ“¦Β 132
Economics πŸ“¦Β 60
Frameworks πŸ“¦Β 198
Games πŸ“¦Β 122
Graphics πŸ“¦Β 103
Hardware πŸ“¦Β 148
Integrated Development Environments πŸ“¦Β 47
Learning Resources πŸ“¦Β 147
Legal πŸ“¦Β 28
Libraries πŸ“¦Β 119
Lists Of Projects πŸ“¦Β 21
Machine Learning πŸ“¦Β 336
Mapping πŸ“¦Β 61
Marketing πŸ“¦Β 15
Mathematics πŸ“¦Β 55
Media πŸ“¦Β 228
Messaging πŸ“¦Β 97
Networking πŸ“¦Β 304
Operating Systems πŸ“¦Β 84
Operations πŸ“¦Β 120
Package Managers πŸ“¦Β 52
Programming Languages πŸ“¦Β 229
Runtime Environments πŸ“¦Β 96
Science πŸ“¦Β 42
Security πŸ“¦Β 375
Social Media πŸ“¦Β 26
Software Architecture πŸ“¦Β 70
Software Development πŸ“¦Β 68
Software Performance πŸ“¦Β 57
Software Quality πŸ“¦Β 127
Text Editors πŸ“¦Β 45
Text Processing πŸ“¦Β 131
User Interface πŸ“¦Β 310
User Interface Components πŸ“¦Β 465
Version Control πŸ“¦Β 29
Virtualization πŸ“¦Β 68
Web Browsers πŸ“¦Β 38
Web Servers πŸ“¦Β 25
Web User Interface πŸ“¦Β 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open SourceΒ 2018-2021