Awesome Open Source
Awesome Open Source
Combined Topics
scraping x
Advertising πŸ“¦Β 9
All Projects
Application Programming Interfaces πŸ“¦Β 120
Applications πŸ“¦Β 181
Artificial Intelligence πŸ“¦Β 72
Blockchain πŸ“¦Β 70
Build Tools πŸ“¦Β 111
Cloud Computing πŸ“¦Β 79
Code Quality πŸ“¦Β 28
Collaboration πŸ“¦Β 30
Command Line Interface πŸ“¦Β 48
Community πŸ“¦Β 81
Companies πŸ“¦Β 60
Compilers πŸ“¦Β 60
Computer Science πŸ“¦Β 74
Configuration Management πŸ“¦Β 39
Content Management πŸ“¦Β 167
Control Flow πŸ“¦Β 197
Data Formats πŸ“¦Β 77
Data Processing πŸ“¦Β 266
Data Storage πŸ“¦Β 132
Economics πŸ“¦Β 60
Frameworks πŸ“¦Β 198
Games πŸ“¦Β 122
Graphics πŸ“¦Β 103
Hardware πŸ“¦Β 148
Integrated Development Environments πŸ“¦Β 47
Learning Resources πŸ“¦Β 147
Legal πŸ“¦Β 28
Libraries πŸ“¦Β 119
Lists Of Projects πŸ“¦Β 21
Machine Learning πŸ“¦Β 336
Mapping πŸ“¦Β 61
Marketing πŸ“¦Β 15
Mathematics πŸ“¦Β 55
Media πŸ“¦Β 228
Messaging πŸ“¦Β 97
Networking πŸ“¦Β 304
Operating Systems πŸ“¦Β 84
Operations πŸ“¦Β 120
Package Managers πŸ“¦Β 52
Programming Languages πŸ“¦Β 229
Runtime Environments πŸ“¦Β 96
Science πŸ“¦Β 42
Security πŸ“¦Β 375
Social Media πŸ“¦Β 26
Software Architecture πŸ“¦Β 70
Software Development πŸ“¦Β 68
Software Performance πŸ“¦Β 57
Software Quality πŸ“¦Β 127
Text Editors πŸ“¦Β 45
Text Processing πŸ“¦Β 131
User Interface πŸ“¦Β 310
User Interface Components πŸ“¦Β 465
Version Control πŸ“¦Β 29
Virtualization πŸ“¦Β 68
Web Browsers πŸ“¦Β 38
Web Servers πŸ“¦Β 25
Web User Interface πŸ“¦Β 194

The Top 1,351 Scraping Open Source Projects on Github

CategoriesΒ >Β Data ProcessingΒ >Β Scraping
Scrapy ⭐ 42,485
Scrapy, a fast high-level web crawling & scraping framework for Python.
Colly ⭐ 15,763
Elegant Scraper and Crawler Framework for Golang
Requests Html ⭐ 12,255
Pythonic HTML Parsing for Humansβ„’
Webmagic ⭐ 10,225
A scalable web crawler framework for Java.
Tabula ⭐ 5,141
Tabula is a tool for liberating data tables trapped inside PDF files
Headless Chrome Crawler ⭐ 5,051
Distributed crawler powered by Headless Chrome
Ferret ⭐ 4,858
Declarative web scraping
Mechanize ⭐ 4,166
Mechanize is a ruby library that makes automated web interaction easy.
Autoscraper ⭐ 3,281
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Apify Js ⭐ 3,187
Apify SDK β€” The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
Panther ⭐ 2,495
A browser testing and web crawling library for PHP and Symfony
Googlescraper ⭐ 2,325
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
Thal ⭐ 2,268
Getting started with Puppeteer and Chrome Headless for Web Scraping
Embed ⭐ 1,813
Get info from any web service or page
Awesome Puppeteer ⭐ 1,673
A curated list of awesome puppeteer resources.
Geziyor ⭐ 1,589
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Data Science ⭐ 1,448
Collection of useful data science topics along with code and articles
Undetected Chromedriver ⭐ 1,394
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
Snoop ⭐ 1,256
Snoop β€” инструмСнт Ρ€Π°Π·Π²Π΅Π΄ΠΊΠΈ Π½Π° основС ΠΎΡ‚ΠΊΡ€Ρ‹Ρ‚Ρ‹Ρ… Π΄Π°Π½Π½Ρ‹Ρ… (OSINT world)
Django Dynamic Scraper ⭐ 1,069
Creating Scrapy scrapers via the Django admin interface
Artoo ⭐ 1,024
artoo.js - the client-side scraping companion.
Facebook Scraper ⭐ 983
Scrape Facebook public pages without an API key
Scrapy Cluster ⭐ 934
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
Instagram Scraper ⭐ 895
Scrape the Instagram frontend. Inspired from twitter-scraper by @kennethreitz.
Till ⭐ 770
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
Lulu ⭐ 752
[Unmaintained] A simple and clean video/music/image downloader πŸ‘Ύ
Parsel ⭐ 727
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
Easy Scraping Tutorial ⭐ 618
Simple but useful Python web scraping tutorial code.
Oj ⭐ 607
Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.
Newcrawler ⭐ 576
Free Web Scraping Tool with Java
Katana ⭐ 576
python script for Google Dorking
Imagescraper ⭐ 572
βœ‚οΈ High performance, multi-threaded image scraper
Iiab ⭐ 569
Internet-in-a-Box - Build your own LIBRARY OF ALEXANDRIA with a Raspberry Pi !
Gazpacho ⭐ 543
πŸ₯« The simple, fast, and modern web scraping library
Crawly ⭐ 534
Crawly, a high-level web crawling & scraping framework for Elixir.
Facebook_data_analyzer ⭐ 507
Analyze facebook copy of your data with ruby language. Download zip file from facebook and get info about friends ranking by message, vocabulary, contacts, friends added statistics and more
Nickjs ⭐ 488
Web scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)
Edu Mail Generator ⭐ 485
Generate Free Edu Mail(s) within minutes
Lookyloo ⭐ 477
Lookyloo is a web interface that allows users to capture a website page and then display a tree of domains that call each other.
Scrapple ⭐ 452
A framework for creating semi-automatic web content extractors
Jekyll ⭐ 443
Jekyll-based static site for The Programming Historian
Clean Text ⭐ 434
🧹 Python package for text cleaning
Loconotion ⭐ 413
πŸ“„ Python tool to turn Notion.so pages into lightweight, customizable static websites
Comic Dl ⭐ 402
Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.
Dataflowkit ⭐ 394
Extract structured data from web sites. Web sites scraping.
Spidermon ⭐ 384
Scrapy Extension for monitoring spiders execution.
Coronadatascraper ⭐ 372
COVID-19 Coronavirus data scraped from government and curated data sources.
Social Media Profiles Regexs ⭐ 365
πŸ“‡ Extract social media profiles and more with regular expressions
Trafilatura ⭐ 358
Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)
Secret Agent ⭐ 347
The web browser that's built for scraping.
Linkedin ⭐ 344
Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy
Tinking ⭐ 337
🧢 Extract data from any website without code, just clicks.
Lambdasoup ⭐ 304
Functional HTML scraping and rewriting with CSS in OCaml
Scrapy Zyte Smartproxy ⭐ 300
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
Elixir Scrape ⭐ 300
Scrape any website, article or RSS/Atom Feed with ease!
Dataengineeringproject ⭐ 299
Example end to end data engineering project.
Kuwala ⭐ 295
Kuwala is a tool for integrating third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
Scrape Linkedin Selenium ⭐ 288
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
Gopa ⭐ 281
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Search Engine Parser ⭐ 278
Lightweight package to query popular search engines and scrape for result titles, links and descriptions
Scweet ⭐ 276
A simple and unlimited twitter scraper : scape tweets, likes, retweets, following, followers, user info, images...
Jikan Rest ⭐ 269
The REST API for Jikan
Post Tuto Deployment ⭐ 269
Build and deploy a machine learning app from scratch πŸš€
Sasila ⭐ 264
δΈ€δΈͺ灡活、友ε₯½ηš„ηˆ¬θ™«ζ‘†ζžΆ
Reaper ⭐ 263
Social media scraping / data collection tool for the Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs
Memorious ⭐ 259
Lightweight web scraping toolkit for documents and structured data.
Musoq ⭐ 250
Use SQL on various data sources
Scrapysharp ⭐ 245
reborn of https://bitbucket.org/rflechner/scrapysharp
Jsoup Annotations ⭐ 244
Jsoup Annotations POJO
Arachnid ⭐ 241
Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites
Socialreaper ⭐ 233
Social media scraping / data collection library for Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs
Juriscraper ⭐ 223
An API to scrape American court websites for metadata.
List Of User Agents ⭐ 219
List of major web + mobile browser user agent strings. +1 Bonus script to scrape :)
Transistor ⭐ 208
Transistor, a Python web scraping framework for intelligent use cases.
Goose Parser ⭐ 208
Universal scrapping tool, which allows you to extract data using multiple environments
Idt ⭐ 206
Image Dataset Tool (idt) is a cli tool designed to make the otherwise repetitive and slow task of creating image datasets into a fast and intuitive process.
Anime Dl ⭐ 196
Anime-dl is a command-line program to download anime from CrunchyRoll and Funimation.
Spatula ⭐ 195
A modern Python library for writing maintainable web scrapers.
Jsonframe Cheerio ⭐ 191
simple multi-level scraper json input/output for Cheerio
Linkedin Profile Scraper ⭐ 191
πŸ•΅οΈβ€β™‚οΈ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.
Phpscraper ⭐ 186
PHP Scraper - an highly opinionated web-interface for PHP
Antch ⭐ 177
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
Educative.io Downloader ⭐ 167
πŸ“– This tool is to download course from educative.io for offline usage. It uses your login credentials and download the course.
Pdf.tocgen ⭐ 166
A CLI toolset to generate table of contents for PDF files automatically.
Fantasy Basketball ⭐ 156
Scraping statistics, predicting NBA player performance with neural networks and boosting algorithms, and optimising lineups for Draft Kings with genetic algorithm. Capstone Project for Machine Learning Engineer Nanodegree by Udacity.
Xquery ⭐ 155
Extract data or evaluate value from HTML/XML documents using XPath
Sqrape ⭐ 145
Simple Query Scraping with CSS and Go Reflection (MOVED to Gitlab)
Scan For Webcams ⭐ 144
scan for webcams on the internet
Souqscraper ⭐ 132
Simple scriptes for Level UP your scraping Skills, and source code for Level UP playlist on Youtube
Jazz ⭐ 130
The Scripting Engine that Combines Speed, Safety, and Simplicity
Grawler ⭐ 128
Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file.
Udemycoursegrabber ⭐ 125
Your will to enroll in Udemy course is here, but the money isn't? Search no more! This python program searches for your desired course in more than [insert big number here] websites, compares the last updated date, and gives you the download link of the latest one back, but you also have the choice to see the other ones as well!
Serpscrap ⭐ 125
SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type from searchresults for given keywords. Detect Ads or make automated screenshots. You can also fetch text content of urls provided in searchresults or by your own. It's usefull for SEO and business related research tasks.
Linkedin Learning Downloader ⭐ 125
Linkedin Learning videos downloader
Shadow Useragent ⭐ 122
Pick the most common user-agents on the Internet πŸ‘»
Htmlsql ⭐ 121
htmlSQL is a experimental PHP library which allows you to access HTML values by an SQL like syntax.
Search Engine Google ⭐ 121
πŸ•· Google client for SERPS
D4n155 ⭐ 117
OWASP D4N155 - Intelligent and dynamic wordlist using OSINT
Od Database ⭐ 113
Distributed crawler, database and web frontend for public directories indexing
Languagepod101 Scraper ⭐ 106
Python scraper for Language Pods such as Japanesepod101.com πŸ‘Ή πŸ—Ύ 🍣 Compatible with Japanese, Chinese, French, German, Italian, Korean, Portuguese, Russian, Spanish and many more! ✨
1-100 of 1,351 projects

Related Projects

Python Scraping Projects (742)
Scraper Scraping Projects (301)
Javascript Scraping Projects (214)
Python3 Scraping Projects (178)
Python Python3 Scraping Projects (161)
Crawler Scraping Projects (151)
Python Scraper Scraping Projects (142)
Webscraping Scraping Projects (142)
Advertising πŸ“¦Β 9
All Projects
Application Programming Interfaces πŸ“¦Β 120
Applications πŸ“¦Β 181
Artificial Intelligence πŸ“¦Β 72
Blockchain πŸ“¦Β 70
Build Tools πŸ“¦Β 111
Cloud Computing πŸ“¦Β 79
Code Quality πŸ“¦Β 28
Collaboration πŸ“¦Β 30
Command Line Interface πŸ“¦Β 48
Community πŸ“¦Β 81
Companies πŸ“¦Β 60
Compilers πŸ“¦Β 60
Computer Science πŸ“¦Β 74
Configuration Management πŸ“¦Β 39
Content Management πŸ“¦Β 167
Control Flow πŸ“¦Β 197
Data Formats πŸ“¦Β 77
Data Processing πŸ“¦Β 266
Data Storage πŸ“¦Β 132
Economics πŸ“¦Β 60
Frameworks πŸ“¦Β 198
Games πŸ“¦Β 122
Graphics πŸ“¦Β 103
Hardware πŸ“¦Β 148
Integrated Development Environments πŸ“¦Β 47
Learning Resources πŸ“¦Β 147
Legal πŸ“¦Β 28
Libraries πŸ“¦Β 119
Lists Of Projects πŸ“¦Β 21
Machine Learning πŸ“¦Β 336
Mapping πŸ“¦Β 61
Marketing πŸ“¦Β 15
Mathematics πŸ“¦Β 55
Media πŸ“¦Β 228
Messaging πŸ“¦Β 97
Networking πŸ“¦Β 304
Operating Systems πŸ“¦Β 84
Operations πŸ“¦Β 120
Package Managers πŸ“¦Β 52
Programming Languages πŸ“¦Β 229
Runtime Environments πŸ“¦Β 96
Science πŸ“¦Β 42
Security πŸ“¦Β 375
Social Media πŸ“¦Β 26
Software Architecture πŸ“¦Β 70
Software Development πŸ“¦Β 68
Software Performance πŸ“¦Β 57
Software Quality πŸ“¦Β 127
Text Editors πŸ“¦Β 45
Text Processing πŸ“¦Β 131
User Interface πŸ“¦Β 310
User Interface Components πŸ“¦Β 465
Version Control πŸ“¦Β 29
Virtualization πŸ“¦Β 68
Web Browsers πŸ“¦Β 38
Web Servers πŸ“¦Β 25
Web User Interface πŸ“¦Β 194
Privacy policy
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open SourceΒ 2018-2022