Awesome Open Source
Awesome Open Source
Combined Topics
web-scraping x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 66 Web Scraping Open Source Projects

Categories > Data Processing > Web Scraping
Awesome Web Scraping ⭐ 3,920
List of libraries, tools and APIs for web scraping and data processing.
Autoscraper ⭐ 3,227
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Php Curl Class ⭐ 2,723
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
Apify Js ⭐ 2,693
Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
Grab ⭐ 2,041
Web Scraping Framework
Dat8 ⭐ 1,485
General Assembly's 2015 Data Science course in Washington, DC
30 Days Of Python ⭐ 1,467
A New Version of 30 Days of Python is nearly here. Get started today.
Rod ⭐ 1,293
A Devtools driver for web automation and scraping
Rvest ⭐ 1,231
Simple web scraping for R
Snoop ⭐ 836
Snoop — инструмент разведки на основе открытых данных (OSINT world)
Youtube_tutorials ⭐ 745
Collection of scripts corresponding to LucidProgramming YouTube tutorials
Spidr ⭐ 649
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Coolqlcool ⭐ 620
Nextjs server to query websites with GraphQL
Faster Than Requests ⭐ 584
Faster requests on Python 3
Scrapy Fake Useragent ⭐ 507
Random User-Agent middleware based on fake-useragent
User Agents ⭐ 463
A JavaScript library for generating random user agents with data that's updated daily.
Scrapple ⭐ 461
A framework for creating semi-automatic web content extractors
Rpa ⭐ 443
UI.Vision: Open-Source RPA Software (formerly Kantu) - Modern Robotic Process Automation with Selenium IDE++
Pythoncode Tutorials ⭐ 426
The Python Code Tutorials
Selectolax ⭐ 358
Python binding to Modest engine (fast HTML5 parser with CSS selectors).
Ache ⭐ 315
ACHE is a web crawler for domain-specific search.
Gopa ⭐ 277
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Basketball_reference_web_scraper ⭐ 265
NBA Stats API via Basketball Reference
Quora Api ⭐ 247
An unofficial API for Quora.
Docbao ⭐ 229
Công cụ quét và phân tích từ khóa các trang báo mạng Việt Nam
Scrape Linkedin Selenium ⭐ 222
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
Wayback Machine Scraper ⭐ 217
A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
City Scrapers ⭐ 213
Scrape, standardize and share public meetings from local government websites
Short Jokes Dataset ⭐ 211
Python scripts for building 'Short Jokes' dataset, featured on Kaggle
R Web Scraping Cheat Sheet ⭐ 199
Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.
Trump Lies ⭐ 194
Tutorial: Web scraping in Python with Beautiful Soup
Bet On Sibyl ⭐ 183
Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)
Twitter Intelligence ⭐ 174
Twitter Intelligence OSINT project performs tracking and analysis of the Twitter
Web Database Analytics ⭐ 168
Web scrapping and related analytics using Python tools
Learnpythonforresearch ⭐ 158
This repository provides everything you need to get started with Python for (social science) research.
Scrapy Training ⭐ 155
Scrapy Training companion code
Juno_crawler ⭐ 150
Scrapy crawler to collect data on the back catalog of songs listed for sale.
Netflix Clone ⭐ 149
Netflix like full-stack application with SPA client and backend implemented in service oriented architecture
Helena ⭐ 146
A Chrome extension for writing custom web scraping programs and web automation programs. Just demonstrate how to collect the first row of data, then let the extension write the program for collecting all rows.
Sqrape ⭐ 144
Simple Query Scraping with CSS and Go Reflection (MOVED to Gitlab)
Zillow ⭐ 139
Zillow Scraper for Python using Selenium
Phpscraper ⭐ 131
PHP Scraper - an highly opinionated web-interface for PHP
Web Scraping ⭐ 129
Python detailed web scraping tutorials for dummies with financial data crawlers on CME (both futures and options), US Treasury, CFTC, LME, SHFE and news data crawlers on BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
Html Metadata ⭐ 128
MetaData html scraper and parser for Node.js (supports Promises and callback style)
Actor Page Analyzer ⭐ 125
Apify actor that opens a web page in headless Chrome and analyzes the HTML and JavaScript objects, looks for schema.org microdata and JSON-LD metadata, analyzes AJAX requests, etc.
Save For Offline ⭐ 114
Android app for saving webpages for offline reading.
Ayakashi ⭐ 112
⚡️ Ayakashi.io - The next generation web scraping framework
Scrapyd Cluster On Heroku ⭐ 103
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉
Pulsar ⭐ 98
Turn large Web sites into tables and charts using simple SQLs.
Hockey Scraper ⭐ 93
Python Package for scraping NHL Play-by-Play and Shift data
Sillynium ⭐ 93
Automate the creation of Python Selenium Scripts by drawing coloured boxes on webpage elements
Splashr ⭐ 92
💦 Tools to Work with the 'Splash' JavaScript Rendering Service in R
Daftlistings ⭐ 83
A library that enables programmatic interaction with daft.ie. Daft.ie has nationwide coverage and contains about 80% of the total available properties in Ireland.
Humanoid ⭐ 82
Node.js package to bypass CloudFlare's anti-bot JavaScript challenges
Detect Cms ⭐ 79
PHP Library for detecting CMS
Reader ⭐ 74
Extract clean(er), readable text from web pages via Mercury Web Parser.
Ping Sm ⭐ 71
Receive an email or Telegram message as soon as Migros Sanalmarket is available for delivery in your neighborhood.
Arachnid ⭐ 68
Powerful web scraping framework for Crystal
Cascadia ⭐ 66
Go cascadia package command line CSS selector
Decapitated ⭐ 65
Headless 'Chrome' Orchestration in R
Instago ⭐ 56
Download/access photos, videos, stories, story highlights, postlives, following and followers of Instagram
Scrapy Craigslist ⭐ 54
Web Scraping Craigslist's Engineering Jobs in NY with Scrapy
Project Tauro ⭐ 51
A Router WiFi key recovery/cracking tool with a twist.
Social Media Profile Scrapers ⭐ 46
Fetch user's data across social media
Uc Davis Cs Exams Analysis ⭐ 33
📈 Regression and Classification with UC Davis student quiz data and exam data
Webmiddle ⭐ 13
Node.js framework for modular web scraping and data extraction
1-66 of 66 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210