Awesome Open Source
Awesome Open Source
Combined Topics
web-scraping x
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210

The Top 68 Web Scraping Open Source Projects

Categories > Data Processing > Web Scraping
Awesome Web Scraping ⭐ 4,106
List of libraries, tools and APIs for web scraping and data processing.
Autoscraper ⭐ 3,458
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Apify Js ⭐ 2,828
Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
Php Curl Class ⭐ 2,780
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
Selenium Python Helium ⭐ 2,466
Selenium-python but lighter: Helium is the best Python library for web automation.
Grab ⭐ 2,080
Web Scraping Framework
30 Days Of Python ⭐ 1,559
Learn Python for the next 30 (or so) Days.
Dat8 ⭐ 1,490
General Assembly's 2015 Data Science course in Washington, DC
Rod ⭐ 1,477
A Devtools driver for web automation and scraping
Rvest ⭐ 1,259
Simple web scraping for R
Snoop ⭐ 918
Snoop — инструмент разведки на основе открытых данных (OSINT world)
Youtube_tutorials ⭐ 779
Collection of scripts corresponding to LucidProgramming YouTube tutorials
Spidr ⭐ 660
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Faster Than Requests ⭐ 660
Faster requests on Python 3
Coolqlcool ⭐ 624
Nextjs server to query websites with GraphQL
Pythoncode Tutorials ⭐ 602
The Python Code Tutorials
Scrapy Fake Useragent ⭐ 532
Random User-Agent middleware based on fake-useragent
User Agents ⭐ 500
A JavaScript library for generating random user agents with data that's updated daily.
Rpa ⭐ 489
UI.Vision: Open-Source RPA Software (formerly Kantu) - Modern Robotic Process Automation with Selenium IDE++
Scrapple ⭐ 470
A framework for creating semi-automatic web content extractors
Selectolax ⭐ 378
Python binding to Modest engine (fast HTML5 parser with CSS selectors).
Ache ⭐ 324
ACHE is a web crawler for domain-specific search.
Basketball_reference_web_scraper ⭐ 283
NBA Stats API via Basketball Reference
Gopa ⭐ 278
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Quora Api ⭐ 254
An unofficial API for Quora.
Scrape Linkedin Selenium ⭐ 249
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
Wayback Machine Scraper ⭐ 240
A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
Docbao ⭐ 233
Công cụ quét và phân tích từ khoá các trang báo mạng Việt Nam
City Scrapers ⭐ 223
Scrape, standardize and share public meetings from local government websites
R Web Scraping Cheat Sheet ⭐ 218
Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.
Short Jokes Dataset ⭐ 216
Python scripts for building 'Short Jokes' dataset, featured on Kaggle
Trump Lies ⭐ 202
Tutorial: Web scraping in Python with Beautiful Soup
Bet On Sibyl ⭐ 189
Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)
Web Scraping ⭐ 183
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, SHFE and news data crawlers on BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
Twitter Intelligence ⭐ 182
Twitter Intelligence OSINT project performs tracking and analysis of the Twitter
Web Database Analytics ⭐ 177
Web scrapping and related analytics using Python tools
Learnpythonforresearch ⭐ 163
This repository provides everything you need to get started with Python for (social science) research.
Netflix Clone ⭐ 162
Netflix like full-stack application with SPA client and backend implemented in service oriented architecture
Scrapy Training ⭐ 157
Scrapy Training companion code
Helena ⭐ 155
A Chrome extension for writing custom web scraping programs and web automation programs. Just demonstrate how to collect the first row of data, then let the extension write the program for collecting all rows.
Phpscraper ⭐ 152
PHP Scraper - an highly opinionated web-interface for PHP
Juno_crawler ⭐ 150
Scrapy crawler to collect data on the back catalog of songs listed for sale.
Sqrape ⭐ 144
Simple Query Scraping with CSS and Go Reflection (MOVED to Gitlab)
Zillow ⭐ 142
Zillow Scraper for Python using Selenium
Html Metadata ⭐ 129
MetaData html scraper and parser for Node.js (supports Promises and callback style)
Actor Page Analyzer ⭐ 124
Apify actor that opens a web page in headless Chrome and analyzes the HTML and JavaScript objects, looks for schema.org microdata and JSON-LD metadata, analyzes AJAX requests, etc.
Ayakashi ⭐ 122
⚡️ Ayakashi.io - The next generation web scraping framework
Save For Offline ⭐ 114
Android app for saving webpages for offline reading.
Scrapyd Cluster On Heroku ⭐ 107
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉
Pulsar ⭐ 104
Turn large Web sites into tables and charts using simple SQLs.
Sillynium ⭐ 101
Automate the creation of Python Selenium Scripts by drawing coloured boxes on webpage elements
Hockey Scraper ⭐ 95
Python Package for scraping NHL Play-by-Play and Shift data
Splashr ⭐ 93
💦 Tools to Work with the 'Splash' JavaScript Rendering Service in R
Humanoid ⭐ 93
Node.js package to bypass CloudFlare's anti-bot JavaScript challenges
Detect Cms ⭐ 80
PHP Library for detecting CMS
Reader ⭐ 76
Extract clean(er), readable text from web pages via Mercury Web Parser.
Ping Sm ⭐ 72
Receive an email or Telegram message as soon as Migros Sanalmarket is available for delivery in your neighborhood.
Arachnid ⭐ 70
Powerful web scraping framework for Crystal
Cascadia ⭐ 68
Go cascadia package command line CSS selector
Decapitated ⭐ 65
Headless 'Chrome' Orchestration in R
Social Media Profile Scrapers ⭐ 64
Fetch user's data across social media
Instago ⭐ 60
Download/access photos, videos, stories, story highlights, postlives, following and followers of Instagram
Scrapy Craigslist ⭐ 54
Web Scraping Craigslist's Engineering Jobs in NY with Scrapy
Project Tauro ⭐ 52
A Router WiFi key recovery/cracking tool with a twist.
Actor Google Search Scraper ⭐ 43
Apify actor that crawls Google Search result pages (SERPs) and extracts a list of organic results, ads, related queries and more. It supports selection of custom country, language and location.
Uc Davis Cs Exams Analysis ⭐ 33
📈 Regression and Classification with UC Davis student quiz data and exam data
Letterboxd_recommendations ⭐ 27
Scraping publicly-accessible Letterboxd data and creating a movie recommendation model with it that can generate recommendations when provided with a Letterboxd username
Webmiddle ⭐ 13
Node.js framework for modular web scraping and data extraction
1-68 of 68 projects
Advertising 📦 10
All Projects
Application Programming Interfaces 📦 124
Applications 📦 192
Artificial Intelligence 📦 78
Blockchain 📦 73
Build Tools 📦 113
Cloud Computing 📦 80
Code Quality 📦 28
Collaboration 📦 32
Command Line Interface 📦 49
Community 📦 83
Companies 📦 60
Compilers 📦 63
Computer Science 📦 80
Configuration Management 📦 42
Content Management 📦 175
Control Flow 📦 213
Data Formats 📦 78
Data Processing 📦 276
Data Storage 📦 135
Economics 📦 64
Frameworks 📦 215
Games 📦 129
Graphics 📦 110
Hardware 📦 152
Integrated Development Environments 📦 49
Learning Resources 📦 166
Legal 📦 29
Libraries 📦 129
Lists Of Projects 📦 22
Machine Learning 📦 347
Mapping 📦 64
Marketing 📦 15
Mathematics 📦 55
Media 📦 239
Messaging 📦 98
Networking 📦 315
Operating Systems 📦 89
Operations 📦 121
Package Managers 📦 55
Programming Languages 📦 245
Runtime Environments 📦 100
Science 📦 42
Security 📦 396
Social Media 📦 27
Software Architecture 📦 72
Software Development 📦 72
Software Performance 📦 58
Software Quality 📦 133
Text Editors 📦 49
Text Processing 📦 136
User Interface 📦 330
User Interface Components 📦 514
Version Control 📦 30
Virtualization 📦 71
Web Browsers 📦 42
Web Servers 📦 26
Web User Interface 📦 210