The Top 115 Scraping Open Source Projects

Awesome Open Source

Awesome Open Source

Combined Topics

Advertising 📦 10

Application Programming Interfaces 📦 124

Applications 📦 192

Artificial Intelligence 📦 78

Blockchain 📦 73

Build Tools 📦 113

Cloud Computing 📦 80

Code Quality 📦 28

Collaboration 📦 32

Command Line Interface 📦 49

Community 📦 83

Companies 📦 60

Compilers 📦 63

Computer Science 📦 80

Configuration Management 📦 42

Content Management 📦 175

Control Flow 📦 213

Data Formats 📦 78

Data Processing 📦 276

Data Storage 📦 135

Economics 📦 64

Frameworks 📦 215

Graphics 📦 110

Hardware 📦 152

Integrated Development Environments 📦 49

Learning Resources 📦 166

Libraries 📦 129

Lists Of Projects 📦 22

Machine Learning 📦 347

Mapping 📦 64

Marketing 📦 15

Mathematics 📦 55

Messaging 📦 98

Networking 📦 315

Operating Systems 📦 89

Operations 📦 121

Package Managers 📦 55

Programming Languages 📦 245

Runtime Environments 📦 100

Science 📦 42

Security 📦 396

Social Media 📦 27

Software Architecture 📦 72

Software Development 📦 72

Software Performance 📦 58

Software Quality 📦 133

Text Editors 📦 49

Text Processing 📦 136

User Interface 📦 330

User Interface Components 📦 514

Version Control 📦 30

Virtualization 📦 71

Web Browsers 📦 42

Web Servers 📦 26

Web User Interface 📦 210

The Top 115 Scraping Open Source Projects

Categories > Data Processing > Scraping

Scrapy ⭐ 40,327

Scrapy, a fast high-level web crawling & scraping framework for Python.

Colly ⭐ 13,620

Elegant Scraper and Crawler Framework for Golang

Requests Html ⭐ 11,779

Pythonic HTML Parsing for Humans™

Webmagic ⭐ 9,726

A scalable web crawler framework for Java.

Tabula ⭐ 5,099

Tabula is a tool for liberating data tables trapped inside PDF files

Headless Chrome Crawler ⭐ 4,950

Distributed crawler powered by Headless Chrome

Ferret ⭐ 4,454

Declarative web scraping

Mechanize ⭐ 4,088

Mechanize is a ruby library that makes automated web interaction easy.

Autoscraper ⭐ 3,440

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Apify Js ⭐ 2,819

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Getting started with Puppeteer and Chrome Headless for Web Scraping

Panther ⭐ 2,303

A browser testing and web crawling library for PHP and Symfony

Googlescraper ⭐ 2,270

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Embed ⭐ 1,730

Get info from any web service or page

Awesome Puppeteer ⭐ 1,556

A curated list of awesome puppeteer resources.

Geziyor ⭐ 1,267

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Artoo ⭐ 1,034

artoo.js - the client-side scraping companion.

Django Dynamic Scraper ⭐ 1,027

Creating Scrapy scrapers via the Django admin interface

Scrapy Cluster ⭐ 938

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Instagram Scraper ⭐ 905

Scrape the Instagram frontend. Inspired from twitter-scraper by @kennethreitz.

[Unmaintained] A simple and clean video/music/image downloader 👾

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

Imagescraper ⭐ 632

✂️ High performance, multi-threaded image scraper

Newcrawler ⭐ 589

Free Web Scraping Tool with Java

Easy Scraping Tutorial ⭐ 588

Simple but useful Python web scraping tutorial code.

Facebook Scraper ⭐ 538

Scrape Facebook public pages without an API key

Gazpacho ⭐ 537

🥫 The simple, fast, and modern web scraping library

Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.

Facebook_data_analyzer ⭐ 515

Analyze facebook copy of your data with ruby language. Download zip file from facebook and get info about friends ranking by message, vocabulary, contacts, friends added statistics and more

Web scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)

Geeksforgeeks.pdf ⭐ 489

Topic wise PDFs of Geeks for Geeks articles. (Last updated in October 2018)

Scrapple ⭐ 469

A framework for creating semi-automatic web content extractors

Dataflowkit ⭐ 458

Extract structured data from web sites. Web sites scraping.

Crawly, a high-level web crawling & scraping framework for Elixir.

Data Science ⭐ 449

Collection of useful data science topics along with code and articles

Undetected Chromedriver ⭐ 416

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

Jekyll-based static site for The Programming Historian

A Python Tool For google Hacking

Lookyloo ⭐ 383

Lookyloo is a web interface that allows users to capture a website page and then display a tree of domains that call each other.

Coronadatascraper ⭐ 372

COVID-19 Coronavirus data scraped from government and curated data sources.

Comic Dl ⭐ 369

Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.

Post Tuto Deployment ⭐ 367

Build and deploy a machine learning app from scratch 🚀

Socialreaper ⭐ 345

Social media scraping / data collection library for Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs

Social Media Profiles Regexs ⭐ 338

📇 Extract social media profiles and more with regular expressions

Tinking ⭐ 336

🧶 Extract data from any website without code, just clicks.

Edu Mail Generator ⭐ 321

Generate Free Edu Mail(s) within minutes

Spidermon ⭐ 319

Scrapy Extension for monitoring spiders execution.

Linkedin ⭐ 315

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

Elixir Scrape ⭐ 307

Scrape any website, article or RSS/Atom Feed with ease!

Clean Text ⭐ 293

🧹 Python package for text cleaning

一个灵活、友好的爬虫框架

Lambdasoup ⭐ 287

Functional HTML scraping and rewriting with CSS in OCaml

Scrapy Crawlera ⭐ 284

Crawlera middleware for Scrapy

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Loconotion ⭐ 263

📄 Python tool to turn Notion.so pages into lightweight, customizable static websites

List Of User Agents ⭐ 259

List of major web + mobile browser user agent strings. +1 Bonus script to scrape :)

Use SQL on various data sources

Memorious ⭐ 250

Distributed crawling framework for documents and structured data.

Scrape Linkedin Selenium ⭐ 247

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

Social media scraping / data collection tool for the Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs

Jsoup Annotations ⭐ 242

Jsoup Annotations POJO

Scrapysharp ⭐ 229

reborn of https://bitbucket.org/rflechner/scrapysharp

Arachnid ⭐ 224

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

Search Engine Parser ⭐ 222

Lightweight package to query popular search engines and scrape for result titles, links and descriptions

Goose Parser ⭐ 211

Universal scrapping tool, which allows you to extract data using multiple environments

Jikan Rest ⭐ 207

The REST API for Jikan

Transistor ⭐ 205

Transistor, a Python web scraping framework for intelligent use cases.

Image Dataset Tool (idt) is a cli tool designed to make the otherwise repetitive and slow task of creating image datasets into a fast and intuitive process.

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Jsonframe Cheerio ⭐ 196

simple multi-level scraper json input/output for Cheerio

Juriscraper ⭐ 195

An API to scrape American court websites for metadata.

Anime Dl ⭐ 192

Anime-dl is a command-line program to download anime from CrunchyRoll and Funimation.

Linkedin Profile Scraper ⭐ 179

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Linkedin Learning Downloader ⭐ 173

Linkedin Learning videos downloader

Secret Agent ⭐ 167

The web browser that's built for scraping.

Extract data or evaluate value from HTML/XML documents using XPath

Serpscrap ⭐ 154

SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type from searchresults for given keywords. Detect Ads or make automated screenshots. You can also fetch text content of urls provided in searchresults or by your own. It's usefull for SEO and business related research tasks.

Fantasy Basketball ⭐ 151

Scraping statistics, predicting NBA player performance with neural networks and boosting algorithms, and optimising lineups for Draft Kings with genetic algorithm. Capstone Project for Machine Learning Engineer Nanodegree by Udacity.

Phpscraper ⭐ 150

PHP Scraper - an highly opinionated web-interface for PHP

Shadow Useragent ⭐ 148

Pick the most common user-agents on the Internet 👻

Simple Query Scraping with CSS and Go Reflection (MOVED to Gitlab)

Educative.io Downloader ⭐ 143

📖 This tool is to download course from educative.io for offline usage. It uses your login credentials and download the course.

Search Engine Google ⭐ 142

🕷 Google client for SERPS

Udemycoursegrabber ⭐ 138

Your will to enroll in Udemy course is here, but the money isn't? Search no more! This python program searches for your desired course in more than [insert big number here] websites, compares the last updated date, and gives you the download link of the latest one back, but you also have the choice to see the other ones as well!

Torchbear ⭐ 129

🔥🐻 The Speakeasy Scripting Engine Which Combines Speed, Safety, and Simplicity

Scan For Webcams ⭐ 128

scan for webcams on the internet

Souqscraper ⭐ 127

Simple scriptes for Level UP your scraping Skills, and source code for Level UP playlist on Youtube

Htmlsql ⭐ 121

htmlSQL is a experimental PHP library which allows you to access HTML values by an SQL like syntax.

Od Database ⭐ 121

Distributed crawler, database and web frontend for public directories indexing

Seleniumcrawler ⭐ 117

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

Laravel Bank Statements ⭐ 105

Laravel package to collect your bank statements history. Currently support for parsing statements history from BCA, Mandiri, BNI, and MUAMALAT e-banking websites.

OWASP D4N155 - Intelligent and dynamic wordlist using OSINT

Dotnetcrawler ⭐ 104

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Languagepod101 Scraper ⭐ 104

Python scraper for Language Pods such as Japanesepod101.com 👹 🗾 🍣 Compatible with Japanese, Chinese, French, German, Italian, Korean, Portuguese, Russian, Spanish and many more! ✨

Grawler ⭐ 102

Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file.

Dataengineeringproject ⭐ 97

Example end to end data engineering project.

Nintendeals ⭐ 96

Library with a set of tools for scraping information about Nintendo games and its prices across all regions (NA, EU and JP).

Humanoid ⭐ 91

Node.js package to bypass CloudFlare's anti-bot JavaScript challenges

Pastepwn ⭐ 87

Python framework to scrape Pastebin pastes and analyze them

legacy backend for Open States

1-100 of 115 projects

Advertising 📦 10

Application Programming Interfaces 📦 124

Applications 📦 192

Artificial Intelligence 📦 78

Blockchain 📦 73

Build Tools 📦 113

Cloud Computing 📦 80

Code Quality 📦 28

Collaboration 📦 32

Command Line Interface 📦 49

Community 📦 83

Companies 📦 60

Compilers 📦 63

Computer Science 📦 80

Configuration Management 📦 42

Content Management 📦 175

Control Flow 📦 213

Data Formats 📦 78

Data Processing 📦 276

Data Storage 📦 135

Economics 📦 64

Frameworks 📦 215

Graphics 📦 110

Hardware 📦 152

Integrated Development Environments 📦 49

Learning Resources 📦 166

Libraries 📦 129

Lists Of Projects 📦 22

Machine Learning 📦 347

Mapping 📦 64

Marketing 📦 15

Mathematics 📦 55

Messaging 📦 98

Networking 📦 315

Operating Systems 📦 89

Operations 📦 121

Package Managers 📦 55

Programming Languages 📦 245

Runtime Environments 📦 100

Science 📦 42

Security 📦 396

Social Media 📦 27

Software Architecture 📦 72

Software Development 📦 72

Software Performance 📦 58

Software Quality 📦 133

Text Editors 📦 49

Text Processing 📦 136

User Interface 📦 330

User Interface Components 📦 514

Version Control 📦 30

Virtualization 📦 71

Web Browsers 📦 42

Web Servers 📦 26

Web User Interface 📦 210