Awesome Open Source
Awesome Open Source
Combined Topics
scraper x

The Top 7,359 Scraper Open Source Projects

Categories > Data Processing > Scraper
Scrapy ⭐ 45,704
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 373total releases 87most recent commit 20 hours agopypi Scrapy} Downloads
Huginn ⭐ 37,411
Create agents that monitor and act on your behalf. Your agents are standing by!
dependent packages 35total releases 8most recent commit 7 days agorubygems huginn_agent} Downloads
Devdocs ⭐ 30,650
API Documentation Browser
most recent commit 3 days ago
Cheerio ⭐ 25,868
Fast, flexible, and lean implementation of core jQuery designed specifically for the server.
dependent packages 16,650total releases 70most recent commit a day agonpm cheerio} Downloads
Lux ⭐ 19,964
👾 Fast and simple video download library and CLI tool written in Go
dependent packages 5total releases 31most recent commit 14 days ago
Colly ⭐ 18,665
Elegant Scraper and Crawler Framework for Golang
dependent packages 201total releases 22most recent commit 11 days ago
Requests Html ⭐ 12,829
Pythonic HTML Parsing for Humans™
dependent packages 166total releases 36most recent commit 2 months agopypi requests-html} Downloads
Newspaper ⭐ 12,252
News, full-text, and article metadata extraction in Python 3. Advanced docs:
dependent packages 48total releases 18most recent commit 2 months agopypi newspaper3k} Downloads
Webmagic ⭐ 10,650
A scalable web crawler framework for Java.
dependent packages 15total releases 21most recent commit a month ago
Jsoup ⭐ 9,951
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
dependent packages 1,305total releases 40most recent commit 3 days ago
Chinese Xinhua ⭐ 9,789
📙 中华新华字典数据库。包括歇后语，成语，词语，汉字。
most recent commit 2 months ago
Avbook ⭐ 8,694
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
most recent commit 23 days ago
Portia ⭐ 8,690
Visual scraping for Scrapy
dependent packages 2total releases 26most recent commit a month agopypi slybot} Downloads
Changedetection.io ⭐ 7,763
The best and simplest self-hosted free open source website change detection, monitor and notification service. Restock Monitor, change detection. Designed for simplicity - the main goal is to simply monitor which websites had a text change for free. Free Open source web page change detection, Restock Monitoring, Visualping and Apify alternative
most recent commit 2 days ago
Crawlee ⭐ 7,509
Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.
dependent packages 16total releases 246most recent commit 21 hours agonpm @crawlee/types} Downloads
Tabula ⭐ 5,629
Tabula is a tool for liberating data tables trapped inside PDF files
most recent commit 9 months ago
Awesome Web Scraping ⭐ 5,576
List of libraries, tools and APIs for web scraping and data processing.
most recent commit a month ago
Awesome Crawler ⭐ 5,308
A collection of awesome web crawler,spider in different languages
most recent commit a month ago
Ferret ⭐ 5,185
Declarative web scraping
dependent packages 3total releases 34most recent commit 15 days ago
Headless Chrome Crawler ⭐ 5,051
Distributed crawler powered by Headless Chrome
dependent packages 9total releases 21most recent commit a year agonpm headless-chrome-crawler} Downloads
Autoscraper ⭐ 4,803
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
dependent packages 1total releases 16most recent commit a month agopypi autoscraper} Downloads
Mechanize ⭐ 4,208
Mechanize is a ruby library that makes automated web interaction easy.
dependent packages 966total releases 84most recent commit 8 months agorubygems mechanize} Downloads
Node Osmosis ⭐ 3,978
Web scraper for NodeJS
dependent packages 56total releases 27most recent commit 2 years agonpm osmosis} Downloads
Scrape It ⭐ 3,880
🔮 A Node.js scraper for humans.
dependent packages 72total releases 42most recent commit 3 months agonpm scrape-it} Downloads
Python Scraping ⭐ 3,757
Code samples from the book Web Scraping with Python http://shop.oreilly.com/product/0636920034391.do
most recent commit 6 months ago
Undetected Chromedriver ⭐ 3,706
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
dependent packages 31total releases 35most recent commit 13 days agopypi undetected-chromedriver} Downloads
Scraperjs ⭐ 3,575
A complete and versatile web scraper.
dependent packages 33total releases 15most recent commit 6 years agonpm scraperjs} Downloads
Node Ytdl Core ⭐ 3,536
YouTube video downloader in javascript.
dependent packages 520total releases 184most recent commit a month agonpm ytdl-core} Downloads
Dotnetspider ⭐ 3,496
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
dependent packages 7total releases 43most recent commit 2 months agonuget DotnetSpider} Downloads
Data Science ⭐ 3,403
Collection of useful data science topics along with articles, videos, and code
most recent commit 11 days ago
Twitter Scraper ⭐ 3,377
Scrape the Twitter Frontend API without authentication.
dependent packages 2total releases 11most recent commit 2 months agopypi twitter-scraper} Downloads
Rod ⭐ 3,335
A Devtools driver for web automation and scraping
dependent packages 58total releases 385most recent commit 13 days ago
Tiktok Scraper ⭐ 3,268
TikTok Scraper. Download video posts, collect user/trend/hashtag/music feed metadata, sign URL and etc.
dependent packages 12total releases 91most recent commit 21 days agonpm tiktok-scraper} Downloads
Onlyfans ⭐ 3,261
Scrape all the media from an OnlyFans account - Updated regularly
most recent commit 11 days ago
Browser Fingerprinting ⭐ 3,170
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?
most recent commit 3 months ago
Saikou ⭐ 3,128
An Android Anilist only client, which lets you stream & download Anime & Manga.
most recent commit 2 days ago
Fake Useragent ⭐ 2,894
Up-to-date simple useragent faker with real world database
most recent commit 16 days ago
Python ⭐ 2,840
Python Books && Courses
most recent commit 2 months ago
Automatic Udemy Course Enroller Get Paid Udemy Courses For Free ⭐ 2,806
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
total releases 10most recent commit 8 days ago
Instagram Php Scraper ⭐ 2,762
Get account information, photos, videos, stories and comments.
dependent packages 17total releases 112most recent commit 4 days agopackagist raiym/instagram-php-scraper} Downloads
Panther ⭐ 2,661
A browser testing and web crawling library for PHP and Symfony
dependent packages 30total releases 21most recent commit 2 months agopackagist symfony/panther} Downloads
Tweets_analyzer ⭐ 2,573
Tweets metadata scraper & activity analyzer
most recent commit 3 years ago
Snscrape ⭐ 2,512
A social networking service scraper in Python
dependent packages 5total releases 11most recent commit 2 days agopypi snscrape} Downloads
Ultimate Facebook Scraper ⭐ 2,479
🤖 A Software that automates your social media interactions to collect posts, photos, videos, interests, friends, followers, and much more on Facebook.
most recent commit a month ago
Emby.plugins.javscraper ⭐ 2,352
Emby/Jellyfin 的一个日本电影刮削器插件，可以从某些网站抓取影片信息。
most recent commit 2 months ago
Querylist ⭐ 2,333
🕷 The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
dependent packages 71total releases 21most recent commit a year agopackagist jaeger/querylist} Downloads
Googlescraper ⭐ 2,325
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
most recent commit 2 years ago
Thal ⭐ 2,268
Getting started with Puppeteer and Chrome Headless for Web Scraping
most recent commit 2 years ago
Weibo_terminater ⭐ 2,265
Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
most recent commit 3 years ago
Grab ⭐ 2,252
Web Scraping Framework
dependent packages 4total releases 120most recent commit a month agopypi grab} Downloads
Facebook Page Post Scraper ⭐ 2,014
Data scraper for Facebook Pages, and also code accompanying the blog post How to Scrape Data From Facebook Page Posts for Statistical Analysis
most recent commit 4 years ago
Embed ⭐ 1,944
Get info from any web service or page
dependent packages 62total releases 160most recent commit a month agopackagist embed/embed} Downloads
Awesome Puppeteer ⭐ 1,881
A curated list of awesome puppeteer resources.
most recent commit 6 months ago
Google Play Scraper ⭐ 1,877
Node.js scraper to get data from Google Play
dependent packages 22total releases 84most recent commit 8 days agonpm google-play-scraper} Downloads
Twitterscraper ⭐ 1,852
Scrape Twitter for Tweets
dependent packages 1total releases 47most recent commit a year agopypi twitterscraper} Downloads
Node.io ⭐ 1,845
dependent packages 8total releases 109most recent commit 7 years agonpm node.io} Downloads
Geziyor ⭐ 1,814
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
dependent packages 11total releases 6most recent commit a month ago
Nyt 2020 Election Scraper ⭐ 1,788
most recent commit 2 years ago
Snoop ⭐ 1,727
Snoop — инструмент разведки на основе открытых данных (OSINT world)
most recent commit a day ago
Content ⭐ 1,711
Official content for Harvard CS109
most recent commit a month ago
Soup ⭐ 1,696
Web Scraper in Go, similar to BeautifulSoup
dependent packages 30total releases 10most recent commit a year ago
Scrapely ⭐ 1,668
A pure-python HTML screen-scraping library
dependent packages 3total releases 13most recent commit 3 years agopypi scrapely} Downloads
Bulk Downloader For Reddit ⭐ 1,661
Downloads and archives content from reddit
total releases 12most recent commit a day ago
Upton ⭐ 1,615
A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
total releases 20most recent commit 4 years ago
Ruia ⭐ 1,611
Async Python 3.6+ web scraping micro-framework based on asyncio
dependent packages 6total releases 54most recent commit 5 months agopypi ruia} Downloads
Freedictionaryapi ⭐ 1,537
There was no free Dictionary API on the web when I wanted one for my friend, so I created one.
dependent packages 1total releases 9most recent commit a month agonpm google-dictionary-api} Downloads
Jobfunnel ⭐ 1,533
Scrape job websites into a single spreadsheet with no duplicates.
most recent commit a year ago
Facebook Scraper ⭐ 1,526
Scrape Facebook public pages without an API key
dependent packages 6total releases 74most recent commit 15 days agopypi facebook-scraper} Downloads
Scrape ⭐ 1,464
A simple, higher level interface for Go web scraping.
dependent packages 16most recent commit 6 years ago
Tomorrow ⭐ 1,463
Magic decorator syntax for asynchronous code in Python
dependent packages 2total releases 5most recent commit 3 years agopypi tomorrow} Downloads
Douyin_tiktok_download_api ⭐ 1,453
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikT
most recent commit 15 days ago
Rvest ⭐ 1,383
Simple web scraping for R
dependent packages 218total releases 9most recent commit 16 days ago
Node Website Scraper ⭐ 1,300
Download website to local directory (including all css, images, js, etc.)
dependent packages 47total releases 59most recent commit 2 months agonpm website-scraper} Downloads
Wombat ⭐ 1,265
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
dependent packages 5total releases 33most recent commit 5 months agorubygems wombat} Downloads
Scraper ⭐ 1,217
HTML parsing and querying with CSS selectors
dependent packages 177total releases 20most recent commit 16 days agocargo scraper} Downloads
Jd Autobuy ⭐ 1,208
Python爬虫，京东自动登录，在线抢购商品
most recent commit 3 years ago
Gdom ⭐ 1,180
DOM Traversing and Scraping using GraphQL
total releases 3most recent commit 3 years ago
Snmp_exporter ⭐ 1,141
SNMP Exporter for Prometheus
total releases 33most recent commit 14 days ago
How To Prevent Scraping ⭐ 1,127
The ultimate guide on preventing Website Scraping
most recent commit 2 years ago
Event Collect ⭐ 1,110
event website listing to Open Event format scraper and converter
most recent commit 3 years ago
Loklak_scraper_js ⭐ 1,094
Scrapers for loklak in javascript
most recent commit 4 years ago
Django Dynamic Scraper ⭐ 1,069
Creating Scrapy scrapers via the Django admin interface
total releases 65most recent commit a year ago
Cinemagoer ⭐ 1,056
Cinemagoer is a Python package useful to retrieve and manage the data of the IMDb (to which we are not affiliated in any way) movie database about movies, people, characters and companies
most recent commit a month ago
Recipe Scrapers ⭐ 1,053
Python package for scraping recipes data
dependent packages 3total releases 141most recent commit 5 days agopypi recipe-scrapers} Downloads
Parliament Scraper ⭐ 1,049
Public Data Scraper for Parliament Data for the EU and other Parliaments
most recent commit 6 years ago
Artoo ⭐ 1,024
artoo.js - the client-side scraping companion.
dependent packages 5total releases 10most recent commit 2 years agonpm artoo-js} Downloads
Scrapy Cluster ⭐ 1,016
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
dependent packages 2total releases 15most recent commit a year agopypi scutils} Downloads
Pjscrape ⭐ 1,003
A web-scraping framework written in Javascript, using PhantomJS and jQuery
most recent commit 8 years ago
Web Scraper Chrome Extension ⭐ 980
Web data extraction tool implemented as chrome extension
most recent commit 4 years ago
Metainspector ⭐ 976
Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
dependent packages 17total releases 112most recent commit a day agorubygems metainspector} Downloads
Aos Avp ⭐ 943
NOVA opeN sOurce Video plAyer: main repository to build them all
most recent commit 4 days ago
World_cup_json ⭐ 937
Rails backend for a scraper that outputs World Cup data as JSON
most recent commit a month ago
Redditdownloader ⭐ 921
Scrapes Reddit to download media of your choice.
most recent commit 12 days ago
Scanless ⭐ 919
online port scan scraper
total releases 17most recent commit a year ago
Crawler User Agents ⭐ 902
Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome ⭐️
dependent packages 3total releases 90most recent commit 2 months agonpm crawler-user-agents} Downloads
Informer ⭐ 902
A Telegram Mass Surveillance Bot in Python
most recent commit a year ago
Linkedin_scraper ⭐ 894
A library that scrapes Linkedin for user data
total releases 35most recent commit 2 months ago
Osi.ig ⭐ 886
Information Gathering Instagram.
most recent commit 2 months ago
Kimuraframework ⭐ 874
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
dependent packages 2total releases 10most recent commit a year agorubygems kimurai} Downloads
Parsel ⭐ 865
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
dependent packages 98total releases 20most recent commit 14 days agopypi parsel} Downloads
1-100 of 7,359 projects
Related Awesome Lists
Python Scraper Projects (3,513)
Javascript Scraper Projects (2,047)
Scraper Scrape Projects (1,534)
Scraper Web Crawler Projects (1,375)
Scraper Scraping Projects (1,097)
Scraper Crawling Projects (866)
Html Scraper Projects (732)
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 72
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 342
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 116
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,154,708)
Python  (888,323)
Java  (411,578)
Typescript  (304,714)
Php  (297,724)
C Plus Plus  (257,665)
Ruby  (225,073)
C  (193,605)
Shell  (182,260)
C Sharp  (178,205)
Golang  (171,741)
Swift  (68,947)
Rust  (68,512)
R  (62,080)
Dart  (61,781)
Objective C  (57,059)
Kotlin  (53,231)
Lua  (38,349)
Matlab  (32,815)
Perl  (30,816)
Language  (29,928)
Scala  (29,370)
Haskell  (20,925)
Powershell  (20,379)
Clojure  (19,555)
Bash  (19,201)
Elixir  (17,444)
Coffeescript  (17,147)
Assembly  (15,654)
Julia  (13,994)
Processing  (13,325)
Emacs Lisp  (10,848)
Basic  (10,674)
Flow  (10,509)
Solidity  (10,029)
Erlang  (9,124)
Groovy  (8,612)
Scheme  (8,437)
Ocaml  (6,951)
Pascal  (6,798)
Hack  (6,596)
Lisp  (6,356)
Fortran  (5,792)
Elm  (5,567)
Common Lisp  (5,467)
Shell Script  (4,835)
Dlang  (4,803)
Python Library  (4,516)
Crystal  (4,443)
F Sharp  (4,260)
Privacy | About | Terms | Follow Us On Twitter | Top Projects

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2023 Awesome Open Source.  All rights reserved.