Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Web Scraper Open Source Projects

Open source projects categorized as Web Scraper
Categories > Data Processing > Scraper
Easyspider ⭐ 20,149
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化
most recent commit 22 days ago
Awesome Web Scraping ⭐ 6,060
List of libraries, tools and APIs for web scraping and data processing.
most recent commit 5 months ago
Awesome Crawler ⭐ 5,859
A collection of awesome web crawler,spider in different languages
most recent commit 5 months ago
Rod ⭐ 4,505
A Devtools driver for web automation and scraping
dependent packages 140total releases 406most recent commit 3 months ago
Mechanize ⭐ 4,338
Mechanize is a ruby library that makes automated web interaction easy.
dependent packages 978total releases 86most recent commit 3 months agorubygems mechanize} Downloads
Python Scraping ⭐ 4,136
Code samples from the book Web Scraping with Python http://shop.oreilly.com/product/0636920034391.do
most recent commit 4 months ago
Browser Fingerprinting ⭐ 3,353
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?
most recent commit a year ago
Panther ⭐ 2,849
A browser testing and web crawling library for PHP and Symfony
dependent packages 50total releases 23most recent commit 3 months agopackagist symfony/panther} Downloads
Trafilatura ⭐ 2,447
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
dependent packages 66total releases 39most recent commit 3 months agopypi trafilatura} Downloads
Grab ⭐ 2,292
Web Scraping Framework
dependent packages 3total releases 120most recent commit 10 months agopypi grab} Downloads
Wombat ⭐ 1,297
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
dependent packages 5total releases 33most recent commit 3 months agorubygems wombat} Downloads
Web Scraper Chrome Extension ⭐ 980
Web data extraction tool implemented as chrome extension
most recent commit 5 years ago
Stealth ⭐ 923
🚀 Stealth - Secure, Peer-to-Peer, Private and Automateable Web Browser/Scraper/Proxy
most recent commit 8 months ago
Awesome Datahoarding ⭐ 892
List of data-hoarding related tools
most recent commit 7 months ago
Spidr ⭐ 775
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
dependent packages 16total releases 26most recent commit 3 months agorubygems spidr} Downloads
Xidel ⭐ 611
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
most recent commit 5 months ago
Dirble ⭐ 592
Fast directory scanning and scraping tool
most recent commit 4 months ago
Websurfx ⭐ 504
🚀 An open source alternative to searx which provides a modern-looking ✨, lightning-fast ⚡, privacy respecting 🥸, secure 🔒 meta search engine
most recent commit 3 months ago
Comic Dl ⭐ 498
Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.
total releases 10most recent commit 8 months ago
Pywebcopy ⭐ 455
Locally saves webpages to your hard disk with images, css, js & links as is.
dependent packages 2total releases 21most recent commit 4 months agopypi pywebcopy} Downloads
Ultimate Web Scraper ⭐ 400
A PHP library/toolkit designed to handle all of your web scraping needs under a MIT or LGPL license. Also has web server and WebSocket server classes for building custom servers.
most recent commit a year ago
Dataflowkit ⭐ 394
Extract structured data from web sites. Web sites scraping.
most recent commit 4 years ago
Advanced Web Scraping Tutorial ⭐ 370
The Zipru scraper developed in the Advanced Web Scraping Tutorial.
most recent commit 7 years ago
The 10 Most Depended On Web Scraper Open Source Projects
Mechanize ⭐ 4,338
Mechanize is a ruby library that makes automated web interaction easy.
dependent packages 978total releases 86latest release April 17, 2023most recent commit 3 months agorubygems mechanize} Downloads
Rod ⭐ 4,505
A Devtools driver for web automation and scraping
dependent packages 140total releases 406latest release November 06, 2023most recent commit 3 months ago
Trafilatura ⭐ 2,447
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
dependent packages 66total releases 39latest release November 29, 2023most recent commit 3 months agopypi trafilatura} Downloads
Web Scraper ⭐ 97
Perl web scraping toolkit
dependent packages 52total releases 38latest release October 20, 2014most recent commit 7 years ago
Panther ⭐ 2,849
A browser testing and web crawling library for PHP and Symfony
dependent packages 50total releases 23latest release December 03, 2023most recent commit 3 months agopackagist symfony/panther} Downloads
Wombat ⭐ 1,297
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
dependent packages 5total releases 33latest release September 03, 2019most recent commit 3 months agorubygems wombat} Downloads
Web Poet ⭐ 85
Web scraping Page Objects core library
dependent packages 4total releases 23latest release November 21, 2023most recent commit 3 months agopypi web-poet} Downloads
Web Master ⭐ 5
Web mastering tools for my personal services
dependent packages 4total releases 17latest release November 01, 2019most recent commit 4 years agonpm @web-master/node-web-scraper} Downloads
Page Previewer ⭐ 23
Node web page preview
dependent packages 4total releases 7latest release February 05, 2014most recent commit 7 years agonpm page-previewer} Downloads
The 10 Latest Releases In Web Scraper Open Source Projects
Pagemunch Php ⭐ 6
A PHP library for the PageMunch web crawler API
latest release December 10, 2023most recent commit 7 years ago
Seer ⭐ 7
A PHP XPath based web scraping framework.
latest release December 07, 2023most recent commit 9 years ago
Panther ⭐ 2,849
A browser testing and web crawling library for PHP and Symfony
dependent packages 50total releases 23latest release December 03, 2023most recent commit 3 months agopackagist symfony/panther} Downloads
Trafilatura ⭐ 2,447
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
dependent packages 66total releases 39latest release November 29, 2023most recent commit 3 months agopypi trafilatura} Downloads
Web Poet ⭐ 85
Web scraping Page Objects core library
dependent packages 4total releases 23latest release November 21, 2023most recent commit 3 months agopypi web-poet} Downloads
Rod ⭐ 4,505
A Devtools driver for web automation and scraping
dependent packages 140total releases 406latest release November 06, 2023most recent commit 3 months ago
Scr Web ⭐ 6
Social media web scraper web browser GUI for post file downloads.
total releases 1latest release August 14, 2023most recent commit 2 years ago
Polite ⭐ 310
Be nice on the web
dependent packages 3total releases 3latest release June 30, 2023most recent commit 8 months ago
Ayakashi ⭐ 177
:zap: Ayakashi.io - The next generation web scraping framework
dependent packages 2total releases 40latest release June 29, 2023most recent commit 10 months agonpm ayakashi} Downloads
Autopahe ⭐ 5
Automatically download your all favorite anime from AnimePahe
total releases 11latest release June 19, 2023most recent commit 10 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.