Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Scraper Crawler Web Crawler Open Source Projects

Open source projects categorized as Scraper Crawler Web Crawler
Categories > Data Processing > Crawler
Categories > Data Processing > Scraper
Categories > Data Processing > Web Crawler
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96most recent commit 3 months agopypi Scrapy} Downloads
Crawlee ⭐ 12,158
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
dependent packages 42total releases 747most recent commit 2 days agonpm crawlee} Downloads
Awesome Web Scraping ⭐ 6,060
List of libraries, tools and APIs for web scraping and data processing.
most recent commit 5 months ago
Awesome Crawler ⭐ 5,859
A collection of awesome web crawler,spider in different languages
most recent commit 5 months ago
Autoscraper ⭐ 5,159
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
dependent packages 1total releases 16most recent commit a year agopypi autoscraper} Downloads
Douyin_tiktok_download_api ⭐ 4,844
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、T
total releases 21most recent commit 5 months ago
Rod ⭐ 4,505
A Devtools driver for web automation and scraping
dependent packages 140total releases 406most recent commit 3 months ago
Browser Fingerprinting ⭐ 3,353
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?
most recent commit a year ago
Trafilatura ⭐ 2,447
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
dependent packages 66total releases 39most recent commit 3 months agopypi trafilatura} Downloads
Grab ⭐ 2,292
Web Scraping Framework
dependent packages 3total releases 120most recent commit 10 months agopypi grab} Downloads
Spidr ⭐ 775
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
dependent packages 16total releases 26most recent commit 3 months agorubygems spidr} Downloads
Till ⭐ 770
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
total releases 37most recent commit 2 years ago
Scrapple ⭐ 452
A framework for creating semi-automatic web content extractors
total releases 10most recent commit 5 years ago
Pulsarrpa ⭐ 413
Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.
dependent packages 3total releases 58most recent commit 4 months ago
Dude ⭐ 397
dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators
total releases 43most recent commit 3 months ago
Hquery.php ⭐ 345
An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.
dependent packages 4total releases 23most recent commit 3 months agonpm hquery.php} Downloads
Polite ⭐ 310
Be nice on the web
dependent packages 3total releases 3most recent commit 8 months ago
Crawler ⭐ 285
Library for Rapid (Web) Crawler and Scraper Development
total releases 25most recent commit 4 months ago
Web Scraping ⭐ 281
Más de 50 ejemplos de web scraping utilizando: Requests | Scrapy | Selenium | LXML | BeautifulSoup
most recent commit 3 months ago
Gopa ⭐ 281
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
total releases 6most recent commit 3 years ago
Youtube Projects ⭐ 272
This repository contains all the code I use in my YouTube tutorials.
most recent commit 3 years ago
Ant ⭐ 271
A web crawler for Go
total releases 8most recent commit a year ago
Rcrawler ⭐ 240
An R web crawler and scraper
total releases 9most recent commit 4 years ago
The 10 Most Depended On Scraper Crawler Web Crawler Open Source Projects
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96latest release September 18, 2023most recent commit 3 months agopypi Scrapy} Downloads
Rod ⭐ 4,505
A Devtools driver for web automation and scraping
dependent packages 140total releases 406latest release November 06, 2023most recent commit 3 months ago
Trafilatura ⭐ 2,447
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
dependent packages 66total releases 39latest release November 29, 2023most recent commit 3 months agopypi trafilatura} Downloads
Robotstxt ⭐ 68
robots.txt file parsing and checking for R
dependent packages 7total releases 10latest release July 25, 2020most recent commit a year ago
Hquery.php ⭐ 345
An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.
dependent packages 4total releases 23latest release July 19, 2019most recent commit 3 months agonpm hquery.php} Downloads
Polite ⭐ 310
Be nice on the web
dependent packages 3total releases 3latest release June 30, 2023most recent commit 8 months ago
Pulsarrpa ⭐ 413
Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.
dependent packages 3total releases 58latest release November 11, 2023most recent commit 4 months ago
Ronin Web ⭐ 40
ronin-web is a collection of useful web helper methods and commands.
dependent packages 3total releases 16latest release April 04, 2023most recent commit 4 months agorubygems ronin-web} Downloads
The 10 Latest Releases In Scraper Crawler Web Crawler Open Source Projects
Pagemunch Php ⭐ 6
A PHP library for the PageMunch web crawler API
latest release December 10, 2023most recent commit 7 years ago
Google News Scraper ⭐ 144
Lightweight scraper for Google News
dependent packages 2total releases 12latest release December 06, 2023most recent commit 4 months agonpm google-news-scraper} Downloads
Crawler ⭐ 285
Library for Rapid (Web) Crawler and Scraper Development
total releases 25latest release December 01, 2023most recent commit 4 months ago
Trafilatura ⭐ 2,447
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
dependent packages 66total releases 39latest release November 29, 2023most recent commit 3 months agopypi trafilatura} Downloads
Pulsarrpa ⭐ 413
Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.
dependent packages 3total releases 58latest release November 11, 2023most recent commit 4 months ago
Rod ⭐ 4,505
A Devtools driver for web automation and scraping
dependent packages 140total releases 406latest release November 06, 2023most recent commit 3 months ago
Python Scrapfly ⭐ 14
Scrapfly Python SDK for headless browsers and proxy rotation
total releases 33latest release October 27, 2023most recent commit 4 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.