Awesome Open Source

Programming Languages

The Top 23 Scraper Web Crawler Open Source Projects

Open source projects categorized as Scraper Web Crawler

Categories > Data Processing > Scraper

Categories > Data Processing > Web Crawler

Scrapy ⭐ 49,918

Scrapy, a fast high-level web crawling & scraping framework for Python.

dependent packages 445total releases 96most recent commit 3 months ago

pypi Scrapy} Downloads

Huginn ⭐ 41,465

Create agents that monitor and act on your behalf. Your agents are standing by!

dependent packages 52total releases 8most recent commit 11 days ago

rubygems huginn_agent} Downloads

Crawlee ⭐ 12,059

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747most recent commit 2 days ago

npm crawlee} Downloads

Awesome Web Scraping ⭐ 6,060

List of libraries, tools and APIs for web scraping and data processing.

most recent commit 5 months ago

Awesome Crawler ⭐ 5,859

A collection of awesome web crawler,spider in different languages

most recent commit 4 months ago

Autoscraper ⭐ 5,159

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

dependent packages 1total releases 16most recent commit a year ago

pypi autoscraper} Downloads

Douyin_tiktok_download_api ⭐ 4,844

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、T

total releases 21most recent commit 5 months ago

A Devtools driver for web automation and scraping

dependent packages 140total releases 406most recent commit 3 months ago

Node Osmosis ⭐ 4,083

Web scraper for NodeJS

dependent packages 58total releases 27most recent commit 9 months ago

npm osmosis} Downloads

Browser Fingerprinting ⭐ 3,353

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

most recent commit a year ago

Automatic Udemy Course Enroller Get Paid Udemy Courses For Free ⭐ 3,010

Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!

total releases 11most recent commit 3 months ago

Snoop ⭐ 2,530

Snoop — инструмент разведки на основе открытых данных (OSINT world)

most recent commit 3 months ago

Trafilatura ⭐ 2,447

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

dependent packages 66total releases 39most recent commit 3 months ago

pypi trafilatura} Downloads

Web Scraping Framework

dependent packages 3total releases 120most recent commit 10 months ago

pypi grab} Downloads

Getting started with Puppeteer and Chrome Headless for Web Scraping

most recent commit 3 years ago

Web Scraper in Go, similar to BeautifulSoup

dependent packages 57total releases 10most recent commit 6 months ago

Tomorrow ⭐ 1,463

Magic decorator syntax for asynchronous code in Python

dependent packages 2total releases 5most recent commit 4 years ago

pypi tomorrow} Downloads

Rvest ⭐ 1,434

Simple web scraping for R

dependent packages 320total releases 13most recent commit 3 months ago

How To Prevent Scraping ⭐ 1,417

The ultimate guide on preventing Website Scraping

most recent commit 4 months ago

Django Dynamic Scraper ⭐ 1,069

Creating Scrapy scrapers via the Django admin interface

total releases 65most recent commit 2 years ago

Stealth ⭐ 923

🚀 Stealth - Secure, Peer-to-Peer, Private and Automateable Web Browser/Scraper/Proxy

most recent commit 7 months ago

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

dependent packages 16total releases 26most recent commit 3 months ago

rubygems spidr} Downloads

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

total releases 37most recent commit 2 years ago

The 10 Most Depended On Scraper Web Crawler Open Source Projects

Scrapy ⭐ 49,918

Scrapy, a fast high-level web crawling & scraping framework for Python.

dependent packages 445total releases 96latest release September 18, 2023most recent commit 3 months ago

pypi Scrapy} Downloads

Rvest ⭐ 1,434

Simple web scraping for R

dependent packages 320total releases 13latest release August 19, 2022most recent commit 3 months ago

A Devtools driver for web automation and scraping

dependent packages 140total releases 406latest release November 06, 2023most recent commit 3 months ago

Google Search Results Python ⭐ 432

Google Search Results via SERP API pip Python Package

dependent packages 99total releases 41latest release March 10, 2023most recent commit 7 months ago

pypi google-search-results} Downloads

Trafilatura ⭐ 2,447

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

dependent packages 66total releases 39latest release November 29, 2023most recent commit 3 months ago

pypi trafilatura} Downloads

Node Osmosis ⭐ 4,083

Web scraper for NodeJS

dependent packages 58total releases 27latest release March 01, 2019most recent commit 9 months ago

npm osmosis} Downloads

Web Scraper in Go, similar to BeautifulSoup

dependent packages 57total releases 10latest release January 16, 2022most recent commit 6 months ago

Huginn ⭐ 41,465

Create agents that monitor and act on your behalf. Your agents are standing by!

dependent packages 52total releases 8latest release September 22, 2017most recent commit 11 days ago

rubygems huginn_agent} Downloads

Crawlee ⭐ 12,059

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747latest release December 10, 2023most recent commit 2 days ago

npm crawlee} Downloads

Webmiddle ⭐ 13

Node.js framework for modular web scraping and data extraction

dependent packages 27total releases 9latest release November 06, 2018most recent commit 4 years ago

npm webmiddle} Downloads

The 10 Latest Releases In Scraper Web Crawler Open Source Projects

Crawlee ⭐ 12,059

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

dependent packages 42total releases 747latest release December 10, 2023most recent commit 2 days ago

npm crawlee} Downloads

Pagemunch Php ⭐ 6

A PHP library for the PageMunch web crawler API

latest release December 10, 2023most recent commit 7 years ago

Google News Scraper ⭐ 144

Lightweight scraper for Google News

dependent packages 2total releases 12latest release December 06, 2023most recent commit 4 months ago

npm google-news-scraper} Downloads

Crawler ⭐ 285

Library for Rapid (Web) Crawler and Scraper Development

total releases 25latest release December 01, 2023most recent commit 3 months ago

Trafilatura ⭐ 2,447

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

dependent packages 66total releases 39latest release November 29, 2023most recent commit 3 months ago

pypi trafilatura} Downloads

A configurable command-line web scraper written in go with auto configuration capability

total releases 66latest release November 28, 2023most recent commit 3 months ago

Mov Cli ⭐ 314

A cli tool to browse and watch Movies/Shows/TV/Sports.

dependent packages 1total releases 103latest release November 25, 2023most recent commit 3 months ago

pypi mov-cli} Downloads

Kenpompy ⭐ 61

A simple yet comprehensive web scraper for kenpom.com.

total releases 9latest release November 22, 2023most recent commit 4 months ago

open source library from digipodium

total releases 29latest release November 22, 2023most recent commit 5 months ago

Web Poet ⭐ 85

Web scraping Page Objects core library

dependent packages 4total releases 23latest release November 21, 2023most recent commit 3 months ago

pypi web-poet} Downloads

Popular Data Processing Categories

Jupyter Notebook

Categories

Application Programming Interfaces

Artificial Intelligence

Cloud Computing

Command Line Interface

Computer Science

Configuration Management

Content Management

Data Processing

Integrated Development Environments

Learning Resources

Lists Of Projects

Machine Learning

Operating Systems

Package Managers

Programming Languages

Runtime Environments

Software Architecture

Software Development

Software Performance

Software Quality

Text Processing

User Interface Components

Version Control

Web User Interface

Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source. All rights reserved.