Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Web Crawler Open Source Projects

Open source projects categorized as Web Crawler
Categories > Data Processing > Web Crawler
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96most recent commit 3 months agopypi Scrapy} Downloads
Huginn ⭐ 41,465
Create agents that monitor and act on your behalf. Your agents are standing by!
dependent packages 52total releases 8most recent commit 15 days agorubygems huginn_agent} Downloads
Changedetection.io ⭐ 13,943
The best and simplest free open source website change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification
most recent commit 3 months ago
Crawlee ⭐ 12,106
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
dependent packages 42total releases 747most recent commit 17 hours agonpm crawlee} Downloads
Crawlab ⭐ 10,521
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
total releases 1most recent commit 3 months ago
Spider Flow ⭐ 8,075
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
most recent commit 10 months ago
Katana ⭐ 7,995
A next-generation crawling and spidering framework.
dependent packages 1total releases 8most recent commit 3 months ago
Awesome Web Scraping ⭐ 6,060
List of libraries, tools and APIs for web scraping and data processing.
most recent commit 5 months ago
Awesome Crawler ⭐ 5,859
A collection of awesome web crawler,spider in different languages
most recent commit 5 months ago
Ani Cli ⭐ 5,724
A cli tool to browse and play anime
most recent commit 3 months ago
Autoscraper ⭐ 5,159
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
dependent packages 1total releases 16most recent commit a year agopypi autoscraper} Downloads
Douyin_tiktok_download_api ⭐ 4,844
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、T
total releases 21most recent commit 5 months ago
Rod ⭐ 4,505
A Devtools driver for web automation and scraping
dependent packages 140total releases 406most recent commit 3 months ago
Hakrawler ⭐ 4,120
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
total releases 11most recent commit 3 months ago
Node Osmosis ⭐ 4,083
Web scraper for NodeJS
dependent packages 58total releases 27most recent commit 9 months agonpm osmosis} Downloads
Helium ⭐ 3,975
Lighter web automation for Python
dependent packages 14total releases 38most recent commit 2 months agopypi helium} Downloads
Browser Fingerprinting ⭐ 3,353
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?
most recent commit a year ago
Php Curl Class ⭐ 3,208
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
dependent packages 362total releases 99most recent commit 3 months agopackagist php-curl-class/php-curl-class} Downloads
Automatic Udemy Course Enroller Get Paid Udemy Courses For Free ⭐ 3,010
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
total releases 11most recent commit 3 months ago
Webcollector ⭐ 2,974
WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
dependent packages 1total releases 22most recent commit a year ago
Nutch ⭐ 2,742
Apache Nutch is an extensible and scalable web crawler
dependent packages 1total releases 26most recent commit 3 months ago
Snoop ⭐ 2,530
Snoop — инструмент разведки на основе открытых данных (OSINT world)
most recent commit 3 months ago
Trafilatura ⭐ 2,447
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
dependent packages 66total releases 39most recent commit 3 months agopypi trafilatura} Downloads
The 10 Most Depended On Web Crawler Open Source Projects
Requests Cache ⭐ 1,244
Persistent HTTP cache for python requests
dependent packages 466total releases 90latest release November 19, 2023most recent commit a month agopypi requests-cache} Downloads
Scrapy ⭐ 49,918
Scrapy, a fast high-level web crawling & scraping framework for Python.
dependent packages 445total releases 96latest release September 18, 2023most recent commit 3 months agopypi Scrapy} Downloads
Php Curl Class ⭐ 3,208
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
dependent packages 362total releases 99latest release September 11, 2023most recent commit 3 months agopackagist php-curl-class/php-curl-class} Downloads
Rvest ⭐ 1,434
Simple web scraping for R
dependent packages 320total releases 13latest release August 19, 2022most recent commit 3 months ago
Rod ⭐ 4,505
A Devtools driver for web automation and scraping
dependent packages 140total releases 406latest release November 06, 2023most recent commit 3 months ago
User Agents ⭐ 871
A JavaScript library for generating random user agents with data that's updated daily.
dependent packages 104total releases 1,550latest release December 10, 2023most recent commit 3 months agonpm user-agents} Downloads
Google Search Results Python ⭐ 432
Google Search Results via SERP API pip Python Package
dependent packages 99total releases 41latest release March 10, 2023most recent commit 7 months agopypi google-search-results} Downloads
Trafilatura ⭐ 2,447
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
dependent packages 66total releases 39latest release November 29, 2023most recent commit 3 months agopypi trafilatura} Downloads
Node Osmosis ⭐ 4,083
Web scraper for NodeJS
dependent packages 58total releases 27latest release March 01, 2019most recent commit 9 months agonpm osmosis} Downloads
Soup ⭐ 2,074
Web Scraper in Go, similar to BeautifulSoup
dependent packages 57total releases 10latest release January 16, 2022most recent commit 6 months ago
The 10 Latest Releases In Web Crawler Open Source Projects
User Agents ⭐ 871
A JavaScript library for generating random user agents with data that's updated daily.
dependent packages 104total releases 1,550latest release December 10, 2023most recent commit 3 months agonpm user-agents} Downloads
Pagemunch Php ⭐ 6
A PHP library for the PageMunch web crawler API
latest release December 10, 2023most recent commit 7 years ago
Helium ⭐ 3,975
Lighter web automation for Python
dependent packages 14total releases 38latest release December 08, 2023most recent commit 2 months agopypi helium} Downloads
Crawley ⭐ 208
The unix-way web crawler
total releases 60latest release December 07, 2023most recent commit 5 months ago
Google News Scraper ⭐ 144
Lightweight scraper for Google News
dependent packages 2total releases 12latest release December 06, 2023most recent commit 4 months agonpm google-news-scraper} Downloads
Crawler ⭐ 285
Library for Rapid (Web) Crawler and Scraper Development
total releases 25latest release December 01, 2023most recent commit 3 months ago
Dmr.py ⭐ 11
Pull data from the danish vehicle registry with dmr.py
total releases 13latest release November 30, 2023most recent commit 5 months ago
Trafilatura ⭐ 2,447
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
dependent packages 66total releases 39latest release November 29, 2023most recent commit 3 months agopypi trafilatura} Downloads
Goskyr ⭐ 27
A configurable command-line web scraper written in go with auto configuration capability
total releases 66latest release November 28, 2023most recent commit 3 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.