Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Web Crawler Webscraper Open Source Projects

Open source projects categorized as Web Crawler Webscraper
Categories > Data Processing > Web Crawler
Categories > Webscraper
Crawlee ⭐ 11,957
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
dependent packages 42total releases 747most recent commit 4 days agonpm crawlee} Downloads
Crawlab ⭐ 10,521
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
total releases 1most recent commit 3 months ago
Spider Flow ⭐ 8,075
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
most recent commit 10 months ago
Awesome Crawler ⭐ 5,859
A collection of awesome web crawler,spider in different languages
most recent commit 4 months ago
Autoscraper ⭐ 5,159
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
dependent packages 1total releases 16most recent commit a year agopypi autoscraper} Downloads
Hakrawler ⭐ 4,120
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
total releases 11most recent commit 3 months ago
Node Osmosis ⭐ 4,083
Web scraper for NodeJS
dependent packages 58total releases 27most recent commit 9 months agonpm osmosis} Downloads
Php Curl Class ⭐ 3,208
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
dependent packages 362total releases 99most recent commit 3 months agopackagist php-curl-class/php-curl-class} Downloads
Webcollector ⭐ 2,974
WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
dependent packages 1total releases 22most recent commit 10 months ago
Nutch ⭐ 2,742
Apache Nutch is an extensible and scalable web crawler
dependent packages 1total releases 26most recent commit 3 months ago
Gecco ⭐ 2,403
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
dependent packages 4total releases 28most recent commit a year ago
Soup ⭐ 2,074
Web Scraper in Go, similar to BeautifulSoup
dependent packages 57total releases 10most recent commit 5 months ago
Abot ⭐ 1,991
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
dependent packages 8total releases 153most recent commit 2 years agonuget Abot} Downloads
Pspider ⭐ 1,675
简单易用的Python爬虫框架，QQ交流群：597510560
most recent commit 2 years ago
Tomorrow ⭐ 1,463
Magic decorator syntax for asynchronous code in Python
dependent packages 2total releases 5most recent commit 4 years agopypi tomorrow} Downloads
100projectsofcode ⭐ 1,293
A list of practical knowledge-building projects.
most recent commit 3 years ago
Lightnovel Crawler ⭐ 1,185
Generate and download e-books from online sources.
dependent packages 1total releases 30most recent commit 3 months agopypi ebook-crawler} Downloads
Faster Than Requests ⭐ 1,061
Faster requests on Python 3
dependent packages 4total releases 18most recent commit 3 months agopypi faster-than-requests} Downloads
Stealth ⭐ 923
🚀 Stealth - Secure, Peer-to-Peer, Private and Automateable Web Browser/Scraper/Proxy
most recent commit 7 months ago
Storm Crawler ⭐ 834
A scalable, mature and versatile web crawler based on Apache Storm
dependent packages 10total releases 36most recent commit 3 months ago
Spidr ⭐ 775
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
dependent packages 16total releases 26most recent commit 3 months agorubygems spidr} Downloads
Fetchbot ⭐ 758
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
dependent packages 3total releases 7most recent commit 3 years ago
Marginaliasearch ⭐ 711
Internet search engine for text-oriented websites. Indexing the small, old and weird web.
most recent commit 3 months ago
The 10 Most Depended On Web Crawler Webscraper Open Source Projects
Php Curl Class ⭐ 3,208
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
dependent packages 362total releases 99latest release September 11, 2023most recent commit 3 months agopackagist php-curl-class/php-curl-class} Downloads
Node Osmosis ⭐ 4,083
Web scraper for NodeJS
dependent packages 58total releases 27latest release March 01, 2019most recent commit 9 months agonpm osmosis} Downloads
Soup ⭐ 2,074
Web Scraper in Go, similar to BeautifulSoup
dependent packages 57total releases 10latest release January 16, 2022most recent commit 5 months ago
Html Metadata ⭐ 115
MetaData html scraper and parser for Node.js (supports Promises and callback style)
dependent packages 26total releases 20latest release November 20, 2019most recent commit 4 years agonpm html-metadata} Downloads
X Ray Crawler ⭐ 39
Friendly web crawler for x-ray
dependent packages 19total releases 7latest release August 26, 2020most recent commit 3 years agonpm x-ray-crawler} Downloads
Storm Crawler ⭐ 834
A scalable, mature and versatile web crawler based on Apache Storm
dependent packages 10total releases 36latest release October 25, 2023most recent commit 3 months ago
Abot ⭐ 1,991
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
dependent packages 8total releases 153latest release August 09, 2021most recent commit 2 years agonuget Abot} Downloads
Crawler Commons ⭐ 217
A set of reusable Java components that implement functionality common to any web crawler
dependent packages 7total releases 10latest release July 13, 2023most recent commit 5 months ago
The 10 Latest Releases In Web Crawler Webscraper Open Source Projects
Pagemunch Php ⭐ 6
A PHP library for the PageMunch web crawler API
latest release December 10, 2023most recent commit 7 years ago
Crawley ⭐ 208
The unix-way web crawler
total releases 60latest release December 07, 2023most recent commit 4 months ago
Google News Scraper ⭐ 144
Lightweight scraper for Google News
dependent packages 2total releases 12latest release December 06, 2023most recent commit 4 months agonpm google-news-scraper} Downloads
Crawler ⭐ 285
Library for Rapid (Web) Crawler and Scraper Development
total releases 25latest release December 01, 2023most recent commit 3 months ago
Dmr.py ⭐ 11
Pull data from the danish vehicle registry with dmr.py
total releases 13latest release November 30, 2023most recent commit 5 months ago
Kenpompy ⭐ 61
A simple yet comprehensive web scraper for kenpom.com.
total releases 9latest release November 22, 2023most recent commit 4 months ago
Basketball_reference_web_scraper ⭐ 382
NBA Stats API via Basketball Reference
total releases 39latest release November 14, 2023most recent commit 3 months ago
Crawler ⭐ 13
Web crawler based on Puppeteer
dependent packages 4total releases 23latest release November 13, 2023most recent commit 5 months agonpm @opd/crawler} Downloads
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.