Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 9 Crawler Warc Web Archiving Open Source Projects

Open source projects categorized as Crawler Warc Web Archiving
Categories > Data Processing > Crawler
Categories > Warc
Categories > Content Management > Web Archiving
Awesome Web Archiving ⭐ 1,669
An Awesome List for getting started with web archiving
most recent commit 3 months ago
Browsertrix Crawler ⭐ 470
Run a high-fidelity browser-based crawler in a single Docker container
most recent commit 3 months ago
Warcdb ⭐ 380
WarcDB: Web crawl data as SQLite databases.
total releases 4most recent commit 5 months ago
Warcreate ⭐ 187
Chrome extension to "Create WARC files from any webpage"
most recent commit 5 months ago
Warc Parquet ⭐ 96
🗄️ A simple CLI for converting WARC to Parquet.
total releases 10most recent commit 3 months ago
Wget Lua ⭐ 72
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
most recent commit 4 months ago
Httrack2warc ⭐ 20
Converts HTTrack crawls to WARC files
most recent commit 2 years ago
Ukwa Manage ⭐ 10
Shepherding our web archives from crawl to access.
most recent commit 6 months ago
Webarchiver ⭐ 9
Decentralized web archiving
most recent commit 6 years ago
The 2 Latest Releases In Crawler Warc Web Archiving Open Source Projects
Warcdb ⭐ 380
WarcDB: Web crawl data as SQLite databases.
total releases 4latest release October 22, 2023most recent commit 5 months ago
Warc Parquet ⭐ 96
🗄️ A simple CLI for converting WARC to Parquet.
total releases 10latest release September 13, 2023most recent commit 3 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.