Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 14 Crawler Web Archiving Open Source Projects

Open source projects categorized as Crawler Web Archiving
Categories > Data Processing > Crawler
Categories > Content Management > Web Archiving
Awesome Web Archiving ⭐ 1,669
An Awesome List for getting started with web archiving
most recent commit 3 months ago
Browsertrix Crawler ⭐ 470
Run a high-fidelity browser-based crawler in a single Docker container
most recent commit 3 months ago
Warcdb ⭐ 380
WarcDB: Web crawl data as SQLite databases.
total releases 4most recent commit 4 months ago
Warcreate ⭐ 187
Chrome extension to "Create WARC files from any webpage"
most recent commit 5 months ago
Squidwarc ⭐ 163
Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
most recent commit 4 years ago
Warc Parquet ⭐ 96
🗄️ A simple CLI for converting WARC to Parquet.
total releases 10most recent commit 3 months ago
Wget Lua ⭐ 72
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
most recent commit 4 months ago
Warcworker ⭐ 33
A dockerized, queued high fidelity web archiver based on Squidwarc
most recent commit 4 years ago
Httrack2warc ⭐ 20
Converts HTTrack crawls to WARC files
most recent commit 2 years ago
Sandcrawler ⭐ 19
Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki
most recent commit a year ago
Ukwa Manage ⭐ 10
Shepherding our web archives from crawl to access.
most recent commit 6 months ago
Webarchiver ⭐ 9
Decentralized web archiving
most recent commit 6 years ago
Chronicrawl ⭐ 8
Experimental continouous web crawler for web archiving
most recent commit a year ago
Bamboo ⭐ 8
Web archive collection manager
most recent commit 5 months ago
The 2 Latest Releases In Crawler Web Archiving Open Source Projects
Warcdb ⭐ 380
WarcDB: Web crawl data as SQLite databases.
total releases 4latest release October 22, 2023most recent commit 4 months ago
Warc Parquet ⭐ 96
🗄️ A simple CLI for converting WARC to Parquet.
total releases 10latest release September 13, 2023most recent commit 3 months ago
Popular Data Processing Categories
Jupyter Notebook
Dataset
Sql
Validation
Pipeline
Translation
Data Science
Classification
Transaction
Scraper
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.