Awesome Open Source
Awesome Open Source
Combined Topics
web-archiving x
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194

The Top 74 Web Archiving Open Source Projects on Github

Categories > Content Management > Web Archiving
Archivebox ⭐ 11,740
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
Conifer ⭐ 1,299
Collect and revisit web pages.
Awesome Web Archiving ⭐ 958
An Awesome List for getting started with web archiving
Pywb ⭐ 866
Core Python Web Archiving Toolkit for replay and recording of web archives
Webrecorder Player ⭐ 381
Webrecorder Player for Desktop (OSX/Windows/Linux). (Built with Electron + Webrecorder)
Ipwb ⭐ 376
InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS
Perma ⭐ 310
Indelible links
Archivenow ⭐ 250
A Tool To Push Web Resources Into Web Archives
Wail ⭐ 249
🐋 Web Archiving Integration Layer: One-Click User Instigated Preservation
Archiveror ⭐ 188
Archiveror will help you preserve the webpages you love. 💾
Warcio ⭐ 173
Streaming WARC/ARC library for fast web archive IO
Warcreate ⭐ 145
Chrome extension to "Create WARC files from any webpage"
Replayweb.page ⭐ 139
Serverless Web Archive Replay directly in the browser
Sfm Ui ⭐ 132
Social Feed Manager user interface application.
Squidwarc ⭐ 131
Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
Archiveweb.page ⭐ 118
A High-Fidelity Web Archiving Extension for Chrome and Chromium based browsers!
Archivespark ⭐ 113
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
Wail ⭐ 97
🐋 One-Click User Instigated Preservation
Fatcat ⭐ 79
Perpetual Access To The Scholarly Record
Cdx_toolkit ⭐ 71
A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
Node Warc ⭐ 62
Parse And Create Web ARChive (WARC) files with node.js
Warrick ⭐ 59
Recover lost websites from the Web Infrastructure
Collect ⭐ 54
A server to collect & archive websites, also supports video downloads
Electron Archivebox ⭐ 46
Desktop Electron app for ArchiveBox internet archiver. (ALPHA: not ready for general use)
Awesome Memento ⭐ 42
A list of things related to software, literature, and other content for 🕣 Memento
Wget Lua ⭐ 40
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
Memgator ⭐ 39
A Memento Aggregator CLI and Server in Go
Warcworker ⭐ 33
A dockerized, queued high fidelity web archiver based on Squidwarc
Outbackcdx ⭐ 20
Web archive index server based on RocksDB
Cdxj Indexer ⭐ 16
CDXJ Indexing of WARC/ARCs
Munin Indexer ⭐ 13
A social media open post web archiving tool
Conifer Deploy ⭐ 11
Conifer setup and deployment via Ansible
Seeder ⭐ 11
Seeder - Czech webarchive curating tool and public site
Httrack2warc ⭐ 11
Converts HTTrack crawls to WARC files
Dat Share ⭐ 10
A prototype server to swarm multiple DATs for Webrecorder
Sandcrawler ⭐ 10
Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki
Ukwa Manage ⭐ 9
Shepherding our web archives from crawl to access.
Webarchiver ⭐ 9
Decentralized web archiving
Cc Notebooks ⭐ 8
Various Jupyter notebooks about Common Crawl data
Internet Archiving Talk ⭐ 8
🎭 An introduction to the Internet Archiving ecosystem, tooling, and some of the ethical dilemmas that the community faces.
Homebrew Archivebox ⭐ 8
Homebrew formula for the ArchiveBox self-hosted internet archiving solution.
Chropro ⭐ 7
Chrome debugging protocol client for Java
Httpreserve ⭐ 7
Digital Preservation of HTTP in documentary heritage.
Hadoopconcatgz ⭐ 7
A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz
Bamboo ⭐ 6
Web archive collection manager
Pywb Recorder Tor ⭐ 6
pywb recorder over tor, anonymously records the web. (docker image)
Chronicrawl ⭐ 6
Experimental continouous web crawler for web archiving
Webrecorder Chrome Extension ⭐ 4
record current active tab on webrecorder.io
Tikalinkextract ⭐ 4
Tika based link extractor for httpreserve
Archiveboxmatic ⭐ 4
ArchiveBoxMatic: configure ArchiveBox with the simplicity of a yaml file.
Linkstat ⭐ 3
CLI implementation of httpreserve that can test links and retrieve internet archive replacements
Linkscanner ⭐ 3
A helper package to tokenize textual content and retrieve hyperlinks
Docs ⭐ 3
Source for the Github Wiki / ReadTheDocs documentation for AchiveBox, the self-hosted internet archiving solution.
Quickcacheandarchivesearch ⭐ 3
Quick Cache and Archive search buttons
Tmvis ⭐ 2
An archival thumbnail visualization server
Moonshine ⭐ 2
Given four bytes, download a random file from web archives implementing the UKWA Shine interface
Workbench ⭐ 2
Client app for httpreserve pkg that generates CSV, JSON, HTTP, and BoltDB
Scrapy Warcio ⭐ 2
Support for writing WARC files with Scrapy
Debian Archivebox ⭐ 2
Home of the official apt/deb package for Ubuntu/Debian-based systems.
Metawarc ⭐ 2
metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)
Pip Archivebox ⭐ 2
Official Python package for ArchiveBox, the self-hosted internet archiving solution.
Phpbbcrawler ⭐ 1
Link crawler for a phpBB forum
Waybacks ⭐ 1
This module builds our Waybacks in the various different configurations we require.
Eprints2archives ⭐ 1
Send records from an EPrints server to the Internet Archive and other web archives
Mdbubing ⭐ 1
From WARC records to MongoDB documents
Memgatorbulkdownload ⭐ 1
Whatdowebpagesdo ⭐ 1
Webarchiver ⭐ 1
A archiving utility with an interface for web servers.
Capture Urls ⭐ 1
Archive a list of URLs using the Wayback Machine
Phantomjsscreenshot ⭐ 1
A wrapper for phantom.js commands for headless screenshots.
Million Dollar Webpage ⭐ 0
HTTPreserve Analysis of Million Dollar Web Page
Warcpartitioner ⭐ 0
Partition (W)ARC Files by MIME Type and Year
Node Cdxj ⭐ 0
Parse CDXJ(https://github.com/oduwsdl/ORS/wiki/CDXJ) files with node.js
Wayback ⭐ 0
A restrictied API in Golang for the (semi)-exposed functions of the internet archive.
1-74 of 74 projects

Related Projects

Python Web Archiving Projects (24)
Web Archiving Warc Projects (22)
Web Archiving Webarchive Projects (18)
Javascript Web Archiving Projects (16)
Python Web Archiving Warc Projects (12)
Archive Web Archiving Projects (11)
Web Archiving Webrecorder Projects (10)
Web Archiving Warc Webarchive Projects (10)
Web Archiving Digipres Projects (10)
Code4lib Web Archiving Projects (9)
Web Archiving Archivebox Projects (9)
Web Archiving Wayback Projects (8)
Java Web Archiving Projects (8)
Python Web Archiving Webarchive Projects (7)
Web Archiving Memento Projects (7)
Internet Archive Web Archiving Projects (7)
Web Archiving Digipres Archivebox Projects (7)
Archive Code4lib Web Archiving Projects (7)
Code4lib Web Archiving Digital Preservation Projects (7)
Web Archiving Digital Preservation Projects (7)
Python Web Archiving Wayback Projects (6)
Docker Web Archiving Projects (6)
Web Archiving Warc Wayback Projects (6)
Web Archiving Pywb Projects (6)
Go Web Archiving Projects (6)
Archive Code4lib Web Archiving Digital Preservation Projects (6)
Archive Web Archiving Digital Preservation Projects (6)
Web Archiving Memento Rfc Projects (5)
Python Web Archiving Warc Wayback Projects (5)
Python Web Archiving Pywb Projects (5)
Web Archiving Warc Pywb Projects (5)
Wayback Machine Web Archiving Projects (5)
Crawler Web Archiving Projects (5)
Web Archiving High Fidelity Preservation Projects (5)
Python Web Archiving Archivebox Projects (5)
Go Code4lib Web Archiving Projects (5)
Python Docker Web Archiving Projects (4)
Python Web Archiving Wayback Pywb Projects (4)
Web Archiving Wayback Pywb Projects (4)
Web Archiving Warc Webrecorder Projects (4)
Web Archiving Warc Webrecorder Pywb Projects (4)
Web Archiving Webrecorder Pywb Projects (4)
Python Docker Web Archiving Warc Projects (4)
Python Docker Web Archiving Warc Wayback Projects (4)
Python Docker Web Archiving Wayback Projects (4)
Python Web Archiving Warc Pywb Projects (4)
Javascript Web Archiving Warc Projects (4)
Chromium Web Archiving Projects (4)
Crawling Web Archiving Projects (4)
Wget Web Archiving Projects (4)
Go Archive Code4lib Web Archiving Projects (4)
Go Archive Web Archiving Projects (4)
Python Web Archiving Digipres Projects (4)
Go Code4lib Web Archiving Digital Preservation Projects (4)
Go Web Archiving Digital Preservation Projects (4)
Code4lib Internet Archive Web Archiving Projects (4)
Code4lib Internet Archive Web Archiving Digital Preservation Projects (4)
Javascript Chrome Extension Web Archiving Projects (3)
Chrome Extension Web Archiving Projects (3)
Javascript Web Archiving Webarchive Projects (3)
Javascript Web Archiving Webrecorder Projects (3)
Html Web Archiving Projects (3)
Self Hosted Web Archiving Projects (3)
Crawler Crawling Web Archiving Projects (3)
Crawler Web Archiving Warc Projects (3)
Web Archiving Memento Timemap Projects (3)
Web Archiving Timemap Projects (3)
Crawling Web Archiving Webarchive Projects (3)
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194
"GitHub" is a registered trademark of GitHub, Inc. Awesome Open Source is not affiliated with GitHub.

All non readme contents or Github based topics or project metadata copyright Awesome Open Source 2018-2021