Awesome Open Source
Awesome Open Source
Combined Topics
web-archiving x

The Top 75 Web Archiving Open Source Projects

Categories > Content Management > Web Archiving
Archivebox ⭐ 13,513
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
dependent packages 1total releases 24most recent commit 21 days agopypi archivebox} Downloads
Conifer ⭐ 1,347
Collect and revisit web pages.
most recent commit 3 months ago
Awesome Web Archiving ⭐ 1,166
An Awesome List for getting started with web archiving
most recent commit a month ago
Pywb ⭐ 951
Core Python Web Archiving Toolkit for replay and recording of web archives
dependent packages 2total releases 91most recent commit 2 months agopypi pywb} Downloads
Ipwb ⭐ 478
InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS
total releases 236most recent commit a month ago
Webrecorder Player ⭐ 381
Webrecorder Player for Desktop (OSX/Windows/Linux). (Built with Electron + Webrecorder)
most recent commit 2 years ago
Perma ⭐ 332
Indelible links
most recent commit 6 days ago
Warcdb ⭐ 276
WarcDB: Web crawl data as SQLite databases.
most recent commit 9 days ago
Wail ⭐ 268
🐋 Web Archiving Integration Layer: One-Click User Instigated Preservation
most recent commit a month ago
Archivenow ⭐ 250
A Tool To Push Web Resources Into Web Archives
most recent commit a year ago
Replayweb.page ⭐ 227
Serverless Web Archive Replay directly in the browser
total releases 53most recent commit 14 days ago
Archiveweb.page ⭐ 203
A High-Fidelity Web Archiving Extension for Chrome and Chromium based browsers!
most recent commit 6 days ago
Archiveror ⭐ 188
Archiveror will help you preserve the webpages you love. 💾
most recent commit 3 years ago
Waybackpy ⭐ 183
Wayback Machine API interface & a command-line tool
dependent packages 3total releases 35most recent commit 3 months agopypi waybackpy} Downloads
Warcio ⭐ 173
Streaming WARC/ARC library for fast web archive IO
dependent packages 17total releases 22most recent commit 2 years agopypi warcio} Downloads
Warcreate ⭐ 163
Chrome extension to "Create WARC files from any webpage"
most recent commit a month ago
Sfm Ui ⭐ 133
Social Feed Manager user interface application.
most recent commit 8 months ago
Squidwarc ⭐ 131
Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
most recent commit 2 years ago
Archivespark ⭐ 118
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
total releases 7most recent commit 9 months ago
Wail ⭐ 97
🐋 One-Click User Instigated Preservation
most recent commit 3 years ago
Cdx_toolkit ⭐ 95
A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
total releases 29most recent commit 3 months ago
Fatcat ⭐ 89
Perpetual Access To The Scholarly Record
dependent packages 1total releases 5most recent commit a month agopypi fatcat-openapi-client} Downloads
Electron Archivebox ⭐ 88
Desktop Electron app for ArchiveBox internet archiver. (ALPHA: not ready for general use)
most recent commit 3 months ago
Warc Parquet ⭐ 68
🗄️ A simple CLI for converting WARC to Parquet.
most recent commit 5 days ago
Node Warc ⭐ 62
Parse And Create Web ARChive (WARC) files with node.js
dependent packages 1total releases 11most recent commit 10 months agonpm node-warc} Downloads
Awesome Memento ⭐ 60
A list of things related to software, literature, and other content for 🕣 Memento
most recent commit 4 months ago
Warrick ⭐ 59
Recover lost websites from the Web Infrastructure
most recent commit a year ago
Collect ⭐ 57
A server to collect & archive websites that also supports video downloads
most recent commit 7 months ago
Wget Lua ⭐ 50
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
most recent commit 22 days ago
Memgator ⭐ 39
A Memento Aggregator CLI and Server in Go
most recent commit a year ago
Warcworker ⭐ 33
A dockerized, queued high fidelity web archiver based on Squidwarc
most recent commit 2 years ago
Outbackcdx ⭐ 21
Web archive index server based on RocksDB
most recent commit a month ago
Cc Notebooks ⭐ 18
Various Jupyter notebooks about Common Crawl data
most recent commit a month ago
Cdxj Indexer ⭐ 17
CDXJ Indexing of WARC/ARCs
most recent commit 4 days ago
Sandcrawler ⭐ 16
Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki
most recent commit 6 months ago
Pdf_trio ⭐ 15
A PDF classifier ensemble with REST API service
most recent commit a year ago
Quickcacheandarchivesearch ⭐ 13
Quick Cache and Archive search buttons
most recent commit a month ago
Munin Indexer ⭐ 13
A social media open post web archiving tool
most recent commit 6 months ago
Seeder ⭐ 13
Seeder - Czech webarchive curating tool and public site
most recent commit 2 months ago
Conifer Deploy ⭐ 11
Conifer setup and deployment via Ansible
most recent commit 2 years ago
Homebrew Archivebox ⭐ 11
Homebrew formula for the ArchiveBox self-hosted internet archiving solution.
most recent commit 6 months ago
Httrack2warc ⭐ 11
Converts HTTrack crawls to WARC files
most recent commit 9 months ago
Dat Share ⭐ 10
A prototype server to swarm multiple DATs for Webrecorder
most recent commit 3 years ago
Webarchiver ⭐ 9
Decentralized web archiving
most recent commit 4 years ago
Ukwa Manage ⭐ 9
Shepherding our web archives from crawl to access.
most recent commit 14 hours ago
Internet Archiving Talk ⭐ 8
🎭 An introduction to the Internet Archiving ecosystem, tooling, and some of the ethical dilemmas that the community faces.
most recent commit 2 years ago
Hadoopconcatgz ⭐ 7
A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz
most recent commit 4 years ago
Httpreserve ⭐ 7
Digital Preservation of HTTP in documentary heritage.
most recent commit 6 months ago
Chropro ⭐ 7
Chrome debugging protocol client for Java
most recent commit 2 years ago
Bamboo ⭐ 6
Web archive collection manager
most recent commit 23 days ago
Chronicrawl ⭐ 6
Experimental continouous web crawler for web archiving
most recent commit a year ago
Metawarc ⭐ 5
metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)
most recent commit 3 months ago
Tikalinkextract ⭐ 4
Tika based link extractor for httpreserve
most recent commit a year ago
Archiveboxmatic ⭐ 4
ArchiveBoxMatic: configure ArchiveBox with the simplicity of a yaml file.
most recent commit a year ago
Linkscanner ⭐ 3
A helper package to tokenize textual content and retrieve hyperlinks
most recent commit a year ago
Linkstat ⭐ 3
CLI implementation of httpreserve that can test links and retrieve internet archive replacements
most recent commit a year ago
Debian Archivebox ⭐ 2
Home of the official apt/deb package for Ubuntu/Debian-based systems.
most recent commit a year ago
Workbench ⭐ 2
Client app for httpreserve pkg that generates CSV, JSON, HTTP, and BoltDB
most recent commit 6 months ago
Tmvis ⭐ 2
An archival thumbnail visualization server
most recent commit 2 years ago
Moonshine ⭐ 2
Given four bytes, download a random file from web archives implementing the UKWA Shine interface
most recent commit 2 years ago
Pip Archivebox ⭐ 2
Official Python package for ArchiveBox, the self-hosted internet archiving solution.
most recent commit a year ago
Scrapy Warcio ⭐ 2
Support for writing WARC files with Scrapy
most recent commit 3 years ago
Capture Urls ⭐ 2
Archive a list of URLs using the Wayback Machine
most recent commit 3 months ago
Scriptor ⭐ 2
Plug-and-play reproducible web analysis.
most recent commit 15 days ago
Bookmarkarchiver ⭐ 2
Python script that archives all of your bookmarks on the Internet Archive. Supports all major browsers.
most recent commit 5 months ago
Eprints2archives ⭐ 1
Send records from an EPrints server to the Internet Archive and other web archives
most recent commit 2 months ago
Memgatorbulkdownload ⭐ 1
most recent commit 4 years ago
Whatdowebpagesdo ⭐ 1
most recent commit 5 years ago
Phpbbcrawler ⭐ 1
Link crawler for a phpBB forum
most recent commit 11 days ago
Pandas4 ⭐ 1
Fourth-generation web archive workflow system
most recent commit 14 hours ago
Phantomjsscreenshot ⭐ 1
A wrapper for phantom.js commands for headless screenshots.
most recent commit a year ago
Mdbubing ⭐ 1
From WARC records to MongoDB documents
most recent commit 2 years ago
Waybacks ⭐ 1
This module builds our Waybacks in the various different configurations we require.
most recent commit 4 years ago
Webarchiver ⭐ 1
A archiving utility with an interface for web servers.
most recent commit a year ago
Openwayback ⭐ 1
(used on swap vm 6/2020) Stanford's fork of iipc/openwayback, which is used on our "swap" (Stanford Web Archiving Portal) machines. (See also sul-dlss/swap which is intended as a replacement)
most recent commit a year ago
1-75 of 75 projects
Categories
Advertising 📦 8
All Projects
Application Programming Interfaces 📦 107
Applications 📦 174
Artificial Intelligence 📦 69
Blockchain 📦 66
Build Tools 📦 105
Cloud Computing 📦 68
Code Quality 📦 24
Collaboration 📦 27
Command Line Interface 📦 38
Community 📦 79
Companies 📦 60
Compilers 📦 59
Computer Science 📦 73
Configuration Management 📦 37
Content Management 📦 153
Control Flow 📦 187
Data Formats 📦 72
Data Processing 📦 249
Data Storage 📦 116
Economics 📦 56
Frameworks 📦 178
Games 📦 110
Graphics 📦 92
Hardware 📦 137
Integrated Development Environments 📦 43
Learning Resources 📦 139
Legal 📦 24
Libraries 📦 117
Lists Of Projects 📦 19
Machine Learning 📦 313
Mapping 📦 57
Marketing 📦 15
Mathematics 📦 54
Media 📦 214
Messaging 📦 96
Networking 📦 292
Operating Systems 📦 71
Operations 📦 114
Package Managers 📦 50
Programming Languages 📦 173
Runtime Environments 📦 90
Science 📦 42
Security 📦 343
Social Media 📦 23
Software Architecture 📦 65
Software Development 📦 60
Software Performance 📦 53
Software Quality 📦 115
Text Editors 📦 40
Text Processing 📦 118
User Interface 📦 291
User Interface Components 📦 440
Version Control 📦 26
Virtualization 📦 63
Web Browsers 📦 35
Web Servers 📦 23
Web User Interface 📦 167
Top Programming Languages
Javascript  (1,059,739)
Python  (807,156)
Java  (388,454)
Php  (285,984)
Typescript  (246,254)
C Plus Plus  (238,787)
Ruby  (221,835)
C  (182,003)
Shell  (171,198)
C Sharp  (164,103)
Golang  (158,248)
Swift  (65,213)
R  (57,905)
Rust  (57,566)
Objective C  (57,184)
Dart  (52,612)
Kotlin  (46,446)
Lua  (33,878)
Matlab  (31,150)
Perl  (30,540)
Language  (30,062)
Scala  (28,665)
Haskell  (20,368)
Clojure  (19,133)
Powershell  (19,013)
Bash  (18,678)
Coffeescript  (17,292)
Elixir  (16,655)
Assembly  (14,679)
Processing  (13,307)
Julia  (12,743)
Basic  (10,739)
Flow  (10,547)
Emacs Lisp  (10,417)
Erlang  (9,065)
Groovy  (8,446)
Scheme  (8,438)
Solidity  (7,954)
Ocaml  (6,703)
Pascal  (6,439)
Hack  (6,398)
Lisp  (6,341)
Fortran  (5,540)
Elm  (5,484)
Common Lisp  (5,298)
Dlang  (4,745)
Shell Script  (4,655)
Python Library  (4,455)
Crystal  (4,390)
Scripting  (4,207)
Top Projects
Freecodecamp ⭐ 348,159
996.icu ⭐ 262,763
Free Programming Books ⭐ 237,303
Coding Interview University ⭐ 222,665
Awesome ⭐ 206,565
Developer Roadmap ⭐ 198,217
Public Apis ⭐ 197,271
Vue ⭐ 197,223
React ⭐ 190,606
System Design Primer ⭐ 184,236
Tensorflow ⭐ 166,046
Bootstrap ⭐ 158,216
You Dont Know Js ⭐ 153,649
Cs Notes ⭐ 151,618
Ohmyzsh ⭐ 146,966
Build Your Own X ⭐ 145,201
Javascript Algorithms ⭐ 143,983
Flutter ⭐ 142,246
Python ⭐ 139,248
Gitignore ⭐ 134,582
Linux ⭐ 133,899
Vscode ⭐ 133,454
Awesome Python ⭐ 131,759
Javascript ⭐ 124,014
Javaguide ⭐ 123,971
Python 100 Days ⭐ 120,066
Computer Science ⭐ 118,096
Youtube Dl ⭐ 111,107
Fucking Algorithm ⭐ 107,431
The Art Of Command Line ⭐ 106,741
React Native ⭐ 103,527
Electron ⭐ 102,301
D3 ⭐ 101,735
Go ⭐ 101,131
30 Seconds Of Code ⭐ 96,728
Create React App ⭐ 95,709
Axios ⭐ 94,209
Free Programming Books Zh_cn ⭐ 93,295
Awesome Selfhosted ⭐ 92,838
Kubernetes ⭐ 89,722
Next.js ⭐ 88,761
Node ⭐ 88,478
Terminal ⭐ 83,789
Deno ⭐ 83,340
Three.js ⭐ 83,174
Awesome Go ⭐ 83,058
Angular ⭐ 82,342
Typescript ⭐ 81,850
Ant Design ⭐ 80,877
Material Ui ⭐ 79,382
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2022 Awesome Open Source.  All rights reserved.