Awesome Open Source
Search
Programming Languages
Languages
All Categories
Categories
About

The Top 23 Python Warc Open Source Projects

Open source projects categorized as Python Warc
Archivebox ⭐ 19,721
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
dependent packages 1total releases 26most recent commit 9 days agopypi archivebox} Downloads
Conifer ⭐ 1,434
Collect and revisit web pages.
most recent commit 6 months ago
Ipwb ⭐ 577
InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS
total releases 241most recent commit 3 months ago
Warcprox ⭐ 348
WARC writing MITM HTTP/S proxy
total releases 24most recent commit 6 months ago
Wail ⭐ 330
🐋 Web Archiving Integration Layer: One-Click User Instigated Preservation
most recent commit 5 months ago
Archivebot ⭐ 328
ArchiveBot, an IRC bot for archiving websites
most recent commit 5 months ago
Cc Pyspark ⭐ 280
Process Common Crawl data with Python and Spark
most recent commit a year ago
Bitextor ⭐ 260
Bitextor generates translation memories from multilingual websites
most recent commit 7 months ago
Zimit ⭐ 209
Make a ZIM file from any Web site and surf offline!
most recent commit 3 months ago
Warc ⭐ 196
Python library for reading and writing warc files
dependent packages 1total releases 3most recent commit 6 years agopypi warc} Downloads
Warcio ⭐ 173
Streaming WARC/ARC library for fast web archive IO
dependent packages 34total releases 22most recent commit 3 years agopypi warcio} Downloads
Cocrawler ⭐ 159
CoCrawler is a versatile web crawler built using modern tools and concurrency.
total releases 8most recent commit 2 years ago
Webarchiveplayer ⭐ 156
NOTE: This project is no longer being actively developed.. Check out Webrecorder Player for the latest player. https://github.com/webrecorder/webrecorderplayer-e (Legacy: Desktop application for browsing web archives (WARC and ARC)
most recent commit 7 years ago
Cdx_toolkit ⭐ 121
A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
total releases 29most recent commit a year ago
Warcat ⭐ 96
Tool and library for handling Web ARChive (WARC) files.
total releases 10most recent commit 3 years ago
Warctools ⭐ 84
Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)
most recent commit 4 years ago
Warcit ⭐ 63
Convert Directories, Files and ZIP Files to Web Archives (WARC)
total releases 4most recent commit 2 years ago
Warc Proxy ⭐ 57
Serving content from a WARC
most recent commit 11 years ago
Warcmiddleware ⭐ 42
WarcMiddleware lets users seamlessly download a mirror copy of a website when running a web crawl with the Python web crawler Scrapy.
most recent commit 6 years ago
Newsgrabber ⭐ 34
Grabbing all news.
most recent commit 4 years ago
Pywb Webrecorder ⭐ 34
Check out https://github.com/webrecorder/webrecorder for newer version matching https://webrecorder.io
most recent commit 9 years ago
Chatnoir Resiliparse ⭐ 33
A robust web archive analytics toolkit
dependent packages 3total releases 71most recent commit 5 months agopypi FastWARC} Downloads
Liveweb ⭐ 32
Liveweb proxy of the Wayback Machine project
total releases 1most recent commit 5 years ago
The 7 Most Depended On Python Warc Open Source Projects
Warcio ⭐ 173
Streaming WARC/ARC library for fast web archive IO
dependent packages 34total releases 22latest release August 11, 2020most recent commit 3 years agopypi warcio} Downloads
Chatnoir Resiliparse ⭐ 33
A robust web archive analytics toolkit
dependent packages 3total releases 71latest release August 11, 2023most recent commit 5 months agopypi FastWARC} Downloads
Cdxj Indexer ⭐ 17
CDXJ Indexing of WARC/ARCs
dependent packages 2total releases 12latest release June 25, 2022most recent commit 2 years agopypi cdxj-indexer} Downloads
Cdx Summary ⭐ 6
Summarize web archive capture index (CDX) files.
dependent packages 2total releases 9latest release July 29, 2022most recent commit 2 years agonpm @internetarchive/cdxsummary} Downloads
Webarticlecurator ⭐ 5
Web Article Curator
dependent packages 1total releases 28latest release February 10, 2022most recent commit 2 years agopypi webarticlecurator} Downloads
Warc ⭐ 196
Python library for reading and writing warc files
dependent packages 1total releases 3latest release May 15, 2012most recent commit 6 years agopypi warc} Downloads
The 10 Latest Releases In Python Warc Open Source Projects
Warc2zim ⭐ 30
Command line tool to convert a file in the WARC format to a file in the ZIM format
total releases 21latest release September 18, 2023most recent commit 3 months ago
Ipwb ⭐ 577
InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS
total releases 241latest release August 16, 2023most recent commit 3 months ago
Chatnoir Resiliparse ⭐ 33
A robust web archive analytics toolkit
dependent packages 3total releases 71latest release August 11, 2023most recent commit 5 months agopypi FastWARC} Downloads
Cdx Summary ⭐ 6
Summarize web archive capture index (CDX) files.
dependent packages 2total releases 9latest release July 29, 2022most recent commit 2 years agonpm @internetarchive/cdxsummary} Downloads
Cdxj Indexer ⭐ 17
CDXJ Indexing of WARC/ARCs
dependent packages 2total releases 12latest release June 25, 2022most recent commit 2 years agopypi cdxj-indexer} Downloads
Cdx_toolkit ⭐ 121
A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
total releases 29latest release March 02, 2022most recent commit a year ago
Webarticlecurator ⭐ 5
Web Article Curator
dependent packages 1total releases 28latest release February 10, 2022most recent commit 2 years agopypi webarticlecurator} Downloads
Liveweb ⭐ 32
Liveweb proxy of the Wayback Machine project
total releases 1latest release December 15, 2021most recent commit 5 years ago
Popular Programming Languages Categories
Python
Javascript
Java
Typescript
C Plus Plus
Php
Golang
C
C Sharp
Shell
Categories
Privacy | About | Terms | Follow Us On Twitter

Downloads, Dependent Repos, Dependent Packages, Total Releases, Latest Releases data powered by Libraries.io.

Copyright 2018-2024 Awesome Open Source.  All rights reserved.