webrecorder/specs

Stars
117
Rank 301,828 (Top 6 %)
Language
HTML
Created over 4 years ago
Updated 8 months ago

webrecorder/specs

webrecorder

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Specifications developed and maintained by the Webrecorder community.

Webrecorder Specifications

This repository contains technical specifications used by the Webrecorder project for building interoperable web archiving tools. Please help us adapt and implement them to ensure that web archives are easier to use, trust, and share on the web.

Use Cases for Decentralized Web Archives: a summary of requirements and potential threat models for distributed web archives
Web Archive Collection Zipped (WACZ): a packaging standard for web archives on the web
WACZ Signing and Verification: the mechanics for signing and verifying WACZ files for proof of authenticity
Crawl Index JSON (CDXJ): an extensible format for WARC index files

Please help us adapt and implement these community standards to help ensure that web archives are easier to use, trust, and share on the web.

We are using Git and GitHub to manage the versioning of these specifications, and also the GitHub Issue Tracker to track ongoing work. If you have questions or suggestions for the specifications, or need implementation guidance please use the issue tracker!

pywb

Core Python Web Archiving Toolkit for replay and recording of web archives

archiveweb.page

A High-Fidelity Web Archiving Extension for Chrome and Chromium based browsers!

replayweb.page

Serverless replay of web archives directly in the browser

browsertrix-crawler

Run a high-fidelity browser-based crawler in a single Docker container

webrecorder-player

Webrecorder Player for Desktop (OSX/Windows/Linux). (Built with Electron + Webrecorder)

warcio

Streaming WARC/ARC library for fast web archive IO

webrecorder-desktop

Webrecorder Desktop App!

browsertrix

Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!

wabac.js

wabac.js - Web Archive Browsing Augmentation Client

browsertrix-old

Browsertrix: Containerized High-Fidelity Browser-Based Automated Crawling + Behavior System

wombat

Wombat.js client-side rewriting library

warcit

Convert Directories, Files and ZIP Files to Web Archives (WARC)

har2warc

Convert HTTP Archive (HAR) -> Web Archive (WARC) format

public-web-archives

A listing of world wide web archives, for humans and machines using Web Archive Manifest (WAM) yaml format

warcio.js

JS Streaming WARC IO optimized for Browser and Node

py-wacz

browsertrix-behaviors

Automated behaviors that run in browser to interact with complex sites automatically. Used by ArchiveWeb.page and Browsertrix Crawler.

archiveweb.page-site

The ArchiveWeb.page Site

wsgiprox

Python WSGI Middleware for adding HTTP/S proxy support to any WSGI Application

cdxj-indexer

CDXJ Indexing of WARC/ARCs

web-replay-gen

Static Site Generator for Viewing Web Archives (in WACZ) format

oembed.link

A Cloudflare Worker to render embeds on a single page using oEmbed

dat-share

A prototype server to swarm multiple DATs for Webrecorder

pywb-remote-browsers

Docker Compose based system for running remote browsers (including Flash and Java support) connected to web archives

markdown-to-respec

A Github Action for turning Markdown into ReSpec HTML

behaviors

Webrecorder Automated In-Page Behavior Framework

dat-s3-hybrid-storage

A S3 hybrid storage interface for dat and hyperdrive

express.archiveweb.page

ArchiveWeb.page Express!

wacz-auth-spec

Specification for authentication and creating signed WACZ Files

platform-spec

Discussion of the broader Webrecorder platform spec

autobrowser

authsign

kubecaptures-backend

awp-sw

sup-digital-web-archives

A collection of self-hostable web archive built for Stanford University Press (SUP)

example-webarchive

This is an example web archive using the ReplayWebPage component.

save-tweet-now

Save Tweet Now (to IPFS)

browserkube

Webrecorder Kubernetes-native Browser Ochestration

webrecorder-tests

QA tests for webrecorder player (WORK IN PROGRESS)

autoscalar

Webrecorder Auto Archiver for Scalar Prototype

browsertrix-browser-base

wacz-uploader

A straightforward single page application for uploading your WACZ archives to IPFS

wabac-cors-proxy

CORS proxy for use with wabac.js-based tools

ipfs-composite-files

CLI and library for create composite files in IPFS

wabac.js-1.0

wacz2car

Conver WACZ files to CAR files for uploading to IPFS.

proxy

(Deprecated) Old Webrecorder proxy component based on mitmproxy

community

Webrecorder Community Info

kubecaptures-ui

mapbox-driver

Browsertrix Crawler driver for Mapbox maps

dashboard-custom-drivers

Custom drivers for browsertrix crawler

sucho-web-archive

functional-spec

A description of the interface and functionalities of webrecorder.io