• This repository has been archived on 16/Feb/2021
  • Stars
    star
    201
  • Rank 194,491 (Top 4 %)
  • Language
    JavaScript
  • Created over 5 years ago
  • Updated over 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Webrecorder Desktop App!

Deprecated The Webrecorder Desktop App is now replaced by the ArchiveWeb.page system, available as an Electron App and Chrome Extension.

The new Electron app is close to supporting all features of Webrecorder Desktop, and should work much better on latest Windows and OS X.

The Chrome extension version is compatible with all Chromium-based browsers, and allowing recording and replay directly in the browser.

Development for ArchiveWeb.page continues at: https://github.com/webrecorder/archiveweb.page.


Webrecorder Desktop App

The Webrecorder Desktop App is a complete packaging of Webrecorder hosted service as an Electron application, with an integrated Chromium browser.

It includes the same functionality available on Webrecorder.io running as a local app, including the new Autopilot feature.

All data captured is stored in a local directory on your machine, in Webrecorder-Data in your Documents directory.

Webrecorder Desktop can be downloaded below or from Releases

OS X Windows (64-bit) Windows (32-bit) Linux
.dmg .exe (64-bit) .exe (32-bit) .AppImage

Note: Running on Linux requires installation of Redis, available as a package on most distros. OS X and Windows versions come with a bundled version of Redis.

Current Features

In addition to the core Webrecorder functionality, the desktop app includes additional features specific to the desktop environmentment. A few of these brand new features are still experimental or in beta, as listed below, so please let us know if anything is not working as expected!

Latest Chromium Browser with Flash Support

Like Webrecorder Player, the Webrecorder Desktop app is built with Electron, and includes the latest release of Chromium, ensuring capture and replay is done with a modern browser. The app also includes a recent Flash plugin to allow for capture and replay of any Flash content. (The App Settings screen includes versions of all components).

Local Storage of All Data

All Webrecorder Data is stored in the <Documents>/Webrecorder-Data directory, with actual WARC files under the storage subdirectory. The Autopilot behaviors are placed in the ‘behaviors’ subdirectory. The directory layout may be updated in the future as we work towards a more standardized directory format for web archives.

Capture, Replay & Curation

The app includes capture, replay, patching as well as curation and collection management features, same as those found on https://webrecorder.io. Existing collections can also be imported (as WARC files) and exported collections can be uploaded to https://webrecorder.io if desired.

Autopilot

The desktop app includes the full Autopilot capabilities for capture of certain dynamic websites, introduced with our last release. Unlike a regular browser, Webrecorder Desktop can run Autopilot in the background and be minimized without affecting the quality of Autopilot capture. For example, users can start Autopilot and have it run in the background while doing other work. (There is an option to mute audio in the Options menu for this use case). There is no limit to how long Autopilot can run locally, and only limits are available network bandwidth and disk storage!

Preview Mode (Beta)

The desktop app includes a new Preview mode that allows browsing content without capture. In particular, this can be used to preview a page before capturing it but also to log in to any sites that require login without capturing the login itself.

After logging to a site in Preview mode, users can then switch to capture mode via the dropdown menu, beginning capture from after the login has completed.

This workflow is recommended for capturing any sites that require a login. To reset all logins, there is also a “Clear Cookies” option in the Options menu. (This feature is currently in beta and we welcome any feedback on this!)

Mobile Device Emulation Mode (Experimental)

The desktop app also includes an experimental mobile device emulation mode, toggleable from the Options menu. With this mode, Webrecorder Desktop will act as a mobile browser and allow for capturing of mobile only content. The window can be resized as needed to support any mobile device. (This feature is currently in beta and we welcome any feedback on this!)

DAT Protocol Support (Experimental)

The app includes our previously-announced approach to sharing web archive collections via the Dat peer-to-peer protocol. To enable sharing of a collection, select Share via Dat from the collection menu. The collection will then have a unique dat:// url, which will allow the full collection (and future updates) to be synched using various tools that use the Dat protocol, to allow for automated backup of local collections, if desired. There is not (yet) a way to import existing collections via Dat, but import is planned for a future update.

Capture Cache (Experimental)

When browsing sites that share resources, Webrecorder Desktop enables the browser cache to avoid capturing the same resources multiple times and writing them to WARC. The cache is reset per recording session, but can also be cleared manually via the Options menu Clear Cache option. The cache should reduce duplicates resources loaded over the network and speed up the browsing and thereby the capture process. This feature is still experimental.

TOR Capture Support (Experimental)

Webrecorder Desktop can capture web content over Tor, including Tor hidden services. However, this requires a bit of manual setup. A local Tor Relay must be installed locally.

Then, via a command-line, set export SOCKS_HOST=localhost before starting Webrecorder Desktop to have it use the Tor SOCKS proxy. Future versions may simplify this process.

Building Webrecorder Desktop

To build Webrecorder Desktop locally, please follow the instructions:

  1. Clone with submodules (the submodule is the main webrecorder/webrecorder, which contains most of the code)
git clone --recurse-submodules https://github.com/webrecorder/webrecorder-desktop.git

This will install the Webrecorder submodule as well

  1. Build Webrecorder Python Binaries and install into python-binaries

This will build the Webrecorder project and install PyInstaller 3.3. Python 3.5 is recommended for now and a separate virtualenv just in case.

./build-wr.sh
  1. Build the Webrecorder frontend
node build-desktop.js
  1. Run in Dev Mode
yarn run start-dev
  1. Build Electron Binary
yarn run dist
  1. If all goes well, the binary image should be placed in ./dist/{mac,linux,win} directory, depending on your platform.

More Repositories

1

pywb

Core Python Web Archiving Toolkit for replay and recording of web archives
JavaScript
1,366
star
2

archiveweb.page

A High-Fidelity Web Archiving Extension for Chrome and Chromium based browsers!
JavaScript
841
star
3

replayweb.page

Serverless replay of web archives directly in the browser
TypeScript
693
star
4

browsertrix-crawler

Run a high-fidelity browser-based crawler in a single Docker container
TypeScript
547
star
5

webrecorder-player

Webrecorder Player for Desktop (OSX/Windows/Linux). (Built with Electron + Webrecorder)
JavaScript
423
star
6

warcio

Streaming WARC/ARC library for fast web archive IO
Python
345
star
7

browsertrix

Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!
TypeScript
178
star
8

specs

Specifications developed and maintained by the Webrecorder community.
HTML
117
star
9

wabac.js

wabac.js - Web Archive Browsing Augmentation Client
JavaScript
98
star
10

browsertrix-old

Browsertrix: Containerized High-Fidelity Browser-Based Automated Crawling + Behavior System
Python
87
star
11

wombat

Wombat.js client-side rewriting library
JavaScript
81
star
12

warcit

Convert Directories, Files and ZIP Files to Web Archives (WARC)
Python
79
star
13

har2warc

Convert HTTP Archive (HAR) -> Web Archive (WARC) format
Python
42
star
14

public-web-archives

A listing of world wide web archives, for humans and machines using Web Archive Manifest (WAM) yaml format
Python
40
star
15

warcio.js

JS Streaming WARC IO optimized for Browser and Node
TypeScript
34
star
16

py-wacz

Python
32
star
17

browsertrix-behaviors

Automated behaviors that run in browser to interact with complex sites automatically. Used by ArchiveWeb.page and Browsertrix Crawler.
TypeScript
30
star
18

archiveweb.page-site

The ArchiveWeb.page Site
HTML
27
star
19

wsgiprox

Python WSGI Middleware for adding HTTP/S proxy support to any WSGI Application
Python
22
star
20

cdxj-indexer

CDXJ Indexing of WARC/ARCs
Python
21
star
21

web-replay-gen

Static Site Generator for Viewing Web Archives (in WACZ) format
JavaScript
19
star
22

oembed.link

A Cloudflare Worker to render embeds on a single page using oEmbed
JavaScript
14
star
23

dat-share

A prototype server to swarm multiple DATs for Webrecorder
JavaScript
12
star
24

pywb-remote-browsers

Docker Compose based system for running remote browsers (including Flash and Java support) connected to web archives
Python
12
star
25

markdown-to-respec

A Github Action for turning Markdown into ReSpec HTML
Python
12
star
26

behaviors

Webrecorder Automated In-Page Behavior Framework
JavaScript
11
star
27

dat-s3-hybrid-storage

A S3 hybrid storage interface for dat and hyperdrive
JavaScript
11
star
28

express.archiveweb.page

ArchiveWeb.page Express!
JavaScript
8
star
29

wacz-auth-spec

Specification for authentication and creating signed WACZ Files
8
star
30

platform-spec

Discussion of the broader Webrecorder platform spec
7
star
31

autobrowser

Python
7
star
32

authsign

Python
7
star
33

kubecaptures-backend

JavaScript
6
star
34

awp-sw

JavaScript
6
star
35

sup-digital-web-archives

A collection of self-hostable web archive built for Stanford University Press (SUP)
HTML
6
star
36

example-webarchive

This is an example web archive using the ReplayWebPage component.
HTML
6
star
37

save-tweet-now

Save Tweet Now (to IPFS)
JavaScript
6
star
38

browserkube

Webrecorder Kubernetes-native Browser Ochestration
JavaScript
5
star
39

webrecorder-tests

QA tests for webrecorder player (WORK IN PROGRESS)
Python
5
star
40

autoscalar

Webrecorder Auto Archiver for Scalar Prototype
Python
5
star
41

browsertrix-browser-base

Dockerfile
5
star
42

wacz-uploader

A straightforward single page application for uploading your WACZ archives to IPFS
JavaScript
5
star
43

wabac-cors-proxy

CORS proxy for use with wabac.js-based tools
JavaScript
4
star
44

ipfs-composite-files

CLI and library for create composite files in IPFS
JavaScript
3
star
45

wabac.js-1.0

JavaScript
3
star
46

wacz2car

Conver WACZ files to CAR files for uploading to IPFS.
JavaScript
3
star
47

proxy

(Deprecated) Old Webrecorder proxy component based on mitmproxy
Python
2
star
48

community

Webrecorder Community Info
2
star
49

kubecaptures-ui

JavaScript
2
star
50

mapbox-driver

Browsertrix Crawler driver for Mapbox maps
JavaScript
2
star
51

dashboard-custom-drivers

Custom drivers for browsertrix crawler
JavaScript
1
star
52

sucho-web-archive

JavaScript
1
star
53

functional-spec

A description of the interface and functionalities of webrecorder.io
1
star