• Stars
    star
    126
  • Rank 284,490 (Top 6 %)
  • Language
    Shell
  • License
    Other
  • Created over 10 years ago
  • Updated over 6 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Hacking the RAVPower Filehub to automate SD card transfers and harddrive backups

RAVPower Automation

This collection of scripts automate functionality for copying and backing up files using a RAVPower Filehub.

  • Change the default password
  • Block external network access
  • Copy files from SD Card to USB drive automatically
  • Rename & organize files using EXIF data
  • Backup / sync between two USB drives
  • Add a swap file on a USB drive
  • Allow import of ODK Collect data from smart phones over USB
  • Allow import of ODK Collect data from smart phones over wifi

How to hack the Filehub embedded Linux

The RAVPower Filehub runs embedded Linux, which is a cut-down version of Linux with a low memory footprint. Most of the filesystem is read-only apart from the contents of /etc and /tmp, but changes are not persisted across reboots.

The easiest way to "hack" / modify the configuration of the embedded Linux is to create a script EnterRouterMode.sh on an SD card and put the card in the Filehub. The current firmware (2.000.004) will execute a script with this name with root permissions when the SD card is mounted.

The EnterRouterMode.sh script modifies scripts within /etc and persists changes by running /usr/sbin/etc_tools p.

To use, download the EnterRouterMode.sh script, copy it to the top-level folder of an SD card, and insert it into the filehub device.

Building from source

git clone https://github.com/digidem/filehub-config.git
make

Change the default password

The default root password on RAVPower Filehub devices is 20080826. This is available on several online forums. Best change it. You can do this by telnet (username: root password: 20080826):

telnet 10.10.10.254
passwd

or create a file EnterRouterMode.sh on an SD card and insert it into the Filehub:

#!/bin/sh
passwd <<'EOF'
newpassword
newpassword
EOF
/usr/sbin/etc_tools p

Block external network access

By default it is possible to telnet into the Filehub from an external network if you know what you are doing. This script adds iptables rules to /etc/rc.local (source)

Copy files from SD card automatically

The script runs when any USB device is attached. It checks whether an SD card is present, and it looks for an external USB drive (can be a thumb drive or a USB disk drive) with a folder /monitoreo/config which contains an rsync binary built for embedded linux. There is not enough memory on the filehub device to store the rsync binary on the device itself.

The script uses rsync to copy files, which should be resilient to interuption mid-copy and resume where it left off. Source files are removed from the SD card as they are copied to the external drive.

A folder is created for each SD card, identified by a UUID. It would be ideal to use the serial number for an SD card for the UUID, but unfortunately it is not possible to access this. udevadm info -a -p $(udevadm info -q path -n /dev/sda) | grep -m 1 "ATTRS{serial}" | cut -d'"' -f2 returns the serial number for the card reader, rather than the SD card. Instead we generate a UUID using cat /proc/sys/kernel/random/uuid and store that on the SD card. Bear in mind if an SD card is re-formatted in the camera then this UUID will be lost, so the card will appear as a new card next time it is inserted. Using a UUID allows for transfers to be interupted and resumed later.

If more than 9999 photos are taken with a camera, filenames will be reused. Similarly if an SD card is used in a different camera, filenames will be repeated. This would lead to overwriting files if we just stored all photos from each SD card in a single folder. Instead we create a subfolder for each import. Ideally this would be named with the date of the import, but the clock on the RavPower device cannot be relied upon without internet access. Instead we use the date of the most recent photo on the SD Card as the name of the subfolder.

When the SD card or USB drive is removed, we kill the rsync process, otherwise it hangs around.

Backup between two USB hard drives

If a second drive is attached with a folder "Backup" then an automatic backup process will begin. Each backup drive is linked to the original drive via the original drive serial number. e.g. if you have two drives with original files and one backup drive for each, it will not backup to the wrong drive because it will detect the serial. The link is created the first time you create the backup, by writing a file ".backup_id" to the backup folder.

At this stage older versions and deleted files are not kept. The backup drive is an exact mirror of the original drive. Backups are created via rsync with the following command:

rsync -vrm --size-only --delete-during --exclude ".?*" --partial-dir "$partial_dir" --exclude "swapfile" --log-file /tmp/rsync_log "$source_dir"/ "$target_dir"

Comments and suggestions for the rsync options are most welcome!

Swap file

The RavPower Filehub only has 28Mb of memory, and about 2Mb of free memory. Rsync needs around 100 bytes for each file. To avoid out of memory issues we create a 64Mb swapfile on the USB drive when it is connected. This appears to speed up rsync and should avoid memory issues. I have not yet tested with thousands of files.

Renaming with EXIF

I would like photo filenames to be unique, so we can use them as a UUID. The best way would be to read the EXIF capture date, and prepend that to the filename. Although it might be possible to do that with just the file creation date and time. To use EXIF we would need to cross-compile an EXIF utility for the MIPS architecture used in the RavPower.

ODK Collect Imports

We are using ODK Collect for data collection. This Android app stores data in a folder on the phone storage, and allows for sending that info via a multi-part form submission. There are 3 options for getting that data onto the filehub:

  1. Modify the form submission code in ODK collect so that instead of a multipart form upload, it uploads the form as an XML file to the WebDav server on the RavPower. The Ravpower can be configured so that the ODK server address will redirect locally when no internet connection is present.

  2. Write a small CGI script that can run on the RavPower to accept a multi-part form submission (containing form XML and associated media/photos). It would need to rename the files with the form submission UUID. Could face memory and processing speed limitations.

  3. Transfer the data via a USB connection. Android >4.0 only connects via MTP, which varies in implmentation in Android. The best seems to be go-mtpfs which would need to be cross-compiled with GO for MIPS architecture, which seems is possible. All libraries would need to be statically linked. This is potentially the most reliable solution.

More Repositories

1

react-dimensions

[Looking for maintainers]
JavaScript
454
star
2

leaflet-side-by-side

A Leaflet control to add a split screen to compare two map overlays
JavaScript
346
star
3

mapeo-desktop

Local-first mapping and monitoring in remote environments
JavaScript
261
star
4

osm-p2p-db

Peer-to-peer database for OpenStreetMap data
JavaScript
231
star
5

mapeo-mobile

Monitor and document the world around you
TypeScript
95
star
6

osm-p2p-server

Peer-to-peer OpenStreetMap API v0.6 Server for osm-p2p-db
JavaScript
84
star
7

mapbox-map-image-export

Export a Mapbox GL map to a hi-res image for printing
JavaScript
83
star
8

leaflet-bing-layer

Bing Maps Layer for Leaflet v1.0.0
JavaScript
83
star
9

osm-p2p

High-level p2p OpenStreetMap database for node and the browser
JavaScript
55
star
10

maplibre-storymap

Storytelling template with MapLibre GL JS, deployed using Node or static html.
HTML
38
star
11

bkd-tree

multi-dimensional spatial database based on the bkd paper
JavaScript
36
star
12

kappa-osm

Peer-to-peer OpenStreetMap database using append-only logs
JavaScript
30
star
13

react-mapfilter

Visualizing, exploring, filtering and printing geographic data and geotagged photos and video
JavaScript
29
star
14

hyperdb-osm

Peer-to-peer OpenStreetMap database over hyperdb.
JavaScript
24
star
15

indexed-tarball

Tar archive with constant-time reads & appends. Supports spanning over multiple archives.
JavaScript
22
star
16

mapeo-core

Library for creating custom geo data and syncronizing via a peer to peer network
JavaScript
22
star
17

geojson-dissolve

Dissolve contiguous GeoJSON (Multi)LineStrings and (Multi)Polygons into single units.
JavaScript
21
star
18

GAS-github-json

Uploads a Google Sheet to a github repository as geojson whenever the spreadsheet is updated
Gosu
21
star
19

unordered-materialized-kv

materialized view key/id store based on unordered log messages
JavaScript
20
star
20

osm2obj

Take a readable stream of OSM XML and output a stream of objects
JavaScript
19
star
21

airtable-github-export

Export airtable tables to json/geojson on Github
JavaScript
18
star
22

simple-odk

A minimal ODK server for receiving form submissions from a mobile device and saving them as JSON on github
JavaScript
18
star
23

cache-blob-store

cache images in local storage for offline web apps
JavaScript
17
star
24

mosaic-image-stream

Streaming mosaic of multiple images into a single image
JavaScript
16
star
25

hyperlog-sneakernet-replicator

peer to peer replication for hyperlog using files you can send around on a USB stick
JavaScript
14
star
26

feature-filter-geojson

Creates filtering function for geojson features
JavaScript
12
star
27

digital-democracy.org

Digital Democracy website
SCSS
12
star
28

map-printer

Export massive hi-res PNGs from Mapbox map styles without memory limits.
JavaScript
11
star
29

github-webhook-middleware

Express middleware for validating Github webhooks
JavaScript
11
star
30

mapeo-server

Mapping web server for managing observations, media, tiles, and various static files.
JavaScript
11
star
31

unordered-materialized-backrefs

materialized view to calculate back-references for unordered log messages
JavaScript
10
star
32

edt-offline

Low-energy device that creates an access-point with the Earth Defender's Toolkit platform running, even without the Internet. It provides applications, documentation, use cases, curated websites, and application bridges.
Shell
10
star
33

multicast-service-discovery

Announce and look up services on a local network using mdns
JavaScript
9
star
34

obj2osm

Generate OSM XML from a stream of objects matching OSM JSON
JavaScript
9
star
35

osm-p2p-syncfile

Replicate OSM data and media using a sneakernet-style sync file.
JavaScript
9
star
36

hyperdb-sneakernet

Peer to peer replication for a hyperdb using files you can send around on a USB stick.
JavaScript
9
star
37

osm-p2p-api

Functions for implementing the OSM API using osm-p2p-db.
JavaScript
9
star
38

xform-to-json

Converts XForm submissions from ODK Collect to json or geojson.
JavaScript
8
star
39

offline-map-editor

offline map editor prototype data model
JavaScript
8
star
40

github-card

Github Card embeds see https://developmentseed.org/blog/2015/05/14/github-cards/
HTML
8
star
41

clearwater-map

An interactive map and stories for ClearWater well installations in the Ecuadorian Amazon. See www.giveclearwater.org for more information.
JavaScript
8
star
42

unordered-materialized-bkd

materialized view spatial tree based on unordered log messages
JavaScript
7
star
43

github-image-resizer

Resizes images pushed to Github and saves them to S3, replacing the original on Github with a lo-res version
JavaScript
7
star
44

tile-downloader

Download tiles for a map region without memory limits.
CSS
6
star
45

osm-p2p-observations

p2p database for monitoring observations
JavaScript
6
star
46

mapeo-docs

Docs site for mapeo
JavaScript
6
star
47

id-presets-builder

Build presets for iD Editor
JavaScript
6
star
48

mapeo-web

A small sync and storage service for Mapeo maps.
JavaScript
6
star
49

satellite-image-processing-environment

Vagrant environment for processing satellite images on AWS or locally
Shell
5
star
50

mapeo-openmaptiles

Lightweight map tiles for mapeo-mobile for global offline map
PLpgSQL
5
star
51

html5-image-cache

Caches images in HTML5 localstorage for offline webapps
JavaScript
5
star
52

vips-resizer

Fast streaming image resizer API using libvips
JavaScript
5
star
53

comapeo-mobile

The next version of Mapeo mobile
TypeScript
5
star
54

mapeo-bridge

A headless Mapeo server that syncronizes with Mapeo devices in order to centralize and manipulate data, exporting to Terrastories csv
JavaScript
5
star
55

mapeo-map-server

Offline map style and tile server
TypeScript
5
star
56

osm-p2p-vector-tile-server

serve vector tiles for an osm-p2p database
JavaScript
4
star
57

hyperdb-osm-server

Peer-to-peer OpenStreetMap server
JavaScript
4
star
58

osm-p2p-vector-tile-index

Vector tile index for osm-p2p-db
JavaScript
4
star
59

HaitiReporter

Database System for reporting gender-based violence in Haiti
4
star
60

react-native-geolocation

GeoLocation module for react-native
JavaScript
4
star
61

ddem-observation-server

server for p2p observations and media
JavaScript
4
star
62

GAS-image-upload

A Google Apps Script for Google Sheets that will scrape any image urls (jpgs only for now) pasted into the spreadsheet and send the image to http://www.blitline.com for resizing and copying to AWS S3
Gosu
4
star
63

mapeo-settings-builder

Build presets and sprites for the Mapeo apps
JavaScript
3
star
64

mapeo-config-deconstructor

JavaScript
3
star
65

openrosa-form-submission-middleware

Express middleware for multiparty to process OpenRosa form submissions from ODK Collect
JavaScript
3
star
66

config-cn

Mapeo config for assisting planning and monitoring community networks
CSS
3
star
67

img-observer

Wrapper for MutationObserver for monitoring added and removed nodes
JavaScript
3
star
68

mapbox-map-image-stream

Create an image stream from a mapbox map any scale
JavaScript
3
star
69

smart-data-table

React component for a table of responses from monitoring data
JavaScript
3
star
70

mapeo-webmaps

Web maps for Mapeo
JavaScript
3
star
71

mapeo-core-next

The upcoming version of Mapeo Core
JavaScript
3
star
72

mapeo-sqlite-indexer

Index Mapeo documents to Sqlite
JavaScript
3
star
73

osm-p2p-mem

Create an osm-p2p-db instance backed entirely by in-memory storage.
JavaScript
3
star
74

mapfilter-desktop

electron app that wraps mapfilter
JavaScript
3
star
75

mapeo-icons

Translate terms, search for icons and download them formated and colored
JavaScript
3
star
76

rpc-reflector

JavaScript
3
star
77

rupununi-mining

Satellite images from mining in the Rupununi, Guyana
JavaScript
3
star
78

digidem-server

Digidem server provisioning and config
Shell
3
star
79

mapeo-crypto

JavaScript
2
star
80

email-templates

Digital Democracy Email Templates for MailChimp
HTML
2
star
81

my-simple-odk

Simple roll-your-own ODK server for receiving ODK Collect form submissions
JavaScript
2
star
82

enketo-noserver

experiments with enketo and ODK forms in the browser
HTML
2
star
83

start-stop-state-machine

A simple state machine for managing a service that asynchronously starts and stops
JavaScript
2
star
84

maptiles

Simple library for writing and reading the maptiles format.
JavaScript
2
star
85

edt-offline-portal

Portal application that links to documentation and applications of Earth Defender's Toolkit Offline
CSS
2
star
86

earthdefenderstoolkit

WordPress Theme for the Earth Defenders Toolkit website, and EDT Toolfinder template
PHP
2
star
87

mapeo-webmaps-public

Publicly shared webmaps from Mapeo
JavaScript
2
star
88

mapeo-config-editor

Work in Progress Mapeo Config Editor
JavaScript
2
star
89

alianza-ceibo-maps

Website maps for Alianza CEIBO
JavaScript
2
star
90

mapa-waorani-block22

Interactive map of Waorani territory showing impact of oil block 22
JavaScript
2
star
91

xhr-offline

Drop-in replacement for XMLHttpRequest adding offline-first caching and retries over slow connections.
JavaScript
2
star
92

mapeo-schema

Data schemas for mapeo data types
JavaScript
2
star
93

mapeo-styles

Default styles & tiles for mapeo backgrounds
JavaScript
2
star
94

osm-p2p-geojson

Export GeoJSON from osm-p2p-db
JavaScript
2
star
95

openrosa-request-middleware

Basic middleware for serving OpenRosa compliant responses for express middleware
JavaScript
2
star
96

guyana-coordinates

convert between coordinate systems commonly used in guyana
HTML
2
star
97

secoya-land-invasion

Interactive map showing a land invasion in indigenous Secoya territory in NE Ecuador
CSS
2
star
98

openrosa-manifest

Create a valid OpenRosa manifest document see https://bitbucket.org/javarosa/javarosa/wiki/FormListAPI
JavaScript
2
star
99

HereSync

Offline Database Desktop Sync
2
star
100

sinangoe-webmap

Sinangoe webmap
JavaScript
2
star