• Stars
    star
    479
  • Rank 91,752 (Top 2 %)
  • Language
    Go
  • License
    BSD 3-Clause "New...
  • Created about 3 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

gokrazy rsync

gokrazy rsync

tests Sourcegraph

Package rsync contains a native Go rsync implementation.

This repository currently contains:

  1. gokr-rsyncd, a read-only rsync daemon sender-only Go implementation of rsyncd. It implements the rsync daemon network protocol (port 873/tcp by default), but can be used over SSH or locally as well.
  2. gokr-rsync is an rsync receiver implementation that can download files via rsync (daemon protocol or SSH).

The following known improvements are not yet implemented:

  • Making gokr-rsync also implement an rsync sender so that it can push (upload) files to a remote rsync server (daemon protocol or SSH).
  • Making gokr-rsync chroot (and/or Linux mount namespaces when available?) into the destination directory to reduce chances of accidental file system manipulation in case of bugs.
  • Making gokr-rsyncd also implement an rsync receiver, so that it can accept files.
  • Merging gokr-rsyncd and gokr-rsync into a single binary.

This project accepts contributions as time permits to merge them (best effort).

How do I know this project won’t eat my data?

This rsync implementation is very fresh. It was started in 2021 and doesn’t have many users yet.

With that warning out of the way, the rsync protocol uses MD4 checksums over file contents, so at least your file contents should never be able to be corrupted.

There is enough other functionality (delta transfers, file metadata, special files like symlinks or devices, directory structures, etc.) in the rsync protocol that provides opportunities for bugs to hide.

I recommend you carefully check that your transfers work, and please do report any issues you run into!

Existing rsync implementation survey

Language URL Note Max Protocol Server mode?
C WayneD/rsync original “tridge” implementation; I found older versions easier to study 31 yes
C kristapsdz/openrsync OpenBSD, good docs 27 yes
Go gokrazy/rsync → you are here ← 27 yes 🎉
Go jbreiding/rsync-go rsync algorithm no
Go kaiakz/rsync-os only client/receiver 27 no
Go knight42 proxy no
Go c4milo/gsync no
Java APNIC-net/repositoryd archived yes
Java JohannesBuchner/Jarsync archived, internet draft RFC “The rsync Network Protocol” yes
Java perlundq/yajsync yes
C++ gilbertchen/acrosync-library commercial no
Rust sourcefrog/rsyn client, “rsyn is rsync with no c” 27 no

Getting started

To serve the current directory via rsync on localhost:8730, use:

go install github.com/gokrazy/rsync/cmd/gokr-rsyncd@latest
gokr-rsyncd --daemon --gokr.listen=localhost:8730 --gokr.modulemap=pwd=$PWD

You can then copy the contents of the current directory with clients such as rsync(1):

% rsync -v --archive --port 8730 rsync://localhost/pwd/ quine
receiving file list ... done
created directory quine
./
.git/
[…]
.github/workflows/main.yml
LICENSE
Makefile
README.md
cmd/gokr-rsyncd/rsyncd.go
doc.go
go.mod
go.sum
internal/rsyncd/connection.go
internal/rsyncd/rsyncd.go
interop_test.go

sent 1,234 bytes  received 5,678 bytes  13,824.00 bytes/sec
total size is 666  speedup is 0.10

…or openrsync(1), shown doing a differential update:

% openrsync -v --archive --port 8730 rsync://localhost/pwd/ quine
socket.c:109: warning: connect refused: ::1, localhost
Transfer starting: 369 files
.git/index (1.1 KB, 100.0% downloaded)
Transfer complete: 5.5 KB sent, 1.2 KB read, 666 B file size

Usage / Setup

setup encrypted authenticated private files? privileges protocol version config required
1. rsync daemon protocol (TCP port 873) no rsync (insecure) only world-readable dropped + namespace negotiated config required
2. anon SSH (daemon) yes rsync only world-readable dropped + namespace negotiated config required
3. SSH (command) yes SSH yes full user assumed no config
4. SSH (daemon) yes SSH (+ rsync) yes full user negotiated ~/.config/gokr-rsyncd.toml required

Regarding protocol version “assumed”: the flags to send over the network are computed before starting SSH and hence the remote rsync process. You might need to specify --protocol=27 explicitly on the client. Once the connection is established, both sides do negotiate the protocol, though.

Setup 1: rsync daemon protocol (TCP port 873)

Serving rsync daemon protocol on TCP port 873 is only safe where the network layer ensures trusted communication, e.g. in a local network (LAN), or when using Tailscale or similar. In untrusted networks, attackers can eavesdrop on file transfers and possibly even modify file contents.

Prefer setup 2 instead.

Example:

  • Server: gokr-rsyncd --daemon --gokr.modulemap=module=/srv/rsync-module
  • Client: rsync rsync://webserver/module/path

Setup 2: anon SSH (daemon)

This setup is well suited for serving world-readable files without authentication.

Example:

  • Server: gokr-rsyncd --daemon --gokr.modulemap=module=/srv/rsync-module --gokr.anonssh_listen=:22873
  • Client: rsync -e ssh rsync://webserver/module/path

Setup 3: SSH (command)

This setup is well suited for interactive one-off transfers or regular backups, and uses SSH for both encryption and authentication.

Note that because gokr-rsyncd is invoked with user privileges (not root privileges), it cannot do namespacing and hence retains more privileges. When serving public data, it is generally preferable to use setup 2 instead.

Note that rsync(1) assumes the server process understands all flags that it sends, i.e. is running the same version on client and server, or at least a compatible-enough version. You can either specify --protocol=27 on the client, or use setup 4, which negotiates the protocol version, side-stepping possible compatibility gaps between rsync clients and gokr-rsyncd.

Example:

  • Server will be started via SSH
  • Client: rsync --rsync-path=gokr-rsyncd webserver:path

Setup 4: SSH (daemon)

This setup is more reliable than setup 3 because the rsync protocol version will be negotiated between client and server. This setup is slightly inconvenient because it requires a config file to be present on the server in ~/.config/gokr-rsyncd.toml.

Example:

  • Server will be started via SSH
  • Client: rsync -e ssh --rsync-path=gokr-rsyncd rsync://webserver/module/path

Limitations

Bandwidth

In my tests, gokr-rsyncd can easily transfer data at > 6 Gbit/s. The current bottleneck is the MD4 algorithm itself (not sure whether in the “tridge” rsync client, or in gokr-rsyncd). Implementing support for more recent protocol versions would help here, as these include hash algorithm negotiation with more recent choices.

Protocol related limitations

  • xattrs (including acls) was introduced in rsync protocol 30, so is currently not supported.

Supported environments and privilege dropping

Supported environments:

  1. systemd (Linux)
  2. Docker (Linux)
  3. privileged Linux
  4. privileged non-Linux

In all environments, the default instructions will take care that:

  • (On Linux only) Only configured rsync modules from the host file system are mounted read-only into a Linux mount namespace for gokr-rsyncd, to guard against data modification and data exfiltration.
  • gokr-rsyncd is running without privileges, as user nobody, to limit the scope of what an attacker can do when exploiting a vulnerability.

Known gaps:

  • gokr-rsyncd does not guard against denial of service attacks, i.e. consuming too many resources (connections, bandwidth, CPU, …).

systemd (unprivileged)

We provide a gokr-rsyncd.socket and gokr-rsyncd.service file for systemd. These files enables most of systemd’s security features. You can check by running systemd-analyze security gokr-rsyncd.service, which should result in an exposure level of “0.2 SAFE” as of systemd 249 (September 2021).

First, configure your server flags by creating a systemd service override file:

systemctl edit gokr-rsyncd.service

In the opened editor, change the file to:

[Service]
ExecStart=
ExecStart=/usr/bin/gokr-rsyncd --gokr.modulemap=pwd=/etc/tmpfiles.d

Close the editor and install the service using:

systemctl enable --now gokr-rsyncd.socket

Additional hardening recommendations:

Docker (unprivileged)

We provide a Dockerfile for gokr-rsyncd.

docker run \
  --read-only \
  -p 127.0.0.1:8730:8730 \
  -v /etc/tmpfiles.d:/srv/rsync:ro \
  stapelberg/gokrazy-rsync:latest \
    --gokr.modulemap=pwd=/srv/rsync

Additional hardening recommendations:

privileged Linux (including gokrazy.org)

When started as root on Linux, gokr-rsyncd will create a Linux mount namespace, mount all configured rsync modules read-only into the namespace, then change into the namespace using chroot(2) and drop privileges using setuid(2).

Tip: you can verify which file system objects the daemon process can see by using ls -l /proc/$(pidof gokr-rsyncd)/root/.

Additional hardening recommendations:

privileged non-Linux (e.g. Mac)

When started as root on non-Linux (e.g. Mac), gokr-rsyncd will drop privileges using setuid(2).

unprivileged with write permission (e.g. from a shell)

To prevent accidental misconfiguration, gokr-rsyncd refuses to start when it detects that it has write permission in any configured rsync module.

More Repositories

1

gokrazy

turn your Go program(s) into an appliance running on the Raspberry Pi 3, Pi 4, Pi 5, Pi Zero 2 W, or amd64 PCs!
Go
3,220
star
2

breakglass

emergency/debugging access for gokrazy installations
Go
68
star
3

kernel

Linux kernel images for gokrazy
Go
57
star
4

tools

this repository contains the gok CLI tool of gokrazy
Go
45
star
5

stat

program to visualize resource usage, very similar to Dag Wieers’s dstat default output
Go
36
star
6

wifi

WiFi daemon for connecting to WiFi networks (encrypted or unencrypted) with gokrazy
Go
35
star
7

fbstatus

fbstatus graphically shows the gokrazy system status on the Linux frame buffer, which is typically available via HDMI when running on a Raspberry Pi or a PC.
Go
25
star
8

firmware

bootloader files for the Raspberry Pi 3, Pi 4 and Pi Zero 2 W
Go
21
star
9

syslogd

gokrazy remote syslog server daemon: writes to disk what it receives via remote syslog
Go
14
star
10

selfupdate

Go
14
star
11

bluetooth

Package for https://gokrazy.org/ to set up Bluetooth
Go
12
star
12

podman

static podman build for gokrazy
Go
11
star
13

mkfs

gokrazy mkfs is a program to create an ext4 file system on the gokrazy perm partition
Go
10
star
14

tsnetdemo

tailscale tsnet demo program for https://gokrazy.org
Go
9
star
15

freeze

Little helper to create tar balls of an executable together with its ELF shared library dependencies. Useful for prototyping with gokrazy: https://gokrazy.org/prototyping/
Go
9
star
16

internal

Go libraries for use within the gokrazy project
Go
8
star
17

gus

⚠ WORK IN PROGRESS ⚠ gokrazy Update System (GUS) ⚠ WORK IN PROGRESS ⚠
Go
7
star
18

serial-busybox

A bundled statically compiled standalone busybox binary, for debugging on the serial port
Dockerfile
6
star
19

rpi-eeprom

EEPROM files for the Raspberry Pi 4
Go
5
star
20

bakery

continuous integration testing infrastructure
Go
5
star
21

gdns

router7 dyndns entries for your gokrazy services
Go
5
star
22

iptables

frozen version of iptables
Makefile
4
star
23

updater

Go package which implements updating the different parts of a running gokrazy installation (boot/root file systems and MBR).
Go
4
star
24

gokapi

OpenAPI descriptions and generated (Go) client for the gokrazy on-device and GUS API
Go
4
star
25

nsenter

Go
3
star
26

hello-gpio

demo program to illustrate GPIO with periph.io on gokrazy
Go
3
star
27

autoupdate

Go
3
star
28

timestamps

Package which provides boot/root build timestamps on :1341/metrics (for Prometheus)
Go
2
star
29

hello

simple hello world program (used as an example)
Go
2
star
30

kernel.rpi

(Raspberry Pi) Linux kernel build for gokrazy
Go
2
star
31

kernel.arm64

(Upstream) Linux kernel build for arm64 (ARM64) PCs/VMs for gokrazy
Go
1
star
32

imaged

Go
1
star
33

gokrazy.github.io

OLD website archive repository. The website is now part of https://github.com/gokrazy/gokrazy, directory website
HTML
1
star
34

kernel.amd64

(Upstream) Linux kernel build for amd64 (x86_64) PCs/VMs for gokrazy
Go
1
star