• Stars
    star
    1,794
  • Rank 25,900 (Top 0.6 %)
  • Language
    Python
  • License
    Mozilla Public Li...
  • Created almost 11 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Run-Your-Own Firefox Sync Server

Run-Your-Own Firefox Sync Server

https://circleci.com/gh/mozilla-services/syncserver/tree/master.svg?style=svg https://img.shields.io/docker/automated/mozilla-services/syncserver.svg?style=flat-square

Note that this repository is no longer being maintained. Use this at your own risk, and with the understanding that it is not being maintained, work is being done on its replacement, and that no support or assistance will be offered.

This is an all-in-one package for running a self-hosted Firefox Sync server. It bundles the "tokenserver" project for authentication and the "syncstorage" project for storage, to produce a single stand-alone webapp.

Complete installation instructions are available at:

https://mozilla-services.readthedocs.io/en/latest/howtos/run-sync-1.5.html

Quickstart

The Sync Server software runs using python 2.7, and the build process requires make and virtualenv. You will need to have the following packages (or similar, depending on your operating system) installed:

  • python2.7
  • python2.7-dev
  • python-virtualenv
  • gcc and g++
  • make
  • libstdc++
  • libffi-dev
  • mysql-dev
  • musl-dev
  • ncurses-dev
  • openssl-dev

Take a checkout of this repository, then run "make build" to pull in the necessary python package dependencies:

$ git clone https://github.com/mozilla-services/syncserver
$ cd syncserver
$ make build

To sanity-check that things got installed correctly, do the following:

$ make test

Now you can run the server:

$ make serve

This should start a server on http://localhost:5000/.

Now go into Firefox's about:config page, search for a setting named "tokenServerURI", and change it to point to your server:

identity.sync.tokenserver.uri:  http://localhost:5000/token/1.0/sync/1.5

(Prior to Firefox 42, the TokenServer preference name for Firefox Desktop was "services.sync.tokenServerURI". While the old preference name will work in Firefox 42 and later, the new preference is recommended as the old preference name will be reset when the user signs out from Sync causing potential confusion.)

Firefox should now sync against your local server rather than the default Mozilla-hosted servers.

For more details on setting up a stable deployment, see:

https://mozilla-services.readthedocs.io/en/latest/howtos/run-sync-1.5.html

Customization

All customization of the server can be done by editing the file "syncserver.ini", which contains lots of comments to help you on your way. Things you might like to change include:

  • The client-visible hostname for your server. Edit the "public_url" key under the [syncerver] section.
  • The database in which to store sync data. Edit the "sqluri" setting under the [syncserver] section.
  • The secret key to use for signing auth tokens. Find the "secret" entry under the [syncserver] section and follow the instructions in the comment to replace it with a strong random key.

Database Backend Modules

If your python installation doesn't provide the "sqlite" module by default, you may need to install it as a separate package:

$ ./local/bin/pip install pysqlite2

Similarly, if you want to use a different database backend you will need to install an appropriate python module, e.g:

$ ./local/bin/pip install PyMySQL
$ ./local/bin/pip install psycopg2

Runner under Docker

Dockerhub Page

There is experimental support for running the server inside a Docker container. The docker image runs with UID/GID 1001/1001. Build the image like this:

$ docker build -t syncserver:latest .

Then you can run the server by passing in configuration options as environment variables, like this:

$ docker run --rm \
    -p 5000:5000 \
    -e SYNCSERVER_PUBLIC_URL=http://localhost:5000 \
    -e SYNCSERVER_SECRET=<PUT YOUR SECRET KEY HERE> \
    -e SYNCSERVER_SQLURI=sqlite:////tmp/syncserver.db \
    -e SYNCSERVER_BATCH_UPLOAD_ENABLED=true \
    -e SYNCSERVER_FORCE_WSGI_ENVIRON=false \
    -e SYNCSERVER_DEBUG_ENABLED=true \
    -e PORT=5000 \
    mozilla/syncserver:latest

or

$ docker run --rm \
    -p 5000:5000 \
    -e SYNCSERVER_PUBLIC_URL=http://localhost:5000 \
    -e SYNCSERVER_SECRET_FILE=<PUT YOUR SECRET KEY FILE LOCATION HERE> \
    -e SYNCSERVER_SQLURI=sqlite:////tmp/syncserver.db \
    -e SYNCSERVER_BATCH_UPLOAD_ENABLED=true \
    -e SYNCSERVER_FORCE_WSGI_ENVIRON=false \
    -e PORT=5000 \
    -v /secret/file/at/host:<PUT YOUR SECRET KEY FILE LOCATION HERE>  \
    mozilla/syncserver:latest

Don't forget to generate a random secret key to use in the SYNCSERVER_SECRET environment variable or mount your secret key file!

And you can test whether it's running correctly by using the builtin function test suite, like so:

$ /usr/local/bin/python -m syncstorage.tests.functional.test_storage \
    --use-token-server http://localhost:5000/token/1.0/sync/1.5

If you'd like a persistent setup, you can mount a volume as well:

$ docker run -d \
    -v /syncserver:/data \
    -p 5000:5000 \
    -e SYNCSERVER_PUBLIC_URL=http://localhost:5000 \
    -e SYNCSERVER_SECRET=<PUT YOUR SECRET KEY HERE> \
    -e SYNCSERVER_SQLURI=sqlite:////data/syncserver.db \
    -e SYNCSERVER_BATCH_UPLOAD_ENABLED=true \
    -e SYNCSERVER_FORCE_WSGI_ENVIRON=false \
    -e PORT=5000 \
    mozilla/syncserver:latest

Make sure that /syncserver is owned by 1001:1001

Docker Compose can also be used for structured deployments:

version: '3.7'
services:
    syncserver:
        container_name: syncserver
        image: mozilla/syncserver:latest
        volumes:
            - /syncserver:/data
        ports:
            - 5000:5000
        environment:
            SYNCSERVER_PUBLIC_URL: 'http://localhost:5000'
            SYNCSERVER_SECRET: '<PUT YOUR SECRET KEY HERE>'
            SYNCSERVER_SQLURI: 'sqlite:////data/syncserver.db'
            SYNCSERVER_BATCH_UPLOAD_ENABLED: 'true'
            SYNCSERVER_FORCE_WSGI_ENVIRON: 'false'
            PORT: '5000'
        restart: always

Removing Mozilla-hosted data

If you have previously uploaded Firefox Sync data to the Mozilla-hosted storage service and would like to remove it, you can use the following script to do so:

$ pip install PyFxA
$ python ./bin/delete_user_data.py [email protected]

Questions, Feedback

More Repositories

1

heka

DEPRECATED: Data collection and processing made easy.
Go
3,408
star
2

syncstorage-rs

Sync Storage server in Rust
Rust
964
star
3

hindsight

Hindsight - light weight data processing skeleton
C
670
star
4

screenshots

Firefox Screenshots: the best way to take screenshots on the web.
FreeMarker
620
star
5

socorro

Socorro is the Mozilla crash ingestion pipeline. It accepts and processes Breakpad-style crash reports. It provides analysis tools.
Python
584
star
6

lua_sandbox

Generic Lua sandbox for dynamic data analysis
C
226
star
7

ios-sync-client

A standalone iOS client for Firefox Sync
C
219
star
8

autopush

Python Web Push Server used by Mozilla
Python
218
star
9

autopush-rs

Push Server in Rust
Rust
196
star
10

Dockerflow

Cloud Services Dockerflow specification
JavaScript
192
star
11

google-cloud-rust

Google Cloud Client Library for Rust
Rust
176
star
12

autograph

Mozilla's digital signature service
Go
141
star
13

shavar-prod-lists

Shavar/tracking protection lists used in prod
Python
139
star
14

GitHub-Audit

INACTIVE - Collection of Tools & Procedures for double checking GitHub configurations
Python
138
star
15

loads

SUPERSEDED BY https://github.com/loads
Go
107
star
16

powerhose

Runs workers via zmq to perform any kind of task
Python
98
star
17

server-syncstorage

The SyncServer server software, as used by Firefox Sync
Python
87
star
18

websec-check

web security checklist for Firefox Services
72
star
19

lua_sandbox_extensions

Extension packages (sandboxes and modules) for the lua_sandbox project
Lua
72
star
20

cliquet

CLIQUET IS NOW DEPRECATED use kinto.core instead
Python
65
star
21

loop-server

The mozilla loop server
JavaScript
61
star
22

tokenserver

The Mozilla Token Server
Python
61
star
23

iprepd

Centralized IP reputation daemon
Go
56
star
24

axe-selenium-python

aXe Selenium Integration python package
Python
56
star
25

android-sync

An outdated mirror of services and related code for Firefox for Android. See gecko-dev/mozilla-central.
Java
52
star
26

tokenlib

generic support library for signed-token-based auth schemes
Python
50
star
27

firefox-send-tab-to-device

OBSOLETE. A Firefox add-on that uses Sync to send tabs to remote devices.
JavaScript
46
star
28

mozilla-pipeline-schemas

Schemas for Mozilla's data ingestion pipeline and data lake outputs
Python
46
star
29

queuey

Mozilla Message Queue
Python
42
star
30

syncclient

Python client for Firefox Sync
Python
42
star
31

pyramid_multiauth

stacked authentication policies for pyramid
Python
41
star
32

guardian-vpn-windows-deprecated

Mozilla VPN for Windows
C#
41
star
33

userplex

Propagate users from Mozilla's Person API to third party systems.
Go
40
star
34

fernet-rs

Fernet implementation in Rust
Rust
40
star
35

go-cose

go library for CBOR Object Signing and Encryption (COSE)
Go
40
star
36

requests-hawk

Hawk authentication strategy for the requests python library.
Python
39
star
37

megaphone

Firefox Global Broadcast API
Rust
38
star
38

python-dockerflow

A Python package to implement tools and helpers for Mozilla Dockerflow
Python
38
star
39

metlog-py

Python library for Services metrics logging
Python
37
star
40

readinglist

Reading List Server
Python
34
star
41

konfig

Yet another configuration object
Python
34
star
42

mozservices

Various utilities for Pyramid-based Mozilla applications
Python
34
star
43

reaper

Reaper culls leftover AWS resources
Go
34
star
44

go-syncstorage

INACTIVE - SyncStorage Server with more golang and less indexes!
Go
32
star
45

merino

Web service for Firefox Suggest
Rust
31
star
46

tecken

Mozilla Symbol Server
Python
31
star
47

data-pipeline

Mozilla Services Data Pipeline
Lua
30
star
48

heka-build

Heka build environment
Python
29
star
49

heka-py

DEPRECATED - Heka Python Library - DEPRECATED
Python
29
star
50

heka-mozsvc-plugins

Set of heka plugins in use by Mozilla Services
Go
28
star
51

aws-signing-proxy

signs http requests using AWS V4 signer
Go
27
star
52

foxsec-pipeline

Log analysis pipeline utilizing Apache Beam
Java
25
star
53

services-central-legacy

Sync working branch. `master` in this repository is stable, and tracks https://hg.mozilla.org/services/services-central. Other branches are owned by developers, and are subject to change.
C++
24
star
54

minidump-stackwalk

Socorro breakpad minidump stackwalker
C++
21
star
55

telescope

A dumb auditing service
Python
21
star
56

antenna

Antenna is the collector for the Socorro crash ingestion pipeline
Python
20
star
57

shavar

Tracking Protection update service for Firefox based on Safe Browsing protocol
Python
20
star
58

zktools

Zookeeper Tools
Python
19
star
59

contile

This is the back-end server for the Mozilla Tile Service (MTS)
Rust
19
star
60

tuxedo

An improved version of Mozilla's download load balancer Bouncer, with a user interface written in Django/Python.
Python
18
star
61

shavar-plugin-blocklist

❌ Firefox plugin blocklist
XSLT
18
star
62

demoapp

An empty app for the next-gen Services app
Python
17
star
63

marteau

client-server on the top of Funkload
Python
16
star
64

msisdn-gateway

An MSISDN based Authentication Server.
JavaScript
16
star
65

docs

Documentation for Mozilla Services Services
Makefile
16
star
66

tigerblood

Deprecated, use https://github.com/mozilla-services/iprepd
Go
15
star
67

readinglist-client

Readinglist client
JavaScript
14
star
68

ldappool

A Pool for python-ldap
14
star
69

buildhub

DEPRECATED: Mozilla Build Metadata Service
Python
13
star
70

skeleton

A Skeleton actix app
Rust
13
star
71

heka-node

DEPRECATED - This repository is no longer maintained. Please go over to : https://github.com/disqus/heka-node
JavaScript
13
star
72

deepspeech-server

Rust
13
star
73

go-bouncer

A Go version of the redirector portion of bouncer.
Go
13
star
74

zipalign

DEPRECATED - Golang implementation of Android's ZipAlign tool
Go
13
star
75

kinto-dist

Kinto Distribution for Mozilla Services
Python
12
star
76

server-full2

Experimental Run-Your-Own Sync2.0 Server
Python
12
star
77

shavar-list-creation

Script to transform the Disconnect block-list into Safebrowsing v2 format for Firefox Tracking Protection
Python
12
star
78

go-mozlogrus

DEPRECATED - A logging library which conforms to Mozilla's logging standard for logrus users.
Go
12
star
79

FindMyDevice

Find My Device - 🚨🚨This server is obsolete and unsupported.🚨🚨
Go
11
star
80

hindsight_admin

Hindsight Administration User Interface
JavaScript
11
star
81

walint

script to validate web apps
Python
11
star
82

pyramid_ipauth

a pyramid authentication policy based on remote ip address
Python
11
star
83

push-service

Top-level repository for the Push Service
11
star
84

macauthlib

low-level library for implementing MAC Access Authentication
Python
11
star
85

pyramid_whoauth

a pyramid authentication policy using repoze.who
Python
10
star
86

merino-py

Web Service for Firefox Suggest
Python
9
star
87

go-mozlog

A logging library which conforms to Mozilla's logging standard.
Go
9
star
88

oidc-gateway

A Docker container and Kubernetes Helm chart to gate access to upstream services.
Lua
9
star
89

redbarrel

JavaScript
9
star
90

iprepd-nginx

Openresty nginx module for integrating with iprepd
Python
9
star
91

services-test

Tools and test scripts used by the Mozilla Cloud Services team
JavaScript
9
star
92

nginx_moz_ingest

HTTP Data Pipeline Ingestion
C
9
star
93

logstash-metlog

Extensions to logstash for metlog
Ruby
9
star
94

shavar-list-creation-config

contains config files needed to run the jenkins task that builds the shavar lists
9
star
95

crashstats-tools

Command line tools and library for interacting with Crash Stats (https://crash-stats.mozilla.org/)
Python
9
star
96

topsites-proxy

Proxy server to track Top Sites default tile campaign attribution
JavaScript
8
star
97

hoverpad

DEPRECATED - Playing around with an Addons to sync some informations.
Elm
8
star
98

absearch

Python
8
star
99

updatebot

Automation for updating third party libraries for Firefox
Python
8
star
100

hawkauthlib

INACTIVE - low-level library for implementing MAC Access Authentication
Python
8
star