README
1.Introduction
This simpe program is for transfering data between two mongo nodes, including:
- data cloning;
- oplog syncing(applying one mongo databbase's oplog to another mongo database) ;
- oplog cloning(not applying, just storing the oplogs from source mongo database to another one)
2.Usage
-
Get the source code
$ git clone https://github.com/Qihoo360/mongosync.git
-
Compile
Enter the source code and typewrite:
$ make
after that, a "mongosync" named execution file appears in the $(MONGOSYNC_ROOT)/output directory, and also a "mongosync.conf" named config file
SP:
a. maybe some depenncies not installed in your machine, install them according to the tips（mostly sons, boost and ssl is needed）
b. don't use c++11 to compile this program, otherwise will occur error
-
Exectution
The program can launches with command line or config file，and the specified usage can be obtained by:
$ cd output $ ./mongosync --help Follow is the mongosync-surpported options: --help to get the help message -c conf.file use config file to start mongosync --src_srv arg the source mongodb server's ip port --src_user arg the source mongodb server's logging user --src_passwd arg the source mongodb server's logging password --src_auth_db arg the source mongodb server's auth db --src_use_mcr force source connection to use MONGODB-CR password machenism --dst_srv arg the destination mongodb server's ip port --dst_user arg the destination mongodb server's logging user --dst_passwd arg the destination mongodb server's logging password --dst_auth_db arg the destination mongodb server's auth db --dst_use_mcr force destination connection to use MONGODB-CR password machenism --db arg the source database to be cloned --dst_db arg the destination database --coll arg the source collection to be cloned --dst_coll arg the destination collection --oplog whether to sync oplog --raw_oplog whether to only clone oplog --op_start arg the start timestamp to sync oplog --op_end arg the start timestamp to sync oplog --dst_op_ns arg the destination namespace for raw oplog mode --no_index whether to clone the db or collection corresponding index --filter arg the bson format string used to filter the records to be transfered
3.Performance
Rough testing under the situation:
- Ping from this program's exectution's machine to the mongo db server's machine(two machines) is about 1.5ms；
- total about 36,400,000 documents；
- database size about 1.09GB
transfering time is about 379 seconds(even qps: 96.04k).
SP: The bottleneck is writing, So using multiple threads(multi-conn) to accelerate the speed of that(but also limited by database server's writing speed).