julien-duponchelle/python-mysql-replication julien-duponchelle/python-mysql-replication

  • Stars
    star
    2,314
  • Rank 19,891 (Top 0.4 %)
  • Language
    Python
  • Created about 12 years ago
  • Updated 4 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
julien-duponchelle/python-mysql-replication
github

julien-duponchelle

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Pure Python Implementation of MySQL replication protocol build on top of PyMYSQL

python-mysql-replication

Β 

Pure Python Implementation of MySQL replication protocol build on top of PyMYSQL. This allow you to receive event like insert, update, delete with their datas and raw SQL queries.

Use cases

  • MySQL to NoSQL database replication
  • MySQL to search engine replication
  • Invalidate cache when something change in database
  • Audit
  • Real time analytics

Documentation

A work in progress documentation is available here: https://python-mysql-replication.readthedocs.org/en/latest/

Instruction about building documentation is available here: https://python-mysql-replication.readthedocs.org/en/latest/developement.html

Installation

pip install mysql-replication

Mailing List

You can get support and discuss about new features on: https://groups.google.com/d/forum/python-mysql-replication

Project status

The project is test with:

  • MySQL 5.5, 5.6 and 5.7
  • Python 3.3, 3.4, 3.5 and 3.6 (3.2 is not supported)
  • PyPy (really faster than the standard Python interpreter)

The project is used in production for critical stuff in some medium internet corporations. But all use case as not been perfectly test in the real world.

Limitations

https://python-mysql-replication.readthedocs.org/en/latest/limitations.html

Projects using this library

MySQL server settings

In your MySQL server configuration file you need to enable replication:

[mysqld]
server-id		 = 1
log_bin			 = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size  = 100M
binlog-format    = row #Very important if you want to receive write, update and delete row events

Examples

All examples are available in the examples directory

This example will dump all replication events to the console:

from pymysqlreplication import BinLogStreamReader

mysql_settings = {'host': '127.0.0.1', 'port': 3306, 'user': 'root', 'passwd': ''}

stream = BinLogStreamReader(connection_settings = mysql_settings, server_id=100)

for binlogevent in stream:
    binlogevent.dump()

stream.close()

For this SQL sessions:

CREATE DATABASE test;
use test;
CREATE TABLE test4 (id int NOT NULL AUTO_INCREMENT, data VARCHAR(255), data2 VARCHAR(255), PRIMARY KEY(id));
INSERT INTO test4 (data,data2) VALUES ("Hello", "World");
UPDATE test4 SET data = "World", data2="Hello" WHERE id = 1;
DELETE FROM test4 WHERE id = 1;

Output will be:

=== RotateEvent ===
Date: 1970-01-01T01:00:00
Event size: 24
Read bytes: 0

=== FormatDescriptionEvent ===
Date: 2012-10-07T15:03:06
Event size: 84
Read bytes: 0

=== QueryEvent ===
Date: 2012-10-07T15:03:16
Event size: 64
Read bytes: 64
Schema: test
Execution time: 0
Query: CREATE DATABASE test

=== QueryEvent ===
Date: 2012-10-07T15:03:16
Event size: 151
Read bytes: 151
Schema: test
Execution time: 0
Query: CREATE TABLE test4 (id int NOT NULL AUTO_INCREMENT, data VARCHAR(255), data2 VARCHAR(255), PRIMARY KEY(id))

=== QueryEvent ===
Date: 2012-10-07T15:03:16
Event size: 49
Read bytes: 49
Schema: test
Execution time: 0
Query: BEGIN

=== TableMapEvent ===
Date: 2012-10-07T15:03:16
Event size: 31
Read bytes: 30
Table id: 781
Schema: test
Table: test4
Columns: 3

=== WriteRowsEvent ===
Date: 2012-10-07T15:03:16
Event size: 27
Read bytes: 10
Table: test.test4
Affected columns: 3
Changed rows: 1
Values:
--
* data : Hello
* id : 1
* data2 : World

=== XidEvent ===
Date: 2012-10-07T15:03:16
Event size: 8
Read bytes: 8
Transaction ID: 14097

=== QueryEvent ===
Date: 2012-10-07T15:03:17
Event size: 49
Read bytes: 49
Schema: test
Execution time: 0
Query: BEGIN

=== TableMapEvent ===
Date: 2012-10-07T15:03:17
Event size: 31
Read bytes: 30
Table id: 781
Schema: test
Table: test4
Columns: 3

=== UpdateRowsEvent ===
Date: 2012-10-07T15:03:17
Event size: 45
Read bytes: 11
Table: test.test4
Affected columns: 3
Changed rows: 1
Affected columns: 3
Values:
--
* data : Hello => World
* id : 1 => 1
* data2 : World => Hello

=== XidEvent ===
Date: 2012-10-07T15:03:17
Event size: 8
Read bytes: 8
Transaction ID: 14098

=== QueryEvent ===
Date: 2012-10-07T15:03:17
Event size: 49
Read bytes: 49
Schema: test
Execution time: 1
Query: BEGIN

=== TableMapEvent ===
Date: 2012-10-07T15:03:17
Event size: 31
Read bytes: 30
Table id: 781
Schema: test
Table: test4
Columns: 3

=== DeleteRowsEvent ===
Date: 2012-10-07T15:03:17
Event size: 27
Read bytes: 10
Table: test.test4
Affected columns: 3
Changed rows: 1
Values:
--
* data : World
* id : 1
* data2 : Hello

=== XidEvent ===
Date: 2012-10-07T15:03:17
Event size: 8
Read bytes: 8
Transaction ID: 14099

Tests

When it's possible we have a unit test.

More information is available here: https://python-mysql-replication.readthedocs.org/en/latest/developement.html

Changelog

https://github.com/noplay/python-mysql-replication/blob/master/CHANGELOG

Similar projects

Special thanks

Contributors

Major contributor:

Other contributors:

Thanks to GetResponse for their support

Licence

Copyright 2012-2023 Julien Duponchelle

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

More Repositories

1

docker-osx

Fast and easy installation of Docker on OS X
Shell
1,028
star
2

scrapy-mongodb

Mongodb support for scrapy
Python
102
star
3

scrapy-elasticsearch

A scrapy pipeline which send items to Elastic Search server
Python
95
star
4

scrapy-graphite

Output scrapy statistics to graphite/carbon
Python
53
star
5

json-api-doc

JSON API parser returning a simple Python dictionary
Python
26
star
6

scrapy-dot

Export a graph of link between crawled items by scrapy in dot file format.
Python
26
star
7

scrapy-couchdb

A scrapy pipeline for couchdb
Python
17
star
8

situation-room

Pilot a tactical dashboard in a browser from another browser
JavaScript
16
star
9

vscode-rainbow-end

This extension allows to identify keyword / end with colours.
TypeScript
14
star
10

sinatra-api-template

Sample REST API build with sinatra and Datamapper
Ruby
7
star
11

norme.py

Controle de la norme de code d'EPITECH
Python
4
star
12

imageclipper

A tool to crop images manually fast
C
2
star
13

awesome-config

My Config File for Awesome windows manager. Optimize for dual screen
Lua
1
star
14

lagune

A zsh theme
1
star