• Stars
    star
    318
  • Rank 127,000 (Top 3 %)
  • Language
    Ruby
  • License
    MIT License
  • Created about 7 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A Gem for creating partial anonymized dumps of your database using your app model relations.

Gem Version Build Status Cult of Martians

EvilSeed

EvilSeed is a tool for creating partial anonymized dump of your database based on your app models.

Sponsored by Evil Martians

Motivation

Using production-like data in your staging environment could be very useful, especially for debugging intricate production bugs.

The easiest way to achieve this is to use production database backups. But that's not an option for rather large applications for two reasons:

  • production dump can be extremely large, and it just can't be dumped and restored in a reasonable time

  • you should care about sensitive data (anonymization).

EvilSeed aims to solve these problems.

Installation

Add this line to your application's Gemfile:

gem 'evil-seed', require: false

And then execute:

$ bundle

Or install it yourself as:

$ gem install evil-seed

Usage

Configuration

require 'evil_seed'

EvilSeed.configure do |config|
  # First, you should specify +root models+ and their +constraints+ to limit the number of dumped records:
  # This is like Forum.where(featured: true).all
  config.root('Forum', featured: true) do |root|
    # It's possible to remove some associations from dumping with pattern of association path to exclude
    #
    # Association path is a dot-delimited string of association chain starting from model itself:
    # example: "forum.users.questions"
    root.exclude(/\btracking_pixels\b/, 'forum.popular_questions')

    # It's possible to limit the number of included into dump has_many and has_one records for every association
    # Note that belongs_to records for all not excluded associations are always dumped to keep referential integrity.
    root.limit_associations_size(100)

    # Or for certain association only
    root.limit_associations_size(10, 'forum.questions')
  end

  # Everything you can pass to +where+ method will work as constraints:
  config.root('User', 'created_at > ?', Time.current.beginning_of_day - 1.day)

  # For some system-wide models you may omit constraints to dump all records
  config.root("Role") do |root|
    # Exclude everything
    root.exclude(/.*/)
  end

  # Transformations allows you to change dumped data e. g. to hide sensitive information
  config.customize("User") do |u|
    # Reset password for all users to the same for ease of debugging on developer's machine
    u["encrypted_password"] = encrypt("qwerty")
    # Reset or mutate other attributes at your convenience
    u["metadata"].merge!("foo" => "bar")
    u["created_at"] = Time.current
    # Please note that there you have only hash of record attributes, not the record itself!
  end

  # Anonymization is a handy DSL for transformations allowing you to transform model attributes in declarative fashion
  # Please note that model setters will NOT be called: results of the blocks will be assigned to
  config.anonymize("User") do
    name  { Faker::Name.name }
    email { Faker::Internet.email }
    login { |login| "#{login}-test" }
  end

  # You can ignore columns for any model. This is specially useful when working
  # with encrypted columns.
  #
  # This will remove the columns even if the model is not a root node and is
  # dumped via an association.
  config.ignore_columns("Profile", :name)
end

Creating dump

Just call the #dump method and pass a path where you want your SQL dump file to appear!

require 'evil_seed'
EvilSeed.dump('path/to/new_dump.sql')

Caveats, tips, and tricks

  1. Specify roots for dictionaries and system-wide models like Role at the top without constraints and with all associations excluded.

  2. Use exclude aggressively. You will be amazed, how much your app's models graph is connected. This, in conjunction with the fact that this gem traverses associations in deep-first fashion, sometimes leads to unwanted results: some records will get into dump even if you don't want them.

  3. Look at the resulted dump: there are some useful debug comments.

Database compatibility

This gem has been tested against:

  • PostgreSQL: any version that works with ActiveRecord should work
  • MySQL: any version that works with ActiveRecord should work
  • SQLite: 3.7.11 or newer is required (with support for inserting multiple rows at a time)

FIXME (help wanted)

  1. has_and_belongs_to_many associations are traversed in a bit nonintuitive way for end user:

    Association path for User.has_and_belongs_to_many :roles is user.users_roles.role, but should be user.roles

  2. Test coverage is poor

  3. Some internal refactoring is required

Standalone usage

If you want to use it as a standalone application, you can place exerything in a single file like this:

#!/usr/bin/env ruby

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'activerecord'
  gem 'evil-seed'
  gem 'mysql2'
end

# Describe your database layout with ActiveRecord models.
# See http://guides.rubyonrails.org/active_record_basics.html

class Category < ActiveRecord::Base
  has_many :translations, class_name: "Category::Translation"
end

class Category::Translation < ActiveRecord::Base
  belongs_to :category, inverse_of: :translations
end

# Configure evil-seed itself
EvilSeed.configure do |config|
  config.root("Category", "id < ?", 1000)
end

# Connect to your database.
# See http://guides.rubyonrails.org/configuring.html#configuring-a-database)
ActiveRecord::Base.establish_connection(ENV.fetch("DATABASE_URL"))

# Create dump in dump.sql file in the same directory as this script
EvilSeed.dump(File.join(__dir__, "dump.sql").to_s)

And launch it like so:

DATABASE_URL=mysql2://user:pass@host/db ruby path/to/your/script.rb

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/palkan/evil-seed.

License

The gem is available as open source under the terms of the MIT License.

More Repositories

1

lefthook

Fast and powerful Git hooks manager for any type of projects.
Go
3,962
star
2

mono

Free and open-source monospaced font from Evil Martians
1,978
star
3

terraforming-rails

Terraforming legacy Rails applications guides and tools
Ruby
724
star
4

oklch-picker

Color Picker for LCH
TypeScript
603
star
5

ruby-on-whales

Ruby on Whales example and templates
Ruby
267
star
6

fullstaq-ruby-docker

Docker image for Ruby build from Fullstaq packages based on Debian 10, 11, and 12.
Dockerfile
161
star
7

harmony

Harmony color palette
TypeScript
143
star
8

evil-client

Human-friendly DSL for writing HTTP(s) clients in Ruby
Ruby
104
star
9

activerecord-slotted_counters

Active Record extension providing slotted counters support
Ruby
98
star
10

fias

Ruby wrapper for the Russian FIAS database (Федеральная Информационная Адресная Система)
Ruby
82
star
11

chronicles-gql-martian-library

Ruby
71
star
12

chef-kubernetes

Google Kubernetes installer for ubuntu >= 16.04
Ruby
69
star
13

liquor

Liquor is a safe sandboxing compiling template language for Ruby
Ruby
59
star
14

figma-polychrom

Figma plugin for displaying the contrast level and font size recommendations according to the APCA method
TypeScript
58
star
15

graphql-connections

Additional implementations of cursor-based paginations for GraphQL Ruby gem.
Ruby
43
star
16

evil_chat

Code for "Evil Front: Modern Front-end in Rails" 3-part tutorial: https://evilmartians.com/chronicles/evil-front-part-1
Ruby
41
star
17

chef-prometheus-exporters

Ruby
38
star
18

chef-nginx

Chef Nginx recipes
Ruby
35
star
19

telephony

Evil Martians' telephony application that run our phone numbers
JavaScript
30
star
20

foundry

Foundry Compiler
OCaml
24
star
21

rubocoping-generator

Ruby application template to configure RuboCop with Standard and plugins
Ruby
14
star
22

furnace-xray

A visualizer for transformations of code in Static Single Assignment form based on the Furnace library.
CoffeeScript
11
star
23

cupid

Create, organize and send emails through ExactTarget SOAP API
Ruby
10
star
24

redis-proxy

Redis Proxy
Go
9
star
25

evil-struct

Nested structure with type constraints, based on the `dry-initializer` DSL
Ruby
9
star
26

omniauth-ebay-oauth

OmniAuth Strategy for eBay Apps (for using with eBay REST APIs)
Ruby
7
star
27

astro-typedoc

A tool for building astro-based documentation sites using Typescript types definitions and TSDocs.
JavaScript
7
star
28

zendesk

Proper Ruby wrapper around the Zendesk API
Ruby
6
star
29

catalyst-tutorial

Beyond Fashion Deep Learning With Catalyst post code
Python
6
star
30

monstro

Cult of Martians screensaver
Objective-C
5
star
31

foundry-lib

Foundry standard library and examples (public)
Fancy
5
star
32

github-tv

Forks your colleagues' open-source repos to your organization page
Ruby
4
star
33

chef-mdadm

mdadm recipe
Ruby
4
star
34

amplifr-node

JavaScript
3
star
35

helm-kubectl-docker

Dockerfile
2
star
36

homebrew-lefthook

Homebrew Formulae to lefthook
Ruby
2
star
37

kibana-logtrail

Kibana docker image with logtrail plugin.
Dockerfile
2
star
38

docker-curl

A simple docker image which contains curl (based on Alpine linux)
Dockerfile
1
star
39

charts

Public Helm charts
Smarty
1
star
40

foundry-web

Foundry website
JavaScript
1
star
41

gcloud-helm-docker

Deploy image for GKE with Helm & kubectl binaries.
Dockerfile
1
star
42

chef-consul-wrapper

Ruby
1
star
43

elasticsearch-k8s

1
star