• Stars
    star
    5
  • Rank 2,777,029 (Top 57 %)
  • Language
    Crystal
  • License
    MIT License
  • Created about 8 years ago
  • Updated over 7 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

import and export data from Redis in TSV format

redis-tsv Build Status

import and export data from Redis in TSV format

% redis-tsv export > backup.tsv
% redis-tsv import backup.tsv
% redis-tsv keys > keys.list

Benchmark

  • Bench: import, export and keys with count option (default: 1000)
COUNT Time QPS Unix Command
import 1000 21.9 sec 457567 redis-tsv import -c 1000 10M.tsv
10000 19.6 sec 509422 redis-tsv import -c 10000 10M.tsv
100000 16.2 sec 617951 redis-tsv import -c 100000 10M.tsv
export 1000 465.1 sec 21500 redis-tsv export -c 1000 > 10M.tsv
10000 90.9 sec 110002 redis-tsv export -c 10000 > 10M.tsv
100000 17.2 sec 582429 redis-tsv export -c 100000 > 10M.tsv
keys 1000 10.9 sec 915983 redis-tsv keys -c 1000 > keys.list
10000 8.9 sec 1124260 redis-tsv keys -c 10000 > keys.list
100000 7.6 sec 1311586 redis-tsv keys -c 100000 > keys.list
  • CPU: Intel(R) Core(TM) i7-4800MQ CPU @ 2.70GHz
  • TSV: 10M.tsv (10M entries where key = val = "%08d")
00000001        00000001
00000002        00000002
...
10000000        10000000

Installation

crystal deps  # for the first time only
make
cp bin/redis-tsv ~/bin/

Usage

Bulk operations

  • import
redis-tsv import foo.tsv
redis-tsv -d, import foo.csv
  • export
redis-tsv export > foo.tsv
redis-tsv -d, export > foo.csv
  • keys
redis-tsv keys > keys.list

options

  • -c grows performance, but uses much memory and locks server.
redis-tsv export -c 1000  > foo.tsv  # 1m13s (default)
redis-tsv export -c 10000 > foo.tsv  # 7m28s (6 times faster)

(entry count: 10M)

information

util commands for easy access to INFO

  • count : show a number of keys
% redis-tsv count
10000000
  • version : show the redis version
% redis-tsv version
3.0.6
  • role : show the role of replication
% redis-tsv role
master
  • ping : execute PING command
% redis-tsv ping
PONG
  • info (INFO result itself)
% redis-tsv info
# Server
redis_version:3.0.6
redis_git_sha1:00000000
...

Test

  • CAUTION: This spec flushes your redis db on localhost:6379.
% crystal spec

TODO

v0.3.0

  • import to redis-cluster

v0.4.0

  • keys from redis-cluster

v0.5.0

  • export from redis-cluster

Contributing

  1. Fork it ( https://github.com/maiha/redis-tsv.cr/fork )
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request

Contributors

  • maiha maiha - creator, maintainer

More Repositories

1

rcm.cr

Redis Cluster Manager in Crystal
Crystal
45
star
2

kafka.cr

kafka utilities written in crystal
Crystal
34
star
3

rocksdb.cr

RocksDB client for Crystal
Crystal
34
star
4

crystal-examples

This has been completely rewritten in 2019. The old implementation has moved to the alpha branch.
Crystal
31
star
5

crt.cr

Bindings for libncursesw and crt class
Crystal
28
star
6

try.cr

Try monad for crystal
Crystal
26
star
7

pcap.cr

Crystal bindings for libpcap
Crystal
24
star
8

crb

A cucumber console that offers cucumber world enviroment on irb
Ruby
20
star
9

xq.cr

Command-line XML processor inspired by `jq`
Crystal
17
star
10

redis-cluster-benchmark.cr

Benchmark utils for Redis Cluster
Crystal
16
star
11

pretty.cr

Something attentive, conservative and pretty stuff for Crystal
Crystal
16
star
12

jq.cr

thin JSON::Any wrapper to emulate jq for crystal
Crystal
16
star
13

must

a runtime specification tool
Ruby
15
star
14

grafana-redis.cr

Grefana Datasource for Redis storage
Crystal
15
star
15

gatling-amqp

Gatling AMQP support
Scala
15
star
16

typed

A Ruby library for Typed variables
Ruby
14
star
17

redis-cluster.cr

redis-cluster library for Crystal
Crystal
13
star
18

htpasswd

ActionPack plugin for authorizing users with HTTP basic/digest authentications
Ruby
12
star
19

memoized.cr

Time-based memoized library for Crystal
Crystal
11
star
20

tokyocabinet

unofficial repository for TokyoCabinet
C
11
star
21

pg-copy-ch

Simply copy the current PostgreSQL data to ClickHouse
Crystal
10
star
22

clickhouse-cluster-examples

Examples of the cluster topologies in ClickHouse
Makefile
10
star
23

facebook.cr

CLI for the Facebook Marketing API.
Crystal
9
star
24

ircbot

old fashioned irc bot
Ruby
9
star
25

rrr

Ruby programming with RR
Ruby
9
star
26

mod_cband

mod_cband Apache2 module
C
8
star
27

dsl_accessor

Ruby
8
star
28

ccp

A Ruby library for Composite Command Programming
Ruby
7
star
29

hq.cr

a simple wrapper for crystal-xml
Crystal
7
star
30

sexy_auto_complete

Rails plugin for better auto_complete, especially it gets along with ActiveScaffold nested forms
Ruby
7
star
31

comment-spec.cr

Comment driven spec builder for Crystal
Crystal
6
star
32

shard.cr

compile-time shard.yml reader for Crystal
Crystal
6
star
33

http-gate

Quite simple http port forwarder
Crystal
6
star
34

auto_nested_layouts

Use multiple nested layout files in Rails
Ruby
6
star
35

curl.cr

high level curl library for crystal
Crystal
6
star
36

open-uri-mapping

a wrapper to open-uri that offers filename mapping, which is useful for test
Ruby
5
star
37

mjs

A Ruby library that offers quite easy-to-use Ajax actions like RJS with jQuery
Ruby
5
star
38

css_parser

hpricot helper that scrapes html easily by parser class defined css selector
Ruby
5
star
39

protobuf-storage.cr

A handy local storage library for Protobuf
Crystal
5
star
40

rows_logger

ActiveRecord plugin for writing rows count information in query log
Ruby
5
star
41

cuke-utils

Goodies for cuke
Ruby
5
star
42

chawan

A cup for chasen that provides an easy to use for extracting Japanese
Ruby
5
star
43

rails_log

Rails log analyzer application with Ext-JS
Ruby
5
star
44

request_id

Rails plugin that adds a new method (Controller#request_id) to distinguish requests
Ruby
4
star
45

ext

ActionPack plugin for using Ext-JS
Ruby
4
star
46

tdiary_google_calendar_plugin

A tdiary plugin that shows event list from google calendar
Ruby
4
star
47

unicode_japanese

Japanese charset converter especially for hankaku and zenkaku characters
Ruby
4
star
48

selenium-chrome.cr

A handy and thin wrapper for `selenium-webdriver-crystal`
Crystal
4
star
49

kafka-utils

Simple Kafka Utils in Scala
Scala
4
star
50

clickhouse.cr

ClickHouse client for Crystal
Crystal
4
star
51

rubyist

A scala goodies for rubyist
Scala
4
star
52

mvc-benchmark-2010

A sample applications to bench MVC frameworks
4
star
53

merb_inspector

no needs to prepare views, scaffold and pagination. just inspect it
Ruby
4
star
54

scoped_access

restrict database access by using with_scope
Ruby
4
star
55

pon.cr

Maiha's private ORM for Crystal
Crystal
4
star
56

tokyotyrant

unofficial repository for TokyoTyrant
4
star
57

ohm-arfreaks

Ohm::Model extensions for AR freaks
Ruby
3
star
58

night-time

A ruby gem that treats over-midnight time easily
Ruby
3
star
59

sexy_actions

defining actions by block
Ruby
3
star
60

hash-path

path accessor to hierarchical hash
Ruby
3
star
61

http-mock

Real http server for stubbing and expectations in Scala
Scala
3
star
62

innodb-cluster-docker

a helper for running InnoDB Cluster on docker
Shell
3
star
63

named_options

Ruby
3
star
64

merb_background

a merb slice for background-fu
Ruby
3
star
65

2ch

local image storage application for 2ch
JavaScript
3
star
66

rack_format_response

A Rack middleware for automatically formatting response body
Ruby
3
star
67

opts.cr

a wrapper for OptionParser to provide default values and handy args
Crystal
3
star
68

acts_with_comma

Rails plugin that accepts numeric value with comma in AR#setter and text_field
Ruby
3
star
69

var.cr

`Object.var` macro for Crystal
Crystal
3
star
70

merb_rjs

Merb plugin that enables you to use 'page' object in your controllers like RJS of Rails
3
star
71

redisniffer

Sniff redis packets and summarize count of commands
Crystal
3
star
72

genspec

A Rails utils that automatically generates a rspec file for the latest action written in a log file
3
star
73

dm-ys

a DataMapper extension that uses html table as its schema and data powerfully like YunkerStar
Ruby
3
star
74

twitter-ads.cr

Twitter Ads API SDK for Crystal
Crystal
2
star
75

crystal-init.sh

a simple wrapper for crystal init command
Shell
2
star
76

crc16.cr

crc16 for Crystal
Crystal
2
star
77

james-bond

James is a bond framework for web development
Ruby
2
star
78

yak-farm

A farm view for Yak
2
star
79

lmdb.cr

Lmdb client for Crystal
Crystal
2
star
80

scalerity

Scalerity = Scala + Celerity
Scala
2
star
81

record_dumper

ActiveRecord plugin for printing record value to various formats
Ruby
2
star
82

hosts_access

Rails/Merb plugin that controls host access like hosts.allow
Ruby
2
star
83

web_api

Basic classes for general Web API
Ruby
2
star
84

toml-config.cr

TOML::Config class for handy use of crystal-toml
Crystal
2
star
85

tokyocabinet.cr

TokyoCabinet client for Crystal
Crystal
2
star
86

clickhouse-table

A standalone ClickHouse table manager that provides partial data updates
Crystal
2
star
87

optionize

method arguments utils
2
star
88

cmds.cr

Yet another CLI Builder library for Crystal
Crystal
2
star
89

base32.cr

Base32 Encoding for Crockford's Base32
Crystal
2
star
90

edd

Error driven development plugin for Rails
2
star
91

wildcard

A ruby library to expand wildcard string like shell command line
2
star
92

active_record_view

Ruby
2
star
93

active_seven

ActiveRecord plugin for 7th normal form
Ruby
2
star
94

include_for

Ruby
2
star
95

sandboxed_methods

Avoid conflicting method and variable names between modules
Ruby
2
star
96

akka-watch-child

study: to know whether a parent actor will watch its child automatically or not.
Scala
2
star
97

neo4j

Neo4j manager that provides many commands such as load,dump,touch,inspect,list
2
star
98

migration2

Yet another migartion library for ActiveRecord
Ruby
2
star
99

habto

ActiveRecord plugin for habtm utils
Ruby
2
star
100

active-scaffold.cr

ActiveScaffold for Amber on Crystal
CSS
2
star