• Stars
    star
    162
  • Rank 232,284 (Top 5 %)
  • Language
    Ruby
  • Created about 9 years ago
  • Updated about 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

ActiveRecord extension to kick the speed of allocating ActiveRecord object

LightRecord

Build Status

ActiveRecord extension to kick the speed of allocating ActiveRecord object, and it allows fetching DB rows sequentially to reduce memory usage

Supports: Rails 5, 6 and 7

Note: it overrides some internal methods of active record, and in some rare cases it not work correctly, please test it well in your project before using it in production

How it works

It provides functionality to load ActiveRecord records with patched attribute related methods. This make AR objects as read-only but it makes up to 5 times less object allocations.

Each time when you retrieve objects via .light_records it will create anonymous class to work with given set of attributes.

  LightRecord Extension Class
              ↓
        Your AR Model
              ↓
      ActiveRecord::Base

Installation

gem 'light_record', github: 'paxa/light_record'

scope.light_records

records = User.limit(1_000_000).light_records
records # => array of records. Very fast and very memory efficient

Idea is to skip all magic related to attributes and object initialization. This creates new class inherited from your model. That allows us to create only one extra object when we initialize new record.

Simply it become something like this:

class User_light_record < User
  def initialize(attributes)
    @attributes = attributes # hash of data "as is" from database library
  end

  def email
    @attributes[:email]
  end
end

scope.light_records_each

Other method: .light_records_each, it will utilize stream: true feature from mysql2 client. So it will initialize objects one by one for every interaction:

User.limit(1_000_000).light_records_each do |user|
  user.do_something
end

This allow you to interate big amount of data without using find_each or find_in_batches because with light_records_each it will use very low memory. Or allow you to use find_in_batches with bigger batch size

* Please note that time will be as a ruby Time object, instead of TimeWithZone. To make it in correct time zone you can call it as:

record.created_at.in_time_zone(Time.zone)

Benchmarks

Still on a way, but I try to use in some project and it gives 3-5 times improvement, and 2-3 times less memory usage


Sometimes this can break functionality because it will override attribute methods and disable some of features in activerecord.

There is mechanism to override attribute methods created by LightRecord:

class User < ActiveRecord::Base
  # this module will be included in extending class when we use light_records and light_records_each
  module LightRecord
    def sometihng
    end

    def success
      attributes[:success] == 1
    end

    def success_time
      return attributes[:success_time] unless attributes[:success_time]
      @success_time ||= attributes[:success_time].in_time_zone(Time.zone)
    end

    def success_time=(val)
      @success_time = nil
      super(val)
    end
  end
end

Note: when you use LightRecord instances it will break type casting

This gem supports MySQL and PostgreSQL

More Repositories

1

postbird

Open source PostgreSQL GUI client for macOS, Linux and Windows
JavaScript
1,537
star
2

fast_excel

Ultra Fast Excel Writer for Ruby
C
314
star
3

green_monkey

Rails 4 & 5 microdata helpers
Ruby
54
star
4

russian_mutators

Гем для автоматического построения формы множественного числа для существительных
Ruby
22
star
5

kt

Docker image for "kt" - kafka command line tool
Shell
18
star
6

mysql.d

mysql library binding for D programming language
D
17
star
7

waithook

Rust
9
star
8

stimulus-rails

JavaScript
8
star
9

mida_vocabulary

Ruby
4
star
10

slimerjs-docker

Docker image for slimerjs + firefox, runs on alpine linux
4
star
11

semantic_data

Microdata online viewer
Ruby
3
star
12

can_write

Set permissions for folders and files
2
star
13

sass-classic-bundle

sass textmate bundle with useful shortcuts
1
star
14

vertexdb_site

Ruby
1
star
15

sunraise_test_app

simple rails (2) application for sunraise testing
Ruby
1
star
16

Boo

Boo
Ruby
1
star
17

sunraise

Super-fast and simple rails deployment
Ruby
1
star
18

HUI_name

generates form elements for accepts_nested_attributes_for in Rails
JavaScript
1
star
19

jsus

Javascript packager & dependency resolver, and G-d knows what's more
Ruby
1
star
20

sass_component

Ruby
1
star
21

server-compare

Compare remote servers configuration and save it to the git
Ruby
1
star
22

process_memory

Get process current memory usage for ruby
Ruby
1
star
23

lsd_rails

Ruby
1
star
24

httpit

HTTP sever for directory index with extra features
Ruby
1
star
25

docker-cake

Command line program to inspect docker images size
Ruby
1
star
26

pg_cluster

vagrant script to setup postgresql cluster
Shell
1
star
27

forefeed

foreigin feed reader
JavaScript
1
star
28

ruby-vertexdb

vertexdb ruby interface
Ruby
1
star