• Stars
    star
    786
  • Rank 57,890 (Top 2 %)
  • Language
    Ruby
  • License
    MIT License
  • Created about 2 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A database-backed ActiveSupport::Cache::Store

Solid Cache

Upgrading from v0.3.0 or earlier? Please see upgrading to version v0.4.x and beyond

Solid Cache is a database-backed Active Support cache store implementation.

Using SQL databases backed by SSDs we can have caches that are much larger and cheaper than traditional memory only Redis or Memcached backed caches.

Usage

To set Solid Cache as your Rails cache, you should add this to your environment config:

config.cache_store = :solid_cache_store

Solid Cache is a FIFO (first in, first out) cache. While this is not as efficient as an LRU cache, this is mitigated by the longer cache lifespan.

A FIFO cache is much easier to manage:

  1. We don't need to track when items are read
  2. We can estimate and control the cache size by comparing the maximum and minimum IDs.
  3. By deleting from one end of the table and adding at the other end we can avoid fragmentation (on MySQL at least).

Installation

Add this line to your application's Gemfile:

gem "solid_cache"

And then execute:

$ bundle

Or install it yourself as:

$ gem install solid_cache

Add the migration to your app:

$ bin/rails solid_cache:install:migrations

Then run it:

$ bin/rails db:migrate

Configuration

Configuration will be read from config/solid_cache.yml. You can change the location of the config file by setting the SOLID_CACHE_CONFIG env variable.

The format of the file is:

default:
  store_options: &default_store_options
    max_age: <%= 60.days.to_i %>
    namespace: <%= Rails.env %>
  size_estimate_samples: 1000

development: &development
  database: development_cache
  store_options:
    <<: *default_store_options
    max_size: <%= 256.gigabytes %>

production: &production
  databases: [production_cache1, production_cache2]
  store_options:
    <<: *default_store_options
    max_entries: <%= 256.gigabytes %>

For the full list of keys for store_options see Cache configuration. Any options passed to the cache lookup will overwrite those specified here.

Connection configuration

You can set one of database, databases and connects_to in the config file. They will be used to configure the cache databases in SolidCache::Record#connects_to.

Setting database to cache_db will configure with:

SolidCache::Record.connects_to database: { writing: :cache_db }

Setting databases to [cache_db, cache_db2] is the equivalent of:

SolidCache::Record.connects_to shards: { cache_db1: { writing: :cache_db1 },  cache_db2: { writing: :cache_db2 } }

If connects_to is set it will be passed directly.

If none of these are set, then Solid Cache will use the ActiveRecord::Base connection pool. This means that cache reads and writes will be part of any wrapping database transaction.

Engine configuration

There are three options that can be set on the engine:

  • executor - the Rails executor used to wrap asynchronous operations, defaults to the app executor
  • connects_to - a custom connects to value for the abstract SolidCache::Record active record model. Required for sharding and/or using a separate cache database to the main app. This will overwrite any value set in config/solid_cache.yml
  • size_estimate_samples - if max_size is set on the cache, the number of the samples used to estimates the size.

These can be set in your Rails configuration:

Rails.application.configure do
  config.solid_cache.size_estimate_samples = 1000
end

Cache configuration

Solid Cache supports these options in addition to the standard ActiveSupport::Cache::Store options.

  • error_handler - a Proc to call to handle any ActiveRecord::ActiveRecordErrors that are raises (default: log errors as warnings)
  • expiry_batch_size - the batch size to use when deleting old records (default: 100)
  • expiry_method - what expiry method to use thread or job (default: thread)
  • expiry_queue - which queue to add expiry jobs to (default: default)
  • max_age - the maximum age of entries in the cache (default: 2.weeks.to_i). Can be set to nil, but this is not recommended unless using max_entries to limit the size of the cache.
  • max_entries - the maximum number of entries allowed in the cache (default: nil, meaning no limit)
  • max_size - the maximum size of the cache entries (default nil, meaning no limit)
  • cluster - a Hash of options for the cache database cluster, e.g { shards: [:database1, :database2, :database3] }
  • clusters - and Array of Hashes for multiple cache clusters (ignored if :cluster is set)
  • active_record_instrumentation - whether to instrument the cache's queries (default: true)
  • clear_with - clear the cache with :truncate or :delete (default truncate, except for when Rails.env.test? then delete)
  • max_key_bytesize - the maximum size of a normalized key in bytes (default 1024)

For more information on cache clusters see Sharding the cache

Cache expiry

Solid Cache tracks writes to the cache. For every write it increments a counter by 1. Once the counter reaches 50% of the expiry_batch_size it adds a task to run on a background thread. That task will:

  1. Check if we have exceeded the max_entries or max_size values (if set) The current entries are estimated by subtracting the max and min IDs from the SolidCache::Entry table. The current size is estimated by sampling the entry byte_size columns.
  2. If we have it will delete expiry_batch_size entries
  3. If not it will delete up to expiry_batch_size entries, provided they are all older than max_age.

Expiring when we reach 50% of the batch size allows us to expire records from the cache faster than we write to it when we need to reduce the cache size.

Only triggering expiry when we write means that the if the cache is idle, the background thread is also idle.

If you want the cache expiry to be run in a background job instead of a thread, you can set expiry_method to :job. This will enqueue a SolidCache::ExpiryJob.

Using a dedicated cache database

Add database configuration to database.yml, e.g.:

development:
  cache:
    database: cache_development
    host: 127.0.0.1
    migrations_paths: "db/cache/migrate"

Create database:

$ bin/rails db:create

Install migrations:

$ bin/rails solid_cache:install:migrations

Move migrations to custom migrations folder:

$ mkdir -p db/cache/migrate
$ mv db/migrate/*.solid_cache.rb db/cache/migrate

Set the engine configuration to point to the new database:

# config/solid_cache.yml
production:
  database: cache

Run migrations:

$ bin/rails db:migrate

Sharding the cache

Solid Cache uses the Maglev consistent hashing scheme to shard the cache across multiple databases.

To shard:

  1. Add the configuration for the database shards to database.yml
  2. Configure the shards via config.solid_cache.connects_to
  3. Pass the shards for the cache to use via the cluster option

For example:

# config/database.yml
production:
  cache_shard1:
    database: cache1_production
    host: cache1-db
  cache_shard2:
    database: cache2_production
    host: cache2-db
  cache_shard3:
    database: cache3_production
    host: cache3-db
# config/solid_cache.yml
production:
  databases: [cache_shard1, cache_shard2, cache_shard3]

Secondary cache clusters

You can add secondary cache clusters. Reads will only be sent to the primary cluster (i.e. the first one listed).

Writes will go to all clusters. The writes to the primary cluster are synchronous, but asynchronous to the secondary clusters.

To specific multiple clusters you can do:

# config/solid_cache.yml
production:
  databases: [cache_primary_shard1, cache_primary_shard2, cache_secondary_shard1, cache_secondary_shard2]
  store_options:
    clusters:
      - shards: [cache_primary_shard1, cache_primary_shard2]
      - shards: [cache_secondary_shard1, cache_secondary_shard2]

Named shard destinations

By default, the node key used for sharding is the name of the database in database.yml.

It is possible to add names for the shards in the cluster config. This will allow you to shuffle or remove shards without breaking consistent hashing.

production:
  databases: [cache_primary_shard1, cache_primary_shard2, cache_secondary_shard1, cache_secondary_shard2]
  store_options:
    clusters:
      - shards:
          cache_primary_shard1: node1
          cache_primary_shard2: node2
      - shards:
          cache_secondary_shard1: node3
          cache_secondary_shard2: node4

Enabling encryption

Add this to an initializer:

ActiveSupport.on_load(:solid_cache_entry) do
  encrypts :value
end

Index size limits

The Solid Cache migrations try to create an index with 1024 byte entries. If that is too big for your database, you should:

  1. Edit the index size in the migration
  2. Set max_key_bytesize on your cache to the new value

Development

Run the tests with bin/rake test. By default, these will run against SQLite.

You can also run the tests against MySQL and PostgreSQL. First start up the databases:

$ docker compose up -d

Next, setup the database schema:

$ TARGET_DB=mysql bin/rails db:setup
$ TARGET_DB=postgres bin/rails db:setup

Then run the tests for the target database:

$ TARGET_DB=mysql bin/rake test
$ TARGET_DB=postgres bin/rake test

Testing with multiple Rails version

Solid Cache relies on appraisal to test multiple Rails version.

To run a test for a specific version run:

bundle exec appraisal rails-7-1 bin/rake test

After updating the dependencies in the Gemfile please run:

$ bundle
$ appraisal update

This ensures that all the Rails versions dependencies are updated.

License

Solid Cache is licensed under MIT.

More Repositories

1

rails

Ruby on Rails
Ruby
55,483
star
2

webpacker

Use Webpack to manage app-like JavaScript modules in Rails
Ruby
5,308
star
3

thor

Thor is a toolkit for building powerful command-line interfaces.
Ruby
5,115
star
4

jbuilder

Jbuilder: generate JSON objects with a Builder-style DSL
Ruby
4,324
star
5

spring

Rails application preloader
Ruby
2,804
star
6

jquery-ujs

Ruby on Rails unobtrusive scripting adapter for jQuery
JavaScript
2,607
star
7

rails-dev-box

A virtual machine for Ruby on Rails core development
Shell
2,050
star
8

solid_queue

Database-backed Active Job backend
Ruby
1,709
star
9

tailwindcss-rails

Ruby
1,388
star
10

kredis

Higher-level data structures built on Redis
Ruby
1,376
star
11

activeresource

Connects business objects and REST web services
Ruby
1,322
star
12

docked

Running Rails from Docker for easy start to development
Dockerfile
1,291
star
13

strong_parameters

Taint and required checking for Action Pack and enforcement in Active Model
Ruby
1,270
star
14

globalid

Identify app models with a URI
Ruby
1,195
star
15

actioncable

Framework for real-time communication over websockets
1,084
star
16

importmap-rails

Use ESM with importmap to manage modern JavaScript in Rails without transpiling or bundling.
Ruby
1,037
star
17

jquery-rails

A gem to automate using jQuery with Rails
Ruby
947
star
18

sprockets

Rack-based asset packaging system
Ruby
937
star
19

sass-rails

Ruby on Rails stylesheet engine for Sass
Ruby
859
star
20

propshaft

Deliver assets for Rails
Ruby
855
star
21

exception_notification

NOTICE: official repository moved to https://github.com/smartinez87/exception_notification
Ruby
841
star
22

sdoc

Standalone sdoc generator
JavaScript
822
star
23

jsbundling-rails

Bundle and transpile JavaScript in Rails with esbuild, rollup.js, or Webpack.
Ruby
819
star
24

rails-perftest

Benchmark and profile your Rails apps
Ruby
780
star
25

activejob

Declare job classes that can be run by a variety of queueing backends
Ruby
744
star
26

activestorage

Store files in Rails applications
734
star
27

pjax_rails

PJAX integration for Rails
Ruby
667
star
28

actioncable-examples

Action Cable Examples
Ruby
663
star
29

cache_digests

Ruby
643
star
30

sprockets-rails

Sprockets Rails integration
Ruby
575
star
31

cssbundling-rails

Bundle and process CSS in Rails with Tailwind, PostCSS, and Sass via Node.js.
Ruby
568
star
32

activerecord-session_store

Active Record's Session Store extracted from Rails
Ruby
539
star
33

execjs

Run JavaScript code from Ruby
Ruby
528
star
34

rails-observers

Rails observer (removed from core in Rails 4.0)
Ruby
516
star
35

mission_control-jobs

Dashboard and Active Job extensions to operate and troubleshoot background jobs
Ruby
491
star
36

actiontext

Edit and display rich text in Rails applications
406
star
37

request.js

JavaScript
386
star
38

acts_as_list

NOTICE: official repository moved to https://github.com/swanandp/acts_as_list
Ruby
385
star
39

marcel

Find the mime type of files, examining file, filename and declared type
Ruby
383
star
40

rubocop-rails-omakase

Omakase Ruby styling for Rails
Ruby
380
star
41

actionpack-page_caching

Static page caching for Action Pack (removed from core in Rails 4.0)
Ruby
347
star
42

commands

Run Rake/Rails commands through the console
Ruby
337
star
43

ssl_requirement

NOTICE: official repository moved to https://github.com/retr0h/ssl_requirement
Ruby
315
star
44

rails-controller-testing

Brings back `assigns` and `assert_template` to your Rails tests
Ruby
303
star
45

rails-html-sanitizer

Ruby
302
star
46

open_id_authentication

NOTICE: official repository moved to https://github.com/Velir/open_id_authentication
Ruby
285
star
47

acts_as_tree

NOTICE: official repository moved to https://github.com/amerine/acts_as_tree
Ruby
281
star
48

actionpack-action_caching

Action caching for Action Pack (removed from core in Rails 4.0)
Ruby
260
star
49

in_place_editing

NOTICE: official repository moved to https://github.com/amerine/in_place_editing
Ruby
230
star
50

protected_attributes

Protect attributes from mass-assignment in ActiveRecord models.
Ruby
229
star
51

journey

A router for rails
Ruby
221
star
52

auto_complete

NOTICE: official repository moved to https://github.com/david-kerins/auto_complete
Ruby
211
star
53

dartsass-rails

Integrate Dart Sass with the asset pipeline in Rails
Ruby
206
star
54

dynamic_form

NOTICE: official repository moved to https://github.com/joelmoss/dynamic_form
Ruby
192
star
55

solid_cable

A database backed ActionCable adapter
Ruby
187
star
56

country_select

NOTICE: official repository moved to https://github.com/stefanpenner/country_select
Ruby
176
star
57

rails-dom-testing

Extracting DomAssertions and SelectorAssertions from ActionView.
Ruby
174
star
58

routing_concerns

Abstract common routing resource concerns to cut down on duplication.
Ruby
154
star
59

esbuild-rails

Bundle and transpile JavaScript in Rails with esbuild
Ruby
147
star
60

rails-contributors

The web application that runs https://contributors.rubyonrails.org
Ruby
138
star
61

rails-new

Create Rails projects with Ruby installed
Rust
125
star
62

actionmailbox

Receive and process incoming emails in Rails
125
star
63

requestjs-rails

JavaScript
119
star
64

activemodel-globalid

Serializing models to a single string makes it easy to pass references around
Ruby
90
star
65

account_location

NOTICE: official repository moved to https://github.com/bbommarito/account_location
Ruby
73
star
66

acts_as_nested_set

NOTICE: official repository moved to https://github.com/bbommarito/acts_as_nested_set
Ruby
71
star
67

iso-3166-country-select

WARNING: this repo is not maintained anymore, if you want to maintain it, please send an mail to rails-core
Ruby
70
star
68

activerecord-deprecated_finders

Ruby
68
star
69

spring-watcher-listen

Ruby
64
star
70

website

HTML
64
star
71

weblog

Superseded by https://github.com/rails/website
HTML
63
star
72

prototype-ujs

JavaScript
62
star
73

prototype_legacy_helper

WARNING: this repo is not maintained anymore, if you want to maintain it, please send an mail to rails-core
Ruby
60
star
74

verification

NOTICE: official repository moved to https://github.com/sikachu/verification
Ruby
58
star
75

prototype-rails

Add RJS, Prototype, and Scriptaculous helpers to Rails 3.1+ apps
Ruby
55
star
76

activemodel-serializers-xml

Ruby
52
star
77

record_tag_helper

ActionView Record Tag Helpers
Ruby
51
star
78

homepage

Superseded by https://github.com/rails/website
HTML
50
star
79

rollupjs-rails

Bundle and transpile JavaScript in Rails with rollup.js
Ruby
49
star
80

actionpack-xml_parser

XML parameters parser for Action Pack (removed from core in Rails 4.0)
Ruby
49
star
81

activesupport-json_encoder

Ruby
48
star
82

etagger

Declare what goes in to your ETags: asset versions, account ID, etc.
Ruby
41
star
83

upload_progress

NOTICE: official repository moved to https://github.com/rishav/upload_progress
Ruby
39
star
84

devcontainer

Shell
38
star
85

atom_feed_helper

NOTICE: official repository moved to https://github.com/TrevorBramble/atom_feed_helper
Ruby
38
star
86

render_component

NOTICE: official repository moved to https://github.com/malev/render_component. Components allow you to call other actions for their rendered response while executing another action
Ruby
38
star
87

gsoc2014

Project website and wiki for Ruby on Rails proposals to Google Summer of Code 2014
37
star
88

gsoc2013

Project website and wiki for Ruby on Rails proposals to Google Summer of Code 2013
31
star
89

ruby-coffee-script

Ruby CoffeeScript Compiler
Ruby
28
star
90

asset_server

NOTICE: official repository moved to https://github.com/andhapp/asset_server
Ruby
27
star
91

homepage-2011

This repo is now legacy. New homepage is at rails/homepage
HTML
27
star
92

deadlock_retry

NOTICE: official repository moved to https://github.com/heaps/deadlock_retry
Ruby
27
star
93

rails-docs-server

Ruby
25
star
94

token_generator

NOTICE: official repository moved to https://github.com/bbommarito/token_generator
Ruby
25
star
95

http_authentication

NOTICE: official repository moved to https://github.com/dshimy/http_authentication
Ruby
22
star
96

irs_process_scripts

WARNING: this repo is not maintained anymore, if you want to maintain it, please send an mail to rails-core. The extracted inspector, reaper, and spawner scripts from script/process/*
22
star
97

javascript_test

WARNING: this repo is not maintained anymore, if you want to maintain it, please send an mail to rails-core
JavaScript
19
star
98

buildkite-config

Fallback configuration for branches that lack a .buildkite/ directory
Ruby
18
star
99

scriptaculous_slider

WARNING: this repo is not maintained anymore, if you want to maintain it, please send an mail to rails-core
JavaScript
18
star
100

request_profiler

WARNING: this repo is not maintained anymore, if you want to maintain it, please send an mail to rails-core. Request profiler based on integration test scripts
Ruby
17
star