• Stars
    star
    705
  • Rank 61,133 (Top 2 %)
  • Language
    Ruby
  • License
    MIT License
  • Created over 3 years ago
  • Updated 10 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A Rails engine for queueing and managing data migrations.

Maintenance Tasks

A Rails engine for queuing and managing maintenance tasks.

By ”maintenance task”, this project means a data migration, i.e. code that changes data in the database, often to support schema migrations. For example, in order to introduce a new NOT NULL column, it has to be added as nullable first, backfilled with values, before finally being changed to NOT NULL. This engine helps with the second part of this process, backfilling.

Maintenance tasks are collection-based tasks, usually using Active Record, that update the data in your database. They can be paused or interrupted. Maintenance tasks can operate in batches and use throttling to control the load on your database.

Maintenance tasks aren't meant to happen on a regular basis. They're used as needed, or as one-offs. Normally maintenance tasks are ephemeral, so they are used briefly and then deleted.

The Rails engine has a web-based UI for listing maintenance tasks, seeing their status, and starting, pausing and restarting them.

Link to demo video

Should I Use Maintenance Tasks?

Maintenance tasks have a limited, specific job UI. While the engine can be used to provide a user interface for other data changes, such as data changes for support requests, we recommend you use regular application code for those use cases instead. These inevitably require more flexibility than this engine will be able to provide.

If your task shouldn't run as an Active Job, it probably isn't a good match for this gem. If your task doesn't need to run in the background, consider a runner script instead. If your task doesn't need to be interruptible, consider a normal Active Job.

Maintenance tasks can be interrupted between iterations. If your task isn't collection-based (no CSV file or database table) or has very large batches, it will get limited benefit from throttling (pausing between iterations) or interrupting. This might be fine, or the added complexity of maintenance Tasks over normal Active Jobs may not be worthwhile.

If your task updates your database schema instead of data, use a migration instead of a maintenance task.

If your task happens regularly, consider Active Jobs with a scheduler or cron, job-iteration jobs and/or custom rails_admin UIs instead of the Maintenance Tasks gem. Maintenance tasks should be ephemeral, to suit their intentionally limited UI.

To create seed data for a new application, use the provided Rails db/seeds.rb file instead.

If your application can't handle a half-completed migration, maintenance tasks are probably the wrong tool. Remember that maintenance tasks are intentionally pausable and can be cancelled halfway.

Installation

To install the gem and run the install generator, execute:

bundle add maintenance_tasks
bin/rails generate maintenance_tasks:install

The generator creates and runs a migration to add the necessary table to your database. It also mounts Maintenance Tasks in your config/routes.rb. By default the web UI can be accessed in the new /maintenance_tasks path.

In case you use an exception reporting service (e.g. Bugsnag) you might want to define an error handler. See Customizing the error handler for more information.

Active Job Dependency

The Maintenance Tasks framework relies on Active Job behind the scenes to run Tasks. The default queuing backend for Active Job is asynchronous. It is strongly recommended to change this to a persistent backend so that Task progress is not lost during code or infrastructure changes. For more information on configuring a queuing backend, take a look at the Active Job documentation.

Autoloading

The Maintenance Tasks framework does not support autoloading in :classic mode. Please ensure your application is using Zeitwerk to load your code. For more information, please consult the Rails guides on autoloading and reloading constants.

Usage

The typical Maintenance Tasks workflow is as follows:

  1. Generate a class describing the Task and the work to be done.
  2. Run the Task
  3. Monitor the Task
    • either by using the included web UI,
    • or by manually checking your task’s run’s status in your database.
  4. Optionally, delete the Task code if you no longer need it.

Creating a Task

A generator is provided to create tasks. Generate a new task by running:

bin/rails generate maintenance_tasks:task update_posts

This creates the task file app/tasks/maintenance/update_posts_task.rb.

The generated task is a subclass of MaintenanceTasks::Task that implements:

  • collection: return an Active Record Relation or an Array to be iterated over.
  • process: do the work of your maintenance task on a single record

Optionally, tasks can also implement a custom #count method, defining the number of elements that will be iterated over. Your task’s tick_total will be calculated automatically based on the collection size, but this value may be overridden if desired using the #count method (this might be done, for example, to avoid the query that would be produced to determine the size of your collection).

Example:

# app/tasks/maintenance/update_posts_task.rb

module Maintenance
  class UpdatePostsTask < MaintenanceTasks::Task
    def collection
      Post.all
    end

    def process(post)
      post.update!(content: "New content!")
    end
  end
end

Creating a CSV Task

You can also write a Task that iterates on a CSV file. Note that writing CSV Tasks requires Active Storage to be configured. Ensure that the dependency is specified in your application’s Gemfile, and that you’ve followed the setup instructions. See also Customizing which Active Storage service to use.

Generate a CSV Task by running:

bin/rails generate maintenance_tasks:task import_posts --csv

The generated task is a subclass of MaintenanceTasks::Task that implements:

  • process: do the work of your maintenance task on a CSV::Row
# app/tasks/maintenance/import_posts_task.rb

module Maintenance
  class ImportPostsTask < MaintenanceTasks::Task
    csv_collection

    def process(row)
      Post.create!(title: row["title"], content: row["content"])
    end
  end
end
# posts.csv
title,content
My Title,Hello World!

The files uploaded to your Active Storage service provider will be renamed to include an ISO 8601 timestamp and the Task name in snake case format. The CSV is expected to have a trailing newline at the end of the file.

Batch CSV Tasks

Tasks can process CSVs in batches. Add the in_batches option to your task’s csv_collection macro:

# app/tasks/maintenance/batch_import_posts_task.rb

module Maintenance
  class BatchImportPostsTask < MaintenanceTasks::Task
    csv_collection(in_batches: 50)

    def process(batch_of_rows)
      Post.insert_all(post_rows.map(&:to_h))
    end
  end
end

As with a regular CSV task, ensure you’ve implemented the following method:

  • process: do the work of your Task on a batch (array of CSV::Row objects).

Note that #count is calculated automatically based on the number of batches in your collection, and your Task’s progress will be displayed in terms of batches (not the total number of rows in your CSV).

Non-batched CSV tasks will have an effective batch size of 1, which can reduce the efficiency of your database operations.

Processing Batch Collections

The Maintenance Tasks gem supports processing Active Records in batches. This can reduce the number of calls your Task makes to the database. Use ActiveRecord::Batches#in_batches on the relation returned by your collection to specify that your Task should process batches instead of records. Active Record defaults to 1000 records by batch, but a custom size can be specified.

# app/tasks/maintenance/update_posts_in_batches_task.rb

module Maintenance
  class UpdatePostsInBatchesTask < MaintenanceTasks::Task
    def collection
      Post.in_batches
    end

    def process(batch_of_posts)
      batch_of_posts.update_all(content: "New content added on #{Time.now.utc}")
    end
  end
end

Ensure that you’ve implemented the following methods:

  • collection: return an ActiveRecord::Batches::BatchEnumerator.
  • process: do the work of your Task on a batch (ActiveRecord::Relation).

Note that #count is calculated automatically based on the number of batches in your collection, and your Task’s progress will be displayed in terms of batches (not the number of records in the relation).

Important! Batches should only be used if #process is performing a batch operation such as #update_all or #delete_all. If you need to iterate over individual records, you should define a collection that returns an ActiveRecord::Relation. This uses batching internally, but loads the records with one SQL query. Conversely, batch collections load the primary keys of the records of the batch first, and then perform an additional query to load the records when calling each (or any Enumerable method) inside #process.

Tasks that don’t need a Collection

Sometimes, you might want to run a Task that performs a single operation, such as enqueuing another background job or querying an external API. The gem supports collection-less tasks.

Generate a collection-less Task by running:

bin/rails generate maintenance_tasks:task no_collection_task --no-collection

The generated task is a subclass of MaintenanceTasks::Task that implements:

  • process: do the work of your maintenance task
# app/tasks/maintenance/no_collection_task.rb

module Maintenance
  class NoCollectionTask < MaintenanceTasks::Task
    no_collection

    def process
      SomeAsyncJob.perform_later
    end
  end
end

Throttling

Maintenance tasks often modify a lot of data and can be taxing on your database. The gem provides a throttling mechanism that can be used to throttle a Task when a given condition is met. If a Task is throttled (the throttle block returns true), it will be interrupted and retried after a backoff period has passed. The default backoff is 30 seconds.

Specify the throttle condition as a block:

# app/tasks/maintenance/update_posts_throttled_task.rb

module Maintenance
  class UpdatePostsThrottledTask < MaintenanceTasks::Task
    throttle_on(backoff: 1.minute) do
      DatabaseStatus.unhealthy?
    end

    def collection
      Post.all
    end

    def process(post)
      post.update!(content: "New content added on #{Time.now.utc}")
    end
  end
end

Note that it’s up to you to define a throttling condition that makes sense for your app. Shopify implements DatabaseStatus.healthy? to check various MySQL metrics such as replication lag, DB threads, whether DB writes are available, etc.

Tasks can define multiple throttle conditions. Throttle conditions are inherited by descendants, and new conditions will be appended without impacting existing conditions.

The backoff can also be specified as a Proc that receives no arguments:

# app/tasks/maintenance/update_posts_throttled_task.rb

module Maintenance
  class UpdatePostsThrottledTask < MaintenanceTasks::Task
    throttle_on(backoff: -> { RandomBackoffGenerator.generate_duration } ) do
      DatabaseStatus.unhealthy?
    end
    ...
  end
end

Custom Task Parameters

Tasks may need additional information, supplied via parameters, to run. Parameters can be defined as Active Model Attributes in a Task, and then become accessible to any of Task’s methods: #collection, #count, or #process.

# app/tasks/maintenance/update_posts_via_params_task.rb

module Maintenance
  class UpdatePostsViaParamsTask < MaintenanceTasks::Task
    attribute :updated_content, :string
    validates :updated_content, presence: true

    def collection
      Post.all
    end

    def process(post)
      post.update!(content: updated_content)
    end
  end
end

Tasks can leverage Active Model Validations when defining parameters. Arguments supplied to a Task accepting parameters will be validated before the Task starts to run. Since arguments are specified in the user interface via text area inputs, it’s important to check that they conform to the format your Task expects, and to sanitize any inputs if necessary.

Using Task Callbacks

The Task provides callbacks that hook into its life cycle.

Available callbacks are:

  • after_start
  • after_pause
  • after_interrupt
  • after_cancel
  • after_complete
  • after_error
module Maintenance
  class UpdatePostsTask < MaintenanceTasks::Task
    after_start :notify

    def notify
      NotifyJob.perform_later(self.class.name)
    end

    # ...
  end
end

Note: The after_error callback is guaranteed to complete, so any exceptions raised in your callback code are ignored. If your after_error callback code can raise an exception, you’ll need to rescue it and handle it appropriately within the callback.

module Maintenance
  class UpdatePostsTask < MaintenanceTasks::Task
    after_error :dangerous_notify

    def dangerous_notify
      # This error is rescued and ignored in favour of the original error causing the error flow.
      raise NotDeliveredError
    end

    # ...
  end
end

If any of the other callbacks cause an exception, it will be handled by the error handler, and will cause the task to stop running.

Callback behaviour can be shared across all tasks using an initializer.

# config/initializer/maintenance_tasks.rb
Rails.autoloaders.main.on_load("MaintenanceTasks::Task") do
  MaintenanceTasks::Task.class_eval do
    after_start(:notify)

    private

    def notify; end
  end
end

Considerations when writing Tasks

Maintenance Tasks relies on the queue adapter configured for your application to run the job which is processing your Task. The guidelines for writing Task may depend on the queue adapter but in general, you should follow these rules:

  • Duration of Task#process: processing a single element of the collection should take less than 25 seconds, or the duration set as a timeout for Sidekiq or the queue adapter configured in your application. Short batches allow the Task to be safely interrupted and resumed.
  • Idempotency of Task#process: it should be safe to run process multiple times for the same element of the collection. Read more in this Sidekiq best practice. It’s important if the Task errors and you run it again, because the same element that caused the Task to give an error may well be processed again. It especially matters in the situation described above, when the iteration duration exceeds the timeout: if the job is re-enqueued, multiple elements may be processed again.

Task object life cycle and memoization

When the Task runs or resumes, the Runner enqueues a job, which processes the Task. That job will instantiate a Task object which will live for the duration of the job. The first time the job runs, it will call count. Every time a job runs, it will call collection on the Task object, and then process for each item in the collection, until the job stops. The job stops when either the collection is finished processing or after the maximum job runtime has expired.

This means memoization can be misleading within process, since the memoized values will be available for subsequent calls to process within the same job. Still, memoization can be used for throttling or reporting, and you can use Task callbacks to persist or log a report for example.

Writing tests for a Task

The task generator will also create a test file for your task in the folder test/tasks/maintenance/. At a minimum, it’s recommended that the #process method in your task be tested. You may also want to test the #collection and #count methods for your task if they are sufficiently complex.

Example:

# test/tasks/maintenance/update_posts_task_test.rb

require "test_helper"

module Maintenance
  class UpdatePostsTaskTest < ActiveSupport::TestCase
    test "#process performs a task iteration" do
      post = Post.new

      Maintenance::UpdatePostsTask.process(post)

      assert_equal "New content!", post.content
    end
  end
end

Writing tests for a CSV Task

You should write tests for your #process method in a CSV Task as well. It takes a CSV::Row as an argument. You can pass a row, or a hash with string keys to #process from your test.

# test/tasks/maintenance/import_posts_task_test.rb

require "test_helper"

module Maintenance
  class ImportPostsTaskTest < ActiveSupport::TestCase
    test "#process performs a task iteration" do
      assert_difference -> { Post.count } do
        Maintenance::UpdatePostsTask.process({
          "title" => "My Title",
          "content" => "Hello World!",
        })
      end

      post = Post.last
      assert_equal "My Title", post.title
      assert_equal "Hello World!", post.content
    end
  end
end

Writing tests for a Task with parameters

Tests for tasks with parameters need to instantiate the task class in order to assign attributes. Once the task instance is setup, you may test #process normally.

# test/tasks/maintenance/update_posts_via_params_task_test.rb

require "test_helper"

module Maintenance
  class UpdatePostsViaParamsTaskTest < ActiveSupport::TestCase
    setup do
      @task = UpdatePostsViaParamsTask.new
      @task.updated_content = "Testing"
    end

    test "#process performs a task iteration" do
      assert_difference -> { Post.first.content } do
        task.process(Post.first)
      end
    end
  end
end

Running a Task

Running a Task from the Web UI

You can run your new Task by accessing the Web UI and clicking on "Run".

Running a Task from the command line

Alternatively, you can run your Task in the command line:

bundle exec maintenance_tasks perform Maintenance::UpdatePostsTask

To run a Task that processes CSVs from the command line, use the --csv option:

bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv "path/to/my_csv.csv"

The --csv option also works with CSV content coming from the standard input:

curl "some/remote/csv" |
  bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv

To run a Task that takes arguments from the command line, use the --arguments option, passing arguments as a set of <key>:<value> pairs:

bundle exec maintenance_tasks perform Maintenance::ParamsTask \
  --arguments post_ids:1,2,3 content:"Hello, World!"

Running a Task from Ruby

You can also run a Task in Ruby by sending run with a Task name to Runner:

MaintenanceTasks::Runner.run(name: "Maintenance::UpdatePostsTask")

To run a Task that processes CSVs using the Runner, provide a Hash containing an open IO object and a filename to run:

MaintenanceTasks::Runner.run(
  name: "Maintenance::ImportPostsTask",
  csv_file: { io: File.open("path/to/my_csv.csv"), filename: "my_csv.csv" }
)

To run a Task that takes arguments using the Runner, provide a Hash containing the set of arguments ({ parameter_name: argument_value }) to run:

MaintenanceTasks::Runner.run(
  name: "Maintenance::ParamsTask",
  arguments: { post_ids: "1,2,3" }
)

Monitoring your Task’s status

The web UI will provide updates on the status of your Task. Here are the states a Task can be in:

  • new: A Task that has not yet been run.
  • enqueued: A Task that is waiting to be performed after a user has instructed it to run.
  • running: A Task that is currently being performed by a job worker.
  • pausing: A Task that was paused by a user, but needs to finish work before stopping.
  • paused: A Task that was paused by a user and is not performing. It can be resumed.
  • interrupted: A Task that has been momentarily interrupted by the job infrastructure.
  • cancelling: A Task that was cancelled by a user, but needs to finish work before stopping.
  • cancelled: A Task that was cancelled by a user and is not performing. It cannot be resumed.
  • succeeded: A Task that finished successfully.
  • errored: A Task that encountered an unhandled exception while performing.

How Maintenance Tasks runs a Task

Maintenance tasks can be running for a long time, and the purpose of the gem is to make it easy to continue running tasks through deploys, Kubernetes Pod scheduling, Heroku dyno restarts or other infrastructure or code changes.

This means a Task can safely be interrupted, re-enqueued and resumed without any intervention at the end of an iteration, after the process method returns.

By default, a running Task will be interrupted after running for more 5 minutes. This is configured in the job-iteration gem and can be tweaked in an initializer if necessary.

Running tasks will also be interrupted and re-enqueued when needed. For example when Sidekiq workers shuts down for a deploy:

  • When Sidekiq receives a TSTP or TERM signal, it will consider itself to be stopping.
  • When Sidekiq is stopping, JobIteration stops iterating over the enumerator. The position in the iteration is saved, a new job is enqueued to resume work, and the Task is marked as interrupted.

When Sidekiq is stopping, it will give workers 25 seconds to finish before forcefully terminating them (this is the default but can be configured with the --timeout option). Before the worker threads are terminated, Sidekiq will try to re-enqueue the job so your Task will be resumed. However, the position in the collection won’t be persisted so at least one iteration may run again.

Help! My Task is stuck

Finally, if the queue adapter configured for your application doesn’t have this property, or if Sidekiq crashes, is forcefully terminated, or is unable to re-enqueue the jobs that were in progress, the Task may be in a seemingly stuck situation where it appears to be running but is not. In that situation, pausing or cancelling it will not result in the Task being paused or cancelled, as the Task will get stuck in a state of pausing or cancelling. As a work-around, if a Task is cancelling for more than 5 minutes, you can cancel it again. It will then be marked as fully cancelled, allowing you to run it again.

Configuring the gem

There are a few configurable options for the gem. Custom configurations should be placed in a maintenance_tasks.rb initializer.

Customizing the error handler

Exceptions raised while a Task is performing are rescued and information about the error is persisted and visible in the UI.

If you want to integrate with an exception monitoring service (e.g. Bugsnag), you can define an error handler:

# config/initializers/maintenance_tasks.rb

MaintenanceTasks.error_handler = ->(error, task_context, _errored_element) do
  Bugsnag.notify(error) do |notification|
    notification.add_metadata(:task, task_context)
  end
end

The error handler should be a lambda that accepts three arguments:

  • error: The exception that was raised.

  • task_context: A hash with additional information about the Task and the error:

    • task_name: The name of the Task that errored
    • started_at: The time the Task started
    • ended_at: The time the Task errored

    Note that task_context may be empty if the Task produced an error before any context could be gathered (for example, if deserializing the job to process your Task failed).

  • errored_element: The element, if any, that was being processed when the Task raised an exception. If you would like to pass this object to your exception monitoring service, make sure you sanitize the object to avoid leaking sensitive data and convert it to a format that is compatible with your bug tracker. For example, Bugsnag only sends the id and class name of Active Record objects in order to protect sensitive data. CSV rows, on the other hand, are converted to strings and passed raw to Bugsnag, so make sure to filter any personal data from these objects before adding them to a report.

Customizing the maintenance tasks module

MaintenanceTasks.tasks_module can be configured to define the module in which tasks will be placed.

# config/initializers/maintenance_tasks.rb

MaintenanceTasks.tasks_module = "TaskModule"

If no value is specified, it will default to Maintenance.

Organizing tasks using namespaces

Tasks may be nested arbitrarily deeply under app/tasks/maintenance, for example given a task file app/tasks/maintenance/team_name/service_name/update_posts_task.rb we can define the task as:

module Maintenance
  module TeamName
    module ServiceName
      class UpdatePostsTask < MaintenanceTasks::Task
        def process(rows)
          # ...
        end
      end
    end
  end
end

Customizing the underlying job class

MaintenanceTasks.job can be configured to define a Job class for your tasks to use. This is a global configuration, so this Job class will be used across all maintenance tasks in your application.

# config/initializers/maintenance_tasks.rb

MaintenanceTasks.job = "CustomTaskJob"

# app/jobs/custom_task_job.rb

class CustomTaskJob < MaintenanceTasks::TaskJob
  queue_as :low_priority
end

The Job class must inherit from MaintenanceTasks::TaskJob.

Note that retry_on is not supported for custom Job classes, so failed jobs cannot be retried.

Customizing the rate at which task progress gets updated

MaintenanceTasks.ticker_delay can be configured to customize how frequently task progress gets persisted to the database. It can be a Numeric value or an ActiveSupport::Duration value.

# config/initializers/maintenance_tasks.rb

MaintenanceTasks.ticker_delay = 2.seconds

If no value is specified, it will default to 1 second.

Customizing which Active Storage service to use

The Active Storage framework in Rails 6.1 and up supports multiple storage services. To specify which service to use, MaintenanceTasks.active_storage_service can be configured with the service’s key, as specified in your application’s config/storage.yml:

# config/storage.yml

user_data:
  service: GCS
  credentials: <%= Rails.root.join("path/to/user/data/keyfile.json") %>
  project: "my-project"
  bucket: "user-data-bucket"

internal:
  service: GCS
  credentials: <%= Rails.root.join("path/to/internal/keyfile.json") %>
  project: "my-project"
  bucket: "internal-bucket"
# config/initializers/maintenance_tasks.rb

MaintenanceTasks.active_storage_service = :internal

There is no need to configure this option if your application uses only one storage service. Rails.configuration.active_storage.service is used by default.

Customizing the backtrace cleaner

MaintenanceTasks.backtrace_cleaner can be configured to specify a backtrace cleaner to use when a Task errors and the backtrace is cleaned and persisted. An ActiveSupport::BacktraceCleaner should be used.

# config/initializers/maintenance_tasks.rb

cleaner = ActiveSupport::BacktraceCleaner.new
cleaner.add_silencer { |line| line =~ /ignore_this_dir/ }

MaintenanceTasks.backtrace_cleaner = cleaner

If none is specified, the default Rails.backtrace_cleaner will be used to clean backtraces.

Customizing the parent controller for the web UI

MaintenanceTasks.parent_controller can be configured to specify a controller class for all of the web UI engine's controllers to inherit from.

This allows applications with common logic in their ApplicationController (or any other controller) to optionally configure the web UI to inherit that logic with a simple assignment in the initializer.

# config/initializers/maintenance_tasks.rb

MaintenanceTasks.parent_controller = "Services::CustomController"

# app/controllers/services/custom_controller.rb

class Services::CustomController < ActionController::Base
  include CustomSecurityThings
  include CustomLoggingThings
  ...
end

The parent controller value must be a string corresponding to an existing controller class which must inherit from ActionController::Base.

If no value is specified, it will default to "ActionController::Base".

Metadata

MaintenanceTasks.metadata can be configured to specify a proc from which to get extra information about the run. Since this proc will be ran in the context of the MaintenanceTasks.parent_controller, it can be used to keep the id or email of the user who performed the maintenance task.

# config/initializers/maintenance_tasks.rb
MaintenanceTasks.metadata = ->() do
  { user_email: current_user.email }
end

Upgrading

Use bundler to check for and upgrade to newer versions. After installing a new version, re-run the install command:

bin/rails generate maintenance_tasks:install

This ensures that new migrations are installed and run as well.

What if I’ve deleted my previous Maintenance Task migrations?

The install command will attempt to reinstall these old migrations and migrating the database will cause problems. Use bin/rails maintenance_tasks:install:migrations to copy the gem’s migrations to your db/migrate folder. Check the release notes to see if any new migrations were added since your last gem upgrade. Ensure that these are kept, but remove any migrations that already ran.

Run the migrations using bin/rails db:migrate.

Contributing

Would you like to report an issue or contribute with code? We accept issues and pull requests. You can find the contribution guidelines on CONTRIBUTING.md.

Releasing new versions

Updates should be added to the latest draft release on GitHub as Pull Requests are merged.

Once a release is ready, follow these steps:

  • Update spec.version in maintenance_tasks.gemspec.
  • Run bundle install to bump the Gemfile.lock version of the gem.
  • Open a PR and merge on approval.
  • Deploy via Shipit and see the new version on https://rubygems.org/gems/maintenance_tasks.
  • Ensure the release has documented all changes and publish it.
  • Create a new draft release on GitHub with the title “Upcoming Release”. The tag version can be left blank. This will be the starting point for documenting changes related to the next release.

More Repositories

1

draggable

The JavaScript Drag & Drop library your grandparents warned you about.
JavaScript
17,350
star
2

dashing

The exceptionally handsome dashboard framework in Ruby and Coffeescript.
JavaScript
11,025
star
3

liquid

Liquid markup language. Safe, customer facing template language for flexible web apps.
Ruby
10,419
star
4

toxiproxy

⏰ 🔥 A TCP proxy to simulate network and system conditions for chaos and resiliency testing
Go
9,412
star
5

react-native-skia

High-performance React Native Graphics using Skia
TypeScript
6,162
star
6

polaris

Shopify’s design system to help us work together to build a great experience for all of our merchants.
TypeScript
5,352
star
7

flash-list

A better list for React Native
TypeScript
4,536
star
8

hydrogen-v1

React-based framework for building dynamic, Shopify-powered custom storefronts.
TypeScript
3,761
star
9

go-lua

A Lua VM in Go
Go
2,773
star
10

bootsnap

Boot large Ruby/Rails apps faster
Ruby
2,614
star
11

graphql-design-tutorial

2,335
star
12

restyle

A type-enforced system for building UI components in React Native with TypeScript.
TypeScript
2,331
star
13

dawn

Shopify's first source available reference theme, with Online Store 2.0 features and performance built-in.
Liquid
2,279
star
14

identity_cache

IdentityCache is a blob level caching solution to plug into Active Record. Don't #find, #fetch!
Ruby
1,874
star
15

shopify_app

A Rails Engine for building Shopify Apps
Ruby
1,649
star
16

kubeaudit

kubeaudit helps you audit your Kubernetes clusters against common security controls
Go
1,624
star
17

quilt

A loosely related set of packages for JavaScript/TypeScript projects at Shopify
TypeScript
1,570
star
18

graphql-batch

A query batching executor for the graphql gem
Ruby
1,388
star
19

shipit-engine

Deployment coordination
Ruby
1,382
star
20

packwerk

Good things come in small packages.
Ruby
1,346
star
21

krane

A command-line tool that helps you ship changes to a Kubernetes namespace and understand the result
Ruby
1,309
star
22

semian

🐒 Resiliency toolkit for Ruby for failing fast
Ruby
1,286
star
23

slate

Slate is a toolkit for developing Shopify themes. It's designed to assist your workflow and speed up the process of developing, testing, and deploying themes.
JavaScript
1,281
star
24

ejson

EJSON is a small library to manage encrypted secrets using asymmetric encryption.
Go
1,246
star
25

superdb

The Super Debugger, a realtime wireless debugger for iOS
Objective-C
1,158
star
26

shopify_python_api

ShopifyAPI library allows Python developers to programmatically access the admin section of stores
Python
1,072
star
27

storefront-api-examples

Example custom storefront applications built on Shopify's Storefront API
JavaScript
1,069
star
28

themekit

Shopify theme development command line tool.
Go
1,068
star
29

Timber

The ultimate Shopify theme framework, built by Shopify.
Liquid
992
star
30

shopify-cli

Shopify CLI helps you build against the Shopify platform faster.
Ruby
987
star
31

shopify-api-ruby

ShopifyAPI is a lightweight gem for accessing the Shopify admin REST and GraphQL web services.
Ruby
982
star
32

hydrogen

Hydrogen is Shopify’s stack for headless commerce. It provides a set of tools, utilities, and best-in-class examples for building dynamic and performant commerce applications. Hydrogen is designed to dovetail with Remix, Shopify’s full stack web framework, but it also provides a React library portable to other supporting frameworks. Demo store 👇🏼
TypeScript
966
star
33

js-buy-sdk

The JS Buy SDK is a lightweight library that allows you to build ecommerce into any website. It is based on Shopify's API and provides the ability to retrieve products and collections from your shop, add products to a cart, and checkout.
JavaScript
932
star
34

job-iteration

Makes your background jobs interruptible and resumable by design.
Ruby
907
star
35

cli-ui

Terminal user interface library
Ruby
869
star
36

ruby-lsp

An opinionated language server for Ruby
Ruby
851
star
37

react-native-performance

Performance monitoring for React Native apps
TypeScript
843
star
38

active_shipping

ActiveShipping is a simple shipping abstraction library extracted from Shopify
Ruby
809
star
39

shopify-api-js

Shopify Admin API Library for Node. Accelerate development with support for authentication, graphql proxy, webhooks
TypeScript
765
star
40

remote-ui

TypeScript
701
star
41

shopify-app-template-node

JavaScript
701
star
42

shopify_theme

A console tool for interacting with Shopify Theme Assets.
Ruby
640
star
43

tapioca

The swiss army knife of RBI generation
Ruby
636
star
44

pitchfork

Ruby
630
star
45

ghostferry

The swiss army knife of live data migrations
Go
596
star
46

yjit

Optimizing JIT compiler built inside CRuby
593
star
47

erb-lint

Lint your ERB or HTML files
Ruby
551
star
48

statsd-instrument

A StatsD client for Ruby apps. Provides metaprogramming methods to inject StatsD instrumentation into your code.
Ruby
546
star
49

shopify.github.com

A collection of the open source projects by Shopify
CSS
505
star
50

theme-scripts

Theme Scripts is a collection of utility libraries which help theme developers with problems unique to Shopify Themes.
JavaScript
470
star
51

livedata-ktx

Kotlin extension for LiveData, chaining like RxJava
Kotlin
467
star
52

starter-theme

The Shopify Themes Team opinionated starting point for new a Slate project
Liquid
459
star
53

ruby-style-guide

Shopify’s Ruby Style Guide
Ruby
446
star
54

shopify-demo-app-node-react

JavaScript
444
star
55

web-configs

Common configurations for building web apps at Shopify
JavaScript
433
star
56

mobile-buy-sdk-ios

Shopify’s Mobile Buy SDK makes it simple to sell physical products inside your mobile app. With a few lines of code, you can connect your app with the Shopify platform and let your users buy your products using Apple Pay or their credit card.
Swift
433
star
57

shopify_django_app

Get a Shopify app up and running with Django and Python Shopify API
Python
425
star
58

deprecation_toolkit

⚒Eliminate deprecations from your codebase ⚒
Ruby
390
star
59

ruby-lsp-rails

A Ruby LSP extension for Rails
Ruby
388
star
60

bootboot

Dualboot your Ruby app made easy
Ruby
374
star
61

FunctionalTableData

Declarative UITableViewDataSource implementation
Swift
365
star
62

shadowenv

reversible directory-local environment variable manipulations
Rust
349
star
63

shopify-node-app

An example app that uses Polaris components and shopify-express
JavaScript
327
star
64

better-html

Better HTML for Rails
Ruby
311
star
65

theme-check

The Ultimate Shopify Theme Linter
Ruby
306
star
66

product-reviews-sample-app

A sample Shopify application that creates and stores product reviews for a store, written in Node.js
JavaScript
300
star
67

tracky

The easiest way to do motion tracking!
Swift
295
star
68

shopify-api-php

PHP
279
star
69

polaris-viz

A collection of React and React native components that compose Shopify's data visualization system
TypeScript
279
star
70

measured

Encapsulate measurements and their units in Ruby.
Ruby
275
star
71

cli

Build apps, themes, and hydrogen storefronts for Shopify
TypeScript
273
star
72

money

Manage money in Shopify with a class that won't lose pennies during division
Ruby
265
star
73

javascript

The home for all things JavaScript at Shopify.
254
star
74

ruvy

Rust
252
star
75

limiter

Simple Ruby rate limiting mechanism.
Ruby
244
star
76

vscode-ruby-lsp

VS Code plugin for connecting with the Ruby LSP
TypeScript
232
star
77

polaris-tokens

Design tokens for Polaris, Shopify’s design system
TypeScript
230
star
78

buy-button-js

BuyButton.js is a highly customizable UI library for adding ecommerce functionality to any website.
JavaScript
230
star
79

android-testify

Add screenshots to your Android tests
Kotlin
225
star
80

turbograft

Hard fork of turbolinks, adding partial page replacement strategies, and utilities.
JavaScript
213
star
81

mobile-buy-sdk-android

Shopify’s Mobile Buy SDK makes it simple to sell physical products inside your mobile app. With a few lines of code, you can connect your app with the Shopify platform and let your users buy your products using their credit card.
Java
202
star
82

spoom

Useful tools for Sorbet enthusiasts
Ruby
192
star
83

graphql-js-client

A Relay compliant GraphQL client.
JavaScript
187
star
84

ruby_memcheck

Use Valgrind memcheck on your native gem without going crazy
Ruby
187
star
85

shopify-app-template-php

PHP
186
star
86

skeleton-theme

A barebones ☠️starter theme with the required files needed to compile with Slate and upload to Shopify.
Liquid
185
star
87

sprockets-commoner

Use Babel in Sprockets to compile JavaScript modules for the browser
Ruby
182
star
88

rotoscope

High-performance logger of Ruby method invocations
Ruby
180
star
89

shopify-app-template-remix

TypeScript
178
star
90

git-chain

Tool to rebase multiple Git branches based on the previous one.
Ruby
176
star
91

verdict

Framework to define and implement A/B tests in your application, and collect data for analysis purposes.
Ruby
176
star
92

hydrogen-react

Reusable components and utilities for building Shopify-powered custom storefronts.
TypeScript
174
star
93

ui-extensions

TypeScript
173
star
94

storefront-api-learning-kit

JavaScript
171
star
95

heap-profiler

Ruby heap profiler
C++
159
star
96

autoload_reloader

Experimental implementation of code reloading using Ruby's autoload
Ruby
158
star
97

app_profiler

Collect performance profiles for your Rails application.
Ruby
157
star
98

graphql-metrics

Extract as much much detail as you want from GraphQL queries, served up from your Ruby app and the graphql gem.
Ruby
157
star
99

active_fulfillment

Active Merchant library for integration with order fulfillment services
Ruby
155
star
100

ci-queue

Distribute tests over many workers using a queue
Ruby
148
star