• Stars
    star
    157
  • Rank 238,399 (Top 5 %)
  • Language
    Ruby
  • License
    MIT License
  • Created almost 5 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Collect performance profiles for your Rails application.

AppProfiler

Profiling is critical to providing an understanding of an application's performance.

AppProfiler aims to provide a common framework for performance profiling for Rails applications.

Installation

To install app_profiler you need to include it in your Gemfile.

Profiling middleware

Configuration

This gem automatically injects the AppProfiler::Middleware middleware into your Rails application.

This middleware can be disabled by using:

AppProfiler.middleware.disabled = true
# OR
Rails.application.config.app_profiler.middleware_disabled = true

Trigger profiling

Profiling can be triggered in one of two ways:

  1. Using the profile key in the query string of the URL.
    • Query string format: /?[key=value]&...
  2. Using the X-Profile key in the request headers.
    • X-Profile header format: [<key>=<value>];...

You can configure the profile header using:

AppProfiler.profile_header = "X-Profile"
# OR
Rails.application.config.app_profiler.profile_header = "X-Profile"

Here are some examples:

  1. /?profile=cpu&interval=2000&autoredirect=1&ignore_gc=1
  2. Set X-Profile to mode=wall;interval=1000;context=test-directory;autoredirect=1

Possible keys:

Key Value Notes
profile/mode Supported profiling modes: cpu, wall, object. Use profile in (1), and mode in (2).
interval Sampling interval in microseconds.
ignore_gc Ignore garbage collection frames
autoredirect Redirect request automatically to Speedscope's page after profiling.
context Directory within the specified bucket in the selected storage where raw profile data should be written. Only supported in (2). Defaults to Rails.env if not specified.

Note that the autoredirect feature can be turned on for all requests by doing the following:

AppProfiler.autoredirect = true
# OR
Rails.application.config.app_profiler.autoredirect = true

To customize the redirect location you can provide a proc:

AppProfiler.profile_url_formatter = ->(upload) { "https://host.com/custom/#{upload.name}" }
# OR
Rails.application.config.app_profiler.profile_url_formatter = ->(upload) { "https://host.com/custom/#{upload.name}" }

When profiling is triggered, the middleware will generate the profile through StackProf and upload the profiles to your specified storage. For example, the default configuration would upload profiles to file storage.

When using a cloud storage provider, you can configure the target bucket name using:

AppProfiler.storage.bucket_name = "new-bucket-name"
# OR
Rails.application.config.app_profiler.storage_bucket_name = "new-bucket-name"

Access control

You may restrict the storing of profiling results by defining your own Middleware based on AppProfiler::Middleware and changing the after_profile hook method to return false for such cases.

For example, the following middleware only stores the profiling results if a disallow_profiling key was not added to the request.env while processing the request.

class AppProfilerAuthorizedMiddleware < AppProfiler::Middleware
  def after_profile(env, params)
    !env.key?("disallow_profiling")
  end
end

You can also restrict running profiling at all by using before_profile. For example you may wish to prevent anonymous users triggering the profiler:

class AppProfilerAuthorizedMiddleware < AppProfiler::Middleware
  def before_profile(env, params)
    current_user.present?
  end
end

The custom middleware can then be configured like the following:

Rails.application.config.app_profiler.middleware = AppProfilerAuthorizedMiddleware

Profile Server

This option allows for profiles to be passively collected via an HTTP endpoint, inspired by golang's built-in pprof server.

A minimal Rack app runs a minimal (non-compliant) HTTP server, which exposes an endpoint that allows for profiling. For security purposes, the server is bound to localhost only. The HTTP server is built using standard library modules only, in order to keep dependencies minimal. Because it is an entirely separate server, listening on an entirely separate socket, this should not interfere with any existing application routes, and should even be usable in non-web apps.

This allows for two main use cases:

  • Passive profile collection in production
    • Periodically profiling production apps to analyze them responding to real workloads
    • Providing a statistical, long-term view of the "hot paths" of the workload
  • Local development profiling
    • Can be used to get a profile "on demand" against a development server

Configuration

If using as a railtie, only a single option needs to be set:

config.app_profiler.server_enabled = true

Alternatively, the server can be directly started by passing in a logger as follows:

AppProfiler::Server.start(logger)

The default duration (in seconds), for requests without a duration parameter, can also be set via the railtie config.

AppProfiler.server.duration = 30
# OR
Rails.application.config.app_profiler.server_duration = 30

The server supports both TCP and Unix sockets for its transport. It is recommended to use TCP for local development, and Unix sockets for production:

AppProfiler.server.transport = AppProfiler::Server::TRANSPORT_UNIX
# OR
Rails.application.config.app_profiler.server_transport = AppProfiler::Server::TRANSPORT_UNIX

It is possible, but not recommended, to hardcode the listen port to be used in TCP server mode with:

AppProfiler.server.port = 8080
# OR
Rails.application.config.app_profiler.server_port = 8080

If this is done in production and it can cause port conflicts with multiple instances of the app, which is another reason why the Unix transport is preferred for production.

Discovering the port or socket path

In general, the server should be run without setting the port, in which case any free TCP port may be used. To determine what the port is, check the application logs, or resolve it from the special "Magic file" which contains a mapping of pid to port:

$ PID=49825
$ port_file=$(ls -1 /tmp/app_profiler/profileserver-$PID-port-*)
$ PORT=$(echo $port_file | sed 's/.*port-\([[:digit:]]*\)-.*/\1/g')
$ echo $PORT
60160

This approach is intended to be "machine friendly" so that an external profiling agent can easily detect what port to profile on.

For the Unix mode, this is even easier as the file simply includes the PID in it, and this will be the file handle to use for the Unix socket:

$ PID=41016
$ SOCK=$(ls -1d /tmp/app_profiler/* | grep $PID.sock)
$ echo $SOCK
/tmp/app_profiler/app-profiler-41016.sock

Collecting a profile

The API is very simple, and passes supported parameters directly to stackprof.

See the possible keys for additional documentation on the supported parameters.

For example, to collect a heap profile for 60 seconds, counting every 10th allocation:

curl "http://127.0.0.1:$PORT/profile?duration=60&mode=object&interval=10"

Usage with speedscope directly

By default the server will allow CORS. This can be disabled if it presents a problem, but it should be generally safe given that the server listens for requests on localhost only, which is already a private network address.

This can be used with a local instance of speedscope to directly initiate profiling from the browser. Assuming speedscope is running locally on port 9292, and the profile server is running on port 57510, the server address can be URL encoded, and passed to speedscope via #profileURL:

http://127.0.0.1:9292/#profileURL=http%3A%2F%2F127.0.0.1%3A57510%2Fprofile%3Fduration%3D1

Profiling manually

AppProfiler can be used more simply to profile blocks of code. Here's how:

report = AppProfiler.run(mode: :cpu) do
  # ...
end

report.view # opens the profile locally in speedscope.

Profile files can be found locally in your rails app at tmp/app_profiler/*.json.

Storage backends

Profiles are stored based on the defined storage class. At the moment, the gem only supports file-based and remote storage via Google Cloud Storage. the default backend is file storage.

You can use a different backend with the following configuration:

AppProfiler.storage = AppProfiler::Storage::GoogleCloudStorage
# OR
Rails.application.config.app_profiler.storage = AppProfiler::Storage::GoogleCloudStorage

Credentials for the selected storage can be set using the following configuration (Google Cloud Storage expects the path to a JSON file, or the JSON contents):

AppProfiler.storage.credentials = { "key" => "value" }
# OR
Rails.application.config.app_profiler.storage_credentials = { "key" => "value" }

Note that in development mode the file isn't uploaded. Instead, it is viewed via the Middleware::ViewAction. If you want to change that, use the middleware_action configuration:

Rails.application.config.app_profiler.middleware_action = AppProfiler::Middleware::UploadAction

Running tests

bin/setup && bundle exec rake

More Repositories

1

draggable

The JavaScript Drag & Drop library your grandparents warned you about.
JavaScript
17,927
star
2

dashing

The exceptionally handsome dashboard framework in Ruby and Coffeescript.
JavaScript
11,025
star
3

liquid

Liquid markup language. Safe, customer facing template language for flexible web apps.
Ruby
10,419
star
4

toxiproxy

⏰ 🔥 A TCP proxy to simulate network and system conditions for chaos and resiliency testing
Go
9,412
star
5

react-native-skia

High-performance React Native Graphics using Skia
TypeScript
6,746
star
6

flash-list

A better list for React Native
TypeScript
5,489
star
7

polaris

Shopify’s design system to help us work together to build a great experience for all of our merchants.
TypeScript
5,352
star
8

hydrogen-v1

React-based framework for building dynamic, Shopify-powered custom storefronts.
TypeScript
3,747
star
9

go-lua

A Lua VM in Go
Go
2,773
star
10

bootsnap

Boot large Ruby/Rails apps faster
Ruby
2,614
star
11

graphql-design-tutorial

2,335
star
12

restyle

A type-enforced system for building UI components in React Native with TypeScript.
TypeScript
2,331
star
13

dawn

Shopify's first source available reference theme, with Online Store 2.0 features and performance built-in.
Liquid
2,279
star
14

identity_cache

IdentityCache is a blob level caching solution to plug into Active Record. Don't #find, #fetch!
Ruby
1,874
star
15

quilt

A loosely related set of packages for JavaScript/TypeScript projects at Shopify
TypeScript
1,703
star
16

shopify_app

A Rails Engine for building Shopify Apps
Ruby
1,649
star
17

kubeaudit

kubeaudit helps you audit your Kubernetes clusters against common security controls
Go
1,624
star
18

shipit-engine

Deployment coordination
Ruby
1,406
star
19

graphql-batch

A query batching executor for the graphql gem
Ruby
1,388
star
20

packwerk

Good things come in small packages.
Ruby
1,346
star
21

krane

A command-line tool that helps you ship changes to a Kubernetes namespace and understand the result
Ruby
1,309
star
22

semian

🐒 Resiliency toolkit for Ruby for failing fast
Ruby
1,286
star
23

slate

Slate is a toolkit for developing Shopify themes. It's designed to assist your workflow and speed up the process of developing, testing, and deploying themes.
JavaScript
1,283
star
24

ejson

EJSON is a small library to manage encrypted secrets using asymmetric encryption.
Go
1,246
star
25

superdb

The Super Debugger, a realtime wireless debugger for iOS
Objective-C
1,158
star
26

shopify_python_api

ShopifyAPI library allows Python developers to programmatically access the admin section of stores
Python
1,072
star
27

storefront-api-examples

Example custom storefront applications built on Shopify's Storefront API
JavaScript
1,069
star
28

themekit

Shopify theme development command line tool.
Go
1,068
star
29

Timber

The ultimate Shopify theme framework, built by Shopify.
Liquid
992
star
30

shopify-cli

Shopify CLI helps you build against the Shopify platform faster.
Ruby
987
star
31

shopify-api-ruby

ShopifyAPI is a lightweight gem for accessing the Shopify admin REST and GraphQL web services.
Ruby
982
star
32

hydrogen

Hydrogen is Shopify’s stack for headless commerce. It provides a set of tools, utilities, and best-in-class examples for building dynamic and performant commerce applications. Hydrogen is designed to dovetail with Remix, Shopify’s full stack web framework, but it also provides a React library portable to other supporting frameworks. Demo store 👇🏼
TypeScript
966
star
33

js-buy-sdk

The JS Buy SDK is a lightweight library that allows you to build ecommerce into any website. It is based on Shopify's API and provides the ability to retrieve products and collections from your shop, add products to a cart, and checkout.
JavaScript
932
star
34

job-iteration

Makes your background jobs interruptible and resumable by design.
Ruby
907
star
35

cli-ui

Terminal user interface library
Ruby
869
star
36

react-native-performance

Performance monitoring for React Native apps
TypeScript
860
star
37

ruby-lsp

An opinionated language server for Ruby
Ruby
851
star
38

active_shipping

ActiveShipping is a simple shipping abstraction library extracted from Shopify
Ruby
809
star
39

shopify-api-js

Shopify Admin API Library for Node. Accelerate development with support for authentication, graphql proxy, webhooks
TypeScript
765
star
40

tapioca

The swiss army knife of RBI generation
Ruby
733
star
41

maintenance_tasks

A Rails engine for queueing and managing data migrations.
Ruby
705
star
42

shopify-app-template-node

JavaScript
701
star
43

remote-ui

TypeScript
701
star
44

erb_lint

Lint your ERB or HTML files
Ruby
651
star
45

shopify_theme

A console tool for interacting with Shopify Theme Assets.
Ruby
640
star
46

pitchfork

Ruby
630
star
47

ghostferry

The swiss army knife of live data migrations
Go
596
star
48

yjit

Optimizing JIT compiler built inside CRuby
593
star
49

statsd-instrument

A StatsD client for Ruby apps. Provides metaprogramming methods to inject StatsD instrumentation into your code.
Ruby
546
star
50

autotuner

Get suggestions to tune Ruby's garbage collector
Ruby
511
star
51

shopify.github.com

A collection of the open source projects by Shopify
CSS
505
star
52

ruby-style-guide

Shopify’s Ruby Style Guide
Ruby
475
star
53

theme-scripts

Theme Scripts is a collection of utility libraries which help theme developers with problems unique to Shopify Themes.
JavaScript
470
star
54

livedata-ktx

Kotlin extension for LiveData, chaining like RxJava
Kotlin
468
star
55

starter-theme

The Shopify Themes Team opinionated starting point for new a Slate project
Liquid
459
star
56

shopify-demo-app-node-react

JavaScript
444
star
57

web-configs

Common configurations for building web apps at Shopify
JavaScript
433
star
58

mobile-buy-sdk-ios

Shopify’s Mobile Buy SDK makes it simple to sell physical products inside your mobile app. With a few lines of code, you can connect your app with the Shopify platform and let your users buy your products using Apple Pay or their credit card.
Swift
433
star
59

shopify_django_app

Get a Shopify app up and running with Django and Python Shopify API
Python
425
star
60

deprecation_toolkit

⚒Eliminate deprecations from your codebase ⚒
Ruby
390
star
61

ruby-lsp-rails

A Ruby LSP extension for Rails
Ruby
388
star
62

bootboot

Dualboot your Ruby app made easy
Ruby
374
star
63

FunctionalTableData

Declarative UITableViewDataSource implementation
Swift
365
star
64

shadowenv

reversible directory-local environment variable manipulations
Rust
349
star
65

shopify-node-app

An example app that uses Polaris components and shopify-express
JavaScript
327
star
66

polaris-viz

A collection of React and React native components that compose Shopify's data visualization system
TypeScript
317
star
67

better-html

Better HTML for Rails
Ruby
311
star
68

theme-check

The Ultimate Shopify Theme Linter
Ruby
306
star
69

product-reviews-sample-app

A sample Shopify application that creates and stores product reviews for a store, written in Node.js
JavaScript
300
star
70

tracky

The easiest way to do motion tracking!
Swift
295
star
71

shopify-api-php

PHP
279
star
72

measured

Encapsulate measurements and their units in Ruby.
Ruby
275
star
73

cli

Build apps, themes, and hydrogen storefronts for Shopify
TypeScript
273
star
74

money

Manage money in Shopify with a class that won't lose pennies during division
Ruby
265
star
75

javascript

The home for all things JavaScript at Shopify.
253
star
76

ruvy

Rust
252
star
77

limiter

Simple Ruby rate limiting mechanism.
Ruby
244
star
78

vscode-ruby-lsp

VS Code plugin for connecting with the Ruby LSP
TypeScript
232
star
79

ruby_memcheck

Use Valgrind memcheck on your native gem without going crazy
Ruby
230
star
80

polaris-tokens

Design tokens for Polaris, Shopify’s design system
TypeScript
230
star
81

buy-button-js

BuyButton.js is a highly customizable UI library for adding ecommerce functionality to any website.
JavaScript
230
star
82

android-testify

Add screenshots to your Android tests
Kotlin
225
star
83

spoom

Useful tools for Sorbet enthusiasts
Ruby
220
star
84

turbograft

Hard fork of turbolinks, adding partial page replacement strategies, and utilities.
JavaScript
213
star
85

mobile-buy-sdk-android

Shopify’s Mobile Buy SDK makes it simple to sell physical products inside your mobile app. With a few lines of code, you can connect your app with the Shopify platform and let your users buy your products using their credit card.
Java
202
star
86

graphql-js-client

A Relay compliant GraphQL client.
JavaScript
187
star
87

shopify-app-template-php

PHP
186
star
88

skeleton-theme

A barebones ☠️starter theme with the required files needed to compile with Slate and upload to Shopify.
Liquid
185
star
89

sprockets-commoner

Use Babel in Sprockets to compile JavaScript modules for the browser
Ruby
182
star
90

rotoscope

High-performance logger of Ruby method invocations
Ruby
180
star
91

shopify-app-template-remix

TypeScript
178
star
92

git-chain

Tool to rebase multiple Git branches based on the previous one.
Ruby
176
star
93

verdict

Framework to define and implement A/B tests in your application, and collect data for analysis purposes.
Ruby
176
star
94

hydrogen-react

Reusable components and utilities for building Shopify-powered custom storefronts.
TypeScript
174
star
95

ui-extensions

TypeScript
173
star
96

storefront-api-learning-kit

JavaScript
171
star
97

heap-profiler

Ruby heap profiler
C++
159
star
98

autoload_reloader

Experimental implementation of code reloading using Ruby's autoload
Ruby
158
star
99

graphql-metrics

Extract as much much detail as you want from GraphQL queries, served up from your Ruby app and the graphql gem.
Ruby
157
star
100

active_fulfillment

Active Merchant library for integration with order fulfillment services
Ruby
155
star