• Stars
    star
    129
  • Rank 273,112 (Top 6 %)
  • Language
    R
  • License
    Other
  • Created about 5 years ago
  • Updated 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Reduce the size of model objects saved to disk

butcher butcher website

R-CMD-check CRAN status Codecov test coverage Lifecycle: stable

Overview

Modeling or machine learning in R can result in fitted model objects that take up too much memory. There are two main culprits:

  1. Heavy usage of formulas and closures that capture the enclosing environment in model training
  2. Lack of selectivity in the construction of the model object itself

As a result, fitted model objects contain components that are often redundant and not required for post-fit estimation activities. The butcher package provides tooling to “axe” parts of the fitted output that are no longer needed, without sacrificing prediction functionality from the original model object.

Installation

Install the released version from CRAN:

install.packages("butcher")

Or install the development version from GitHub:

# install.packages("pak")
pak::pak("tidymodels/butcher")

Butchering

As an example, let’s wrap an lm model so it contains a lot of unnecessary stuff:

library(butcher)
our_model <- function() {
  some_junk_in_the_environment <- runif(1e6) # we didn't know about
  lm(mpg ~ ., data = mtcars) 
}

This object is unnecessarily large:

library(lobstr)
obj_size(our_model())
#> 8.02 MB

When, in fact, it should only be:

small_lm <- lm(mpg ~ ., data = mtcars) 
obj_size(small_lm)
#> 22.22 kB

To understand which part of our original model object is taking up the most memory, we leverage the weigh() function:

big_lm <- our_model()
weigh(big_lm)
#> # A tibble: 25 × 2
#>    object            size
#>    <chr>            <dbl>
#>  1 terms         8.05    
#>  2 qr.qr         0.00666 
#>  3 residuals     0.00286 
#>  4 fitted.values 0.00286 
#>  5 effects       0.0014  
#>  6 coefficients  0.00109 
#>  7 call          0.000728
#>  8 model.mpg     0.000304
#>  9 model.cyl     0.000304
#> 10 model.disp    0.000304
#> # ℹ 15 more rows

The problem here is in the terms component of our big_lm. Because of how lm() is implemented in the stats package, the environment in which our model was made is carried along in the fitted output. To remove the (mostly) extraneous component, we can use butcher():

cleaned_lm <- butcher(big_lm, verbose = TRUE)
#> ✔ Memory released: 8.03 MB
#> ✖ Disabled: `print()`, `summary()`, and `fitted()`

Comparing it against our small_lm, we find:

weigh(cleaned_lm)
#> # A tibble: 25 × 2
#>    object           size
#>    <chr>           <dbl>
#>  1 terms        0.00771 
#>  2 qr.qr        0.00666 
#>  3 residuals    0.00286 
#>  4 effects      0.0014  
#>  5 coefficients 0.00109 
#>  6 model.mpg    0.000304
#>  7 model.cyl    0.000304
#>  8 model.disp   0.000304
#>  9 model.hp     0.000304
#> 10 model.drat   0.000304
#> # ℹ 15 more rows

And now it will take up about the same memory on disk as small_lm:

weigh(small_lm)
#> # A tibble: 25 × 2
#>    object            size
#>    <chr>            <dbl>
#>  1 terms         8.06    
#>  2 qr.qr         0.00666 
#>  3 residuals     0.00286 
#>  4 fitted.values 0.00286 
#>  5 effects       0.0014  
#>  6 coefficients  0.00109 
#>  7 call          0.000728
#>  8 model.mpg     0.000304
#>  9 model.cyl     0.000304
#> 10 model.disp    0.000304
#> # ℹ 15 more rows

To make the most of your memory available, this package provides five S3 generics for you to remove parts of a model object:

  • axe_call(): To remove the call object.
  • axe_ctrl(): To remove controls associated with training.
  • axe_data(): To remove the original training data.
  • axe_env(): To remove environments.
  • axe_fitted(): To remove fitted values.

When you run butcher(), you execute all of these axing functions at once. Any kind of axing on the object will append a butchered class to the current model object class(es) as well as a new attribute named butcher_disabled that lists any post-fit estimation functions that are disabled as a result.

Model Object Coverage

Check out the vignette("available-axe-methods") to see butcher’s current coverage. If you are working with a new model object that could benefit from any kind of axing, we would love for you to make a pull request! You can visit the vignette("adding-models-to-butcher") for more guidelines, but in short, to contribute a set of axe methods:

  1. Run new_model_butcher(model_class = "your_object", package_name = "your_package")
  2. Use butcher helper functions weigh() and locate() to decide what to axe
  3. Finalize edits to R/your_object.R and tests/testthat/test-your_object.R
  4. Make a pull request!

Contributing

This project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

More Repositories

1

broom

Convert statistical analysis objects from R into tidy format
R
1,424
star
2

tidymodels

Easily install and load the tidymodels packages
R
738
star
3

infer

An R package for tidyverse-friendly statistical inference
R
707
star
4

corrr

Explore correlations in R
R
585
star
5

TMwR

Code and content for "Tidy Modeling with R"
RMarkdown
569
star
6

parsnip

A tidy unified interface to models
R
563
star
7

recipes

Pipeable steps for feature engineering and data preprocessing to prepare for modeling
R
546
star
8

yardstick

Tidy methods for measuring model performance
R
363
star
9

rsample

Classes and functions to create and summarize resampling objects
R
328
star
10

stacks

An R package for tidy stacked ensemble modeling
R
285
star
11

tune

Tools for tidy parameter tuning
R
260
star
12

tidypredict

Run predictions inside the database
R
259
star
13

workflows

Modeling Workflows
R
198
star
14

textrecipes

Extra recipes for Text Processing
R
157
star
15

embed

Extra recipes for predictor embeddings
R
141
star
16

themis

Extra recipes steps for dealing with unbalanced data
R
139
star
17

censored

Parsnip wrappers for survival models
R
123
star
18

dials

Tools for creating tuning parameter values
R
111
star
19

probably

Tools for post-processing class probability estimates
R
110
star
20

tidyclust

A tidy unified interface to clustering models
R
104
star
21

tidyposterior

Bayesian comparisons of models using resampled statistics
R
101
star
22

hardhat

Construct Modeling Packages
R
100
star
23

tidymodels.org-legacy

Legacy Source of tidymodels.org
HTML
100
star
24

aml-training

The most recent version of the Applied Machine Learning notes
HTML
100
star
25

workflowsets

Create a collection of modeling workflows
R
88
star
26

usemodels

Boilerplate Code for tidymodels
R
85
star
27

modeldb

Run models inside a database using R
R
80
star
28

workshops

Website and materials for tidymodels workshops
JavaScript
76
star
29

multilevelmod

Parsnip wrappers for mixed-level and hierarchical models
R
74
star
30

spatialsample

Create and summarize spatial resampling objects 🗺
R
70
star
31

learntidymodels

Learn tidymodels with interactive learnr primers
R
66
star
32

brulee

High-Level Modeling Functions with 'torch'
R
62
star
33

finetune

Additional functions for model tuning
R
61
star
34

shinymodels

R
46
star
35

applicable

Quantify extrapolation of new samples given a training set
R
44
star
36

model-implementation-principles

recommendations for creating R modeling packages
HTML
41
star
37

bonsai

parsnip wrappers for tree-based models
R
41
star
38

rules

parsnip extension for rule-based models
R
39
star
39

planning

Documents to plan and discuss future development
37
star
40

discrim

Wrappers for discriminant analysis and naive Bayes models for use with the parsnip package
R
28
star
41

baguette

parsnip Model Functions for Bagging
R
24
star
42

modeldata

Data Sets Used by tidymodels Packages
R
22
star
43

poissonreg

parsnip wrappers for Poisson regression
R
22
star
44

agua

Create and evaluate models using 'tidymodels' and 'h2o'
R
21
star
45

extratests

Integration and other testing for tidymodels
R
20
star
46

tidymodels.org

Source of tidymodels.org
JavaScript
16
star
47

plsmod

Model Wrappers for Projection Methods
R
14
star
48

cloudstart

RStudio Cloud ☁️ resources to accompany tidymodels.org
12
star
49

modeldatatoo

More Data Sets Useful for Modeling Examples
R
7
star
50

desirability2

Desirability Functions for Multiparameter Optimization
R
7
star
51

.github

GitHub contributing guidelines for tidymodels packages
4
star
52

modelenv

Provide Tools to Register Models for use in Tidymodels
R
4
star
53

tailor

Sandbox for a postprocessor object.
R
2
star
54

survivalauc

What the Package Does (One Line, Title Case)
C
2
star