modelstore
modelstore
is a Python library that allows you to version, export, save and download machine learning models in your choice of storage.
modelstore is an open source model registry
- Store models on a local file system or in a bucket
- Support for multiple clouds (AWS, GCP, Azure)
- Models are versioned on each upload
- Replaces all the boiler plate code you need to save models
- List models in a domain
- Create model states and manage which state a model is in
- Download models by id
- Load models straight from your storage back into memory
- Download models from the command line
For more details, please refer to the documentation.
modelstore is being built in the open
#oss-modelstore
channel.
Installation
pip install modelstore
Supported storage types
- AWS S3 Bucket (example)
- Azure Blob Storage (example)
- Google Cloud Storage Bucket (example)
- Any s3-compatible object storage that you can access via MinIO
- A filesystem directory (example)
Supported machine learning libraries
- Annoy
- Catboost
- Fast.AI
- Gensim
- Keras
- LightGBM
- Mxnet
- Onnx
- Prophet
- PyTorch
- PyTorch Lightning
- Scikit Learn
- Skorch
- Shap
- Spark ML Lib
- Tensorflow
- Transformers
- XGBoost
Is there a machine learning framework that is missing?
- Save your model and then upload it as a raw file.
- Feel free to open an issue
Read more about modelstore
- Evidently.AI AMA with Neal Lathia, January 2023
- MLOps Model Stores: Definition, Functionality, Tools Review, January 2023
- Monzo's machine learning stack, April 2022
- Model arterfacts: the war stories, September 2020
Example Usage
Colab Notebook
There is a full example in this Colab notebook.
Python Script
from modelstore import ModelStore
#Β And your other imports
# Train your model
clf = RandomForestClassifier(n_estimators=10)
clf = clf.fit(X, Y)
# Create a model store that uses a one of the storage options
# In this example, the model store is created with a GCP bucket
model_store = ModelStore.from_gcloud(
project_name="my-project",
bucket_name="my-bucket",
)
# Upload the archive to your model store
domain = "example-model"
meta_data = model_store.upload(domain, model=clf)
# Print the meta-data about the model
print(json.dumps(meta_data, indent=4))
# Load the model back!
clf = model_store.load(domain=model_domain, model_id=meta["model"]["model_id"])
Find out more
Watch an interview and demo, recorded with Alexey from the Data Talks Club in July 2021, is based on modelstore==0.0.6
, on YouTube. Note the talk below is based on an older version of modelstore
and the API has been simplified since then.
License
Copyright 2020 Neal Lathia
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.