BentoML: The Unified AI Application Framework
BentoML is a framework for building reliable, scalable, and cost-efficient AI applications. It comes with everything you need for model serving, application packaging, and production deployment.
Highlights
🍱 Bento is the container for AI apps
- Open standard and SDK for AI apps, pack your code, inference pipelines, model files, dependencies, and runtime configurations in a Bento.
- Auto-generate API servers, supporting REST API, gRPC, and long-running inference jobs.
- Auto-generate Docker container images.
🏄 Freedom to build with any AI models
- Import from any model hub or bring your own models built with frameworks like PyTorch, TensorFlow, Keras, Scikit-Learn, XGBoost and many more.
- Native support for LLM inference, generative AI, embedding creation, and multi-modal AI apps.
- Run and debug your BentoML apps locally on Mac, Windows, or Linux.
🍭 Simplify modern AI application architecture
- Python-first! Effortlessly scale complex AI workloads.
- Enable GPU inference without the headache.
- Compose multiple models to run concurrently or sequentially, over multiple GPUs or on a Kubernetes Cluster.
- Natively integrates with MLFlow, LangChain, Kubeflow, Triton, Spark, Ray, and many more to complete your production AI stack.
🚀 Deploy Anywhere
- One-click deployment to
☁️ BentoCloud, the Serverless platform made for hosting and operating AI apps. - Scalable BentoML deployment with
🦄️ Yatai on Kubernetes. - Deploy auto-generated container images anywhere docker runs.
Documentation
- Installation:
pip install bentoml - Full Documentation: docs.bentoml.com
- Tutorial: Intro to BentoML
🛠️ What you can build with BentoML
- OpenLLM - An open platform for operating large language models (LLMs) in production.
- StableDiffusion - Create your own text-to-image service with any diffusion models.
- CLIP-API-service - Embed images and sentences, object recognition, visual reasoning, image classification, and reverse image search.
- Transformer NLP Service - Online inference API for Transformer NLP models.
- Distributed TaskMatrix(Visual ChatGPT) - Scalable deployment for TaskMatrix from Microsoft.
- Fraud Detection - Online model serving with custom XGBoost model.
- OCR as a Service - Turn any OCR models into online inference API endpoints.
- Replace Anything - Combining the power of Segment Anything and Stable Diffusion.
- DeepFloyd IF Multi-GPU serving - Serve IF models easily across multiple GPUs
- Check out more examples here.
Getting Started
Save or import models in BentoML local model store:
import bentoml
import transformers
pipe = transformers.pipeline("text-classification")
bentoml.transformers.save_model(
"text-classification-pipe",
pipe,
signatures={
"__call__": {"batchable": True} # Enable dynamic batching for model
}
)View all models saved locally:
$ bentoml models list
Tag Module Size Creation Time
text-classification-pipe:kn6mr3aubcuf… bentoml.transformers 256.35 MiB 2023-05-17 14:36:25Define how your model runs in a service.py file:
import bentoml
model_runner = bentoml.models.get("text-classification-pipe").to_runner()
svc = bentoml.Service("text-classification-service", runners=[model_runner])
@svc.api(input=bentoml.io.Text(), output=bentoml.io.JSON())
async def classify(text: str) -> str:
results = await model_runner.async_run([text])
return results[0]Now, run the API service locally:
bentoml serve service.py:svcSent a prediction request:
$ curl -X POST -H "Content-Type: text/plain" --data "BentoML is awesome" http://localhost:3000/classify
{"label":"POSITIVE","score":0.9129443168640137}%Define how a Bento can
be built for deployment, with bentofile.yaml:
service: 'service.py:svc'
name: text-classification-svc
include:
- 'service.py'
python:
packages:
- torch>=2.0
- transformersBuild a Bento and generate a docker image:
$ bentoml build
...
Successfully built Bento(tag="text-classification-svc:mc322vaubkuapuqj").$ bentoml containerize text-classification-svc
Building OCI-compliant image for text-classification-svc:mc322vaubkuapuqj with docker
...
Successfully built Bento container for "text-classification-svc" with tag(s) "text-classification-svc:mc322vaubkuapuqj"$ docker run -p 3000:3000 text-classification-svc:mc322vaubkuapuqjFor a more detailed user guide, check out the BentoML Tutorial.
Community
BentoML supports billions of model runs per day and is used by thousands of organizations around the globe.
Join our Community Slack
To report a bug or suggest a feature request, use GitHub Issues.
Contributing
There are many ways to contribute to the project:
- Report bugs and "Thumbs up" on issues that are relevant to you.
- Investigate issues and review other developers' pull requests.
- Contribute code or documentation to the project by submitting a GitHub pull request.
- Check out the Contributing Guide and Development Guide to learn more
- Share your feedback and discuss roadmap plans in the
#bentoml-contributorschannel here.
Thanks to all of our amazing contributors!
Usage Reporting
BentoML collects usage data that helps our team to improve the product. Only
BentoML's internal API calls are being reported. We strip out as much
potentially sensitive information as possible, and we will never collect user
code, model data, model names, or stack traces. Here's the
code for usage
tracking. You can opt-out of usage tracking by the --do-not-track CLI option:
bentoml [command] --do-not-trackOr by setting environment variable BENTOML_DO_NOT_TRACK=True:
export BENTOML_DO_NOT_TRACK=TrueLicense
Citation
If you use BentoML in your research, please cite using the following [citation](./CITATION.cff:
@software{Yang_BentoML_The_framework,
author = {Yang, Chaoyu and Sheng, Sean and Pham, Aaron and Zhao, Shenyang and Lee, Sauyon and Jiang, Bo and Dong, Fog and Guan, Xipeng and Ming, Frost},
license = {Apache-2.0},
title = {{BentoML: The framework for building reliable, scalable and cost-efficient AI application}},
url = {https://github.com/bentoml/bentoml}
}