BentoML

Founded 2019

Package, deploy, and scale machine learning models across any cloud infrastructure

You can use BentoML to package, serve, and run machine learning models in production environments. It supports popular open-source models like Llama, DeepSeek, and Flux, as well as your own custom models built with PyTorch, vLLM, or Transformers. The service lets you deploy directly on your cloud infrastructure or on managed hardware with automatic scaling, cold-start reduction, and scale-to-zero capabilities. You can also chain multiple models together for complex workflows, monitor system performance metrics, and manage deployments with built-in version control and rollback mechanisms.

What BentoML does

Package custom PyTorch models into standardized container imagesConfigure auto-scaling policies based on incoming trafficDeploy pre-built open-source models like Llama and DeepSeekChain multiple AI models for retrieval-augmented generationTrack hardware and latency metrics across active deploymentsRoll back model versions when encountering production issuesDeploy open-source and custom models using a unified Python frameworkScale compute resources down to zero when idle to reduce costsDeploy on any cloud provider, on-premises Kubernetes, or managed cloud GPUsChain multiple models together for complex retrieval-augmented generation pipelinesSupport specialized runtimes including vLLM, TRT-LLM, and SGLangMonitor hardware utilization, latency, and inference metrics in real timeRoll back releases with canary and shadow deployment testing