triton-inference-server/core

Stars
78
Rank 398,001 (Top 9 %)
Language
C++
License
BSD 3-Clause "New...
Created over 3 years ago
Updated 2 months ago

triton-inference-server/core

triton-inference-server

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

The core library and APIs implementing the Triton Inference Server.

server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

pytriton

PyTriton is a Flask/FastAPI-like interface that simplifies Triton's deployment in Python environments.

client

Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.

python_backend

Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.

tensorrtllm_backend

The Triton TensorRT-LLM Backend

fastertransformer_backend

tutorials

This repository contains tutorials and examples for Triton Inference Server

model_analyzer

Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models.

backend

Common source, scripts and utilities for creating Triton backends.

model_navigator

Triton Model Navigator is a tool that provides the ability to automate the process of model deployment on the Triton Inference Server.

dali_backend

The Triton backend that allows running GPU-accelerated data pre-processing pipelines implemented in DALI's python API.

onnxruntime_backend

The Triton backend for the ONNX Runtime.

pytorch_backend

The Triton backend for the PyTorch TorchScript models.

vllm_backend

fil_backend

FIL backend for the Triton Inference Server

Jupyter Notebook

common

Common source, scripts and utilities shared across all Triton repositories.

hugectr_backend

Jupyter Notebook

tensorrt_backend

The Triton backend for TensorRT.

tensorflow_backend

The Triton backend for TensorFlow.

paddlepaddle_backend

openvino_backend

OpenVINO backend for Triton.

developer_tools

stateful_backend

Triton backend for managing the model state tensors automatically in sequence batcher

contrib

Community contributions to Triton that are not officially supported or maintained by the Triton project.

third_party

Third-party source packages that are modified for use in Triton.

checksum_repository_agent

The Triton repository agent that verifies model checksums.

identity_backend

Example Triton backend that demonstrates most of the Triton Backend API.

redis_cache

TRITONCACHE implementation of a Redis cache

repeat_backend

An example Triton backend that demonstrates sending zero, one, or multiple responses for each request.

local_cache

Implementation of a local in-memory cache for Triton Inference Server's TRITONCACHE API

square_backend

Simple Triton backend used for testing.