• Stars
    star
    8,784
  • Rank 3,882 (Top 0.08 %)
  • Language
    C++
  • License
    Apache License 2.0
  • Created almost 5 years ago
  • Updated about 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

License Documentation

TensorRT Open Source Software

This repository contains the Open Source Software (OSS) components of NVIDIA TensorRT. It includes the sources for TensorRT plugins and parsers (Caffe and ONNX), as well as sample applications demonstrating usage and capabilities of the TensorRT platform. These open source software components are a subset of the TensorRT General Availability (GA) release with some extensions and bug-fixes.

Need enterprise support? NVIDIA global support is available for TensorRT with the NVIDIA AI Enterprise software suite. Check out NVIDIA LaunchPad for free access to a set of hands-on labs with TensorRT hosted on NVIDIA infrastructure.

Join the TensorRT and Triton community and stay current on the latest product updates, bug fixes, content, best practices, and more.

Prebuilt TensorRT Python Package

We provide the TensorRT Python package for an easy installation.
To install:

pip install tensorrt

You can skip the Build section to enjoy TensorRT with Python.

Build

Prerequisites

To build the TensorRT-OSS components, you will first need the following software packages.

TensorRT GA build

System Packages

Optional Packages

Downloading TensorRT Build

  1. Download TensorRT OSS

    git clone -b main https://github.com/nvidia/TensorRT TensorRT
    cd TensorRT
    git submodule update --init --recursive
  2. (Optional - if not using TensorRT container) Specify the TensorRT GA release build path

    If using the TensorRT OSS build container, TensorRT libraries are preinstalled under /usr/lib/x86_64-linux-gnu and you may skip this step.

    Else download and extract the TensorRT GA build from NVIDIA Developer Zone.

    Example: Ubuntu 20.04 on x86-64 with cuda-12.0

    cd ~/Downloads
    tar -xvzf TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0.tar.gz
    export TRT_LIBPATH=`pwd`/TensorRT-8.6.1.6
  3. (Optional - for Jetson builds only) Download the JetPack SDK

    1. Download and launch the JetPack SDK manager. Login with your NVIDIA developer account.
    2. Select the platform and target OS (example: Jetson AGX Xavier, Linux Jetpack 5.0), and click Continue.
    3. Under Download & Install Options change the download folder and select Download now, Install later. Agree to the license terms and click Continue.
    4. Move the extracted files into the <TensorRT-OSS>/docker/jetpack_files folder.

Setting Up The Build Environment

For Linux platforms, we recommend that you generate a docker container for building TensorRT OSS as described below. For native builds, please install the prerequisite System Packages.

  1. Generate the TensorRT-OSS build container.

    The TensorRT-OSS build container can be generated using the supplied Dockerfiles and build scripts. The build containers are configured for building TensorRT OSS out-of-the-box.

    Example: Ubuntu 20.04 on x86-64 with cuda-12.0 (default)

    ./docker/build.sh --file docker/ubuntu-20.04.Dockerfile --tag tensorrt-ubuntu20.04-cuda12.0

    Example: CentOS/RedHat 7 on x86-64 with cuda-11.8

    ./docker/build.sh --file docker/centos-7.Dockerfile --tag tensorrt-centos7-cuda11.8 --cuda 11.8.0

    Example: Ubuntu 20.04 cross-compile for Jetson (aarch64) with cuda-11.4.2 (JetPack SDK)

    ./docker/build.sh --file docker/ubuntu-cross-aarch64.Dockerfile --tag tensorrt-jetpack-cuda11.4

    Example: Ubuntu 20.04 on aarch64 with cuda-11.8

    ./docker/build.sh --file docker/ubuntu-20.04-aarch64.Dockerfile --tag tensorrt-aarch64-ubuntu20.04-cuda11.8 --cuda 11.8.0
  2. Launch the TensorRT-OSS build container.

    Example: Ubuntu 20.04 build container

    ./docker/launch.sh --tag tensorrt-ubuntu20.04-cuda12.0 --gpus all

    NOTE:
    1. Use the --tag corresponding to build container generated in Step 1.
    2. NVIDIA Container Toolkit is required for GPU access (running TensorRT applications) inside the build container.
    3. sudo password for Ubuntu build containers is 'nvidia'.
    4. Specify port number using --jupyter <port> for launching Jupyter notebooks.

Building TensorRT-OSS

  • Generate Makefiles and build.

    Example: Linux (x86-64) build with default cuda-12.0

     cd $TRT_OSSPATH
     mkdir -p build && cd build
     cmake .. -DTRT_LIB_DIR=$TRT_LIBPATH -DTRT_OUT_DIR=`pwd`/out
     make -j$(nproc)

    NOTE: On CentOS7, the default g++ version does not support C++14. For native builds (not using the CentOS7 build container), first install devtoolset-8 to obtain the updated g++ toolchain as follows:

    yum -y install centos-release-scl
    yum-config-manager --enable rhel-server-rhscl-7-rpms
    yum -y install devtoolset-8
    export PATH="/opt/rh/devtoolset-8/root/bin:${PATH}"

    Example: Linux (aarch64) build with default cuda-12.0

     cd $TRT_OSSPATH
     mkdir -p build && cd build
     cmake .. -DTRT_LIB_DIR=$TRT_LIBPATH -DTRT_OUT_DIR=`pwd`/out -DCMAKE_TOOLCHAIN_FILE=$TRT_OSSPATH/cmake/toolchains/cmake_aarch64-native.toolchain
     make -j$(nproc)

    Example: Native build on Jetson (aarch64) with cuda-11.4

    cd $TRT_OSSPATH
    mkdir -p build && cd build
    cmake .. -DTRT_LIB_DIR=$TRT_LIBPATH -DTRT_OUT_DIR=`pwd`/out -DTRT_PLATFORM_ID=aarch64 -DCUDA_VERSION=11.4
    CC=/usr/bin/gcc make -j$(nproc)

    NOTE: C compiler must be explicitly specified via CC= for native aarch64 builds of protobuf.

    Example: Ubuntu 20.04 Cross-Compile for Jetson (aarch64) with cuda-11.4 (JetPack)

    cd $TRT_OSSPATH
    mkdir -p build && cd build
    cmake .. -DCMAKE_TOOLCHAIN_FILE=$TRT_OSSPATH/cmake/toolchains/cmake_aarch64.toolchain -DCUDA_VERSION=11.4 -DCUDNN_LIB=/pdk_files/cudnn/usr/lib/aarch64-linux-gnu/libcudnn.so -DCUBLAS_LIB=/usr/local/cuda-11.4/targets/aarch64-linux/lib/stubs/libcublas.so -DCUBLASLT_LIB=/usr/local/cuda-11.4/targets/aarch64-linux/lib/stubs/libcublasLt.so -DTRT_LIB_DIR=/pdk_files/tensorrt/lib
    
    make -j$(nproc)

    NOTE: The latest JetPack SDK v5.1 only supports TensorRT 8.5.2.

    NOTE:
    1. The default CUDA version used by CMake is 12.0.1. To override this, for example to 11.8, append -DCUDA_VERSION=11.8 to the cmake command.
    2. If samples fail to link on CentOS7, create this symbolic link: ln -s $TRT_OUT_DIR/libnvinfer_plugin.so $TRT_OUT_DIR/libnvinfer_plugin.so.8

  • Required CMake build arguments are:

    • TRT_LIB_DIR: Path to the TensorRT installation directory containing libraries.
    • TRT_OUT_DIR: Output directory where generated build artifacts will be copied.
  • Optional CMake build arguments:

    • CMAKE_BUILD_TYPE: Specify if binaries generated are for release or debug (contain debug symbols). Values consists of [Release] | Debug
    • CUDA_VERSION: The version of CUDA to target, for example [11.7.1].
    • CUDNN_VERSION: The version of cuDNN to target, for example [8.6].
    • PROTOBUF_VERSION: The version of Protobuf to use, for example [3.0.0]. Note: Changing this will not configure CMake to use a system version of Protobuf, it will configure CMake to download and try building that version.
    • CMAKE_TOOLCHAIN_FILE: The path to a toolchain file for cross compilation.
    • BUILD_PARSERS: Specify if the parsers should be built, for example [ON] | OFF. If turned OFF, CMake will try to find precompiled versions of the parser libraries to use in compiling samples. First in ${TRT_LIB_DIR}, then on the system. If the build type is Debug, then it will prefer debug builds of the libraries before release versions if available.
    • BUILD_PLUGINS: Specify if the plugins should be built, for example [ON] | OFF. If turned OFF, CMake will try to find a precompiled version of the plugin library to use in compiling samples. First in ${TRT_LIB_DIR}, then on the system. If the build type is Debug, then it will prefer debug builds of the libraries before release versions if available.
    • BUILD_SAMPLES: Specify if the samples should be built, for example [ON] | OFF.
    • GPU_ARCHS: GPU (SM) architectures to target. By default we generate CUDA code for all major SMs. Specific SM versions can be specified here as a quoted space-separated list to reduce compilation time and binary size. Table of compute capabilities of NVIDIA GPUs can be found here. Examples:
      • NVidia A100: -DGPU_ARCHS="80"
      • Tesla T4, GeForce RTX 2080: -DGPU_ARCHS="75"
      • Titan V, Tesla V100: -DGPU_ARCHS="70"
      • Multiple SMs: -DGPU_ARCHS="80 75"
    • TRT_PLATFORM_ID: Bare-metal build (unlike containerized cross-compilation) on non Linux/x86 platforms must explicitly specify the target platform. Currently supported options: x86_64 (default), aarch64

References

TensorRT Resources

Known Issues

More Repositories

1

nvidia-docker

Build and run Docker containers leveraging NVIDIA GPUs
16,896
star
2

open-gpu-kernel-modules

NVIDIA Linux open GPU kernel module source
C
13,784
star
3

DeepLearningExamples

State-of-the-Art Deep Learning scripts organized by models - easy to train and deploy with reproducible accuracy and performance on enterprise-grade infrastructure.
Jupyter Notebook
12,414
star
4

FastPhotoStyle

Style transfer, deep learning, feature transform
Python
11,020
star
5

NeMo

NeMo: a framework for generative AI
Python
9,773
star
6

vid2vid

Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic video-to-video translation.
Python
8,482
star
7

Megatron-LM

Ongoing research training transformer models at scale
Python
8,169
star
8

apex

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch
Python
7,915
star
9

pix2pixHD

Synthesizing and manipulating 2048x1024 images with conditional GANs
Python
6,488
star
10

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
C++
5,923
star
11

FasterTransformer

Transformer related optimization, including BERT, GPT
C++
5,313
star
12

thrust

[ARCHIVED] The C++ parallel algorithms library. See https://github.com/NVIDIA/cccl
C++
4,843
star
13

DALI

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
C++
4,839
star
14

cuda-samples

Samples for CUDA Developers which demonstrates features in CUDA Toolkit
C
4,710
star
15

tacotron2

Tacotron 2 - PyTorch implementation with faster-than-realtime inference
Jupyter Notebook
4,562
star
16

cutlass

CUDA Templates for Linear Algebra Subroutines
C++
4,278
star
17

DIGITS

Deep Learning GPU Training System
HTML
4,105
star
18

NeMo-Guardrails

NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.
Python
3,165
star
19

flownet2-pytorch

Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Python
2,938
star
20

nccl

Optimized primitives for collective multi-GPU communication
C++
2,730
star
21

libcudacxx

[ARCHIVED] The C++ Standard Library for your entire system. See https://github.com/NVIDIA/cccl
C++
2,292
star
22

k8s-device-plugin

NVIDIA device plugin for Kubernetes
Go
2,269
star
23

waveglow

A Flow-based Generative Network for Speech Synthesis
Python
2,133
star
24

trt-llm-rag-windows

A developer reference project for creating Retrieval Augmented Generation (RAG) chatbots on Windows using TensorRT-LLM
Python
2,011
star
25

MinkowskiEngine

Minkowski Engine is an auto-diff neural network library for high-dimensional sparse tensors
Python
2,007
star
26

semantic-segmentation

Nvidia Semantic Segmentation monorepo
Python
1,746
star
27

DeepRecommender

Deep learning for recommender systems
Python
1,662
star
28

Stable-Diffusion-WebUI-TensorRT

TensorRT Extension for Stable Diffusion Web UI
Python
1,660
star
29

cub

[ARCHIVED] Cooperative primitives for CUDA C++. See https://github.com/NVIDIA/cccl
Cuda
1,639
star
30

warp

A Python framework for high performance GPU simulation and graphics
Python
1,573
star
31

OpenSeq2Seq

Toolkit for efficient experimentation with Speech Recognition, Text2Speech and NLP
Python
1,511
star
32

TransformerEngine

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilization in both training and inference.
Python
1,350
star
33

nvidia-container-toolkit

Build and run containers leveraging NVIDIA GPUs
Go
1,239
star
34

GenerativeAIExamples

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
Python
1,238
star
35

VideoProcessingFramework

Set of Python bindings to C++ libraries which provides full HW acceleration for video decoding, encoding and GPU-accelerated color space and pixel format conversions
C++
1,221
star
36

trt-samples-for-hackathon-cn

Simple samples for TensorRT programming
Python
1,211
star
37

open-gpu-doc

Documentation of NVIDIA chip/hardware interfaces
C
1,193
star
38

Q2RTX

NVIDIA’s implementation of RTX ray-tracing in Quake II
C
1,182
star
39

stdexec

`std::execution`, the proposed C++ framework for asynchronous and parallel programming.
C++
1,182
star
40

partialconv

A New Padding Scheme: Partial Convolution based Padding
Python
1,145
star
41

deepops

Tools for building GPU clusters
Shell
1,142
star
42

CUDALibrarySamples

CUDA Library Samples
Cuda
1,122
star
43

gpu-operator

NVIDIA GPU Operator creates/configures/manages GPUs atop Kubernetes
Go
1,117
star
44

MatX

An efficient C++17 GPU numerical computing library with Python-like syntax
C++
1,095
star
45

sentiment-discovery

Unsupervised Language Modeling at scale for robust sentiment classification
Python
1,055
star
46

aistore

AIStore: scalable storage for AI applications
Go
1,052
star
47

nvidia-container-runtime

NVIDIA container runtime
Makefile
1,035
star
48

gpu-monitoring-tools

Tools for monitoring NVIDIA GPUs on Linux
C
974
star
49

retinanet-examples

Fast and accurate object detection with end-to-end GPU optimization
Python
876
star
50

flowtron

Flowtron is an auto-regressive flow-based generative network for text to speech synthesis with control over speech variation and style transfer
Jupyter Notebook
867
star
51

mellotron

Mellotron: a multispeaker voice synthesis model based on Tacotron 2 GST that can make a voice emote and sing without emotive or singing training data
Jupyter Notebook
842
star
52

jetson-gpio

A Python library that enables the use of Jetson's GPIOs
Python
834
star
53

gdrcopy

A fast GPU memory copy library based on NVIDIA GPUDirect RDMA technology
C++
743
star
54

nv-wavenet

Reference implementation of real-time autoregressive wavenet inference
Cuda
728
star
55

tensorflow

An Open Source Machine Learning Framework for Everyone
C++
719
star
56

spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
Scala
704
star
57

cuda-python

CUDA Python Low-level Bindings
Python
695
star
58

libnvidia-container

NVIDIA container runtime library
C
679
star
59

cccl

CUDA C++ Core Libraries
C++
676
star
60

MAXINE-AR-SDK

NVIDIA AR SDK - API headers and sample applications
C
671
star
61

nvvl

A library that uses hardware acceleration to load sequences of video frames to facilitate machine learning training
C++
665
star
62

gvdb-voxels

Sparse volume compute and rendering on NVIDIA GPUs
C
643
star
63

BigVGAN

Official PyTorch implementation of BigVGAN (ICLR 2023)
Python
633
star
64

runx

Deep Learning Experiment Management
Python
630
star
65

nccl-tests

NCCL Tests
Cuda
600
star
66

DLSS

NVIDIA DLSS is a new and improved deep learning neural network that boosts frame rates and generates beautiful, sharp images for your games
C
588
star
67

modulus

Open-source deep-learning framework for building, training, and fine-tuning deep learning models using state-of-the-art Physics-ML methods
Python
556
star
68

dcgm-exporter

NVIDIA GPU metrics exporter for Prometheus leveraging DCGM
Go
551
star
69

Dataset_Synthesizer

NVIDIA Deep learning Dataset Synthesizer (NDDS)
C++
530
star
70

nvcomp

Repository for nvCOMP docs and examples. nvCOMP is a library for fast lossless compression/decompression on the GPU that can be downloaded from https://developer.nvidia.com/nvcomp.
C++
510
star
71

jitify

A single-header C++ library for simplifying the use of CUDA Runtime Compilation (NVRTC).
C++
495
star
72

NVFlare

NVIDIA Federated Learning Application Runtime Environment
Python
486
star
73

libglvnd

The GL Vendor-Neutral Dispatch library
C
462
star
74

enroot

A simple yet powerful tool to turn traditional container/OS images into unprivileged sandboxes.
Shell
459
star
75

MDL-SDK

NVIDIA Material Definition Language SDK
C++
438
star
76

PyProf

A GPU performance profiling tool for PyTorch models
Python
437
star
77

AMGX

Distributed multigrid linear solver library on GPU
Cuda
434
star
78

multi-gpu-programming-models

Examples demonstrating available options to program multiple GPUs in a single node or a cluster
Cuda
422
star
79

gpu-rest-engine

A REST API for Caffe using Docker and Go
C++
421
star
80

framework-reproducibility

Providing reproducibility in deep learning frameworks
Python
412
star
81

hpc-container-maker

HPC Container Maker
Python
404
star
82

cuCollections

C++
400
star
83

nvbench

CUDA Kernel Benchmarking Library
Cuda
391
star
84

NvPipe

NVIDIA-accelerated zero latency video compression library for interactive remoting applications
Cuda
384
star
85

NeMo-Megatron-Launcher

NeMo Megatron launcher and tools
Python
365
star
86

cuda-quantum

C++ and Python support for the CUDA Quantum programming model for heterogeneous quantum-classical workflows
C++
331
star
87

data-science-stack

NVIDIA Data Science stack tools
Shell
317
star
88

video-sdk-samples

Samples demonstrating how to use various APIs of NVIDIA Video Codec SDK
C++
301
star
89

ai-assisted-annotation-client

Client side integration example source code and libraries for AI-Assisted Annotation SDK
C++
301
star
90

cuQuantum

Home for cuQuantum Python & NVIDIA cuQuantum SDK C++ samples
Jupyter Notebook
298
star
91

nvidia-settings

NVIDIA driver control panel
C
284
star
92

DCGM

NVIDIA Data Center GPU Manager (DCGM) is a project for gathering telemetry and measuring the health of NVIDIA GPUs
C++
282
star
93

cnmem

A simple memory manager for CUDA designed to help Deep Learning frameworks manage memory
C++
280
star
94

radtts

Provides training, inference and voice conversion recipes for RADTTS and RADTTS++: Flow-based TTS models with Robust Alignment Learning, Diverse Synthesis, and Generative Modeling and Fine-Grained Control over of Low Dimensional (F0 and Energy) Speech Attributes.
Roff
269
star
95

fsi-samples

A collection of open-source GPU accelerated Python tools and examples for quantitative analyst tasks and leverages RAPIDS AI project, Numba, cuDF, and Dask.
Jupyter Notebook
262
star
96

tensorrt-laboratory

Explore the Capabilities of the TensorRT Platform
C++
259
star
97

CleanUNet

Official PyTorch Implementation of CleanUNet (ICASSP 2022)
Python
258
star
98

gpu-feature-discovery

GPU plugin to the node feature discovery for Kubernetes
Go
255
star
99

egl-wayland

The EGLStream-based Wayland external platform
C
243
star
100

torch-harmonics

Differentiable spherical harmonic transforms and spherical convolutions in PyTorch
Jupyter Notebook
240
star